Repository: metron Updated Branches: refs/heads/master 2c56a13e5 -> 2ae1f5adf
METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/2ae1f5ad Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/2ae1f5ad Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/2ae1f5ad Branch: refs/heads/master Commit: 2ae1f5adf8428832550f99bcc4cc76645597851a Parents: 2c56a13 Author: JonZeolla <zeo...@gmail.com> Authored: Wed Sep 13 16:58:11 2017 -0700 Committer: cstella <ceste...@gmail.com> Committed: Wed Sep 13 16:58:11 2017 -0700 ---------------------------------------------------------------------- metron-analytics/metron-profiler-client/README.md | 4 ++-- .../src/main/config/zeppelin/metron/metron-pcap.json | 2 +- metron-sensors/bro-plugin-kafka/README.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/2ae1f5ad/metron-analytics/metron-profiler-client/README.md ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-profiler-client/README.md b/metron-analytics/metron-profiler-client/README.md index 27aa3f4..63d0cd4 100644 --- a/metron-analytics/metron-profiler-client/README.md +++ b/metron-analytics/metron-profiler-client/README.md @@ -305,7 +305,7 @@ The person doing the querying must carry forward the knowledge of the Profiler c ### Examples -The following are usage examples that show how the Stellar API can be used to read profiles generated by the [Metron Profiler](../metron-profiler). This API would be used in conjunction with other Stellar functions like [`MAAS_MODEL_APPLY`](../../metron-platform/metron-common#maas_model_apply) to perform model scoring on streaming data. +The following are usage examples that show how the Stellar API can be used to read profiles generated by the [Metron Profiler](../metron-profiler). This API would be used in conjunction with other Stellar functions like [`MAAS_MODEL_APPLY`](../../metron-stellar/stellar-common#maas_model_apply) to perform model scoring on streaming data. These examples assume a profile has been defined called 'snort-alerts' that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following. @@ -462,4 +462,4 @@ Follow these steps in the Stellar REPL to see how it can be used to help create This profile simply counts the number of messages by IP source address. Notice that the value is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr' of '10.0.0.1'. There will always be one measurement for each [profile, entity] pair. -1. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat' this process. Once you are happy with the profile that was created, follow the [Getting Started](../metron-profiler#getting-started) guide to use the profile against your live, streaming data in a Metron cluster. \ No newline at end of file +1. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat' this process. Once you are happy with the profile that was created, follow the [Getting Started](../metron-profiler#getting-started) guide to use the profile against your live, streaming data in a Metron cluster. http://git-wip-us.apache.org/repos/asf/metron/blob/2ae1f5ad/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json ---------------------------------------------------------------------- diff --git a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json index c18b5fb..447056f 100644 --- a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json +++ b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json @@ -1 +1 @@ -{"paragraphs":[{"text":"%md\n# Execute Packet Capture Queries\n\nSpecify the following to filter the packet capture query:\n* *end time* - The ending time of the query in yyyyMMdd format (e.g. 20170428)\n* *start time* - The starting time of the query in yyyyMMdd format (e.g. 20170428)\n* *query* - The [Stellar](https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language) query (i.e. a Stellar expression that returns `true` or `false`) to specify the packets.\n\nThe available fields to use in the queries are as follows:\n* `ip_src_addr` - The source IP address of the packets filtered\n* `ip_src_port` - The source port of the packets filtered\n* `ip_dst_addr` - The destination IP address of the packets filtered\n* `ip_dst_port` - The destination port of the packets filtered\n* `packet` - The raw packet (for use with the `BYTEARRAY_MATCHER` function)\n\nYou can use any [Stellar function](https://github.com/apache/metron/tree/master/metron-platform/me tron-common#stellar-core-functions)\n\n## Simple Boolean Expressions\n\nFor example:\n* `ip_dst_port == 8080` would return all packets where the destination port is `8080`\n* `ip_dst_port in [ 8080, 80 ]` would return all packets where the destination port is either `8080` or `80`\n\n## Common Network Functions on Metadata\n\nFor example:\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24')` would return all packets whose destination conforms to the CIDR `192.168.0.0/24`\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080` would return all packets matching the CIDR and whose destination port is `8080`\n \n## Filtering based on the Packet Contents\n\nWe use byteseek regular expressions to filter packets. The syntax for these is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).\n* `BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` in them anywhere.\n* `ip_dst_port==8080 && BYTE ARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` and have a destination port of `8080`\n* `BYTEARRAY_MATCHER('ff(.){5}ff', packet)` would return all packets containing a binary regex with `0xff` followed by any 5 bytes and then `0xff`\n\n# The Output\nThe output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.","dateUpdated":"2017-05-01T17:36:27+0000","config":{"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"query":"ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\\\'/api/v1/\\\\\\'', packet)"},"forms":{}},"jobName":"paragraph_1493658709556_1010155640","id":"20170428-183346_777875025","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Execute Packet Capture Queries</h1>\n<p>Specify the following to filter th e packet capture query:</p>\n<ul>\n<li><em>end time</em> - The ending time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>start time</em> - The starting time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>query</em> - The <a href=\"https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language\";>Stellar</a> query (i.e. a Stellar expression that returns <code>true</code> or <code>false</code>) to specify the packets.</li>\n</ul>\n<p>The available fields to use in the queries are as follows:</p>\n<ul>\n<li><code>ip_src_addr</code> - The source IP address of the packets filtered</li>\n<li><code>ip_src_port</code> - The source port of the packets filtered</li>\n<li><code>ip_dst_addr</code> - The destination IP address of the packets filtered</li>\n<li><code>ip_dst_port</code> - The destination port of the packets filtered</li>\n<li><code>packet</code> - The raw packet (for use with the <code>BYTEARRAY_MATCHER</code> function)</l i>\n</ul>\n<p>You can use any <a href=\"https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-core-functions\";>Stellar function</a></p>\n<h2>Simple Boolean Expressions</h2>\n<p>For example:</p>\n<ul>\n<li><code>ip_dst_port == 8080</code> would return all packets where the destination port is <code>8080</code></li>\n<li><code>ip_dst_port in [ 8080, 80 ]</code> would return all packets where the destination port is either <code>8080</code> or <code>80</code></li>\n</ul>\n<h2>Common Network Functions on Metadata</h2>\n<p>For example:</p>\n<ul>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24')</code> would return all packets whose destination conforms to the CIDR <code>192.168.0.0/24</code></li>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080</code> would return all packets matching the CIDR and whose destination port is <code>8080</code></li>\n</ul>\n<h2>Filtering based on the Packet Contents</h2>\n<p>We use byteseek reg ular expressions to filter packets. The syntax for these is described <a href=\"https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md\";>here</a>.</p>\n<ul>\n<li><code>BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> in them anywhere.</li>\n<li><code>ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> and have a destination port of <code>8080</code></li>\n<li><code>BYTEARRAY_MATCHER('ff(.){5}ff', packet)</code> would return all packets containing a binary regex with <code>0xff</code> followed by any 5 bytes and then <code>0xff</code></li>\n</ul>\n<h1>The Output</h1>\n<p>The output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.</p>\n"},"dateCreated":"2017-05-01T17:11:49+0000","status":"FINISHED","progressUpdateIntervalMs ":500,"$$hashKey":"object:1724","dateFinished":"2017-05-01T17:36:26+0000","dateStarted":"2017-05-01T17:36:26+0000","focus":true},{"text":"%sh\nexport PCAP_ZEPPELIN_RUN=$(find /usr -name pcap_zeppelin_run.sh)\nexport RECORDS_PER_FILE=10000\nexport NUMBER_OF_REDUCERS=10\nexport DATE_FORMAT=\"yyyyMMdd\"\nexport PCAP_DATA_PATH=\"/apps/metron/pcap\"\n\n$PCAP_ZEPPELIN_RUN \"${query}\" \"${start time}\" \"${optional end time}\"","dateUpdated":"2017-05-01T17:35:04+0000","config":{"colWidth":12,"editorMode":"ace/mode/sh","editorHide":false,"graph":{"mode":"table","height":164,"optionOpen":false,"keys":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"values":[],"groups":[],"scatter":{"xAxis":{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}}},"enabled":true},"settings":{"params":{"sta rt time":"20170428","end time":"","query":"BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet) && ip_dst_port == 8080","optional end time":""},"forms":{"optional end time":{"name":"optional end time","defaultValue":"","hidden":false},"query":{"name":"query","defaultValue":"","hidden":false},"start time":{"name":"start time","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493658709562_1009386142","id":"20170428-181957_829993114","result":{"code":"SUCCESS","type":"TABLE","msg":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now\n%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\";>pcap-data-20170501173442578+0000.pcap</a>\n","comment":"","msgTable":[[{"value":"%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\";>pcap-data-2017050 1173442578+0000.pcap</a>"}]],"columnNames":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"rows":[["%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\";>pcap-data-20170501173442578+0000.pcap</a>"]]},"dateCreated":"2017-05-01T17:11:49+0000","dateStarted":"2017-05-01T17:35:04+0000","dateFinished":"2017-05-01T17:34:55+0000","status":"RUNNING","progressUpdateIntervalMs":500,"$$hashKey":"object:1725","focus":true},{"dateUpdated":"2017-05-01T17:34:55+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493658709563_1009001393","id":"20170428-184335_1604096389","dateCreated":"2017-05-01T17 :11:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1726","dateFinished":"2017-05-01T17:34:54+0000","dateStarted":"2017-05-01T17:34:54+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Troubleshooting</h1>\n<p>If you are having problems with the above form, the following may help.</p>\n<h2>I see <code>Terminated by SIGINTERRUPT</code> or something similar in the output!</h2>\n<p>PCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\n<br />They can do this by changing the <code>shell.command.timeout.millisecs</code> property for the <code>sh</code> interpreter to a larger value, likely <code>100000</code>.</p>\n<h2>I do not see a table of URLs to pcap files in my output, what happened?</h2>\n<p>If an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.</p>\n"},"text":"%md\n# Troubleshooting\n\nIf you are having problems with the above form, the following may help.\n\n## I see `Terminated by SIGINTERRUPT` or something similar in the output!\n\nPCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\nThey can do this by changing the `shell.command.timeout.millisecs` property for the `sh` interpreter to a larger value, likely `100000`.\n\n## I do not see a table of URLs to pcap files in my output, what happened?\n\nIf an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.","focus":true},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493659778398_1082698332 ","id":"20170501-172938_213921861","dateCreated":"2017-05-01T17:29:38+0000","status":"READY","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:1972"}],"name":"metron/pcap","id":"2CEEXVR1W","angularObjects":{"2CFNGE6TP:shared_process":[],"2CGUX5TSW:shared_process":[],"2CETVR2AB:shared_process":[],"2CF163WFX:shared_process":[],"2CG4YXKUV:shared_process":[],"2CG6QDFF7:shared_process":[]},"config":{"looknfeel":"default"},"info":{}} \ No newline at end of file +{"paragraphs":[{"text":"%md\n# Execute Packet Capture Queries\n\nSpecify the following to filter the packet capture query:\n* *end time* - The ending time of the query in yyyyMMdd format (e.g. 20170428)\n* *start time* - The starting time of the query in yyyyMMdd format (e.g. 20170428)\n* *query* - The [Stellar](https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language) query (i.e. a Stellar expression that returns `true` or `false`) to specify the packets.\n\nThe available fields to use in the queries are as follows:\n* `ip_src_addr` - The source IP address of the packets filtered\n* `ip_src_port` - The source port of the packets filtered\n* `ip_dst_addr` - The destination IP address of the packets filtered\n* `ip_dst_port` - The destination port of the packets filtered\n* `packet` - The raw packet (for use with the `BYTEARRAY_MATCHER` function)\n\nYou can use any [Stellar function](https://github.com/apache/metron/tree/master/metron-stellar/ste llar-common#stellar-core-functions)\n\n## Simple Boolean Expressions\n\nFor example:\n* `ip_dst_port == 8080` would return all packets where the destination port is `8080`\n* `ip_dst_port in [ 8080, 80 ]` would return all packets where the destination port is either `8080` or `80`\n\n## Common Network Functions on Metadata\n\nFor example:\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24')` would return all packets whose destination conforms to the CIDR `192.168.0.0/24`\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080` would return all packets matching the CIDR and whose destination port is `8080`\n \n## Filtering based on the Packet Contents\n\nWe use byteseek regular expressions to filter packets. The syntax for these is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).\n* `BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` in them anywhere.\n* `ip_dst_port==8080 && BYTE ARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` and have a destination port of `8080`\n* `BYTEARRAY_MATCHER('ff(.){5}ff', packet)` would return all packets containing a binary regex with `0xff` followed by any 5 bytes and then `0xff`\n\n# The Output\nThe output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.","dateUpdated":"2017-05-01T17:36:27+0000","config":{"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"query":"ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\\\'/api/v1/\\\\\\'', packet)"},"forms":{}},"jobName":"paragraph_1493658709556_1010155640","id":"20170428-183346_777875025","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Execute Packet Capture Queries</h1>\n<p>Specify the following to filter th e packet capture query:</p>\n<ul>\n<li><em>end time</em> - The ending time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>start time</em> - The starting time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>query</em> - The <a href=\"https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language\";>Stellar</a> query (i.e. a Stellar expression that returns <code>true</code> or <code>false</code>) to specify the packets.</li>\n</ul>\n<p>The available fields to use in the queries are as follows:</p>\n<ul>\n<li><code>ip_src_addr</code> - The source IP address of the packets filtered</li>\n<li><code>ip_src_port</code> - The source port of the packets filtered</li>\n<li><code>ip_dst_addr</code> - The destination IP address of the packets filtered</li>\n<li><code>ip_dst_port</code> - The destination port of the packets filtered</li>\n<li><code>packet</code> - The raw packet (for use with the <code>BYTEARRAY_MATCHER</code> function)</l i>\n</ul>\n<p>You can use any <a href=\"https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#stellar-core-functions\";>Stellar function</a></p>\n<h2>Simple Boolean Expressions</h2>\n<p>For example:</p>\n<ul>\n<li><code>ip_dst_port == 8080</code> would return all packets where the destination port is <code>8080</code></li>\n<li><code>ip_dst_port in [ 8080, 80 ]</code> would return all packets where the destination port is either <code>8080</code> or <code>80</code></li>\n</ul>\n<h2>Common Network Functions on Metadata</h2>\n<p>For example:</p>\n<ul>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24')</code> would return all packets whose destination conforms to the CIDR <code>192.168.0.0/24</code></li>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080</code> would return all packets matching the CIDR and whose destination port is <code>8080</code></li>\n</ul>\n<h2>Filtering based on the Packet Contents</h2>\n<p>We use byteseek reg ular expressions to filter packets. The syntax for these is described <a href=\"https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md\";>here</a>.</p>\n<ul>\n<li><code>BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> in them anywhere.</li>\n<li><code>ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> and have a destination port of <code>8080</code></li>\n<li><code>BYTEARRAY_MATCHER('ff(.){5}ff', packet)</code> would return all packets containing a binary regex with <code>0xff</code> followed by any 5 bytes and then <code>0xff</code></li>\n</ul>\n<h1>The Output</h1>\n<p>The output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.</p>\n"},"dateCreated":"2017-05-01T17:11:49+0000","status":"FINISHED","progressUpdateIntervalMs ":500,"$$hashKey":"object:1724","dateFinished":"2017-05-01T17:36:26+0000","dateStarted":"2017-05-01T17:36:26+0000","focus":true},{"text":"%sh\nexport PCAP_ZEPPELIN_RUN=$(find /usr -name pcap_zeppelin_run.sh)\nexport RECORDS_PER_FILE=10000\nexport NUMBER_OF_REDUCERS=10\nexport DATE_FORMAT=\"yyyyMMdd\"\nexport PCAP_DATA_PATH=\"/apps/metron/pcap\"\n\n$PCAP_ZEPPELIN_RUN \"${query}\" \"${start time}\" \"${optional end time}\"","dateUpdated":"2017-05-01T17:35:04+0000","config":{"colWidth":12,"editorMode":"ace/mode/sh","editorHide":false,"graph":{"mode":"table","height":164,"optionOpen":false,"keys":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"values":[],"groups":[],"scatter":{"xAxis":{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}}},"enabled":true},"settings":{"params":{"sta rt time":"20170428","end time":"","query":"BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet) && ip_dst_port == 8080","optional end time":""},"forms":{"optional end time":{"name":"optional end time","defaultValue":"","hidden":false},"query":{"name":"query","defaultValue":"","hidden":false},"start time":{"name":"start time","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493658709562_1009386142","id":"20170428-181957_829993114","result":{"code":"SUCCESS","type":"TABLE","msg":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now\n%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\";>pcap-data-20170501173442578+0000.pcap</a>\n","comment":"","msgTable":[[{"value":"%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\";>pcap-data-2017050 1173442578+0000.pcap</a>"}]],"columnNames":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"rows":[["%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\";>pcap-data-20170501173442578+0000.pcap</a>"]]},"dateCreated":"2017-05-01T17:11:49+0000","dateStarted":"2017-05-01T17:35:04+0000","dateFinished":"2017-05-01T17:34:55+0000","status":"RUNNING","progressUpdateIntervalMs":500,"$$hashKey":"object:1725","focus":true},{"dateUpdated":"2017-05-01T17:34:55+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493658709563_1009001393","id":"20170428-184335_1604096389","dateCreated":"2017-05-01T17 :11:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1726","dateFinished":"2017-05-01T17:34:54+0000","dateStarted":"2017-05-01T17:34:54+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Troubleshooting</h1>\n<p>If you are having problems with the above form, the following may help.</p>\n<h2>I see <code>Terminated by SIGINTERRUPT</code> or something similar in the output!</h2>\n<p>PCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\n<br />They can do this by changing the <code>shell.command.timeout.millisecs</code> property for the <code>sh</code> interpreter to a larger value, likely <code>100000</code>.</p>\n<h2>I do not see a table of URLs to pcap files in my output, what happened?</h2>\n<p>If an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.</p>\n"},"text":"%md\n# Troubleshooting\n\nIf you are having problems with the above form, the following may help.\n\n## I see `Terminated by SIGINTERRUPT` or something similar in the output!\n\nPCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\nThey can do this by changing the `shell.command.timeout.millisecs` property for the `sh` interpreter to a larger value, likely `100000`.\n\n## I do not see a table of URLs to pcap files in my output, what happened?\n\nIf an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.","focus":true},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493659778398_1082698332 ","id":"20170501-172938_213921861","dateCreated":"2017-05-01T17:29:38+0000","status":"READY","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:1972"}],"name":"metron/pcap","id":"2CEEXVR1W","angularObjects":{"2CFNGE6TP:shared_process":[],"2CGUX5TSW:shared_process":[],"2CETVR2AB:shared_process":[],"2CF163WFX:shared_process":[],"2CG4YXKUV:shared_process":[],"2CG6QDFF7:shared_process":[]},"config":{"looknfeel":"default"},"info":{}} http://git-wip-us.apache.org/repos/asf/metron/blob/2ae1f5ad/metron-sensors/bro-plugin-kafka/README.md ---------------------------------------------------------------------- diff --git a/metron-sensors/bro-plugin-kafka/README.md b/metron-sensors/bro-plugin-kafka/README.md index e219360..427988b 100644 --- a/metron-sensors/bro-plugin-kafka/README.md +++ b/metron-sensors/bro-plugin-kafka/README.md @@ -141,7 +141,7 @@ event bro_init() &priority=-5 #### Notes * `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`. - * You can also filter IPv6 logs from within your Metron cluster [using Stellar](../../metron-platform/metron-common#IS_IP). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron. + * You can also filter IPv6 logs from within your Metron cluster [using Stellar](../../metron-stellar/stellar-common#IS_IP). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron. * It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin. Settings
