[jira] [Commented] (METRON-660) [Umbrella] up-to-date versioned documentation
[ https://issues.apache.org/jira/browse/METRON-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848045#comment-15848045 ] ASF GitHub Bot commented on METRON-660: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/429 Okay, @JonZeolla , @justinleet , and others, the awk script has been replaced by a python script. The results of the dialect fix-up are identical to the awk script's results across all 37 .md files. Thanks. > [Umbrella] up-to-date versioned documentation > - > > Key: METRON-660 > URL: https://issues.apache.org/jira/browse/METRON-660 > Project: Metron > Issue Type: Improvement >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: METRON-660-screenshot2.png, > site-book_0.3.0_20170130.tar.gz > > > We currently have three forms of documentation, with the following advantages > and disadvantages: > || Docs || Pro || Con || > | CWiki | Easy to edit, no special tools required, don't have to be a > developer to contribute, google and wiki search | Not versioned, no review > process, distant from the code, obsolete content tends to accumulate | > | Site | Versioned and reviewed, only committers can edit, google search | > Yet another arcane toolset must be learned, only web programmers feel > comfortable contributing, "asf-site" branch not related to code versions, > distant from the code, tends to go obsolete due to non-maintenance | > | README.md | Versioned and reviewed, only committers can edit, tied to code > versions, highly local to the code being documented | Non-developers don't > know about them, may be scared by github, poor scoring in google search, no > high-level presentation | > Various discussion threads indicate the developer community likes > README-based docs, and it's easy to see why from the above. I propose this > extension to the README-based documentation, to address their disadvantages: > # Produce a script that runs at build time and gathers the README.md files > from all code subdirectories into a hierarchical tree. The script would have > an exclusion list for non-user-content, which at this point would consist of > \[site/\*, build_utils/\*\]. > # Utilize standard toolsets to publish the hierarchical collection of .md > files as a "book", at least as far as the end user sees. Put the web gateway > for this book in the public Metron Site, preferably with a "version" > pull-down menu that allows access to older versions too. > # Deprecate the use of cwiki for anything but long-lived > demonstrations/tutorials that are unlikely to go obsolete. Move appropriate > content from the cwiki into versioned .md files. > Extensive email discussion of these ideas occurred in thread > https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201701.mbox/%3c74a5a05b-b42d-482c-ae58-a0be06770...@apache.org%3E -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-503) Metron REST API
[ https://issues.apache.org/jira/browse/METRON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847745#comment-15847745 ] ASF GitHub Bot commented on METRON-503: --- Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/316 It just occurred to me that we don't have an HdfsController. Do we even need an HdfsService or could we just use WebHDFS (or Knox) instead? > Metron REST API > --- > > Key: METRON-503 > URL: https://issues.apache.org/jira/browse/METRON-503 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman > Attachments: Metron REST API.docx > > > As discussed on the dev list ([DISCUSS] Metron REST API Requirements), this > Jira includes adding a REST API to Metron. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-503) Metron REST API
[ https://issues.apache.org/jira/browse/METRON-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847716#comment-15847716 ] ASF GitHub Bot commented on METRON-503: --- Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/316 @jjmeyer0, merged your PR in with the new tests. Everything looked good, thanks for helping with that. I just submitted another commit with tests for the remaining services excluding TransformationValidationService (still thinking about how to rename that). Will have that one done soon. I also refactored the SensorParserConfigServiceImpl by removing all the Grok functions with the exception of the isGrokParser method. The read/write methods for Grok statements are exactly the same as the read/write methods in the HdfsServiceImpl class and are not needed so I removed them. I feel like the saveTemporary method should be in GrokServiceImpl so I moved it there. With this change, SensorParserConfigServiceImpl more closely matches the other CRUD interfaces. Let me know what you think about that. As for the error you ran into, I have hit that as well many times while working on Metron. The underlying problem is that the shade plugin is not correctly configured on the metron-statistics module. The guava library is not shaded and the uber jar is installed instead of the regular jar. This means you cannot exclude guava from the metron-statistics dependency and the version mismatch tends to show up at runtime. I believe this problem is outside the scope of this PR and should be fixed separately. There are other modules that have the same problem so those need to be fixed as well. Speaking of guava, you might want to consider not using it in KafkaServiceImplTest. That library is deprecated aggressively and has caused us a lot of problems because we depend on so many different projects that all use different guava versions. Not a big deal if we do use it (we're using it directly in other Metron modules) but we might want to call that dependency out and use the Metron global guava version. It looks like you're just using it for syntactical convenience and isn't critical so my preference would be to take it out. I could be convinced otherwise if you think we'll need it down the road. > Metron REST API > --- > > Key: METRON-503 > URL: https://issues.apache.org/jira/browse/METRON-503 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman > Attachments: Metron REST API.docx > > > As discussed on the dev list ([DISCUSS] Metron REST API Requirements), this > Jira includes adding a REST API to Metron. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-658) Invalid Stellar Predicate Statement
[
https://issues.apache.org/jira/browse/METRON-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847590#comment-15847590
]
ASF GitHub Bot commented on METRON-658:
---
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/430
I'm good with this split as separate commits.
> Invalid Stellar Predicate Statement
> ---
>
> Key: METRON-658
> URL: https://issues.apache.org/jira/browse/METRON-658
> Project: Metron
> Issue Type: Bug
>Reporter: Josh Meyer
> Labels: grammar, stellar
>
> Currently there seems to be an issue in the way Stellar defines predicate
> statements in its grammar. This may not necessarily be limited to predicate
> statements, but currently this is where the issue is showing itself. For
> example, the following expressions should be valid, but each of them fails
> during parsing.
> {code}
> ('grok' not in 'foobar') == true
> 'grok' not in ('foobar' == true)
> 'grok' not in 'foobar' == true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (METRON-658) Invalid Stellar Predicate Statement
[
https://issues.apache.org/jira/browse/METRON-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847583#comment-15847583
]
ASF GitHub Bot commented on METRON-658:
---
Github user jjmeyer0 commented on the issue:
https://github.com/apache/incubator-metron/pull/430
@nickwallen personally I don't think it should be a separate PR. To me it
is just one of those things that should be cleaned up as we go. My reasoning is
that I think we should encourage clean up as we go with minimal overhead.
However, I understand that it is difficult to read on github. I do agree that I
could have been more conscious of the way I made the commits though. I changed
this PR to have two commits,1 with the changes and 1 with the cleanup. If this
is still hard to review, I can split them into two PRs.
> Invalid Stellar Predicate Statement
> ---
>
> Key: METRON-658
> URL: https://issues.apache.org/jira/browse/METRON-658
> Project: Metron
> Issue Type: Bug
>Reporter: Josh Meyer
> Labels: grammar, stellar
>
> Currently there seems to be an issue in the way Stellar defines predicate
> statements in its grammar. This may not necessarily be limited to predicate
> statements, but currently this is where the issue is showing itself. For
> example, the following expressions should be valid, but each of them fails
> during parsing.
> {code}
> ('grok' not in 'foobar') == true
> 'grok' not in ('foobar' == true)
> 'grok' not in 'foobar' == true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (METRON-658) Invalid Stellar Predicate Statement
[
https://issues.apache.org/jira/browse/METRON-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847527#comment-15847527
]
ASF GitHub Bot commented on METRON-658:
---
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/430
Is there a way to split out the core changes that you made from the
whitespace changes? Were the whitespace changes required? Or should those be
a separate "clean-up" PR?
> Invalid Stellar Predicate Statement
> ---
>
> Key: METRON-658
> URL: https://issues.apache.org/jira/browse/METRON-658
> Project: Metron
> Issue Type: Bug
>Reporter: Josh Meyer
> Labels: grammar, stellar
>
> Currently there seems to be an issue in the way Stellar defines predicate
> statements in its grammar. This may not necessarily be limited to predicate
> statements, but currently this is where the issue is showing itself. For
> example, the following expressions should be valid, but each of them fails
> during parsing.
> {code}
> ('grok' not in 'foobar') == true
> 'grok' not in ('foobar' == true)
> 'grok' not in 'foobar' == true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (METRON-675) Make Threat Triage rules able to be assigned names and comments
[ https://issues.apache.org/jira/browse/METRON-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847494#comment-15847494 ] ASF GitHub Bot commented on METRON-675: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/426 +1 Works great. Spun everything up, followed your script, created my own triage rules and validated the scoring. The 'RiskLevelRule' POJO certainly makes things a little cleaner. As a random side note, will be really cool when the aggregation of the scores is just Stellar code, rather than MAX or SUM. This would allow us to plug-in a real model for scoring the alerts. > Make Threat Triage rules able to be assigned names and comments > --- > > Key: METRON-675 > URL: https://issues.apache.org/jira/browse/METRON-675 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > > There may be many, many threat triage rules. To help organize these, we > should make them slightly more complex than a simple key/value as we have it > now. We should add optional name and optional comment fields. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-678) Multithread the flat file loader
[ https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847439#comment-15847439 ] ASF GitHub Bot commented on METRON-678: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/428 > Multithread the flat file loader > > > Key: METRON-678 > URL: https://issues.apache.org/jira/browse/METRON-678 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > > Currently the flat file loader is single threaded in its writing to HBase. > We could make this a lot faster by multithreading the HBase puts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847399#comment-15847399 ] ASF GitHub Bot commented on METRON-676: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/427 > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > Attachments: Metron - YAF Telemetry.png, Metron - YAF Telemetry - > Report Mode.png > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-660) [Umbrella] up-to-date versioned documentation
[
https://issues.apache.org/jira/browse/METRON-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847275#comment-15847275
]
ASF GitHub Bot commented on METRON-660:
---
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/429#discussion_r98739265
--- Diff: site-book/bin/generate-md.sh ---
@@ -190,7 +190,13 @@ cd site-book/src/site/markdown
# Rewrite hrefs for resource references, using table provided by
Maintainers
for (( i=0; i<${#HREF_REWRITE_LIST[@]} ; i+=2 )) ; do
echo rewriting href in "${HREF_REWRITE_LIST[$i]}" :
"${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
-sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
+if [[ "$OSTYPE" =~ "darwin" ]] ; then
+# Macs need an argument after -i option
+sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
+else
--- End diff --
@JonZeolla thanks for the example pointer. Fixed.
@james-sirota , yes maybe sorta. But that's work too, and the on-going
nuisance for people trying to use it isn't adequately balanced for the effort.
I'm almost done with the conversion to python.
> [Umbrella] up-to-date versioned documentation
> -
>
> Key: METRON-660
> URL: https://issues.apache.org/jira/browse/METRON-660
> Project: Metron
> Issue Type: Improvement
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: METRON-660-screenshot2.png,
> site-book_0.3.0_20170130.tar.gz
>
>
> We currently have three forms of documentation, with the following advantages
> and disadvantages:
> || Docs || Pro || Con ||
> | CWiki | Easy to edit, no special tools required, don't have to be a
> developer to contribute, google and wiki search | Not versioned, no review
> process, distant from the code, obsolete content tends to accumulate |
> | Site | Versioned and reviewed, only committers can edit, google search |
> Yet another arcane toolset must be learned, only web programmers feel
> comfortable contributing, "asf-site" branch not related to code versions,
> distant from the code, tends to go obsolete due to non-maintenance |
> | README.md | Versioned and reviewed, only committers can edit, tied to code
> versions, highly local to the code being documented | Non-developers don't
> know about them, may be scared by github, poor scoring in google search, no
> high-level presentation |
> Various discussion threads indicate the developer community likes
> README-based docs, and it's easy to see why from the above. I propose this
> extension to the README-based documentation, to address their disadvantages:
> # Produce a script that runs at build time and gathers the README.md files
> from all code subdirectories into a hierarchical tree. The script would have
> an exclusion list for non-user-content, which at this point would consist of
> \[site/\*, build_utils/\*\].
> # Utilize standard toolsets to publish the hierarchical collection of .md
> files as a "book", at least as far as the end user sees. Put the web gateway
> for this book in the public Metron Site, preferably with a "version"
> pull-down menu that allows access to older versions too.
> # Deprecate the use of cwiki for anything but long-lived
> demonstrations/tutorials that are unlikely to go obsolete. Move appropriate
> content from the cwiki into versioned .md files.
> Extensive email discussion of these ideas occurred in thread
> https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201701.mbox/%3c74a5a05b-b42d-482c-ae58-a0be06770...@apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (METRON-660) [Umbrella] up-to-date versioned documentation
[
https://issues.apache.org/jira/browse/METRON-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847213#comment-15847213
]
ASF GitHub Bot commented on METRON-660:
---
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/429#discussion_r98728815
--- Diff: site-book/bin/generate-md.sh ---
@@ -190,7 +190,13 @@ cd site-book/src/site/markdown
# Rewrite hrefs for resource references, using table provided by
Maintainers
for (( i=0; i<${#HREF_REWRITE_LIST[@]} ; i+=2 )) ; do
echo rewriting href in "${HREF_REWRITE_LIST[$i]}" :
"${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
-sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
+if [[ "$OSTYPE" =~ "darwin" ]] ; then
+# Macs need an argument after -i option
+sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
+else
--- End diff --
Is the path of least resistance here just to provide a docker container
with the right version of awk?
> [Umbrella] up-to-date versioned documentation
> -
>
> Key: METRON-660
> URL: https://issues.apache.org/jira/browse/METRON-660
> Project: Metron
> Issue Type: Improvement
>Reporter: Matt Foley
>Assignee: Matt Foley
> Attachments: METRON-660-screenshot2.png,
> site-book_0.3.0_20170130.tar.gz
>
>
> We currently have three forms of documentation, with the following advantages
> and disadvantages:
> || Docs || Pro || Con ||
> | CWiki | Easy to edit, no special tools required, don't have to be a
> developer to contribute, google and wiki search | Not versioned, no review
> process, distant from the code, obsolete content tends to accumulate |
> | Site | Versioned and reviewed, only committers can edit, google search |
> Yet another arcane toolset must be learned, only web programmers feel
> comfortable contributing, "asf-site" branch not related to code versions,
> distant from the code, tends to go obsolete due to non-maintenance |
> | README.md | Versioned and reviewed, only committers can edit, tied to code
> versions, highly local to the code being documented | Non-developers don't
> know about them, may be scared by github, poor scoring in google search, no
> high-level presentation |
> Various discussion threads indicate the developer community likes
> README-based docs, and it's easy to see why from the above. I propose this
> extension to the README-based documentation, to address their disadvantages:
> # Produce a script that runs at build time and gathers the README.md files
> from all code subdirectories into a hierarchical tree. The script would have
> an exclusion list for non-user-content, which at this point would consist of
> \[site/\*, build_utils/\*\].
> # Utilize standard toolsets to publish the hierarchical collection of .md
> files as a "book", at least as far as the end user sees. Put the web gateway
> for this book in the public Metron Site, preferably with a "version"
> pull-down menu that allows access to older versions too.
> # Deprecate the use of cwiki for anything but long-lived
> demonstrations/tutorials that are unlikely to go obsolete. Move appropriate
> content from the cwiki into versioned .md files.
> Extensive email discussion of these ideas occurred in thread
> https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201701.mbox/%3c74a5a05b-b42d-482c-ae58-a0be06770...@apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847208#comment-15847208 ] ASF GitHub Bot commented on METRON-676: --- Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/427 @nickwallen Thanks for closing off that discussion. This has percolated awhile, and I'm +1 on it. > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > Attachments: Metron - YAF Telemetry.png, Metron - YAF Telemetry - > Report Mode.png > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847195#comment-15847195 ] ASF GitHub Bot commented on METRON-676: --- Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/427 +1 Great job > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > Attachments: Metron - YAF Telemetry.png, Metron - YAF Telemetry - > Report Mode.png > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city We actually use the field directly for the "Unique-Location(s)" visualization in Kibana. How we choose to populate the field changes the meaning of the count, which is where that question comes from. If we're fine counting only the city's geoname_id for that field, the switch to city is a one line change + test fix. The swapped data is in the same format as the current data, so the change to city is something I could have a PR for very quickly. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title James Sirota commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city The kibana dashboards feed off the lat/lon coordinates. This field is used primarily for log correlation. We include the locID for convenience, but it's not a hard requirement to have it. We definitely don't want use city else use country. We should be consistent. If city doesn't exist then we should just not have anything there Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title James Sirota updated an issue Metron / METRON-680 GeoLiteDatabase incorrectly using country geoname_id instead of city Change By: James Sirota Priority: Major Minor Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
Title: Message Title ASF GitHub Bot commented on METRON-676 Re: Create Zeppelin Notebook for YAF Telemetry Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/427 @justinleet Sorry, I originally responded to the dev list and not here. I sent a link to the PR and JIRA, mentioning the screenshots, to the original discussion thread. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
Title: Message Title Nick Allen commented on METRON-676 Re: Create Zeppelin Notebook for YAF Telemetry Sure, I'll just send a link to this JIRA. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city The part about including the country's code as the city isn't always true. See: 23.129.1.0/24,,6252001,,0,0 It's the U.S. (http://www.geonames.org/6252001/united-states.html), but doesn't attach any city information to it. Do we need to be doing a fallback of "Use city if available, else use country"? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-660) [Umbrella] up-to-date versioned documentation
Title: Message Title ASF GitHub Bot commented on METRON-660 Re: [Umbrella] up-to-date versioned documentation Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/429 Ok, I'm going to hold off on any additional review until the move to a python script. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-660) [Umbrella] up-to-date versioned documentation
Title: Message Title
ASF GitHub Bot commented on METRON-660
Re: [Umbrella] up-to-date versioned documentation
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/429#discussion_r98674950
— Diff: site-book/bin/generate-md.sh — @@ -190,7 +190,13 @@ cd site-book/src/site/markdown
Rewrite hrefs for resource references, using table provided by Maintainers for (( i=0; i<$ {#HREF_REWRITE_LIST[@]}
; i+=2 )) ; do echo rewriting href in "$ {HREF_REWRITE_LIST[$i]}" : "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}" - sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}" "${HREF_REWRITE_LIST[$i]}
" + if [[ "$OSTYPE" =~ "darwin" ]] ; then + # Macs need an argument after -i option + sed -i '' -e "$ {HREF_REWRITE_LIST[ $(( i + 1 )) ]}
" "$ {HREF_REWRITE_LIST[$i]}
" + else
End diff –
I would suggest handling this slightly more explicitly. My two thoughts are: 1. Approach it similar to [here](https://github.com/anandsubbu/incubator-metron/blob/dc3fa9739bad53da7931d992cdfd499abba5387d/metron-deployment/scripts/platform-info.sh#L68), using a `case "$ {OSTYPE}"` to watch `darwin*` and `linux*`, and then a `*)` that alerts the user. 1. By adding an elif for either `"${OSTYPE}
" == linux*` or `"$ {OSTYPE}
" =~ "linux"`, and then an `else` that alerts the user.
Add Comment
This message was sent by Atlassian JIRA
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title
Justin Leet created an issue
Metron / METRON-680
GeoLiteDatabase incorrectly using country geoname_id instead of city
Issue Type:
Bug
Assignee:
Justin Leet
Created:
31/Jan/17 14:12
Priority:
Major
Reporter:
Justin Leet
Due to misunderstanding exactly how things tied together with the updated database, the wrong field is used for the locId. Instead of using the city's geoname_id, we are using the country's.
This will effect Kibana dashboards and anything that depends on the locId, because it will be retrieved at the country level instead of the city level. The change will not break anything (anything not at the city level uses the country's code, e.g. if the IP is for Japan in general, the city code is 1861060, not empty or null). This example from the plaintext database can be seen in the second and third fields at:
1.112.0.0/15,1861060,1861060,,0,0,,35.6900,139.6900,500
The offending code is in `GeoLiteDatabase.java` and should be `geoInfo.put("locID", convertNullToEmptyString(country.getGeoNameId()));`
This should be updated to grab the city's geoname, and tests should be updated to reflect this (they didn't catch this error because of the misunderstood data change, not an error in coding).
Ideally, this field is renamed and better documented as part of METRON-679
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet commented on METRON-679 Re: Investigate including additional GeoIP fields James Sirota You're more familiar with what's potentially useful than I am, any insight here? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet created an issue Metron / METRON-679 Investigate including additional GeoIP fields Issue Type: Improvement Assignee: Unassigned Created: 31/Jan/17 13:56 Priority: Minor Reporter: Justin Leet The newer downloadable database we're using provides a wider array of fields than our old solution. We may also want to update some naming (we've been using metro code and calling it dma code, even in the old solution, for example). Data pulled from plaintext download at: http://dev.maxmind.com/geoip/geoip2/geolite2/ Currently used fields: geoname_id (renaming it LocId). city_name country_iso_code metro_code (but renaming it dma code) postal_code latitude longitude network (indirectly, it's the field used for pulling back data for IP and is
[jira] (METRON-503) Metron REST API
Title: Message Title ASF GitHub Bot commented on METRON-503 Re: Metron REST API Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr Today I decided to run through the documentation to see how it works out. When trying to run the spring boot app through the `start.sh` script using an h2 database I got the error below. I did had h2 to my classpath instead of mysql. I did some investigating and it seems like this is an issue with having multiple versions of guava. I used the following code to print which dependency guava is coming from. ``` System.out.println(FluentIterable.class.getProtectionDomain().getCodeSource().getLocation().toExternalForm()); ``` It prints out the line below. It looks like it is getting the dependency from metron-statistics which uses version `12.0`. I believe spring needs `16.01`. ` jar:file:/home/choosetwo/workspace/incubator-metron/metron-interface/metron-rest/target/lib/metron-rest-0.3.0.jar!/BOOT-INF/lib/metron-statistics-0.3.0.jar!/ ` After manually adding guava version `16.01` to the end of the classpath in `start.sh` the error below goes away, and everything works just fine. I think we may need to clean up the dependencies in `metron-rest`'s pom file. ``` 2017-01-31T07:13:11.909 ERROR [org.springframework.boot.SpringApplication] - Application startup failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'requestMappingHandlerAdapter' defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Initialization of bean failed; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.FluentIterable.toList()Lcom/google/common/collect/ImmutableList; at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371) at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175) at org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:32) at
