[jira] [Commented] (METRON-2155) Cache Maven in Travis CI Builds
[ https://issues.apache.org/jira/browse/METRON-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856024#comment-16856024 ] Michael Miklavcic commented on METRON-2155: --- [~nickwallen] I'm unclear whether #1 will help considering the nature of the failures I've seen - doesn't even appear to be a problem with the download, the Travis process just goes AWOL and eventually bombs out with a timeout. Even so, it might alleviate our problem if there's an issue with how the download mirrors are accessed in Travis by internalizing the endpoint that we download from. It won't do anything to address performance, as you mentioned the Travis docs clearly state - "Large files that are quick to install but slow to download do not benefit from caching, as they take as long to download from the cache as from the original source" - but it *may* help with stability. I'd be in favor of trying this approach first. #3 - it looks like that Maven wrapper effectively does what we're doing with curl and unzip - [https://github.com/takari/maven-wrapper/blob/master/src/main/java/org/apache/maven/wrapper/Installer.java]. If curl is running into issues, this doesn't seem, on the surface, like it's going to add much value to addressing that problem. > Cache Maven in Travis CI Builds > --- > > Key: METRON-2155 > URL: https://issues.apache.org/jira/browse/METRON-2155 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > In the Travis CI builds, we download Maven and even retry up to 10 times > should this download fail. We continue to see some failures when downloading > Maven. > [https://api.travis-ci.org/v3/job/540955869/log.txt] > > This could be a problem within Travis' internal networks or this could be an > intermittent issue with the Apache mirrors. We have no way of knowing and no > good way to resolve these sorts of problems. > We could cache the Maven dependency to mitigate this problem. It would not > truly resolve the problem, but should help mitigate it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] mmiklavc commented on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table
mmiklavc commented on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table URL: https://github.com/apache/metron/pull/1431#issuecomment-498797705 @tiborm The follow on Jiras make sense. Thanks for responding to these questions and adding the documentation and tests. You probably don't want every edge case validation conditional spelled out in the docs, but I think it's worth letting the user know what to expect, e.g. "Be aware that if you misspell a property, it will be treated as non-existent and the default value of 'X' will be used. You will not get an exception/error." As a general soft rule around adding features, they can typically come in these flavors, in quasi-descending order of desirability: 1. Feature added with full Management UI or Ambari support for enabling/disabling and configuring in single PR. (if it's a huge PR, it may be better to go with option 2) 2. Feature added with UI config support, but split into multiple PRs. Indicate in the description what you're doing and link the PRs to one another in the description. You can request individual review of each PR and say something like "don't merge until all K PRs in this work have been +1'ed," and then commit each PR in sequence. Ideally, the last PR will include all K-1 PRs and will have been tested to ensure the entire unit of work works. See parser chaining as an example. https://github.com/apache/metron/pull/1084. I would take this a step further and explicitly add the links to the PRs. > Note, this PR depends on METRON-1643 and METRON-1642, so those should be reviewed prior to this. 3. Feature added without UI support. **MUST** have documentation for manual configuration. Create Jiras to track the follow-on work and link them to the original. 4. Feature flags - yet to be formally adopted as an approach in Metron, but there is open discussion on it. Not really applicable here, but one other option in situations like this is a feature branch. We typically use that for more disruptive and/or larger changes that we want more flexibility with on the individual commits, but it's worth mentioning. In this case, the gating factor for the individual PRs in the feature branch is a bit lower. You might provide a PR that breaks something temporarily in the FB, for example. Or you save documentation until after you've iterated on the feature enough to be comfortable with the implementation. You indicate what the necessary follow-on work will be arising from the PR against the feature branch, create a sub-task for it on the feature Jira, and it becomes a gating factor for final vote and acceptance on merging the FB into master. This PR fits option 3, which is perfectly fine. We just need a bit of expansion on testing and documentation as we've already touched on. Apache Metron community standards around automated and manual testing apply the same in all cases, with the exception of feature branches where only the final unit of work is reviewed by that standard. Any committer can veto a PR and the approach taken at any time if they think it will not work well for the product and its users, so ymmv. Communicating the full feature delivery lifecycle (e.g. justifying/explaining the why and how of follow-ons) obviously helps grease the skids on this and increases the likelihood of an expedient review and ultimately a +1. Ok, enough on dev guidelines for now. I should probably get these thoughts into a discuss thread and our dev guidelines at some point, but it seemed useful here as we discussed the implementation choices. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-2145) Clarify RPM build documentation
[ https://issues.apache.org/jira/browse/METRON-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2145: -- Fix Version/s: Next + 1 > Clarify RPM build documentation > --- > > Key: METRON-2145 > URL: https://issues.apache.org/jira/browse/METRON-2145 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 20m > Remaining Estimate: 0h > > The RPM doc quick start currently tells users to run the command from the > project root. This works, however it also runs a full packaging for the > entire project. This may not be what users want. The full dev build, for > example, runs from /metron-deployment and will only build the > rpm artifacts. This effectively cuts down 20 minutes from the rpm build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] sardell commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default
sardell commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default URL: https://github.com/apache/metron/pull/1393#issuecomment-498736259 @mmiklavc That makes sense. I figured the name of the field might have been misleading me, but I wanted to check before merging. You answered my question. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] asfgit closed pull request #1434: METRON-2145 Clarify RPM build documentation
asfgit closed pull request #1434: METRON-2145 Clarify RPM build documentation URL: https://github.com/apache/metron/pull/1434 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-2083) Fix broken links in root metron README
[ https://issues.apache.org/jira/browse/METRON-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2083: -- Fix Version/s: Next + 1 > Fix broken links in root metron README > -- > > Key: METRON-2083 > URL: https://issues.apache.org/jira/browse/METRON-2083 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > [https://github.com/apache/metron/blob/master/README.md] > Broken links: > # > [Parsers|https://github.com/apache/metron/blob/master/metron-platform/metron-parsers-common] > # [Using Metron with Elasticsearch > 2.x|https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] asfgit closed pull request #1435: METRON-2083 Fix broken links in root metron README
asfgit closed pull request #1435: METRON-2083 Fix broken links in root metron README URL: https://github.com/apache/metron/pull/1435 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default
mmiklavc commented on issue #1393: METRON-2092: Config UI does not require you
to set a grok timestamp field by default
URL: https://github.com/apache/metron/pull/1393#issuecomment-498733822
@sardell This comes from the grok pattern used. The value is an indirection
to another variable that references the actual timestamp the user wants to use.
For example, in full dev:
**YAF**
- `"timestampField":"start_time"`
- `"dateFormat":"-MM-dd HH:mm:ss.S"` - need this in order for the parser
to translate the non-timestamp-fromatted date/time into a proper numeric
timestamp.
- You can find the grok expressions loaded in HDFS at `/patterns/yaf`.
They're also staged from the RPM installs on the local file system at
`$METRON_HOME/patterns`. Anyhow, here's what YAF's Grok patterns file looks
like:
```
YAF_TIME_FORMAT %{YEAR:UNWANTED}-%{MONTHNUM:UNWANTED}-%{MONTHDAY:UNWANTED}[T
]%{HOUR:UNWANTED}:%{MINUTE:UNWANTED}:%{SECOND:UNWANTED}
YAF_DELIMITED
%{YAF_TIME_FORMAT:start_time}\|%{YAF_TIME_FORMAT:end_time}\|%{SPACE:UNWANTED}%{BASE10NUM:duration}\|%{SPACE:UNWANTED}%{BASE10NUM:rtt}\|%{SPACE:UNWANTED}%{INT:protocol}\|%{SPACE:UNWANTED}%{IP:ip_src_addr}\|%{SPACE:UNWANTED}%{INT:ip_src_port}\|%{SPACE:UNWANTED}%{IP:ip_dst_addr}\|%{SPACE:UNWANTED}%{INT:ip_dst_port}\|%{SPACE:UNWANTED}%{DATA:iflags}\|%{SPACE:UNWANTED}%{DATA:uflags}\|%{SPACE:UNWANTED}%{DATA:riflags}\|%{SPACE:UNWANTED}%{DATA:ruflags}\|%{SPACE:UNWANTED}%{WORD:isn}\|%{SPACE:UNWANTED}%{DATA:risn}\|%{SPACE:UNWANTED}%{DATA:tag}\|%{GREEDYDATA:rtag}\|%{SPACE:UNWANTED}%{INT:pkt}\|%{SPACE:UNWANTED}%{INT:oct}\|%{SPACE:UNWANTED}%{INT:rpkt}\|%{SPACE:UNWANTED}%{INT:roct}\|%{SPACE:UNWANTED}%{INT:app}\|%{GREEDYDATA:end_reason}
```
Notice the `start_time` field - `%{YAF_TIME_FORMAT:start_time}`
For an original message `"original_string": "2019-06-03
20:38:27.000|2019-06-03 20:38:27.000| 0.000| 0.000| 6| 192.168.138.158|49189|
62.75.195.236| 80| A| 0| 0| 0|9dfb1927||000|000| 1| 40| 0| 0| 0|idle
",`, this results in a Metron `timestamp` field published to the index as
`"timestamp": 1559594307000`
**Squid**
- `"timestampField": "timestamp"`
- No need for dateFormat bc the timestamp is in the right format for Metron
OOTB.
```
SQUID_DELIMITED %{NUMBER:timestamp}[^0-9]*%{INT:elapsed} %{IP:ip_src_addr}
%{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method}
%{NOTSPACE:url}[^0-9]*(%{IP:ip_dst_addr})?
```
Does that answer your question?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1438: METRON-2153: ParserIntegrationTest should print failed messages
mmiklavc commented on a change in pull request #1438: METRON-2153:
ParserIntegrationTest should print failed messages
URL: https://github.com/apache/metron/pull/1438#discussion_r290361824
##
File path:
metron-platform/metron-parsing/metron-parsing-storm/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
##
@@ -326,6 +326,10 @@ protected void handleError(String sensorType, byte[]
originalMessage, Tuple tupl
.withThrowable(ex)
.withSensorType(Collections.singleton(sensorType))
.addRawMessage(originalMessage);
+handleError(collector, error);
+ }
+
+ protected void handleError(OutputCollector collector, MetronError error) {
Review comment:
Hey, I like this pattern you used here of pushing the external dep into a
function. Any future refactoring on code that does this becomes a bit easier.
Also:
- Easier to read
- Gives a more consistent abstraction level within the main method
- Encourages good composition patterns, which is great architecturally as
well as for unit testing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (METRON-2152) Add debug logging for when sensor batchTimeout exceeds the calculated maximum
[ https://issues.apache.org/jira/browse/METRON-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2152: -- Fix Version/s: Next + 1 > Add debug logging for when sensor batchTimeout exceeds the calculated maximum > - > > Key: METRON-2152 > URL: https://issues.apache.org/jira/browse/METRON-2152 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > This detail is helpful when tuning the writer performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] asfgit closed pull request #1437: METRON-2152 Add debug logging for when sensor batchTimeout exceeds the calculated maximum
asfgit closed pull request #1437: METRON-2152 Add debug logging for when sensor batchTimeout exceeds the calculated maximum URL: https://github.com/apache/metron/pull/1437 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (METRON-2155) Cache Maven in Travis CI Builds
[ https://issues.apache.org/jira/browse/METRON-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855793#comment-16855793 ] Nick Allen commented on METRON-2155: (1) One way would be to cache the Maven zip file. Travis says this won't help performance, but at least it might help with the build failures. [https://docs.travis-ci.com/user/caching/#things-not-to-cache] (2) Another way would be to cache the directory containing the unzipped maven binaries. But how does Travis know not to re-download the Maven binary in that case? (3) Would using something like mvnw help? [https://github.com/takari/maven-wrapper] This would make Maven installation simpler for both users and Travis. I am not sure if this would help the current build failures that we are seeing though. > Cache Maven in Travis CI Builds > --- > > Key: METRON-2155 > URL: https://issues.apache.org/jira/browse/METRON-2155 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > In the Travis CI builds, we download Maven and even retry up to 10 times > should this download fail. We continue to see some failures when downloading > Maven. > [https://api.travis-ci.org/v3/job/540955869/log.txt] > > This could be a problem within Travis' internal networks or this could be an > intermittent issue with the Apache mirrors. We have no way of knowing and no > good way to resolve these sorts of problems. > We could cache the Maven dependency to mitigate this problem. It would not > truly resolve the problem, but should help mitigate it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-2155) Cache Maven in Travis CI Builds
[ https://issues.apache.org/jira/browse/METRON-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855739#comment-16855739 ] Nick Allen commented on METRON-2155: This suggestion originally came from [~justinleet], I believe. > Cache Maven in Travis CI Builds > --- > > Key: METRON-2155 > URL: https://issues.apache.org/jira/browse/METRON-2155 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > In the Travis CI builds, we download Maven and even retry up to 10 times > should this download fail. We continue to see some failures when downloading > Maven. > [https://api.travis-ci.org/v3/job/540955869/log.txt] > > This could be a problem within Travis' internal networks or this could be an > intermittent issue with the Apache mirrors. We have no way of knowing and no > good way to resolve these sorts of problems. > We could cache the Maven dependency to mitigate this problem. It would not > truly resolve the problem, but should help mitigate it. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2155) Cache Maven in Travis CI Builds
Nick Allen created METRON-2155: -- Summary: Cache Maven in Travis CI Builds Key: METRON-2155 URL: https://issues.apache.org/jira/browse/METRON-2155 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen In the Travis CI builds, we download Maven and even retry up to 10 times should this download fail. We continue to see some failures when downloading Maven. [https://api.travis-ci.org/v3/job/540955869/log.txt] This could be a problem within Travis' internal networks or this could be an intermittent issue with the Apache mirrors. We have no way of knowing and no good way to resolve these sorts of problems. We could cache the Maven dependency to mitigate this problem. It would not truly resolve the problem, but should help mitigate it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [metron] nickwallen commented on issue #1435: METRON-2083 Fix broken links in root metron README
nickwallen commented on issue #1435: METRON-2083 Fix broken links in root metron README URL: https://github.com/apache/metron/pull/1435#issuecomment-498663337 +1 Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1434: METRON-2145 Clarify RPM build documentation
nickwallen commented on issue #1434: METRON-2145 Clarify RPM build documentation URL: https://github.com/apache/metron/pull/1434#issuecomment-498662850 +1 Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1437: METRON-2152 Add debug logging for when sensor batchTimeout exceeds the calculated maximum
nickwallen commented on issue #1437: METRON-2152 Add debug logging for when sensor batchTimeout exceeds the calculated maximum URL: https://github.com/apache/metron/pull/1437#issuecomment-498662519 +1 Very helpful addition This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (METRON-2154) Metron Indexing can't start
nickjames created METRON-2154:
-
Summary: Metron Indexing can't start
Key: METRON-2154
URL: https://issues.apache.org/jira/browse/METRON-2154
Project: Metron
Issue Type: Bug
Affects Versions: 0.7.1
Reporter: nickjames
Ambari Version 2.4.3.0
Centos 7
Metron mpack versiong 0.7.2
Errors
{quote} Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/indexing_master.py",
line 244, in
Indexing().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 285, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/indexing_master.py",
line 155, in restart
commands.restart_indexing_topology(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/indexing_commands.py",
line 342, in restart_indexing_topology
self.restart_batch_indexing_topology(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/indexing_commands.py",
line 361, in restart_batch_indexing_topology
self.start_batch_indexing_topology(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.7.2/package/scripts/indexing_commands.py",
line 268, in start_batch_indexing_topology
Execute(start_cmd, user=self.__params.metron_user, tries=3, try_sleep=5,
logoutput=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line
155, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 293, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of
'/usr/metron/0.7.2/bin/start_hdfs_topology.sh' returned 1. Running:
/usr/jdk64/jdk1.8.0_77/bin/java -server -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/usr/hdp/2.5.3.0-37/storm -Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
-cp
/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.5.3.0-37/storm/extlib/atlas-plugin-classloader-0.7.0.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/extlib/storm-bridge-shim-0.7.0.2.5.3.0-37.jar
org.apache.storm.daemon.ClientJarTransformerRunner
org.apache.storm.hack.StormShadeTransformer
/usr/metron/0.7.2/lib/metron-elasticsearch-storm-0.7.2-uber.jar
/tmp/6544563086c211e98906005056b9b3fb.jar
Exception in thread "main" java.lang.IllegalAccessError: tried to access method
org.apache.logging.log4j.core.lookup.MapLookup.newMap(I)Ljava/util/HashMap;
from class org.apache.logging.log4j.core.lookup.MainMapLookup
at
org.apache.logging.log4j.core.lookup.MainMapLookup.(MainMapLookup.java:37)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.logging.log4j.core.util.ReflectionUtil.instantiate(ReflectionUtil.java:185)
[GitHub] [metron] tiborm edited a comment on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table
tiborm edited a comment on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table URL: https://github.com/apache/metron/pull/1431#issuecomment-498642471 > @tiborm this looks like a pretty cool feature, thanks for the contribution! I'd like to address a couple things before we commit this. I am on board with providing a flag for this feature and leaving it disabled by default - that looks reasonable to me, and I also coincidentally just kicked off a DISCUSS thread along these lines as well - https://lists.apache.org/thread.html/e24e4709fbc06473977a0a7f0c0ce92eb02cfcc8c1c2bc41be08a698@%3Cdev.metron.apache.org%3E. I do think we need to resolve some outstanding questions about how we handle that as a community - see below. > > I looked over your instructions on the Jira ticket and I think we're pretty close. I'd like to see the following: > > 1. What's the plan for handling this feature flag? We haven't had any discussion in the community about enabling feature flags to this point. I suspect we might consider them an option for lowering the bar to testing, but we should address this. I don't see any non-happy path tests on this PR, for example. I think that either we need to put this PR through more rigorous testing or come to a community understanding (and document it in the dev guidelines) about what a reasonable level of quality and feature polish will be for code committed under a feature flag. I agree with @sardell on this. The isEnable config parameter here is not a feature flag. IMHO feature flags are for development teams. In this case, the isEnabled parameter is for our end users (more precisely for those who maintain deployed Metron on a service level). Even if the click through navigation was put behind a feature flag, I would keep the isEnabled flags for our clients to turn on and off based according to their preference. Having this feature on by default would weaken the user experience if users don’t want to use it. I know you already agreed with @sardell briefly pointing this out, but I wanted to clarify my use of the isEnabled parameter. On a related note, I added these two followup Jira here: [Create REST endpoint to persist Click Through navigation configuration](https://issues.apache.org/jira/browse/METRON-2104) [Impl Click Through navigation admin interface on Management UI](https://issues.apache.org/jira/browse/METRON-2103) With these two additional changes, isEnabled would be fully exposed to our end users (trough a toggle control) and they would be able to turn it on or off through the Management UI. > 2. Non-happy path test scenarios? e.g. > >1. What happens to the UI if you misspell a field name? What errors/indication does the user get? Will the UI fail to come up? Will it only fail when a field is visible in the UI column listing? If the isEnabled field name is misspelled then the isEnabled field is considered as missing. It will evaluate as not true, therefore the feature stays turned off. I’ll extend the implementation with warnings in the console log if any required configuration is missing. I’m also going to cover these scenarios with unit tests. If a field name is misspelled it will be considered missing. None of these scenarios causing errors but I’ll cover them with unit tests. Visibility of a column has no impact on this. I checked it and /search request returns with all the available fields. Visibility of the columns has no impact on the result set. >2. What happens if you type "fasle" for `isEnabled`? Again, what's the default behavior for misconfiguration and where can operations people managing this feature look for logs that detail the exceptions? That would actually evaluate as true because of Javascript’s auto type coercion. I’m going to fix this and cover it with unit tests. >3. What happens for fields not present in a particular record that are used for token replacement? e.g. per the Jira instructions - "But in the configuration, any available alert property field could be referenced like the following:" Also, is this impacted in any way by what fields are currently displayed/enabled in the alerts UI column listing? If I've hidden ip_src_addr from display, will it still render the URL correctly? Visibility of a column has no impact on the token replacement. Referencing a non-existing token causes no error. However, the URL will not be resolved properly. I’ll cover this scenario with unit tests as well. And add log messages to help configuration. > 3. These instructions should have a dedicated section in the documentation/README for the Alerts UI. > >1. Please provide a detailed explanation of the configuration lifecycle - will the changes be picked up automatically, or only after a restart? Provide instructions for whatever those steps are (e.g.
[GitHub] [metron] tiborm edited a comment on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table
tiborm edited a comment on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table URL: https://github.com/apache/metron/pull/1431#issuecomment-498642471 > @tiborm this looks like a pretty cool feature, thanks for the contribution! I'd like to address a couple things before we commit this. I am on board with providing a flag for this feature and leaving it disabled by default - that looks reasonable to me, and I also coincidentally just kicked off a DISCUSS thread along these lines as well - https://lists.apache.org/thread.html/e24e4709fbc06473977a0a7f0c0ce92eb02cfcc8c1c2bc41be08a698@%3Cdev.metron.apache.org%3E. I do think we need to resolve some outstanding questions about how we handle that as a community - see below. > > I looked over your instructions on the Jira ticket and I think we're pretty close. I'd like to see the following: > > 1. What's the plan for handling this feature flag? We haven't had any discussion in the community about enabling feature flags to this point. I suspect we might consider them an option for lowering the bar to testing, but we should address this. I don't see any non-happy path tests on this PR, for example. I think that either we need to put this PR through more rigorous testing or come to a community understanding (and document it in the dev guidelines) about what a reasonable level of quality and feature polish will be for code committed under a feature flag. I agree with @sardell on this. The isEnable config parameter here is not a feature flag. IMHO feature flags are for development teams. In this case, the isEnabled parameter is for our end users (more precisely for those who maintain deployed Metron on a service level). Even if the click through navigation was put behind a feature flag, I would keep the isEnabled flags for our clients to turn on and off based according to their preference. Having this feature on by default would weaken the user experience if users don’t want to use it. I know you already agreed with @sardell briefly pointing this out, but I wanted to clarify my use of the isEnabled parameter. On a related note, I added these two followup Jira here: [Create REST endpoint to persist Click Through navigation configuration](https://issues.apache.org/jira/browse/METRON-2104) [Impl Click Through navigation admin interface on Management UI](https://issues.apache.org/jira/browse/METRON-2103) With these two additional changes, isEnabled would be fully exposed to our end users (trough a toggle control) and they would be able to turn it on or off through the Management UI. > 2. Non-happy path test scenarios? e.g. > >1. What happens to the UI if you misspell a field name? What errors/indication does the user get? Will the UI fail to come up? Will it only fail when a field is visible in the UI column listing? If the isEnabled field name is misspelled then the isEnabled field is considered as missing. It will evaluate as not true, therefore the feature stays turned off. I’ll extend the implementation with warnings in the console log if any required configuration is missing. I’m also going to cover these scenarios with unit tests. If a field name is misspelled it will be considered missing. None of these scenarios causing errors but I’ll cover them with unit tests. Visibility of a column has no impact on this. As far as I know REST returns with all the available fields. But let me double check this. >2. What happens if you type "fasle" for `isEnabled`? Again, what's the default behavior for misconfiguration and where can operations people managing this feature look for logs that detail the exceptions? That would actually evaluate as true because of Javascript’s auto type coercion. I’m going to fix this and cover it with unit tests. >3. What happens for fields not present in a particular record that are used for token replacement? e.g. per the Jira instructions - "But in the configuration, any available alert property field could be referenced like the following:" Also, is this impacted in any way by what fields are currently displayed/enabled in the alerts UI column listing? If I've hidden ip_src_addr from display, will it still render the URL correctly? Visibility of a column has no impact on the token replacement. Referencing a non-existing token causes no error. However, the URL will not be resolved properly. I’ll cover this scenario with unit tests as well. And add log messages to help configuration. > 3. These instructions should have a dedicated section in the documentation/README for the Alerts UI. > >1. Please provide a detailed explanation of the configuration lifecycle - will the changes be picked up automatically, or only after a restart? Provide instructions for whatever those steps are (e.g. restart Alerts UI using command foo)
[GitHub] [metron] tiborm commented on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table
tiborm commented on issue #1431: METRON-2102: [UI] Adding click-through navigation to Alerts table URL: https://github.com/apache/metron/pull/1431#issuecomment-498642471 > @tiborm this looks like a pretty cool feature, thanks for the contribution! I'd like to address a couple things before we commit this. I am on board with providing a flag for this feature and leaving it disabled by default - that looks reasonable to me, and I also coincidentally just kicked off a DISCUSS thread along these lines as well - https://lists.apache.org/thread.html/e24e4709fbc06473977a0a7f0c0ce92eb02cfcc8c1c2bc41be08a698@%3Cdev.metron.apache.org%3E. I do think we need to resolve some outstanding questions about how we handle that as a community - see below. > > I looked over your instructions on the Jira ticket and I think we're pretty close. I'd like to see the following: > > 1. What's the plan for handling this feature flag? We haven't had any discussion in the community about enabling feature flags to this point. I suspect we might consider them an option for lowering the bar to testing, but we should address this. I don't see any non-happy path tests on this PR, for example. I think that either we need to put this PR through more rigorous testing or come to a community understanding (and document it in the dev guidelines) about what a reasonable level of quality and feature polish will be for code committed under a feature flag. I agree with @sardell on this. The isEnable config parameter here is not a feature flag. IMHO feature flags are for development teams. In this case, the isEnabled parameter is for our end users (more precisely for those who maintain deployed Metron on a service level). Even if the click through navigation was put behind a feature flag, I would keep the isEnabled flags for our clients to turn on and off based according to their preference. Having this feature on by default would weaken the user experience if users don’t want to use it. I know you already agreed with @sardell briefly pointing this out, but I wanted to clarify my use of the isEnabled parameter. On a related note, I added these two followup Jira here: [Create REST endpoint to persist Click Through navigation configuration](https://issues.apache.org/jira/browse/METRON-2104) [Impl Click Through navigation admin interface on Management UI](https://issues.apache.org/jira/browse/METRON-2103) With these two additional changes, isEnabled would be fully exposed to our end users (trough a toggle control) and they would be able to turn it on or off through the Management UI. > 2. Non-happy path test scenarios? e.g. > >1. What happens to the UI if you misspell a field name? What errors/indication does the user get? Will the UI fail to come up? Will it only fail when a field is visible in the UI column listing? If the field name is misspelled then the isEnabled field is missing. It will evaluate as not true, therefore the feature stays turned off. I’ll extend the implementation with warnings in the console log if any required configuration is missing. I’m also going to cover these scenarios with unit tests. If a field name is misspelled it will be considered missing. None of these scenarios causing errors but I’ll cover them with unit tests. >2. What happens if you type "fasle" for `isEnabled`? Again, what's the default behavior for misconfiguration and where can operations people managing this feature look for logs that detail the exceptions? That would actually evaluate as true because of Javascript’s auto type coercion. I’m going to fix this and cover it with unit tests. >3. What happens for fields not present in a particular record that are used for token replacement? e.g. per the Jira instructions - "But in the configuration, any available alert property field could be referenced like the following:" Also, is this impacted in any way by what fields are currently displayed/enabled in the alerts UI column listing? If I've hidden ip_src_addr from display, will it still render the URL correctly? Visibility of a column has no impact on the token replacement. Referencing a non-existing token causes no error. However, the URL will not be resolved properly. I’ll cover this scenario with unit tests as well. And add log messages to help configuration. > 3. These instructions should have a dedicated section in the documentation/README for the Alerts UI. > >1. Please provide a detailed explanation of the configuration lifecycle - will the changes be picked up automatically, or only after a restart? Provide instructions for whatever those steps are (e.g. restart Alerts UI using command foo) >2. Provide a detailed schema description for the JSON you provided in your testing examples. I think you've done a decent job of explaining what `alertEntry` and
[GitHub] [metron] ottobackwards commented on issue #1438: METRON-2153: ParserIntegrationTest should print failed messages
ottobackwards commented on issue #1438: METRON-2153: ParserIntegrationTest should print failed messages URL: https://github.com/apache/metron/pull/1438#issuecomment-498621136 +1 by inspection This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
