[GitHub] metron issue #782: METRON-1222
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/782 Please fill out the PR description ---
[GitHub] metron pull request #664: METRON-1059 address AvoidStarImport checkstyle war...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/664 ---
[GitHub] metron issue #737: METRON-1161: Add ability to edit parser command line opti...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/737 I moved the default values to the storm config and changed getNumWorkers and getNumAckers to return the values in the storm config if numWorkers/numAckers is null. Is this what you were thinking? I had to set the defaults in SensorParserConfig because I could not find a way to get these defaults from the Storm Java API. Is there a way and I just missed it? ---
[GitHub] metron issue #664: METRON-1059 address AvoidStarImport checkstyle warning in...
Github user dbist commented on the issue: https://github.com/apache/metron/pull/664 I have a failure locally, will re-do the commit. ---
[GitHub] metron pull request #779: METRON-1218: Metron REST should return better erro...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/779#discussion_r142237205 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/RestExceptionHandler.java --- @@ -35,7 +36,7 @@ @ResponseBody ResponseEntity handleControllerException(HttpServletRequest request, Throwable ex) { HttpStatus status = getStatus(request); -return new ResponseEntity<>(new RestError(status.value(), ex.getMessage(), getFullMessage(ex)), status); +return new ResponseEntity<>(new RestError(status.value(), ex.getMessage(), ExceptionUtils.getStackTrace(ex)), status); --- End diff -- I would prefer just root cause but I had the impression the full stack trace was wanted. Doesn't make that much difference to me so we just need to agree on what it should be. ---
[GitHub] metron pull request #779: METRON-1218: Metron REST should return better erro...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/779#discussion_r142236713 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/RestExceptionHandler.java --- @@ -35,7 +36,7 @@ @ResponseBody ResponseEntity handleControllerException(HttpServletRequest request, Throwable ex) { HttpStatus status = getStatus(request); -return new ResponseEntity<>(new RestError(status.value(), ex.getMessage(), getFullMessage(ex)), status); +return new ResponseEntity<>(new RestError(status.value(), ex.getMessage(), ExceptionUtils.getStackTrace(ex)), status); --- End diff -- ok, but I thought you wanted to show the root cause to the user, **and** have the stack trace that casey wanted. Not just the stack trace. What is the user going to see now? ---
[GitHub] metron pull request #779: METRON-1218: Metron REST should return better erro...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/779#discussion_r142235450 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/RestExceptionHandler.java --- @@ -35,7 +36,7 @@ @ResponseBody ResponseEntity handleControllerException(HttpServletRequest request, Throwable ex) { HttpStatus status = getStatus(request); -return new ResponseEntity<>(new RestError(status.value(), ex.getMessage(), getFullMessage(ex)), status); +return new ResponseEntity<>(new RestError(status.value(), ex.getMessage(), ExceptionUtils.getStackTrace(ex)), status); --- End diff -- Because @cestella requested it earlier. ---
[GitHub] metron pull request #771: METRON-1204: UI does not time out after being idle...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/771 ---
[GitHub] metron issue #779: METRON-1218: Metron REST should return better error messa...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/779 The latest commit replaces the getFullMessage call with ExceptionUtils.getStackTrace. Hopefully that addresses the recursion issue. As for security @ottobackwards, I'm not sure what the requirements are. ---
[GitHub] metron issue #772: METRON-1209: Make stellar repl take logging properties, l...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/772 It does not and should not. If that's what you're seeing, @nickwallen then it's a bug and I should fix it. ---
[GitHub] metron issue #772: METRON-1209: Make stellar repl take logging properties, l...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/772 Does this change the default log level of the REPL when run in a deployed Metron cluster? ---
Re: [DISCUSS] How should Management UI save changes?
Maybe we should use DEPLOY as the final stage? On October 2, 2017 at 14:31:08, Nick Allen (n...@nickallen.org) wrote: > Maybe change the text on the button on the primary panel to "write" instead of "save"? Another option would be to call it "Apply". > Also, I want wider child panels in the management UI if at all possible. Especially the "RAW JSON" feels cramped. Yes, I agree. It seems to odd to me that the main work area in the Management UI is presented as a side panel that consumes a minority portion of the screen real estate. On Thu, Sep 28, 2017 at 12:30 PM Laurens Vetswrote: > Maybe change the text on the button on the primary panel to "write" > instead of "save"? > > Also, I want wider child panels in the management UI if at all possible. > Especially the "RAW JSON" feels cramped. > > On 2017-09-20 14:37, Ryan Merriman wrote: > > Recently @nickwallen brought up some good points about the usability of > > the > > Management UI here: > > https://github.com/apache/metron/pull/737#issuecomment-330632113. The > > issues he brings up apply to all child panels so I think it makes sense > > to > > agree on a common approach and apply it to all of them. > > > > Most child panels have a save button that saves changes to the local > > (browser) copy of the config. The save button on the primary panel > > persists the changes to zookeeper and closes all panels. Should we > > change > > the buttons or button text? What should the different buttons do? One > > idea could be to just skip saving to a local copy, meaning hitting the > > save > > button persists changes in that panel to zookeeper. Another idea could > > be > > to get rid of the save buttons on child panels and changes to the form > > would immediately update the local copy. In this case we would likely > > need > > an indicator that there are changes to be saved (or should we have that > > no > > matter what?). Other ideas? > > > > There is also the issue of being able to discard changes and go back to > > what they were before. Now you can close a child or primary panel but > > you > > discard all changes in that panel and all changes period in the case of > > the > > primary panel. We could be to expose a revert link or button for each > > form > > input (a lot of work probably). Other ideas? > > > > Ryan >
[GitHub] metron issue #772: METRON-1209: Make stellar repl take logging properties, l...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/772 +1 ---
[GitHub] metron issue #772: METRON-1209: Make stellar repl take logging properties, l...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/772 As a note : ```bash -> % mvn exec:java \ -Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell" -Dexec.args="-l src/test/resources/log4j.properties" ``` Is how to run this with mv:exec from the stellar-common directory. I also changed the log4j.properties to DEBUG from error. ---
Re: [DISCUSS] Is there a reason for separate Management & Alerts UIs?
I think the main reason historically is that each UI has different use cases and user roles. The Management UI will mainly be used by an Security Platform Engineer, while the Alerts UI will be used by a SOC Analyst, Investigator or Manager. That being said, I am not against a single, unified UI, as long as it is paired with appropriate role based access controls. On Thu, Sep 28, 2017 at 12:10 PM Laurens Vetswrote: > As the subject says, is there a specific reason to have the Management & > Alerts UI separate? > > Having another option under "Operations" called "Alerts" in the > Management UI seems to make more sense to me... If it's because they are > called Management UI and Alerts UI, maybe we should make it more general > and name it Metron UI? >
Re: [DISCUSS] How should Management UI save changes?
> Maybe change the text on the button on the primary panel to "write" instead of "save"? Another option would be to call it "Apply". > Also, I want wider child panels in the management UI if at all possible. > Especially the "RAW JSON" feels cramped. Yes, I agree. It seems to odd to me that the main work area in the Management UI is presented as a side panel that consumes a minority portion of the screen real estate. On Thu, Sep 28, 2017 at 12:30 PM Laurens Vetswrote: > Maybe change the text on the button on the primary panel to "write" > instead of "save"? > > Also, I want wider child panels in the management UI if at all possible. > Especially the "RAW JSON" feels cramped. > > On 2017-09-20 14:37, Ryan Merriman wrote: > > Recently @nickwallen brought up some good points about the usability of > > the > > Management UI here: > > https://github.com/apache/metron/pull/737#issuecomment-330632113. The > > issues he brings up apply to all child panels so I think it makes sense > > to > > agree on a common approach and apply it to all of them. > > > > Most child panels have a save button that saves changes to the local > > (browser) copy of the config. The save button on the primary panel > > persists the changes to zookeeper and closes all panels. Should we > > change > > the buttons or button text? What should the different buttons do? One > > idea could be to just skip saving to a local copy, meaning hitting the > > save > > button persists changes in that panel to zookeeper. Another idea could > > be > > to get rid of the save buttons on child panels and changes to the form > > would immediately update the local copy. In this case we would likely > > need > > an indicator that there are changes to be saved (or should we have that > > no > > matter what?). Other ideas? > > > > There is also the issue of being able to discard changes and go back to > > what they were before. Now you can close a child or primary panel but > > you > > discard all changes in that panel and all changes period in the case of > > the > > primary panel. We could be to expose a revert link or button for each > > form > > input (a lot of work probably). Other ideas? > > > > Ryan >
[GitHub] metron pull request #780: METRON-1220: Create documentation around alert nes...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/780#discussion_r142215715 --- Diff: metron-platform/metron-indexing/README.md --- @@ -163,6 +163,36 @@ Both of these functions are handled under the hood. In addition, an API endpoint is added for the meta alert specific features of creation and going from meta alert to alert. The denormalization handles the case of going from meta alert to alert automatically. +With Elasticsearch 2.x, there is an additional requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See [Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields) for more information + +Definition of the expected field: +``` + "alert": { +"type": "nested" + } +``` + +Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor): + +Exception seen: +``` +QueryParsingException[[nested] failed to find nested object under path [alert]]; +``` + +To put a new template into Elasticsearch to resolve this issue, update the template with the new field: +``` +curl -XPUT 'http://node1:9200/$SENSOR*/_mapping/$SENSOR_doc' -d ' --- End diff -- Can we make it more clear that `node1:9200` is the Elasticsearch API host:port that may be different outside of Full Dev? There may be a better way, but maybe something like the following. If I install Metron using the MPack, do we call the Elasticsearch REST API something in there that might make sense here? ``` export ELASTICSEARCH="node1:9200" curl -XPUT 'http://$ELASTICSEARCH/$SENSOR*/_mapping/$SENSOR_doc ... ``` ---
[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/780 > Honestly, I think it should at the very least be linked from the documentation which tells people how to create their own parsers...which we do not seem to have. ;) Yes @cestella, I was thinking the same thing too. Right now, I think that means just updating the Squid Getting Started blog series in the Metron Wiki/Blogs. We really should capture that entire Squid Getting Started Guide in our version controlled docs at some point. ---
[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/780 Also, should be linked from metron-parsers makes sense IMO as any time you are adding a new sensor, you're going to need to add this field. This is the use-case that keeps biting us in the butt over the last week. ---
[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/780 Honestly, I think it should at the very least be linked from the documentation which tells people how to create their own parsers...which we do not seem to have. ;) ---
[GitHub] metron issue #780: METRON-1220: Create documentation around alert nested fie...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/780 > Where should this live? I have this put up, so people can see what it says, but I'm pretty sure it should live somewhere else, under a different heading. I'm just not sure what the best fit is. Whichever README this lands in, I would prefer to see it in a section entitled something like "Using Metron with Elasticsearch 2.x". Does this make sense in the (currently non-existent) `metron-elasticsearch/README.md`? We could link to `metron-elasticsearch/README.md` from the current location (**The MetaAlertDao** section) if you think that it makes sense. My thought is that a user encountering this problem, won't even know what the `MetaAlertDao` is, so I think the current location is not very user friendly. ---
[GitHub] metron issue #737: METRON-1161: Add ability to edit parser command line opti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/737 So, I do believe the default values for num workers and ackers should be taken from the storm config if unspecified by us. Providing defaults when storm already has defaults for these properties seems wrong IMO. I think that's what you are asking. ---
Re: [DISCUSS] Build broken due to transitive dependencies
I might have spoken too soon. This is what I see now on 0.4.1-release: ... [INFO] metron-contrib . SUCCESS [ 0.006 s] [INFO] metron-docker .. SUCCESS [ 3.088 s] [INFO] metron-interface ... SUCCESS [ 0.057 s] [INFO] metron-config .. FAILURE [06:54 min] [INFO] metron-alerts .. SUCCESS [03:44 min] [INFO] metron-rest-client . SUCCESS [ 0.411 s] [INFO] metron-rest SUCCESS [ 26.628 s] [INFO] site-book .. SUCCESS [ 1.136 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 06:56 min (Wall Clock) [INFO] Finished at: 2017-10-02T16:33:39+00:00 [INFO] Final Memory: 240M/3203M [INFO] [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (ng build) on project metron-config: Failed to run task: 'npm run build' failed. (error code 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :metron-config On 2017-10-02 08:16, Laurens Vets wrote: I can confirm 0.4.1 (on CentOS 6!) builds for me as well. Are we sure it isn't due to the version of node shipped with the OS? On 2017-10-02 08:04, zeo...@gmail.com wrote: Hmm, 0.4.1 built fine for me. Jon On Mon, Oct 2, 2017 at 10:44 AM Casey Stellawrote: Ok, the build is broken in metron-config due to some transitive changes that happened in npm-land: [INFO] /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) [INFO] ^ [INFO] Error: Cyclic dependency: "[object Object]" [INFO] at visit (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13) [INFO] at visit (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9) Evidently one of our transitive dependencies has changed and we have ended up with a cyclic dependency. I'm not sure where or why yet, but I believe this breaks both master and our 0.4.1 release (I haven't confirmed this yet, but I strongly suspect). While the good work of tracking down this specific error is done, I'd like to bring up a broader discussion point: our practice of not fixing versions for our node dependencies. This is, in effect, causing a few problems: - We do not have a consistent, repeatable build. - We set ourselves up for possible license violation without knowing about it (a transitive dependency changes its license) As we stand, we have a release which doesn't not build after we have released it and tested it. It seems to me that we should at a minimum as a stopgap: - fix the versions of our dependencies so that they are in a working state - consider a point release to get a working build. I guess my questions to those of us with more javascript and UI experience is as follows: - Does fixing the version of our dependencies actually fix the problem transitively? - IF not, then how do we get a version of a build which is consistent and repeatable and does not expose us to downstream licensing issues? Thanks, Casey
Re: [DISCUSS] Build broken due to transitive dependencies
Well, that's a prescient and didn't seem to get any love. On Mon, Oct 2, 2017 at 11:53 AM, Otto Fowlerwrote: > http://mail-archives.apache.org/mod_mbox/metron-dev/ > 201708.mbox/%3cCAGh_R=xvLB3tg_j3FhupyMz4iUsiXmNbagHKZ= > gtfdc8u4b...@mail.gmail.com%3e > > > On October 2, 2017 at 11:50:48, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > 1. My failing builds are working again, both my branch off of master and > my pr checkouts, all that was required was a mvn clean package > 2. There was some discussion on yarn recently in a pr, though I can’t > remember which one > > > On October 2, 2017 at 11:45:48, Casey Stella (ceste...@gmail.com) wrote: > > Yeah, seems like it might be worth while to seriously investigate > migrating to yarn if that gets us a consistent build. > > On Mon, Oct 2, 2017 at 11:37 AM, Otto Fowler > wrote: > >> [WARNING] npm WARN deprecated bower@1.8.2: ...psst! Your project can >> stop working at any moment because its dependencies can change. Prevent >> this by migrating to Yarn: https://bower.io/blog/2017/how >> -to-migrate-away-from-bower/ >> >> >> On October 2, 2017 at 11:29:50, Casey Stella (ceste...@gmail.com) wrote: >> >> Ok, I can verify that 0.4.1 did build for me (took forever with vagrant >> running too). It's probably due to a dependency that was added after the >> release (whew!). Hmm, now that I've build master again, the problem seems >> to have gone away and the build is working again. >> >> On Mon, Oct 2, 2017 at 11:04 AM, zeo...@gmail.com >> wrote: >> >> > Hmm, 0.4.1 built fine for me. >> > >> > Jon >> > >> > On Mon, Oct 2, 2017 at 10:44 AM Casey Stella >> wrote: >> > >> > > Ok, the build is broken in metron-config due to some transitive >> changes >> > > that happened in npm-land: >> > > >> > > [INFO] >> > > >> > > /Users/cstella/Documents/workspace/metron/fork/incubator- >> metron/metron- >> > interface/metron-config/node_modules/toposort/index.js:32 >> > > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) >> > > [INFO] ^ >> > > [INFO] Error: Cyclic dependency: "[object Object]" >> > > [INFO] at visit >> > > >> > > (/Users/cstella/Documents/workspace/metron/fork/incubator- >> metron/metron- >> > interface/metron-config/node_modules/toposort/index.js:32:13) >> > > [INFO] at visit >> > > >> > > (/Users/cstella/Documents/workspace/metron/fork/incubator- >> metron/metron- >> > interface/metron-config/node_modules/toposort/index.js:48:9) >> > > >> > > Evidently one of our transitive dependencies has changed and we have >> > ended >> > > up with a cyclic dependency. I'm not sure where or why yet, but I >> > believe >> > > this breaks both master and our 0.4.1 release (I haven't confirmed >> this >> > > yet, but I strongly suspect). >> > > >> > > While the good work of tracking down this specific error is done, I'd >> > like >> > > to bring up a broader discussion point: our practice of not fixing >> > versions >> > > for our node dependencies. This is, in effect, causing a few problems: >> > > >> > > - We do not have a consistent, repeatable build. >> > > - We set ourselves up for possible license violation without knowing >> > > about it (a transitive dependency changes its license) >> > > >> > > As we stand, we have a release which doesn't not build after we have >> > > released it and tested it. It seems to me that we should at a minimum >> > as a >> > > stopgap: >> > > >> > > - fix the versions of our dependencies so that they are in a working >> > > state >> > > - consider a point release to get a working build. >> > > >> > > I guess my questions to those of us with more javascript and UI >> > experience >> > > is as follows: >> > > >> > > - Does fixing the version of our dependencies actually fix the problem >> > > transitively? >> > > - IF not, then how do we get a version of a build which is consistent >> > > and repeatable and does not expose us to downstream licensing issues? >> > > >> > > Thanks, >> > > >> > > Casey >> > > >> > -- >> > >> > Jon >> > >> >> >
Re: [DISCUSS] Build broken due to transitive dependencies
http://mail-archives.apache.org/mod_mbox/metron-dev/201708.mbox/%3cCAGh_R=xvLB3tg_j3FhupyMz4iUsiXmNbagHKZ=gtfdc8u4b...@mail.gmail.com%3e On October 2, 2017 at 11:50:48, Otto Fowler (ottobackwa...@gmail.com) wrote: 1. My failing builds are working again, both my branch off of master and my pr checkouts, all that was required was a mvn clean package 2. There was some discussion on yarn recently in a pr, though I can’t remember which one On October 2, 2017 at 11:45:48, Casey Stella (ceste...@gmail.com) wrote: Yeah, seems like it might be worth while to seriously investigate migrating to yarn if that gets us a consistent build. On Mon, Oct 2, 2017 at 11:37 AM, Otto Fowlerwrote: > [WARNING] npm WARN deprecated bower@1.8.2: ...psst! Your project can stop > working at any moment because its dependencies can change. Prevent this by > migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from- > bower/ > > > On October 2, 2017 at 11:29:50, Casey Stella (ceste...@gmail.com) wrote: > > Ok, I can verify that 0.4.1 did build for me (took forever with vagrant > running too). It's probably due to a dependency that was added after the > release (whew!). Hmm, now that I've build master again, the problem seems > to have gone away and the build is working again. > > On Mon, Oct 2, 2017 at 11:04 AM, zeo...@gmail.com > wrote: > > > Hmm, 0.4.1 built fine for me. > > > > Jon > > > > On Mon, Oct 2, 2017 at 10:44 AM Casey Stella wrote: > > > > > Ok, the build is broken in metron-config due to some transitive changes > > > that happened in npm-land: > > > > > > [INFO] > > > > > > /Users/cstella/Documents/workspace/metron/fork/ > incubator-metron/metron- > > interface/metron-config/node_modules/toposort/index.js:32 > > > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) > > > [INFO] ^ > > > [INFO] Error: Cyclic dependency: "[object Object]" > > > [INFO] at visit > > > > > > (/Users/cstella/Documents/workspace/metron/fork/ > incubator-metron/metron- > > interface/metron-config/node_modules/toposort/index.js:32:13) > > > [INFO] at visit > > > > > > (/Users/cstella/Documents/workspace/metron/fork/ > incubator-metron/metron- > > interface/metron-config/node_modules/toposort/index.js:48:9) > > > > > > Evidently one of our transitive dependencies has changed and we have > > ended > > > up with a cyclic dependency. I'm not sure where or why yet, but I > > believe > > > this breaks both master and our 0.4.1 release (I haven't confirmed this > > > yet, but I strongly suspect). > > > > > > While the good work of tracking down this specific error is done, I'd > > like > > > to bring up a broader discussion point: our practice of not fixing > > versions > > > for our node dependencies. This is, in effect, causing a few problems: > > > > > > - We do not have a consistent, repeatable build. > > > - We set ourselves up for possible license violation without knowing > > > about it (a transitive dependency changes its license) > > > > > > As we stand, we have a release which doesn't not build after we have > > > released it and tested it. It seems to me that we should at a minimum > > as a > > > stopgap: > > > > > > - fix the versions of our dependencies so that they are in a working > > > state > > > - consider a point release to get a working build. > > > > > > I guess my questions to those of us with more javascript and UI > > experience > > > is as follows: > > > > > > - Does fixing the version of our dependencies actually fix the problem > > > transitively? > > > - IF not, then how do we get a version of a build which is consistent > > > and repeatable and does not expose us to downstream licensing issues? > > > > > > Thanks, > > > > > > Casey > > > > > -- > > > > Jon > > > >
Re: [DISCUSS] Build broken due to transitive dependencies
1. My failing builds are working again, both my branch off of master and my pr checkouts, all that was required was a mvn clean package 2. There was some discussion on yarn recently in a pr, though I can’t remember which one On October 2, 2017 at 11:45:48, Casey Stella (ceste...@gmail.com) wrote: Yeah, seems like it might be worth while to seriously investigate migrating to yarn if that gets us a consistent build. On Mon, Oct 2, 2017 at 11:37 AM, Otto Fowlerwrote: > [WARNING] npm WARN deprecated bower@1.8.2: ...psst! Your project can stop > working at any moment because its dependencies can change. Prevent this by > migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from- > bower/ > > > On October 2, 2017 at 11:29:50, Casey Stella (ceste...@gmail.com) wrote: > > Ok, I can verify that 0.4.1 did build for me (took forever with vagrant > running too). It's probably due to a dependency that was added after the > release (whew!). Hmm, now that I've build master again, the problem seems > to have gone away and the build is working again. > > On Mon, Oct 2, 2017 at 11:04 AM, zeo...@gmail.com > wrote: > > > Hmm, 0.4.1 built fine for me. > > > > Jon > > > > On Mon, Oct 2, 2017 at 10:44 AM Casey Stella wrote: > > > > > Ok, the build is broken in metron-config due to some transitive changes > > > that happened in npm-land: > > > > > > [INFO] > > > > > > /Users/cstella/Documents/workspace/metron/fork/ > incubator-metron/metron- > > interface/metron-config/node_modules/toposort/index.js:32 > > > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) > > > [INFO] ^ > > > [INFO] Error: Cyclic dependency: "[object Object]" > > > [INFO] at visit > > > > > > (/Users/cstella/Documents/workspace/metron/fork/ > incubator-metron/metron- > > interface/metron-config/node_modules/toposort/index.js:32:13) > > > [INFO] at visit > > > > > > (/Users/cstella/Documents/workspace/metron/fork/ > incubator-metron/metron- > > interface/metron-config/node_modules/toposort/index.js:48:9) > > > > > > Evidently one of our transitive dependencies has changed and we have > > ended > > > up with a cyclic dependency. I'm not sure where or why yet, but I > > believe > > > this breaks both master and our 0.4.1 release (I haven't confirmed this > > > yet, but I strongly suspect). > > > > > > While the good work of tracking down this specific error is done, I'd > > like > > > to bring up a broader discussion point: our practice of not fixing > > versions > > > for our node dependencies. This is, in effect, causing a few problems: > > > > > > - We do not have a consistent, repeatable build. > > > - We set ourselves up for possible license violation without knowing > > > about it (a transitive dependency changes its license) > > > > > > As we stand, we have a release which doesn't not build after we have > > > released it and tested it. It seems to me that we should at a minimum > > as a > > > stopgap: > > > > > > - fix the versions of our dependencies so that they are in a working > > > state > > > - consider a point release to get a working build. > > > > > > I guess my questions to those of us with more javascript and UI > > experience > > > is as follows: > > > > > > - Does fixing the version of our dependencies actually fix the problem > > > transitively? > > > - IF not, then how do we get a version of a build which is consistent > > > and repeatable and does not expose us to downstream licensing issues? > > > > > > Thanks, > > > > > > Casey > > > > > -- > > > > Jon > > > >
Re: [DISCUSS] Build broken due to transitive dependencies
Yeah, seems like it might be worth while to seriously investigate migrating to yarn if that gets us a consistent build. On Mon, Oct 2, 2017 at 11:37 AM, Otto Fowlerwrote: > [WARNING] npm WARN deprecated bower@1.8.2: ...psst! Your project can stop > working at any moment because its dependencies can change. Prevent this by > migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from- > bower/ > > > On October 2, 2017 at 11:29:50, Casey Stella (ceste...@gmail.com) wrote: > > Ok, I can verify that 0.4.1 did build for me (took forever with vagrant > running too). It's probably due to a dependency that was added after the > release (whew!). Hmm, now that I've build master again, the problem seems > to have gone away and the build is working again. > > On Mon, Oct 2, 2017 at 11:04 AM, zeo...@gmail.com > wrote: > > > Hmm, 0.4.1 built fine for me. > > > > Jon > > > > On Mon, Oct 2, 2017 at 10:44 AM Casey Stella > wrote: > > > > > Ok, the build is broken in metron-config due to some transitive > changes > > > that happened in npm-land: > > > > > > [INFO] > > > > > > /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron- > > > interface/metron-config/node_modules/toposort/index.js:32 > > > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) > > > [INFO] ^ > > > [INFO] Error: Cyclic dependency: "[object Object]" > > > [INFO] at visit > > > > > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron- > > > interface/metron-config/node_modules/toposort/index.js:32:13) > > > [INFO] at visit > > > > > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron- > > > interface/metron-config/node_modules/toposort/index.js:48:9) > > > > > > Evidently one of our transitive dependencies has changed and we have > > ended > > > up with a cyclic dependency. I'm not sure where or why yet, but I > > believe > > > this breaks both master and our 0.4.1 release (I haven't confirmed > this > > > yet, but I strongly suspect). > > > > > > While the good work of tracking down this specific error is done, I'd > > like > > > to bring up a broader discussion point: our practice of not fixing > > versions > > > for our node dependencies. This is, in effect, causing a few problems: > > > > > > - We do not have a consistent, repeatable build. > > > - We set ourselves up for possible license violation without knowing > > > about it (a transitive dependency changes its license) > > > > > > As we stand, we have a release which doesn't not build after we have > > > released it and tested it. It seems to me that we should at a minimum > > as a > > > stopgap: > > > > > > - fix the versions of our dependencies so that they are in a working > > > state > > > - consider a point release to get a working build. > > > > > > I guess my questions to those of us with more javascript and UI > > experience > > > is as follows: > > > > > > - Does fixing the version of our dependencies actually fix the problem > > > transitively? > > > - IF not, then how do we get a version of a build which is consistent > > > and repeatable and does not expose us to downstream licensing issues? > > > > > > Thanks, > > > > > > Casey > > > > > -- > > > > Jon > > > >
Re: [DISCUSS] Build broken due to transitive dependencies
[WARNING] npm WARN deprecated bower@1.8.2: ...psst! Your project can stop working at any moment because its dependencies can change. Prevent this by migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from-bower/ On October 2, 2017 at 11:29:50, Casey Stella (ceste...@gmail.com) wrote: Ok, I can verify that 0.4.1 did build for me (took forever with vagrant running too). It's probably due to a dependency that was added after the release (whew!). Hmm, now that I've build master again, the problem seems to have gone away and the build is working again. On Mon, Oct 2, 2017 at 11:04 AM, zeo...@gmail.comwrote: > Hmm, 0.4.1 built fine for me. > > Jon > > On Mon, Oct 2, 2017 at 10:44 AM Casey Stella wrote: > > > Ok, the build is broken in metron-config due to some transitive changes > > that happened in npm-land: > > > > [INFO] > > > > /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron- > interface/metron-config/node_modules/toposort/index.js:32 > > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) > > [INFO] ^ > > [INFO] Error: Cyclic dependency: "[object Object]" > > [INFO] at visit > > > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron- > interface/metron-config/node_modules/toposort/index.js:32:13) > > [INFO] at visit > > > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron- > interface/metron-config/node_modules/toposort/index.js:48:9) > > > > Evidently one of our transitive dependencies has changed and we have > ended > > up with a cyclic dependency. I'm not sure where or why yet, but I > believe > > this breaks both master and our 0.4.1 release (I haven't confirmed this > > yet, but I strongly suspect). > > > > While the good work of tracking down this specific error is done, I'd > like > > to bring up a broader discussion point: our practice of not fixing > versions > > for our node dependencies. This is, in effect, causing a few problems: > > > > - We do not have a consistent, repeatable build. > > - We set ourselves up for possible license violation without knowing > > about it (a transitive dependency changes its license) > > > > As we stand, we have a release which doesn't not build after we have > > released it and tested it. It seems to me that we should at a minimum > as a > > stopgap: > > > > - fix the versions of our dependencies so that they are in a working > > state > > - consider a point release to get a working build. > > > > I guess my questions to those of us with more javascript and UI > experience > > is as follows: > > > > - Does fixing the version of our dependencies actually fix the problem > > transitively? > > - IF not, then how do we get a version of a build which is consistent > > and repeatable and does not expose us to downstream licensing issues? > > > > Thanks, > > > > Casey > > > -- > > Jon >
Re: [DISCUSS] Build broken due to transitive dependencies
We download and install node locally. But you can test this by : > git fetch apache > git checkout -b build-test apache/master > mvn clean package -DskipTests On October 2, 2017 at 11:17:03, Laurens Vets (laur...@daemon.be) wrote: I can confirm 0.4.1 (on CentOS 6!) builds for me as well. Are we sure it isn't due to the version of node shipped with the OS? On 2017-10-02 08:04, zeo...@gmail.com wrote: > Hmm, 0.4.1 built fine for me. > > Jon > > On Mon, Oct 2, 2017 at 10:44 AM Casey Stella> wrote: > >> Ok, the build is broken in metron-config due to some transitive >> changes >> that happened in npm-land: >> >> [INFO] >> >> /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 >> [INFO] throw new Error('Cyclic dependency: >> '+JSON.stringify(node)) >> [INFO] ^ >> [INFO] Error: Cyclic dependency: "[object Object]" >> [INFO] at visit >> >> (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13) >> [INFO] at visit >> >> (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9) >> >> Evidently one of our transitive dependencies has changed and we have >> ended >> up with a cyclic dependency. I'm not sure where or why yet, but I >> believe >> this breaks both master and our 0.4.1 release (I haven't confirmed >> this >> yet, but I strongly suspect). >> >> While the good work of tracking down this specific error is done, I'd >> like >> to bring up a broader discussion point: our practice of not fixing >> versions >> for our node dependencies. This is, in effect, causing a few >> problems: >> >> - We do not have a consistent, repeatable build. >> - We set ourselves up for possible license violation without >> knowing >> about it (a transitive dependency changes its license) >> >> As we stand, we have a release which doesn't not build after we have >> released it and tested it. It seems to me that we should at a minimum >> as a >> stopgap: >> >> - fix the versions of our dependencies so that they are in a >> working >> state >> - consider a point release to get a working build. >> >> I guess my questions to those of us with more javascript and UI >> experience >> is as follows: >> >> - Does fixing the version of our dependencies actually fix the >> problem >> transitively? >> - IF not, then how do we get a version of a build which is >> consistent >> and repeatable and does not expose us to downstream licensing >> issues? >> >> Thanks, >> >> Casey >>
Re: [DISCUSS] Build broken due to transitive dependencies
I can confirm 0.4.1 (on CentOS 6!) builds for me as well. Are we sure it isn't due to the version of node shipped with the OS? On 2017-10-02 08:04, zeo...@gmail.com wrote: Hmm, 0.4.1 built fine for me. Jon On Mon, Oct 2, 2017 at 10:44 AM Casey Stellawrote: Ok, the build is broken in metron-config due to some transitive changes that happened in npm-land: [INFO] /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) [INFO] ^ [INFO] Error: Cyclic dependency: "[object Object]" [INFO] at visit (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13) [INFO] at visit (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9) Evidently one of our transitive dependencies has changed and we have ended up with a cyclic dependency. I'm not sure where or why yet, but I believe this breaks both master and our 0.4.1 release (I haven't confirmed this yet, but I strongly suspect). While the good work of tracking down this specific error is done, I'd like to bring up a broader discussion point: our practice of not fixing versions for our node dependencies. This is, in effect, causing a few problems: - We do not have a consistent, repeatable build. - We set ourselves up for possible license violation without knowing about it (a transitive dependency changes its license) As we stand, we have a release which doesn't not build after we have released it and tested it. It seems to me that we should at a minimum as a stopgap: - fix the versions of our dependencies so that they are in a working state - consider a point release to get a working build. I guess my questions to those of us with more javascript and UI experience is as follows: - Does fixing the version of our dependencies actually fix the problem transitively? - IF not, then how do we get a version of a build which is consistent and repeatable and does not expose us to downstream licensing issues? Thanks, Casey
Re: [DISCUSS] Build broken due to transitive dependencies
Maybe we added a dependency or something since? On October 2, 2017 at 11:04:16, zeo...@gmail.com (zeo...@gmail.com) wrote: Hmm, 0.4.1 built fine for me. Jon On Mon, Oct 2, 2017 at 10:44 AM Casey Stellawrote: > Ok, the build is broken in metron-config due to some transitive changes > that happened in npm-land: > > [INFO] > > /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) > [INFO] ^ > [INFO] Error: Cyclic dependency: "[object Object]" > [INFO] at visit > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13) > [INFO] at visit > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9) > > Evidently one of our transitive dependencies has changed and we have ended > up with a cyclic dependency. I'm not sure where or why yet, but I believe > this breaks both master and our 0.4.1 release (I haven't confirmed this > yet, but I strongly suspect). > > While the good work of tracking down this specific error is done, I'd like > to bring up a broader discussion point: our practice of not fixing versions > for our node dependencies. This is, in effect, causing a few problems: > > - We do not have a consistent, repeatable build. > - We set ourselves up for possible license violation without knowing > about it (a transitive dependency changes its license) > > As we stand, we have a release which doesn't not build after we have > released it and tested it. It seems to me that we should at a minimum as a > stopgap: > > - fix the versions of our dependencies so that they are in a working > state > - consider a point release to get a working build. > > I guess my questions to those of us with more javascript and UI experience > is as follows: > > - Does fixing the version of our dependencies actually fix the problem > transitively? > - IF not, then how do we get a version of a build which is consistent > and repeatable and does not expose us to downstream licensing issues? > > Thanks, > > Casey > -- Jon
Re: [DISCUSS] Build broken due to transitive dependencies
Hmm, 0.4.1 built fine for me. Jon On Mon, Oct 2, 2017 at 10:44 AM Casey Stellawrote: > Ok, the build is broken in metron-config due to some transitive changes > that happened in npm-land: > > [INFO] > > /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 > [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) > [INFO] ^ > [INFO] Error: Cyclic dependency: "[object Object]" > [INFO] at visit > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13) > [INFO] at visit > > (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9) > > Evidently one of our transitive dependencies has changed and we have ended > up with a cyclic dependency. I'm not sure where or why yet, but I believe > this breaks both master and our 0.4.1 release (I haven't confirmed this > yet, but I strongly suspect). > > While the good work of tracking down this specific error is done, I'd like > to bring up a broader discussion point: our practice of not fixing versions > for our node dependencies. This is, in effect, causing a few problems: > >- We do not have a consistent, repeatable build. >- We set ourselves up for possible license violation without knowing >about it (a transitive dependency changes its license) > > As we stand, we have a release which doesn't not build after we have > released it and tested it. It seems to me that we should at a minimum as a > stopgap: > >- fix the versions of our dependencies so that they are in a working >state >- consider a point release to get a working build. > > I guess my questions to those of us with more javascript and UI experience > is as follows: > >- Does fixing the version of our dependencies actually fix the problem >transitively? >- IF not, then how do we get a version of a build which is consistent >and repeatable and does not expose us to downstream licensing issues? > > Thanks, > > Casey > -- Jon
[DISCUSS] Build broken due to transitive dependencies
Ok, the build is broken in metron-config due to some transitive changes that happened in npm-land: [INFO] /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 [INFO] throw new Error('Cyclic dependency: '+JSON.stringify(node)) [INFO] ^ [INFO] Error: Cyclic dependency: "[object Object]" [INFO] at visit (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13) [INFO] at visit (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9) Evidently one of our transitive dependencies has changed and we have ended up with a cyclic dependency. I'm not sure where or why yet, but I believe this breaks both master and our 0.4.1 release (I haven't confirmed this yet, but I strongly suspect). While the good work of tracking down this specific error is done, I'd like to bring up a broader discussion point: our practice of not fixing versions for our node dependencies. This is, in effect, causing a few problems: - We do not have a consistent, repeatable build. - We set ourselves up for possible license violation without knowing about it (a transitive dependency changes its license) As we stand, we have a release which doesn't not build after we have released it and tested it. It seems to me that we should at a minimum as a stopgap: - fix the versions of our dependencies so that they are in a working state - consider a point release to get a working build. I guess my questions to those of us with more javascript and UI experience is as follows: - Does fixing the version of our dependencies actually fix the problem transitively? - IF not, then how do we get a version of a build which is consistent and repeatable and does not expose us to downstream licensing issues? Thanks, Casey
[GitHub] metron issue #771: METRON-1204: UI does not time out after being idle, but s...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/771 +1 by inspection, great job @merrimanr ---
[GitHub] metron pull request #779: METRON-1218: Metron REST should return better erro...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/779#discussion_r142149285 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/RestExceptionHandler.java --- @@ -45,4 +45,14 @@ private HttpStatus getStatus(HttpServletRequest request) { } return HttpStatus.valueOf(statusCode); } + + private String getFullMessage(Throwable ex) { +String fullMessage = ex.getMessage(); +Throwable cause = ex.getCause(); --- End diff -- This is a blocker imo. ---
[GitHub] metron pull request #779: METRON-1218: Metron REST should return better erro...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/779#discussion_r142149083 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/RestExceptionHandler.java --- @@ -45,4 +45,14 @@ private HttpStatus getStatus(HttpServletRequest request) { } return HttpStatus.valueOf(statusCode); } + + private String getFullMessage(Throwable ex) { +String fullMessage = ex.getMessage(); +Throwable cause = ex.getCause(); --- End diff -- I think so. The get cause recursion is an oldie but a goodie. ---
[GitHub] metron pull request #779: METRON-1218: Metron REST should return better erro...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/779#discussion_r142147088 --- Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/RestExceptionHandler.java --- @@ -45,4 +45,14 @@ private HttpStatus getStatus(HttpServletRequest request) { } return HttpStatus.valueOf(statusCode); } + + private String getFullMessage(Throwable ex) { +String fullMessage = ex.getMessage(); +Throwable cause = ex.getCause(); --- End diff -- Yeah, I think this method is just `ExceptionUtils.getRootCause.getMessage()`, isn't it? ---
[GitHub] metron issue #772: METRON-1209: Make stellar repl take logging properties, l...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/772 Oh, sorry, forgot to respond here @ottobackwards You can test it by passing in a log4j properties (let's say something that turns on DEBUG logging) file via -l and ensure that the REPL shows debug logging. Perhaps something like: ``` log4j.rootLogger=DEBUG,stdout log4j.threshhold=ALL log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2} (%F:%M(%L)) - %m%n log4j.appender.stdout.filter.1=org.apache.log4j.varia.StringMatchFilter log4j.appender.stdout.filter.1.StringToMatch=ExpiredTokenRemover log4j.appender.stdout.filter.1.AcceptOnMatch=false log4j.appender.stdout.filter.2=org.apache.log4j.varia.StringMatchFilter log4j.appender.stdout.filter.2.StringToMatch=interrupted log4j.appender.stdout.filter.2.AcceptOnMatch=false ``` ---
[GitHub] metron pull request #732: METRON-632: Added validation of "shew.enrichmentTy...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/732 ---