Note that I cleaned up the ansible scripts that install C++ 11 in my latest PR <https://github.com/apache/metron/pull/847/files>, but it's not super relevant to this conversation.
Jon On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com <zeo...@gmail.com> wrote: > That was also required for bro 2.5.2, so I did that here > <https://github.com/apache/metron/commit/59fe1b453279bf5c7df627ea656c762b3a98e777>. > Feel free to reuse the approach elsewhere > > Jon > > On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> First issue is that we need c++ 11 on centos 6.8 >> >> >> >> On November 27, 2017 at 09:53:55, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >> Well, that’s good news on that issue. Reproducing the problem is half way >> to solving it, right? >> >> I would still say there are some systemic things going on that have >> manifested in a variety of ways on both the users and dev list, so it’s >> worth us having a good look at a more robust approach to node dependencies >> (both npm ones, and the native ones) >> >> Simon >> >> On 27 Nov 2017, at 13:30, Otto Fowler <ottobackwa...@gmail.com> wrote: >> >> I can reproduce the failure in out ansible docker build container, which >> is >> also centos. >> The issue is building our node on centos in all these cases. >> >> >> >> On November 27, 2017 at 07:02:51, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >> Thinking about this, doesn’t our build plugin explicitly install it’s own >> node? So actually all the node version things may be a red herring, since >> this is under our control through the pom. Not sure if we actually >> exercising this control. It seems that some of the errors people report >> are >> more to do with compilation failures for native node modules, which is >> doesn’t pin (i.e. things like system library dependencies). I’m not sure >> what we have in the dependency tree that requires complex native >> dependencies, but this might just be one of those node things we could doc >> around. >> >> This scenario would be fixed by standardising the build container. >> >> Yarn’s big thing is that it enables faster dependency resolution and local >> caching, right? It does not seem to address any of the problems we see, >> but >> sure, it’s the shiny new dependency system for node modules, which might >> make npm less horrible to deal with, so worth looking into. >> >> The other issue that I’ve seen people run into a lot is flat out download >> errors. This could be helped by finding our versions, maybe with yarn, but >> let’s face it, package-lock.json could also do that with npm, albeit with >> a >> slightly slower algorithm. However, short of bundling and hosting deps >> ourselves, I suspect the download errors are beyond our control, and >> certainly beyond the scope of this project (fix maven, fix npm, fix all >> the >> node hosting servers…) >> >> Simon >> >> >> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda < >> raghumitra....@gmail.com> >> wrote: >> > >> > Looking at some of the build failure emails and past experience i >> > would suggest having a node & npm version check in our build scripts >> > and moving dependency management to yarn. >> > >> > We need not restrict the build to a specific version of node & npm but >> > we can surely suggest a min version required to build UI successfully. >> > >> > -Raghu >> > >> > >> > >> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball >> > <si...@simonellistonball.com> wrote: >> >> Agreeing with Nick, it seems like the main reason people are building >> themselves, and hitting all these environmental issues, is that we do not >> as a project produce binary release artefacts (the rpms which users could >> just install) and instead leave that for the commercial distributors to >> do. >> >> >> >> Yarn may help with some of the dependency version issues we’re having, >> but not afaik with the core missing library headers / build tools / node >> and npm version issue, those would seem to fit a documentation fix and >> improvements to platform-info to flag the problems, so this can then be a >> pre-flight check tool as well as a diagnostic tool. >> >> >> >> Another option I would put on the table is to standardise our build >> environment, so that the non-java bits are run in a standard docker image >> or something fo the sort, that way we can take control of all the >> environmental and OS dependent pieces, much as we do right now with the >> rpm >> build sections of the mpack build. >> >> >> >> The challenge here will be adding the relevant maven support. At the >> moment we’re relying on the maven npm and node build plugins, this would >> likely need replacing with something custom and a challenge to support to >> go dow this route. >> >> >> >> Perhaps the real answer here is to push people who are just kicking the >> tyres towards a binary distribution, or at least rpm artefacts as part of >> the Apache release to give them a head start for a happy path on a known >> good OS environment. >> >> >> >> Simon >> >> >> >>> On 24 Nov 2017, at 16:01, Nick Allen <n...@nickallen.org> wrote: >> >>> >> >>> Yes, it is a problem. I think you've identified a couple important >> things >> >>> that we could address in parallel. I see these as challenges we need >> to >> >>> solve for the dev community. >> >>> >> >>> (1) NPM is causing us some major headaches. Which version do we >> require? >> >> >>> How do I install that version (on Mac, Windows, Linux)? Does YARN help >> >>> here at all? >> >>> >> >>> (2) Can we automate the prerequisite checks that we currently do >> manually >> >>> with `platform-info.sh`? An automated check could run and fail as part >> of >> >>> the build or deployment process. >> >>> >> >>> >> >>> >> >>> More importantly though is that users should not have to build Metron >> at >> >> >>> all. They should not have to worry about installing NPM and the rest >> of >> >>> the development tooling. Here are some options that are not mutually >> >>> exclusive. >> >>> >> >>> >> >>> (1) Create an image in Atlas that has Metron fully installed. A new >> user >> >> >>> could run single node Metron on their laptop with a single command and >> the >> >>> only prereqs would be Vagrant and Virtualbox. We could cut new images >> for >> >>> each Metron release. Or selectively cut new dev images from master as >> we >> >> >>> see fit. >> >>> >> >>> (2) Distribute the Metron RPMs (and the MPack tarball?) so that users >> can >> >>> install Metron on a cluster without having to build it. >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Fri, Nov 24, 2017 at 10:11 AM, Otto Fowler < >> ottobackwa...@gmail.com> >> >>> wrote: >> >>> >> >>>> It seems like it is getting *very* common for people to have trouble >> >>>> building recently. Errors with NPM and Node seen common, with fixes >> ranging >> >>>> from updating c/c++ libs to the version of npm/node. >> >>>> >> >>>> There has to be a better way to do this. >> >>>> >> >>>> - >> >>>> >> >>>> Are we out of date or missing requirements in our documentation? >> >>>> - >> >>>> >> >>>> Does our documentation need to be updated for building? >> >>>> - >> >>>> >> >>>> Is there a better way in maven to check the versions required for >> some >> >>>> of these things and fail faster with a better message? >> >>>> - >> >>>> >> >>>> Are we building correctly or are we asking for trouble? >> >>>> >> >>>> The ability to build metron is pretty important, and it seems that >> people >> >>>> are having a lot of trouble related to the new technologies in alerts >> and >> >>>> config ui. >> >>>> >> >> >> > -- > > Jon > -- Jon
