Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler 
Also, since I changed the profiles to not run the rpm docker if you are in
docker already ( and put the rpm tools into the ansible docker ) a while ago
we may be able to build world in the ansible image, and point folks having
issues to that….


On November 27, 2017 at 10:57:03, Otto Fowler (ottobackwa...@gmail.com)
wrote:

OK,
So I have >mvn clean package working in docker.
I want to try a couple of things and maybe I can throw a pr together.



On November 27, 2017 at 10:03:31, Otto Fowler (ottobackwa...@gmail.com)
wrote:

First issue is that we need c++ 11 on centos 6.8



On November 27, 2017 at 09:53:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Well, that’s good news on that issue. Reproducing the problem is half way
to solving it, right?

I would still say there are some systemic things going on that have
manifested in a variety of ways on both the users and dev list, so it’s
worth us having a good look at a more robust approach to node dependencies
(both npm ones, and the native ones)

Simon

On 27 Nov 2017, at 13:30, Otto Fowler  wrote:

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler 
OK,
So I have >mvn clean package working in docker.
I want to try a couple of things and maybe I can throw a pr together.



On November 27, 2017 at 10:03:31, Otto Fowler (ottobackwa...@gmail.com)
wrote:

First issue is that we need c++ 11 on centos 6.8



On November 27, 2017 at 09:53:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Well, that’s good news on that issue. Reproducing the problem is half way
to solving it, right?

I would still say there are some systemic things going on that have
manifested in a variety of ways on both the users and dev list, so it’s
worth us having a good look at a more robust approach to node dependencies
(both npm ones, and the native ones)

Simon

On 27 Nov 2017, at 13:30, Otto Fowler  wrote:

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve for the dev community.
>>>
>>> (1) NPM is causing us some major headaches. Which version do we require?

>>> How do I install that version (on Mac, Windows, Linux)? Does YARN help
>>> here at all?
>>>
>>> (2) Can we automate the prerequisite checks that we currently do
manually
>>> with `platform-info.sh`? An automated check

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread zeo...@gmail.com 
Note that I cleaned up the ansible scripts that install C++ 11 in my latest
PR , but it's not super
relevant to this conversation.

Jon

On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com  wrote:

> That was also required for bro 2.5.2, so I did that here
> .
> Feel free to reuse the approach elsewhere
>
> Jon
>
> On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
> wrote:
>
>> First issue is that we need c++ 11 on centos 6.8
>>
>>
>>
>> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Well, that’s good news on that issue. Reproducing the problem is half way
>> to solving it, right?
>>
>> I would still say there are some systemic things going on that have
>> manifested in a variety of ways on both the users and dev list, so it’s
>> worth us having a good look at a more robust approach to node dependencies
>> (both npm ones, and the native ones)
>>
>> Simon
>>
>> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>>
>> I can reproduce the failure in out ansible docker build container, which
>> is
>> also centos.
>> The issue is building our node on centos in all these cases.
>>
>>
>>
>> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Thinking about this, doesn’t our build plugin explicitly install it’s own
>> node? So actually all the node version things may be a red herring, since
>> this is under our control through the pom. Not sure if we actually
>> exercising this control. It seems that some of the errors people report
>> are
>> more to do with compilation failures for native node modules, which is
>> doesn’t pin (i.e. things like system library dependencies). I’m not sure
>> what we have in the dependency tree that requires complex native
>> dependencies, but this might just be one of those node things we could doc
>> around.
>>
>> This scenario would be fixed by standardising the build container.
>>
>> Yarn’s big thing is that it enables faster dependency resolution and local
>> caching, right? It does not seem to address any of the problems we see,
>> but
>> sure, it’s the shiny new dependency system for node modules, which might
>> make npm less horrible to deal with, so worth looking into.
>>
>> The other issue that I’ve seen people run into a lot is flat out download
>> errors. This could be helped by finding our versions, maybe with yarn, but
>> let’s face it, package-lock.json could also do that with npm, albeit with
>> a
>> slightly slower algorithm. However, short of bundling and hosting deps
>> ourselves, I suspect the download errors are beyond our control, and
>> certainly beyond the scope of this project (fix maven, fix npm, fix all
>> the
>> node hosting servers…)
>>
>> Simon
>>
>>
>> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
>> raghumitra@gmail.com>
>> wrote:
>> >
>> > Looking at some of the build failure emails and past experience i
>> > would suggest having a node & npm version check in our build scripts
>> > and moving dependency management to yarn.
>> >
>> > We need not restrict the build to a specific version of node & npm but
>> > we can surely suggest a min version required to build UI successfully.
>> >
>> > -Raghu
>> >
>> >
>> >
>> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>> >  wrote:
>> >> Agreeing with Nick, it seems like the main reason people are building
>> themselves, and hitting all these environmental issues, is that we do not
>> as a project produce binary release artefacts (the rpms which users could
>> just install) and instead leave that for the commercial distributors to
>> do.
>> >>
>> >> Yarn may help with some of the dependency version issues we’re having,
>> but not afaik with the core missing library headers / build tools / node
>> and npm version issue, those would seem to fit a documentation fix and
>> improvements to platform-info to flag the problems, so this can then be a
>> pre-flight check tool as well as a diagnostic tool.
>> >>
>> >> Another option I would put on the table is to standardise our build
>> environment, so that the non-java bits are run in a standard docker image
>> or something fo the sort, that way we can take control of all the
>> environmental and OS dependent pieces, much as we do right now with the
>> rpm
>> build sections of the mpack build.
>> >>
>> >> The challenge here will be adding the relevant maven support. At the
>> moment we’re relying on the maven npm and node build plugins, this would
>> likely need replacing with something custom and a challenge to support to
>> go dow this route.
>> >>
>> >> Perhaps the real answer here is to push people who are just kicking the
>> tyres towards a binary distribution, or at least rpm artefacts as part of
>> the Apache release to give them a

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread zeo...@gmail.com 
That was also required for bro 2.5.2, so I did that here
.
Feel free to reuse the approach elsewhere

Jon

On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
wrote:

> First issue is that we need c++ 11 on centos 6.8
>
>
>
> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Well, that’s good news on that issue. Reproducing the problem is half way
> to solving it, right?
>
> I would still say there are some systemic things going on that have
> manifested in a variety of ways on both the users and dev list, so it’s
> worth us having a good look at a more robust approach to node dependencies
> (both npm ones, and the native ones)
>
> Simon
>
> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>
> I can reproduce the failure in out ansible docker build container, which is
> also centos.
> The issue is building our node on centos in all these cases.
>
>
>
> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Thinking about this, doesn’t our build plugin explicitly install it’s own
> node? So actually all the node version things may be a red herring, since
> this is under our control through the pom. Not sure if we actually
> exercising this control. It seems that some of the errors people report are
> more to do with compilation failures for native node modules, which is
> doesn’t pin (i.e. things like system library dependencies). I’m not sure
> what we have in the dependency tree that requires complex native
> dependencies, but this might just be one of those node things we could doc
> around.
>
> This scenario would be fixed by standardising the build container.
>
> Yarn’s big thing is that it enables faster dependency resolution and local
> caching, right? It does not seem to address any of the problems we see, but
> sure, it’s the shiny new dependency system for node modules, which might
> make npm less horrible to deal with, so worth looking into.
>
> The other issue that I’ve seen people run into a lot is flat out download
> errors. This could be helped by finding our versions, maybe with yarn, but
> let’s face it, package-lock.json could also do that with npm, albeit with a
> slightly slower algorithm. However, short of bundling and hosting deps
> ourselves, I suspect the download errors are beyond our control, and
> certainly beyond the scope of this project (fix maven, fix npm, fix all the
> node hosting servers…)
>
> Simon
>
>
> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
> raghumitra@gmail.com>
> wrote:
> >
> > Looking at some of the build failure emails and past experience i
> > would suggest having a node & npm version check in our build scripts
> > and moving dependency management to yarn.
> >
> > We need not restrict the build to a specific version of node & npm but
> > we can surely suggest a min version required to build UI successfully.
> >
> > -Raghu
> >
> >
> >
> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
> >  wrote:
> >> Agreeing with Nick, it seems like the main reason people are building
> themselves, and hitting all these environmental issues, is that we do not
> as a project produce binary release artefacts (the rpms which users could
> just install) and instead leave that for the commercial distributors to do.
> >>
> >> Yarn may help with some of the dependency version issues we’re having,
> but not afaik with the core missing library headers / build tools / node
> and npm version issue, those would seem to fit a documentation fix and
> improvements to platform-info to flag the problems, so this can then be a
> pre-flight check tool as well as a diagnostic tool.
> >>
> >> Another option I would put on the table is to standardise our build
> environment, so that the non-java bits are run in a standard docker image
> or something fo the sort, that way we can take control of all the
> environmental and OS dependent pieces, much as we do right now with the rpm
> build sections of the mpack build.
> >>
> >> The challenge here will be adding the relevant maven support. At the
> moment we’re relying on the maven npm and node build plugins, this would
> likely need replacing with something custom and a challenge to support to
> go dow this route.
> >>
> >> Perhaps the real answer here is to push people who are just kicking the
> tyres towards a binary distribution, or at least rpm artefacts as part of
> the Apache release to give them a head start for a happy path on a known
> good OS environment.
> >>
> >> Simon
> >>
> >>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
> >>>
> >>> Yes, it is a problem. I think you've identified a couple important
> things
> >>> that we could address in parallel. I see these as challenges we need to
> >>> solve for the dev community.
> >>>
> >>> (1) NPM is causing us some

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler 
First issue is that we need c++ 11 on centos 6.8



On November 27, 2017 at 09:53:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Well, that’s good news on that issue. Reproducing the problem is half way
to solving it, right?

I would still say there are some systemic things going on that have
manifested in a variety of ways on both the users and dev list, so it’s
worth us having a good look at a more robust approach to node dependencies
(both npm ones, and the native ones)

Simon

On 27 Nov 2017, at 13:30, Otto Fowler  wrote:

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve for the dev community.
>>>
>>> (1) NPM is causing us some major headaches. Which version do we require?

>>> How do I install that version (on Mac, Windows, Linux)? Does YARN help
>>> here at all?
>>>
>>> (2) Can we automate the prerequisite checks that we currently do
manually
>>> with `platform-info.sh`? An automated check could run and fail as part
of
>>> the build or deployment process.
>>>
>>>
>>>
>>> More importantly though is that users should not have to build Metron at

>>> all. They should not have to worry about

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Simon Elliston Ball 
Well, that’s good news on that issue. Reproducing the problem is half way to 
solving it, right? 

I would still say there are some systemic things going on that have manifested 
in a variety of ways on both the users and dev list, so it’s worth us having a 
good look at a more robust approach to node dependencies (both npm ones, and 
the native ones) 

Simon

> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
> 
> I can reproduce the failure in out ansible docker build container, which is 
> also centos.
> The issue is building our node on centos in all these cases.
> 
> 
> 
> On November 27, 2017 at 07:02:51, Simon Elliston Ball 
> (si...@simonellistonball.com ) wrote:
> 
>> Thinking about this, doesn’t our build plugin explicitly install it’s own 
>> node? So actually all the node version things may be a red herring, since 
>> this is under our control through the pom. Not sure if we actually 
>> exercising this control. It seems that some of the errors people report are 
>> more to do with compilation failures for native node modules, which is 
>> doesn’t pin (i.e. things like system library dependencies). I’m not sure 
>> what we have in the dependency tree that requires complex native 
>> dependencies, but this might just be one of those node things we could doc 
>> around.  
>> 
>> This scenario would be fixed by standardising the build container. 
>> 
>> Yarn’s big thing is that it enables faster dependency resolution and local 
>> caching, right? It does not seem to address any of the problems we see, but 
>> sure, it’s the shiny new dependency system for node modules, which might 
>> make npm less horrible to deal with, so worth looking into.  
>> 
>> The other issue that I’ve seen people run into a lot is flat out download 
>> errors. This could be helped by finding our versions, maybe with yarn, but 
>> let’s face it, package-lock.json could also do that with npm, albeit with a 
>> slightly slower algorithm. However, short of bundling and hosting deps 
>> ourselves, I suspect the download errors are beyond our control, and 
>> certainly beyond the scope of this project (fix maven, fix npm, fix all the 
>> node hosting servers…) 
>> 
>> Simon 
>> 
>> 
>> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda > > > wrote: 
>> >  
>> > Looking at some of the build failure emails and past experience i 
>> > would suggest having a node & npm version check in our build scripts 
>> > and moving dependency management to yarn. 
>> >  
>> > We need not restrict the build to a specific version of node & npm but 
>> > we can surely suggest a min version required to build UI successfully. 
>> >  
>> > -Raghu 
>> >  
>> >  
>> >  
>> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball 
>> > > wrote: 
>> >> Agreeing with Nick, it seems like the main reason people are building 
>> >> themselves, and hitting all these environmental issues, is that we do not 
>> >> as a project produce binary release artefacts (the rpms which users could 
>> >> just install) and instead leave that for the commercial distributors to 
>> >> do. 
>> >>  
>> >> Yarn may help with some of the dependency version issues we’re having, 
>> >> but not afaik with the core missing library headers / build tools / node 
>> >> and npm version issue, those would seem to fit a documentation fix and 
>> >> improvements to platform-info to flag the problems, so this can then be a 
>> >> pre-flight check tool as well as a diagnostic tool. 
>> >>  
>> >> Another option I would put on the table is to standardise our build 
>> >> environment, so that the non-java bits are run in a standard docker image 
>> >> or something fo the sort, that way we can take control of all the 
>> >> environmental and OS dependent pieces, much as we do right now with the 
>> >> rpm build sections of the mpack build. 
>> >>  
>> >> The challenge here will be adding the relevant maven support. At the 
>> >> moment we’re relying on the maven npm and node build plugins, this would 
>> >> likely need replacing with something custom and a challenge to support to 
>> >> go dow this route. 
>> >>  
>> >> Perhaps the real answer here is to push people who are just kicking the 
>> >> tyres towards a binary distribution, or at least rpm artefacts as part of 
>> >> the Apache release to give them a head start for a happy path on a known 
>> >> good OS environment. 
>> >>  
>> >> Simon 
>> >>  
>> >>> On 24 Nov 2017, at 16:01, Nick Allen > >>> > wrote: 
>> >>>  
>> >>> Yes, it is a problem. I think you've identified a couple important 
>> >>> things 
>> >>> that we could address in parallel. I see these as challenges we need to 
>> >>> solve for the dev community. 
>> >>>  
>> >>> (1) NPM is causing us some major headaches. Which version do we require?

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler 
I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve for the dev community.
>>>
>>> (1) NPM is causing us some major headaches. Which version do we
require?
>>> How do I install that version (on Mac, Windows, Linux)? Does YARN help
>>> here at all?
>>>
>>> (2) Can we automate the prerequisite checks that we currently do
manually
>>> with `platform-info.sh`? An automated check could run and fail as part
of
>>> the build or deployment process.
>>>
>>>
>>>
>>> More importantly though is that users should not have to build Metron
at
>>> all. They should not have to worry about installing NPM and the rest of
>>> the development tooling. Here are some options that are not mutually
>>> exclusive.
>>>
>>>
>>> (1) Create an image in Atlas that has Metron fully installed. A new
user
>>> could run single node Metron on their laptop with a single command and
the
>>> only prereqs would be Vagrant and Virtualbox. We could cut new images
for
>>> each Metron release. Or selectively cut new dev images from master as
we
>>> see fit.
>>>
>>> (2) Distribute the Metron RPMs (and the MPack tarball?) so that users
can
>>> install Metron on a cluster without

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Simon Elliston Ball 
Thinking about this, doesn’t our build plugin explicitly install it’s own node? 
So actually all the node version things may be a red herring, since this is 
under our control through the pom. Not sure if we actually exercising this 
control. It seems that some of the errors people report are more to do with 
compilation failures for native node modules, which is doesn’t pin (i.e. things 
like system library dependencies). I’m not sure what we have in the dependency 
tree that requires complex native dependencies, but this might just be one of 
those node things we could doc around. 

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local 
caching, right? It does not seem to address any of the problems we see, but 
sure, it’s the shiny new dependency system for node modules, which might make 
npm less horrible to deal with, so worth looking into. 

The other issue that I’ve seen people run into a lot is flat out download 
errors. This could be helped by finding our versions, maybe with yarn, but 
let’s face it, package-lock.json could also do that with npm, albeit with a 
slightly slower algorithm. However, short of bundling and hosting deps 
ourselves, I suspect the download errors are beyond our control, and certainly 
beyond the scope of this project (fix maven, fix npm, fix all the node hosting 
servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda  
> wrote:
> 
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
> 
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
> 
> -Raghu
> 
> 
> 
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building 
>> themselves, and hitting all these environmental issues, is that we do not as 
>> a project produce binary release artefacts (the rpms which users could just 
>> install) and instead leave that for the commercial distributors to do.
>> 
>> Yarn may help with some of the dependency version issues we’re having, but 
>> not afaik with the core missing library headers / build tools / node and npm 
>> version issue, those would seem to fit a documentation fix and improvements 
>> to platform-info to flag the problems, so this can then be a pre-flight 
>> check tool as well as a diagnostic tool.
>> 
>> Another option I would put on the table is to standardise our build 
>> environment, so that the non-java bits are run in a standard docker image or 
>> something fo the sort, that way we can take control of all the environmental 
>> and OS dependent pieces, much as we do right now with the rpm build sections 
>> of the mpack build.
>> 
>> The challenge here will be adding the relevant maven support. At the moment 
>> we’re relying on the maven npm and node build plugins, this would likely 
>> need replacing with something custom and a challenge to support to go dow 
>> this route.
>> 
>> Perhaps the real answer here is to push people who are just kicking the 
>> tyres towards a binary distribution, or at least rpm artefacts as part of 
>> the Apache release to give them a head start for a happy path on a known 
>> good OS environment.
>> 
>> Simon
>> 
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>> 
>>> Yes, it is a problem.  I think you've identified a couple important things
>>> that we could address in parallel.  I see these as challenges we need to
>>> solve for the dev community.
>>> 
>>> (1) NPM is causing us some major headaches.  Which version do we require?
>>> How do I install that version (on Mac, Windows, Linux)?  Does YARN help
>>> here at all?
>>> 
>>> (2) Can we automate the prerequisite checks that we currently do manually
>>> with `platform-info.sh`?  An automated check could run and fail as part of
>>> the build or deployment process.
>>> 
>>> 
>>> 
>>> More importantly though is that users should not have to build Metron at
>>> all.  They should not have to worry about installing NPM and the rest of
>>> the development tooling.   Here are some options that are not mutually
>>> exclusive.
>>> 
>>> 
>>> (1) Create an image in Atlas that has Metron fully installed.  A new user
>>> could run single node Metron on their laptop with a single command and the
>>> only prereqs would be Vagrant and Virtualbox.  We could cut new images for
>>> each Metron release.  Or selectively cut new dev images from master as we
>>> see fit.
>>> 
>>> (2) Distribute the Metron RPMs (and the MPack tarball?) so that users can
>>> install Metron on a cluster without having to build it.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Nov 24, 2017 at 10:11 AM, Otto Fowler 
>>>

Re: [DISCUSS] NPM / Node Problems

2017-11-26 Thread RaghuMitra Kandikonda 
Looking at some of the build failure emails and past experience i
would suggest having a node & npm version check in our build scripts
and moving dependency management to yarn.

We need not restrict the build to a specific version of node & npm but
we can surely suggest a min version required to build UI successfully.

-Raghu



On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
 wrote:
> Agreeing with Nick, it seems like the main reason people are building 
> themselves, and hitting all these environmental issues, is that we do not as 
> a project produce binary release artefacts (the rpms which users could just 
> install) and instead leave that for the commercial distributors to do.
>
> Yarn may help with some of the dependency version issues we’re having, but 
> not afaik with the core missing library headers / build tools / node and npm 
> version issue, those would seem to fit a documentation fix and improvements 
> to platform-info to flag the problems, so this can then be a pre-flight check 
> tool as well as a diagnostic tool.
>
> Another option I would put on the table is to standardise our build 
> environment, so that the non-java bits are run in a standard docker image or 
> something fo the sort, that way we can take control of all the environmental 
> and OS dependent pieces, much as we do right now with the rpm build sections 
> of the mpack build.
>
> The challenge here will be adding the relevant maven support. At the moment 
> we’re relying on the maven npm and node build plugins, this would likely need 
> replacing with something custom and a challenge to support to go dow this 
> route.
>
> Perhaps the real answer here is to push people who are just kicking the tyres 
> towards a binary distribution, or at least rpm artefacts as part of the 
> Apache release to give them a head start for a happy path on a known good OS 
> environment.
>
> Simon
>
>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>
>> Yes, it is a problem.  I think you've identified a couple important things
>> that we could address in parallel.  I see these as challenges we need to
>> solve for the dev community.
>>
>> (1) NPM is causing us some major headaches.  Which version do we require?
>> How do I install that version (on Mac, Windows, Linux)?  Does YARN help
>> here at all?
>>
>> (2) Can we automate the prerequisite checks that we currently do manually
>> with `platform-info.sh`?  An automated check could run and fail as part of
>> the build or deployment process.
>>
>>
>>
>> More importantly though is that users should not have to build Metron at
>> all.  They should not have to worry about installing NPM and the rest of
>> the development tooling.   Here are some options that are not mutually
>> exclusive.
>>
>>
>> (1) Create an image in Atlas that has Metron fully installed.  A new user
>> could run single node Metron on their laptop with a single command and the
>> only prereqs would be Vagrant and Virtualbox.  We could cut new images for
>> each Metron release.  Or selectively cut new dev images from master as we
>> see fit.
>>
>> (2) Distribute the Metron RPMs (and the MPack tarball?) so that users can
>> install Metron on a cluster without having to build it.
>>
>>
>>
>>
>>
>>
>> On Fri, Nov 24, 2017 at 10:11 AM, Otto Fowler 
>> wrote:
>>
>>> It seems like it is getting *very* common for people to have trouble
>>> building recently. Errors with NPM and Node seen common, with fixes ranging
>>> from updating c/c++ libs to the version of npm/node.
>>>
>>> There has to be a better way to do this.
>>>
>>>   -
>>>
>>>   Are we out of date or missing requirements in our documentation?
>>>   -
>>>
>>>   Does our documentation need to be updated for building?
>>>   -
>>>
>>>   Is there a better way in maven to check the versions required for some
>>>   of these things and fail faster with a better message?
>>>   -
>>>
>>>   Are we building correctly or are we asking for trouble?
>>>
>>> The ability to build metron is pretty important, and it seems that people
>>> are having a lot of trouble related to the new technologies in alerts and
>>> config ui.
>>>
>

Re: [DISCUSS] NPM / Node Problems

2017-11-24 Thread Nick Allen 
Yes, it is a problem.  I think you've identified a couple important things
that we could address in parallel.  I see these as challenges we need to
solve for the dev community.

(1) NPM is causing us some major headaches.  Which version do we require?
How do I install that version (on Mac, Windows, Linux)?  Does YARN help
here at all?

(2) Can we automate the prerequisite checks that we currently do manually
with `platform-info.sh`?  An automated check could run and fail as part of
the build or deployment process.



More importantly though is that users should not have to build Metron at
all.  They should not have to worry about installing NPM and the rest of
the development tooling.   Here are some options that are not mutually
exclusive.


(1) Create an image in Atlas that has Metron fully installed.  A new user
could run single node Metron on their laptop with a single command and the
only prereqs would be Vagrant and Virtualbox.  We could cut new images for
each Metron release.  Or selectively cut new dev images from master as we
see fit.

(2) Distribute the Metron RPMs (and the MPack tarball?) so that users can
install Metron on a cluster without having to build it.






On Fri, Nov 24, 2017 at 10:11 AM, Otto Fowler 
wrote:

> It seems like it is getting *very* common for people to have trouble
> building recently. Errors with NPM and Node seen common, with fixes ranging
> from updating c/c++ libs to the version of npm/node.
>
> There has to be a better way to do this.
>
>-
>
>Are we out of date or missing requirements in our documentation?
>-
>
>Does our documentation need to be updated for building?
>-
>
>Is there a better way in maven to check the versions required for some
>of these things and fail faster with a better message?
>-
>
>Are we building correctly or are we asking for trouble?
>
> The ability to build metron is pretty important, and it seems that people
> are having a lot of trouble related to the new technologies in alerts and
> config ui.
>

[DISCUSS] NPM / Node Problems

2017-11-24 Thread Otto Fowler 
It seems like it is getting *very* common for people to have trouble
building recently. Errors with NPM and Node seen common, with fixes ranging
from updating c/c++ libs to the version of npm/node.

There has to be a better way to do this.

   -

   Are we out of date or missing requirements in our documentation?
   -

   Does our documentation need to be updated for building?
   -

   Is there a better way in maven to check the versions required for some
   of these things and fail faster with a better message?
   -

   Are we building correctly or are we asking for trouble?

The ability to build metron is pretty important, and it seems that people
are having a lot of trouble related to the new technologies in alerts and
config ui.