Re: [DISCUSS] Are/how are you using the ES data pruner?

[Ali Nazemian]

Sorry, Michael. I am having some issues to share any code right now. It
seems we need to go through an internal verification of anything we want to
share. BTW, the curator script I mentioned is very simple and nothing
special. It doesn't worth to waste any time on waiting for it.

Cheers,
Ali

On Tue, Nov 28, 2017 at 9:56 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> It's a worthy mention. Our existing pruner wouldn't be able to handle Solr
> without modification, so we'd either need something native to Solr or
> something custom.
>
> Mike
>
> On Mon, Nov 27, 2017 at 3:46 PM, James Sirota  wrote:
>
> > One thing to keep in mind, as we will be introducing Solr shortly, is to
> > find if something similar to curator exists for Solr.  But we'll cross
> that
> > bridge when we get there
> >
> > 22.11.2017, 22:58, "Ali Nazemian" :
> > > Sure. I will have a chat internally and come back to you shortly. It
> was
> > a
> > > quick and dirty work actually just to fix this temporarily. However, it
> > > might be a good starting point.
> > >
> > > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >>  Thanks Ali, that's good feedback. Would you be willing to share any
> of
> > your
> > >>  Curator calls/config and use cases with the community? I'd love to
> add
> > it
> > >>  to a document around ES pruning in the short term, and maybe we could
> > look
> > >>  at how to build this into indexing at some point.
> > >>
> > >>  Cheers,
> > >>  Mike
> > >>
> > >>  On Nov 22, 2017 8:53 PM, "Ali Nazemian" 
> wrote:
> > >>
> > >>  > We tried to use it, but we had the same issue. It was not
> > documented. We
> > >>  > tried to use it, and we had some issues. It also was not exactly
> > what we
> > >>  > wanted, so we decided to create something from scratch by using
> > >>  > Elasticsearch Curator. We wanted to have an ability to manage
> > different
> > >>  > prune mechanism for different feeds. Having a hard threshold to
> > remove
> > >>  > index and Soft threshold to close that index. Maybe it can be a
> > feature
> > >>  to
> > >>  > add to the indexing JSON config file per feed.
> > >>  >
> > >>  > Cheers,
> > >>  > Ali
> > >>  >
> > >>  > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic <
> > >>  > michael.miklav...@gmail.com> wrote:
> > >>  >
> > >>  > > From what I can tell, the data pruner isn't documented anywhere,
> > so I'm
> > >>  > > curious if anybody is using this, and if so, how are you using
> it?
> > >>  > >
> > >>  > > -
> > >>  > > https://github.com/apache/metron/blob/master/metron-
> > >>  > > platform/metron-data-management/README.md
> > >>  > > -
> > >>  > > https://github.com/apache/metron/blob/master/metron-
> > >>  > > platform/metron-data-management/src/main/java/org/
> > >>  > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java
> > >>  > > -
> > >>  > > https://github.com/apache/metron/blob/master/metron-
> > >>  > > platform/metron-data-management/src/main/java/org/
> > >>  > > apache/metron/dataloads/bulk/DataPruner.java
> > >>  > >
> > >>  > > It looks to me that it allows you to specify the start date and a
> > >>  number
> > >>  > of
> > >>  > > days for lookback from the start date to purge along with a regex
> > >>  pattern
> > >>  > > to match the index name. It also does not look like it has any
> > built-in
> > >>  > > scheduling semantics, so I assume this was a cron job. I think
> that
> > >>  about
> > >>  > > covers it. Anything I've missed?
> > >>  > >
> > >>  > > I'm adding a quick doc write-up to METRON-939 (
> > >>  > > https://github.com/apache/metron/pull/840) for using Curator to
> > prune
> > >>  > > indices from Elasticsearch. It is desirable to make sure I've
> > covered
> > >>  > > existing use cases.
> > >>  > >
> > >>  > > Best,
> > >>  > > Mike
> > >>  > >
> > >>  >
> > >>  >
> > >>  >
> > >>  > --
> > >>  > A.Nazemian
> > >>  >
> > >
> > > --
> > > A.Nazemian
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
>



-- 
A.Nazemian

[GitHub] metron pull request #850: METRON-1335: Install metron-maas-service RPM as a ...

[anandsubbu]

GitHub user anandsubbu opened a pull request:

https://github.com/apache/metron/pull/850

METRON-1335: Install metron-maas-service RPM as a part of the full-dev 
deployment

## Contributor Comments
Modified mpack metainfo.xml to install metron-maas-service RPM as well.

**Testing Done**
- [x] Deployed full dev and validated that the metron-maas-service RPM is 
shown as installed from the Ambari UI 
- [x] Did `vagrant ssh` to the full dev box. Confirmed that the RPM is 
installed.
```
[vagrant@node1 ~]$ rpm -qa | grep metron-maas
metron-maas-service-0.4.2-201711271930.noarch
```

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-1335

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/850.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #850


commit 313b6e611a9ed27fc54800c0c54a0ac6e93f959a
Author: Anand Subramanian 
Date:   2017-11-28T06:01:08Z

Install metron-maas-service RPM




---

[GitHub] metron-bro-plugin-kafka issue #4: METRON-1329: Simplify metron-bro-plugin-ka...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/4
  
+1
-  review looks good
- ran through test steps, everything works as described

Great job!


---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/836
  
@cestella I know you were out recently, just wanted to bring this one to 
the top of your inbox.  Would like to have this in the upcoming release, but 
also want to get your input.


---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

In an attempt to keep this from becoming unbearably long, I will try
to keep my responses short, but I would be happy to elaborate.  That's a
fairly good timeline and summary, but here are some clarifications in
corresponding order:

- The plugin history is quite short and you can probably get a good bit of
context just by looking at the commits
.
- The plugin is only useful to the bro community, but it is rather popular.
- The Bro team created the idea of bro packages, which can include bro
plugins, bro scripts, or BroControl plugins.  So, instead of having a
'plugins' repo, they moved to have a 'packages' repo which is by default
referenced by a bro-pkg tool they wrote for package management.
- I believe I kicked this off (or at least I did in my head) when I started
complaining 
about the plugin divergence that occurred due to the move to bro/plugins
(the right move at the time), but Metron's use of a local directory that
hadn't been kept up to date.  My current efforts are an attempt to make
sure this doesn't happen again, and to take advantage of the bro-pkg
benefits.
- The gap between ~3/31 and actual progress on 11/12 is completely on me -
I had intentions of doing this work sooner but failed to do so.
- You can most definitely still install/use the bro plugin without using
bro-pkg.  In fact, the README in my PR

still has the instructions on how to do so.

Q1:  The simple explanation is that the only thing that makes a plugin a
bro package is the inclusion of a bro-pkg.meta file
,
and it includes a build_command which could easily be manually performed to
install by hand (assuming dependencies are met).

I've worked with other projects that use submodules and while I'm fine
discussing it, I suggest that we don't implement it.  I put together a
quick example of why here[1], using the bro project as an example since
it's top of mind.

Q2:   I think the answer to Q1 answers this.  There is absolutely nothing
stopping a git clone && cd $dir && configure && make && make install, but
using bro-pkg to install/load takes into account dependencies

and
unit tests

when it is loaded (and thus fails early and more intuitively).  It only
must be a separate repo (or, more technically correct, a git branch that
includes only the package) because of how bro-pkg works.  If you'd like to
get an idea of how this would work in application for Bro users, you can
see my test instructions here
 (specifically
step #3).  If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka, the
command could be `bro-pkg install metron-bro-plugin-kafka --version 0.1` or
`bro-pkg install apache/metron-bro-plugin-kafka --version 0.1` due to this
 (the
--force is just to remove user interaction, for an ansible spin-up).


1:  To clone the Bro git repo, you must run `git clone --recursive
https://github.com/bro/bro` (note the --recursive).  Not too big of a deal,
but requires that you remember it and existing instructions/blog posts may
give users inaccurate steps.  Let's make this worse and try to checkout
their latest release, v2.5.2, and automatically update the submodules
appropriately via `git checkout v2.5.2 --recurse-submodules`.  This fails
because aux/plugins (https://github.com/bro/plugins) was removed since
their latest release.  Okay, we can work around this using `git checkout
v2.5.2` and then remember to `git submodule update` every time you checkout
a release or branch.  But because they have nested submodules, we actually
need to run `git submodule update --recursive`.  I can't imagine opting
*into* a workflow anything like this.  There are other options as well,
such as git subtrees, but those I am less familiar with.

Jon

On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler  wrote:

> I am not sure that our use of the plugin necessarily equates to it being
> implicitly coupled to Metron.  It seems like the Right Thing To Do[image:
> ™], esp.
>  for an Apache project would be to make this available for use by the
> greater bro community.
> Unless we expect to do extensive iterative work on the plugin, which would
> then make the decision to spin it out now premature.
>
> Then again, I might be wrong ;)
>
>
> On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote:
>
> [Please pardon me that the below is a little labored. I’m trying to
> understand the implications for both release and use, which 

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/848
  
Tested via
```
cd /Users/jzeolla/metron-pr848
docker run -it -v /Users/jzeolla/metron-pr848:/root/metron 
ansible-docker:2.0.0.2 bash
cd /root/metron
mvn clean package -DskipTests # Failure
```

Ran into
```
[ERROR] npm ERR! Linux 4.9.49-moby
[ERROR] npm ERR! argv 
"/root/metron/metron-interface/metron-config/node/node" 
"/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
 "run" "build"
[ERROR] npm ERR! node v6.2.0
[ERROR] npm ERR! npm  v3.8.9
[ERROR] npm ERR! code ELIFECYCLE
[ERROR] npm ERR! metron-management-ui@0.4.2 build: 
`./node_modules/angular-cli/bin/ng build -prod`
[ERROR] npm ERR! Exit status 1
[ERROR] npm ERR!
[ERROR] npm ERR! Failed at the metron-management-ui@0.4.2 build script 
'./node_modules/angular-cli/bin/ng build -prod'.
[ERROR] npm ERR! Make sure you have the latest version of node.js and npm 
installed.
[ERROR] npm ERR! If you do, this is most likely a problem with the 
metron-management-ui package,
[ERROR] npm ERR! not with npm itself.
[ERROR] npm ERR! Tell the author that this fails on your system:
[ERROR] npm ERR! ./node_modules/angular-cli/bin/ng build -prod
[ERROR] npm ERR! You can get information on how to open an issue for this 
project with:
[ERROR] npm ERR! npm bugs metron-management-ui
[ERROR] npm ERR! Or if that isn't available, you can get their info via:
[ERROR] npm ERR! npm owner ls metron-management-ui
[ERROR] npm ERR! There is likely additional logging output above.
[ERROR]
[ERROR] npm ERR! Please include the following file with any support request:
[ERROR] npm ERR! 
/root/metron/metron-interface/metron-config/npm-debug.log
[INFO] 

[INFO] Reactor Summary:
[INFO]
[INFO] Metron . SUCCESS [ 
18.059 s]
[INFO] metron-stellar . SUCCESS [  
7.965 s]
[INFO] stellar-common . SUCCESS [01:27 
min]
[INFO] metron-analytics ... SUCCESS [  
0.012 s]
[INFO] metron-maas-common . SUCCESS [ 
10.648 s]
[INFO] metron-platform  SUCCESS [  
0.031 s]
[INFO] metron-zookeeper ... SUCCESS [  
1.542 s]
[INFO] metron-test-utilities .. SUCCESS [ 
42.995 s]
[INFO] metron-integration-test  SUCCESS [ 
40.759 s]
[INFO] metron-maas-service  SUCCESS [ 
10.431 s]
[INFO] metron-common .. SUCCESS [ 
46.535 s]
[INFO] metron-statistics .. SUCCESS [ 
22.078 s]
[INFO] metron-writer .. SUCCESS [ 
33.132 s]
[INFO] metron-storm-kafka-override  SUCCESS [  
4.151 s]
[INFO] metron-storm-kafka . SUCCESS [  
2.083 s]
[INFO] metron-hbase ... SUCCESS [ 
16.480 s]
[INFO] metron-profiler-common . SUCCESS [  
5.086 s]
[INFO] metron-profiler-client . SUCCESS [ 
26.173 s]
[INFO] metron-profiler  SUCCESS [ 
56.438 s]
[INFO] metron-hbase-client  SUCCESS [ 
16.283 s]
[INFO] metron-enrichment .. SUCCESS [01:01 
min]
[INFO] metron-indexing  SUCCESS [ 
22.402 s]
[INFO] metron-solr  SUCCESS [ 
56.301 s]
[INFO] metron-pcap  SUCCESS [  
5.602 s]
[INFO] metron-parsers . SUCCESS [01:09 
min]
[INFO] metron-pcap-backend  SUCCESS [ 
35.876 s]
[INFO] metron-data-management . SUCCESS [01:46 
min]
[INFO] metron-api . SUCCESS [ 
57.116 s]
[INFO] metron-management .. SUCCESS [ 
14.077 s]
[INFO] elasticsearch-shaded ... SUCCESS [ 
11.380 s]
[INFO] metron-elasticsearch ... SUCCESS [01:02 
min]
[INFO] metron-deployment .. SUCCESS [  
0.006 s]
[INFO] Metron Ambari Management Pack .. SUCCESS [  
4.145 s]
[INFO] metron-contrib . SUCCESS [  
0.015 s]
[INFO] metron-docker .. SUCCESS [  
5.673 s]
[INFO] metron-interface 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Otto Fowler]

I am not sure that our use of the plugin necessarily equates to it being
implicitly coupled to Metron.  It seems like the Right Thing To Do™, esp.
 for an Apache project would be to make this available for use by the
greater bro community.
Unless we expect to do extensive iterative work on the plugin, which would
then make the decision to spin it out now premature.

Then again, I might be wrong ;)


On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote:

[Please pardon me that the below is a little labored. I’m trying to
understand the implications for both release and use, which requires some
explanation as well as the two questions needed. Q1 and Q2 below are
probably the same question, asked in slightly different contexts. Please
consider them together.]

So this made me go back and look at the history that caused us to put the
bro plugin in a separate repo. As best I can see, this was in
https://issues.apache.org/jira/browse/METRON-813 , which cites an email
discussion thread. Also please see
https://issues.apache.org/jira/browse/METRON-883 for background on the
plugin itself.

As best I can assemble the many bits brought up in the threads, the reasons
to put it in a separate repo was:
- The plugin was thought to be useful to multiple clients of bro and kafka,
including Storm and Spark, as well as Metron.
- Originally the bro project was maintaining bro plugins and it was thought
they might adopt this one.
- Bro then formalized their plugin framework BUT dumped all plugins out of
their sphere of maintenance.
- As of 3/31/2017, Nick said that “the [bro] package mechanism requires
that a package live within its own repo”. Jon said “the bro packages model
doesn't allow colocation with anything else.”
- So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
was created a few days later. But Metron wasn’t actually modified to remove
the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
– with https://issues.apache.org/jira/browse/METRON-1309 .
- Presumably the need to have metron-bro-plugin-kafka in a separate repo
remain valid, if the bro plugin mechanism is used. But obviously there are
(non-conforming) ways to build the plugin as part of metron, and install it
in a way that works.

Q1. I think that last statement needs some explanation. Nick or Jon, can
you please expand on it, especially wrt how the end user installs the
plugin once the plugin is built the two different ways? And whether it’s
still valuable to have a separate repo for the plugin?

Nick suggests using a submodule approach to managing the bro plugin, for
Metron versioning purposes. As I understand it, this would continue the
existence of the metron-bro-plugin-kafka repo, but copy it into the metron
code tree for building, versioning, and release purposes. Git submodules
are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
We would use the submodule capability to clone the metron-bro-plugin-kafka
source code into a subdirectory of Metron at the time one clones the metron
repo. It would then be released with Metron as part of the source code
release for a given version of Metron. Part of the way submodules are
managed, is that git stores the SHA1 hash of the submodule into a file
named .gitmodules, which in turn gets saved when you do a git push. So
indeed submodules would ensure that everyone cloning a given version of
metron would get the expected “version” (sha, actually) of
metron-bro-plugin-kafka.

This sounds like a good idea, although it isn’t without cost. Submodules
impose the need for additional commands to actually get a copy of the
submodule source, and if the plugin repo advanced beyond the version in a
metron repo, it causes some ‘git status’ artifacts that could be confusing
to folks who aren’t familiar with submodules. But these can be documented.

Q2. Nick, what I’m not clear about is the process by which the
metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
developers, and (b) end users. If it “must” be in a separate repo to be
successfully built and managed by the bro plugin mechanism, does that mean
it can’t be built from the copy in the Metron source tree? Yet until
November, that’s exactly what we were doing. Do we go back to doing that?
What does that mean wrt users installing the plugin?

Thanks for your patience in reading this far.
--Matt


On 11/27/17, 2:58 PM, "James Sirota"  wrote:

I agree with Nick. Since the plugin is tightly coupled with Metron why not
just pull it into the main repo and version it with the rest of the code?
Do we really need the second repo for the plug-in?

Thanks,
James



16.11.2017, 08:06, "Nick Allen" :
>> I would suggest that we institute a release procedure for the package
>> itself, but I don't think it necessarily has to line up with metron
>> releases (happy to be persuaded 

[GitHub] metron issue #847: METRON-1313: Update metron-deployment to use bro-pkg to i...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/847
  
+1, nice work


---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/849
  
I think that is valid


---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley]

[Please pardon me that the below is a little labored.  I’m trying to understand 
the implications for both release and use, which requires some explanation as 
well as the two questions needed.  Q1 and Q2 below are probably the same 
question, asked in slightly different contexts.  Please consider them together.]

So this made me go back and look at the history that caused us to put the bro 
plugin in a separate repo.  As best I can see, this was in 
https://issues.apache.org/jira/browse/METRON-813 , which cites an email 
discussion thread.  Also please see 
https://issues.apache.org/jira/browse/METRON-883 for background on the plugin 
itself.

As best I can assemble the many bits brought up in the threads, the reasons to 
put it in a separate repo was:
-  The plugin was thought to be useful to multiple clients of bro and kafka, 
including Storm and Spark, as well as Metron.
-  Originally the bro project was maintaining bro plugins and it was thought 
they might adopt this one.
-  Bro then formalized their plugin framework BUT dumped all plugins out of 
their sphere of maintenance.
-  As of 3/31/2017, Nick said that “the [bro] package mechanism requires that a 
package live within its own repo”.  Jon said “the bro packages model doesn't 
allow colocation with anything else.”
-  So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo was 
created a few days later.  But Metron wasn’t actually modified to remove the 
metron-sensors/bro-plugin-kafka/ subdirectory and start using the plugin from 
the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago! – with 
https://issues.apache.org/jira/browse/METRON-1309 .
-  Presumably the need to have metron-bro-plugin-kafka in a separate repo 
remain valid, if the bro plugin mechanism is used.  But obviously there are 
(non-conforming) ways to build the plugin as part of metron, and install it in 
a way that works.

Q1.  I think that last statement needs some explanation.  Nick or Jon, can you 
please expand on it, especially wrt how the end user installs the plugin once 
the plugin is built the two different ways?  And whether it’s still valuable to 
have a separate repo for the plugin?

Nick suggests using a submodule approach to managing the bro plugin, for Metron 
versioning purposes.  As I understand it, this would continue the existence of 
the metron-bro-plugin-kafka repo, but copy it into the metron code tree for 
building, versioning, and release purposes.  Git submodules are documented 
here: https://git-scm.com/book/en/v2/Git-Tools-Submodules . We would use the 
submodule capability to clone the metron-bro-plugin-kafka source code into a 
subdirectory of Metron at the time one clones the metron repo.  It would then 
be released with Metron as part of the source code release for a given version 
of Metron.  Part of the way submodules are managed, is that git stores the SHA1 
hash of the submodule into a file named .gitmodules, which in turn gets saved 
when you do a git push.  So indeed submodules would ensure that everyone 
cloning a given version of metron would get the expected “version” (sha, 
actually) of metron-bro-plugin-kafka.

This sounds like a good idea, although it isn’t without cost.  Submodules 
impose the need for additional commands to actually get a copy of the submodule 
source, and if the plugin repo advanced beyond the version in a metron repo, it 
causes some ‘git status’ artifacts that could be confusing to folks who aren’t 
familiar with submodules.  But these can be documented.

Q2.  Nick, what I’m not clear about is the process by which the 
metron-bro-plugin-kafka would be built and “plugged in” by (a) metron 
developers, and (b) end users.  If it “must” be in a separate repo to be 
successfully built and managed by the bro plugin mechanism, does that mean it 
can’t be built from the copy in the Metron source tree?  Yet until November, 
that’s exactly what we were doing.  Do we go back to doing that?  What does 
that mean wrt users installing the plugin?

Thanks for your patience in reading this far.
--Matt


On 11/27/17, 2:58 PM, "James Sirota"  wrote:

I agree with Nick.  Since the plugin is tightly coupled with Metron why not 
just pull it into the main repo and version it with the rest of the code?  Do 
we really need the second repo for the plug-in?

Thanks,
James 



16.11.2017, 08:06, "Nick Allen" :
>>  I would suggest that we institute a release procedure for the package
>>  itself, but I don't think it necessarily has to line up with metron
>>  releases (happy to be persuaded otherwise). Then we can just link metron
>>  to metron-bro-plugin-kafka by pointing to specific
>>  metron-bro-plugin-kafka releases (git tags
>>  
>  versioning>
>>  ).
>>  Right now, full-dev spins up against the
>>  

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

The reason we decided to do that was because it is the best way for it to
be used (and thus improved on and quality tested) by the broader bro
community.  If it's any indication of it's popularity, there was just an
email on the bro mailing list about the plugin a few days ago, and I've
already received a PR against my temporary holding place for this plugin to
improve it (GitHub.com/jonzeolla/jzeolla-metron-bro-plugin-kafka), which
would never have happened if it wasn't a standalone repo.  Previously Nick
pushed the code into a bro/plugins repo, which is now deprecated in favor
of bro packages, but while it was there the community was able to identify
issues which helped improve its quality.

By putting it in its own repo we were also able to turn it into a bro
package, similar to debs or rpms but slightly different.  This has numerous
benefits, including turning setup into single command that installs and
loads the plugin, in addition to handling version management and performing
on installation unit tests to identify issues before use.

I would advocate to treat this code as a dependancy, but not necessarily a
core part of Metron, like we currently do.  This also allows us to maintain
separate versioning and release cycles (i.e. not necessarily lined up with
Metron releases), which I expect will be very simple, infrequent, and
easily tested.  Happy to discuss this further - thanks,

Jon

On Mon, Nov 27, 2017, 17:58 James Sirota  wrote:

> I agree with Nick.  Since the plugin is tightly coupled with Metron why
> not just pull it into the main repo and version it with the rest of the
> code?  Do we really need the second repo for the plug-in?
>
> Thanks,
> James
>
>
>
> 16.11.2017, 08:06, "Nick Allen" :
> >>  I would suggest that we institute a release procedure for the package
> >>  itself, but I don't think it necessarily has to line up with metron
> >>  releases (happy to be persuaded otherwise). Then we can just link
> metron
> >>  to metron-bro-plugin-kafka by pointing to specific
> >>  metron-bro-plugin-kafka releases (git tags
> >>  <
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> >>  versioning>
> >>  ).
> >>  Right now, full-dev spins up against the
> >>  apache/metron-bro-plugin-kafka master branch, which is not a good idea
> to
> >>  have in place for an upcoming release. That is the crux of why I think
> we
> >>  need to finalize the move to bro 2.5.2 and the plugin packaging before
> our
> >>  next release (working on it as we speak).
> >>  Jon
> >
> > ​I replayed Jon's comments from the other thread above.​
> >
> > My initial thought, is that I would not want to manage two separate
> release
> > processes. I don't want to have a roll call, cut release candidates and
> > test both.
> >
> > I was thinking we would just need to change some of the behind-the-scenes
> > processes handled by the release manager. This is one area where I had
> > thought using a submodule in Git would help.
> >
> > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen  wrote:
> >
> >>  + Restarting the thread to include mentors.
> >>
> >>  The code of the 'Kafka Plugin for Bro' is now maintained in the
> external
> >>  repository that we set up a while back.
> >>
> >> - Metron Core: git://git.apache.org/metron.git
> >> - Kafka Plugin for Bro: git://git.apache.org/
> >> metron-bro-plugin-kafka.git
> >>
> >>  (Q) Do we need to change anything in the release procedure to account
> for
> >>  this?
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[James Sirota]

I agree with Nick.  Since the plugin is tightly coupled with Metron why not 
just pull it into the main repo and version it with the rest of the code?  Do 
we really need the second repo for the plug-in?

Thanks,
James 



16.11.2017, 08:06, "Nick Allen" :
>>  I would suggest that we institute a release procedure for the package
>>  itself, but I don't think it necessarily has to line up with metron
>>  releases (happy to be persuaded otherwise). Then we can just link metron
>>  to metron-bro-plugin-kafka by pointing to specific
>>  metron-bro-plugin-kafka releases (git tags
>>  >  versioning>
>>  ).
>>  Right now, full-dev spins up against the
>>  apache/metron-bro-plugin-kafka master branch, which is not a good idea to
>>  have in place for an upcoming release. That is the crux of why I think we
>>  need to finalize the move to bro 2.5.2 and the plugin packaging before our
>>  next release (working on it as we speak).
>>  Jon
>
> ​I replayed Jon's comments from the other thread above.​
>
> My initial thought, is that I would not want to manage two separate release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen  wrote:
>
>>  + Restarting the thread to include mentors.
>>
>>  The code of the 'Kafka Plugin for Bro' is now maintained in the external
>>  repository that we set up a while back.
>>
>> - Metron Core: git://git.apache.org/metron.git
>> - Kafka Plugin for Bro: git://git.apache.org/
>> metron-bro-plugin-kafka.git
>>
>>  (Q) Do we need to change anything in the release procedure to account for
>>  this?

--- 
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Re: [DISCUSS] Are/how are you using the ES data pruner?

[Michael Miklavcic]

It's a worthy mention. Our existing pruner wouldn't be able to handle Solr
without modification, so we'd either need something native to Solr or
something custom.

Mike

On Mon, Nov 27, 2017 at 3:46 PM, James Sirota  wrote:

> One thing to keep in mind, as we will be introducing Solr shortly, is to
> find if something similar to curator exists for Solr.  But we'll cross that
> bridge when we get there
>
> 22.11.2017, 22:58, "Ali Nazemian" :
> > Sure. I will have a chat internally and come back to you shortly. It was
> a
> > quick and dirty work actually just to fix this temporarily. However, it
> > might be a good starting point.
> >
> > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >>  Thanks Ali, that's good feedback. Would you be willing to share any of
> your
> >>  Curator calls/config and use cases with the community? I'd love to add
> it
> >>  to a document around ES pruning in the short term, and maybe we could
> look
> >>  at how to build this into indexing at some point.
> >>
> >>  Cheers,
> >>  Mike
> >>
> >>  On Nov 22, 2017 8:53 PM, "Ali Nazemian"  wrote:
> >>
> >>  > We tried to use it, but we had the same issue. It was not
> documented. We
> >>  > tried to use it, and we had some issues. It also was not exactly
> what we
> >>  > wanted, so we decided to create something from scratch by using
> >>  > Elasticsearch Curator. We wanted to have an ability to manage
> different
> >>  > prune mechanism for different feeds. Having a hard threshold to
> remove
> >>  > index and Soft threshold to close that index. Maybe it can be a
> feature
> >>  to
> >>  > add to the indexing JSON config file per feed.
> >>  >
> >>  > Cheers,
> >>  > Ali
> >>  >
> >>  > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic <
> >>  > michael.miklav...@gmail.com> wrote:
> >>  >
> >>  > > From what I can tell, the data pruner isn't documented anywhere,
> so I'm
> >>  > > curious if anybody is using this, and if so, how are you using it?
> >>  > >
> >>  > > -
> >>  > > https://github.com/apache/metron/blob/master/metron-
> >>  > > platform/metron-data-management/README.md
> >>  > > -
> >>  > > https://github.com/apache/metron/blob/master/metron-
> >>  > > platform/metron-data-management/src/main/java/org/
> >>  > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java
> >>  > > -
> >>  > > https://github.com/apache/metron/blob/master/metron-
> >>  > > platform/metron-data-management/src/main/java/org/
> >>  > > apache/metron/dataloads/bulk/DataPruner.java
> >>  > >
> >>  > > It looks to me that it allows you to specify the start date and a
> >>  number
> >>  > of
> >>  > > days for lookback from the start date to purge along with a regex
> >>  pattern
> >>  > > to match the index name. It also does not look like it has any
> built-in
> >>  > > scheduling semantics, so I assume this was a cron job. I think that
> >>  about
> >>  > > covers it. Anything I've missed?
> >>  > >
> >>  > > I'm adding a quick doc write-up to METRON-939 (
> >>  > > https://github.com/apache/metron/pull/840) for using Curator to
> prune
> >>  > > indices from Elasticsearch. It is desirable to make sure I've
> covered
> >>  > > existing use cases.
> >>  > >
> >>  > > Best,
> >>  > > Mike
> >>  > >
> >>  >
> >>  >
> >>  >
> >>  > --
> >>  > A.Nazemian
> >>  >
> >
> > --
> > A.Nazemian
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>

Re: [DISCUSS] Are/how are you using the ES data pruner?

[James Sirota]

One thing to keep in mind, as we will be introducing Solr shortly, is to find 
if something similar to curator exists for Solr.  But we'll cross that bridge 
when we get there

22.11.2017, 22:58, "Ali Nazemian" :
> Sure. I will have a chat internally and come back to you shortly. It was a
> quick and dirty work actually just to fix this temporarily. However, it
> might be a good starting point.
>
> On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>>  Thanks Ali, that's good feedback. Would you be willing to share any of your
>>  Curator calls/config and use cases with the community? I'd love to add it
>>  to a document around ES pruning in the short term, and maybe we could look
>>  at how to build this into indexing at some point.
>>
>>  Cheers,
>>  Mike
>>
>>  On Nov 22, 2017 8:53 PM, "Ali Nazemian"  wrote:
>>
>>  > We tried to use it, but we had the same issue. It was not documented. We
>>  > tried to use it, and we had some issues. It also was not exactly what we
>>  > wanted, so we decided to create something from scratch by using
>>  > Elasticsearch Curator. We wanted to have an ability to manage different
>>  > prune mechanism for different feeds. Having a hard threshold to remove
>>  > index and Soft threshold to close that index. Maybe it can be a feature
>>  to
>>  > add to the indexing JSON config file per feed.
>>  >
>>  > Cheers,
>>  > Ali
>>  >
>>  > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic <
>>  > michael.miklav...@gmail.com> wrote:
>>  >
>>  > > From what I can tell, the data pruner isn't documented anywhere, so I'm
>>  > > curious if anybody is using this, and if so, how are you using it?
>>  > >
>>  > > -
>>  > > https://github.com/apache/metron/blob/master/metron-
>>  > > platform/metron-data-management/README.md
>>  > > -
>>  > > https://github.com/apache/metron/blob/master/metron-
>>  > > platform/metron-data-management/src/main/java/org/
>>  > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java
>>  > > -
>>  > > https://github.com/apache/metron/blob/master/metron-
>>  > > platform/metron-data-management/src/main/java/org/
>>  > > apache/metron/dataloads/bulk/DataPruner.java
>>  > >
>>  > > It looks to me that it allows you to specify the start date and a
>>  number
>>  > of
>>  > > days for lookback from the start date to purge along with a regex
>>  pattern
>>  > > to match the index name. It also does not look like it has any built-in
>>  > > scheduling semantics, so I assume this was a cron job. I think that
>>  about
>>  > > covers it. Anything I've missed?
>>  > >
>>  > > I'm adding a quick doc write-up to METRON-939 (
>>  > > https://github.com/apache/metron/pull/840) for using Curator to prune
>>  > > indices from Elasticsearch. It is desirable to make sure I've covered
>>  > > existing use cases.
>>  > >
>>  > > Best,
>>  > > Mike
>>  > >
>>  >
>>  >
>>  >
>>  > --
>>  > A.Nazemian
>>  >
>
> --
> A.Nazemian

--- 
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/803


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Great.  I will get this PR merged.  I am glad to see that this one is ready 
to go.


---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/849
  
+1, Tested on OSX and on centos 6 with and without compliant g++ installed
Thanks Nick!


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153336117
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

No worries.  Glad to have a second set of eyes on it.


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153334325
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

OK, I ran it, I did mis-understand.  sorry.



---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r15358
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/README.md ---
@@ -1,17 +1,34 @@
 # Overview
-The Metron ansible-docker container is provided in an effort reduce the 
installation burden of deploying Metron in a live envirionment.
-It is provisioned with software required to sucessfully run the deployment 
scripts.
+The Metron ansible-docker container is provided in an effort reduce the 
installation burden of building Metron.
+It may also be used to deploy Metron in a live environment.
+It is provisioned with software required to sucessfully build metron run 
the deployment scripts.
--- End diff --

thanks!


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153332780
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile ---
@@ -14,13 +14,18 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-FROM centos:centos6
+FROM centos:centos6.9
 MAINTAINER Apache Metron
 
 RUN yum install -y tar
 RUN yum install -y wget
+# base development tools required
 RUN yum groupinstall -y "Development tools"
+# newer cpp 11 support required for building node modules
+RUN wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O 
/etc/yum.repos.d/devtools-2.repo
--- End diff --

yeah, thanks.  Nick also pointed this out in a jira ( referring to your 
steps ), i'll have this in in a moment


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/803
  
I agree. I'm fine with going ahead with this, but I'd like to see end to 
end stability being addressed as the next UI priority, which I believe 
@iraghumitra is already doing some work on.

+1


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153328847
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

i agree about it not being worth the trouble.

as far as g++, I would expect us to check (pseudo)

if c++ in path then
build test program
else
echo No Cpp, you are in trouble!
fi

Maybe I'm mis-reading this, if there is no cpp, it will still try to build 
etc won't it?




---

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
Yes, I agree. It completely makes sense to minimize the scope and work on 
stabilizing this version at this moment. 


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153327669
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/README.md ---
@@ -1,17 +1,34 @@
 # Overview
-The Metron ansible-docker container is provided in an effort reduce the 
installation burden of deploying Metron in a live envirionment.
-It is provisioned with software required to sucessfully run the deployment 
scripts.
+The Metron ansible-docker container is provided in an effort reduce the 
installation burden of building Metron.
+It may also be used to deploy Metron in a live environment.
+It is provisioned with software required to sucessfully build metron run 
the deployment scripts.
--- End diff --

`s/metron run/metron and run/`


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153326885
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile ---
@@ -14,13 +14,18 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-FROM centos:centos6
+FROM centos:centos6.9
 MAINTAINER Apache Metron
 
 RUN yum install -y tar
 RUN yum install -y wget
+# base development tools required
 RUN yum groupinstall -y "Development tools"
+# newer cpp 11 support required for building node modules
+RUN wget http://people.centos.org/tru/devtools-2/devtools-2.repo -O 
/etc/yum.repos.d/devtools-2.repo
--- End diff --

Please upgrade to devtoolset-4.  
```
yum -y install centos-release-scl
yum -y install devtoolset-4-gcc devtoolset-4-gcc-c++ devtoolset-4-binutils
```
or similar.


---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/848#discussion_r153326917
  
--- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile ---
@@ -33,18 +38,28 @@ RUN tar xvf setuptools-11.3.tar.gz
 WORKDIR /usr/src/setuptools-11.3
 RUN python2.7 setup.py install
 RUN easy_install-2.7 pip
+# install ansible and set the configuration var
 RUN pip2.7 install ansible==2.0.0.2
 RUN pip2.7 install boto
 COPY ansible.cfg /root/
 ENV ANSIBLE_CONFIG /root/ansible.cfg
+# java
 RUN yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
 RUN yum install -y which
 RUN yum install -y nss
 WORKDIR /usr/src
+# setup maven
 RUN wget 
http://apache.cs.utah.edu/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
 RUN tar xzvf apache-maven-3.3.9-bin.tar.gz
 RUN mv apache-maven-3.3.9 /opt/maven
 RUN ln -s /opt/maven/bin/mvn /usr/bin/mvn
-RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint 
&& yum clean all
+# install rpm tools required to build rpms
+RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint 
make && yum clean all
+# create a .bashrc for root, enabling the cpp 11 toolset
+RUN touch /root/.bashrc \
+ && cat '/opt/rh/devtoolset-2/enable' >> /root/.bashrc
--- End diff --

Same as above


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153325047
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

> A standard way in the language is to check for the value of __cplusplus

Thanks, I saw that.  I didn't think it was worth the trouble.  IMHO


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153324714
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

> Do we have to account for platform_info on a non-dev machine? If there is 
no g++ at all?

The platform-info script is there to tell us whether a machine can be used 
to deploy metron.  Right now, you have to build Metron to deploy it, so the 
script checks all dev dependencies.

If g++ is not in the path, then the script will tell us that.  And it would 
carry on its merry way.  That's the behavior that I would expect.

Maybe I am not following you. Can you describe the scenario you are 
thinking of more?


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153322856
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

Do we have to account for platform_info on a non-dev machine?  If there is 
no g++ at all?


---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/849#discussion_r153322627
  
--- Diff: metron-deployment/scripts/platform-info.sh ---
@@ -73,6 +73,30 @@ echo "--"
 echo "npm"
 npm --version
 
+# C++ compiler
+echo "--"
+echo "g++"
+g++ --version
+
+# C++11 compliant compiler
+echo "--"
+OBJFILE=/tmp/test
+CPPFILE=/tmp/test.cpp
+cat > $CPPFILE <<- EOM
+#include 
+using namespace std;
+int main() {
+cout << "Hello World!" << endl;
+return 0;
+}
+EOM
+g++ -std=c++11 $CPPFILE -o $OBJFILE
+if [ $? -eq 0 ]; then
+echo "Compiler is C++11 compliant"
+else
+echo "Warning: Compiler is NOT C++11 compliant"
+fi
+rm -f $CPPFILE $OBJFILE
 
--- End diff --

A standard way in the language is to check for the value of __cplusplus

201402L  is CPP 14
201103L is CPP 11

I don't have a preference, just throwing this out there.


---

[GitHub] metron pull request #849: METRON-1320 Add C++11 Compliance Check to platform...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/849

METRON-1320 Add C++11 Compliance Check to platform-info

Some of the module dependencies for the Management and Alerts UI must be 
built natively on the host.  This requires a C/C++ compiler.  In addition, some 
of the dependencies require a C++11 compliant compiler.  This is causing 
problems for users who attempt to build Metron on a system with an older 
version of GCC, like CentOS 6.

Not having a C++11 compliant compiler can cause some non-obvious error 
messages when the build fails.  This adds a version check for GCC and also a 
C++11 compliance check.  The compiler itself must be on the user's PATH, which 
is what the Node modules also require.

If the compiler is C++11 compliant, you will see the following.
```
--
g++
g++ (GCC) 5.3.1 20160406 (Red Hat 5.3.1-6)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

--
Compiler is C++11 compliant
```

If the compiler is NOT C++11 compliant, you will see the following.
```
--
g++
g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

--
cc1plus: error: unrecognized command line option "-std=c++11"
Warning: Compiler is NOT C++11 compliant
```

This was tested manually on Mac OS X, CentOS 6, CentOS 7, and Ubuntu Trusty 
64.


## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1320

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/849.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #849


commit 4ca17d5780729a61615e7ee6bc86e6ddf86c339e
Author: Nick Allen 
Date:   2017-11-27T20:29:38Z

METRON-1320 Cannot perform a bare-metal installation




---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/848

METRON-1333 Ensure that ansible-docker can be used to build metron ( 
including rpms )

The ansible-docker container could be used to build metron ( even the rpms 
) and run the ansible scripts. This is no longer true, as the new node ui 
projects have native compilation that requires c++ 11 support in gcc, and rpm 
build requires node to be installed ( not local installed as in the mvn 
packaging ).
Many users are having problems building metron on centos platforms, having 
this container available and functioning would be a great help.

This PR addresses this by:

- installing devtoolset-2 into the container to get cpp 11
- creating a .bash_profile for root that enables the devtoolset environment
- installing node into the container since building rpms now requires node 
( and not just the local node used during building in maven )
- documenting building in container, as well as mapping local .m2 repo into 
the container

You can now build and test metron, including building the rpms.


## Testing

Follow the steps in the read me and build metron:

`mvn clean package`
`mvn clean package -DskipTests`
`mvn clean install && cd metron-deployment && mvn package -P build-rpms`


## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x ] Have you run a build, test, build rpm set of builds in the container?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron docker_build_env

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/848.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #848


commit 3b9f0e9aa13f3140433af1c9fe84ef87ac9016ff
Author: Otto Fowler 
Date:   2017-11-27T16:18:28Z

Refactor and document the ansible-docker container such that it now 
supports building metron and it's rpms
Also updated to centos 6.9




---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/803
  
I agree with @nickwallen.  I think we're good to merge this as long as e2e 
tests are being addressed in a separate PR.  +1


---

[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/840#discussion_r153289006
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template
 ---
@@ -102,13 +94,25 @@
   "match_mapping_type": "*"
 }
   },
-  {
-"threat_triage_reason": {
-  "mapping": {
-"type": "string"
-  },
-  "match": "threat:triage:rules:*:reason",
-  "match_mapping_type": "*"
+{
+  "threat_triage_reason": {
+"mapping": {
+  "type": "text",
+  "fielddata": "true"
+},
+"match": "threat.triage.rules:*:reason",
+"match_mapping_type": "*"
+  }
+},
+{
+  "threat_triage_name": {
+"mapping": {
+  "type": "text",
+  "fielddata": "true"
+},
+"match": "threat.triage.rules:*:name",
+"match_mapping_type": "*"
+  }
 }
   },
--- End diff --

This brace is extraneous, I'm guessing a merge broke it. Drop it, but keep 
the comma and we should be good.  I'd just get the formatting lined back up 
while you're in there.


---

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
For reference, here is a list of some of the follow-on work we should 
consider:

- Improvements to Kibana dashboard
- Add new timestamp field to parsers and index templates to take place of 
_timestamp
- Fix Log4j logging problem - classpath issues
- Upgrade to ES 6.x
- Migrate to new ES REST client
- Improved config management for ES 5.x+ in Ambari
- New field name conventions?



---

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/840
  
@mraliagha I do think we should consider revisiting the field name 
conventions, but I'd push for that as a follow-on task. As discussed in other 
points on this thread, e.g. going straight to ES 6.x, this PR is already 
massive in scope, and adding further non-critical improvements will increase 
the surface area of those changes. 

Now that we have the meta alerts fixes in master, I strongly encourage the 
community to stabilize around as minimal an ES upgrade as possible with 
incremental improvements and feature enhancements once we've been able to 
convince ourselves through the unit and integration test suites and functional 
testing that we're in a healthy and stable place with minimal risk of adding 
regressions.


---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
+1  

I'd like to see sign-off from at least one other committer (if not more) 
before this gets merged.

I previously outlined the manual functional testing that I performed.  All 
the core functionality is there.  We also have sufficient test coverage for the 
functionality that was added.

I have been able to get the e2e tests to run good enough (IMO) to move 
forward with the PR.  Based on the experimentation that I describe below, I 
believe that the test failures are due to the tests and test infrastructure and 
not the core functionality in the PR.  

The intermittent failures are a pre-existing condition to this PR.  This 
makes me comfortable enough to merge this PR and then follow-on with test 
fixes.  @iraghumitra mentioned that he is currently working on improving the 
tests and will open a separate PR.  

1. I used Brew to install the following versions of Node/NPM.
```
$ node --version
v8.9.1

$ npm --version
5.5.1
```

2. Before running the tests, truncate the `metron_update` table.  The tests 
cannot currently clear out old test data stored in HBase.

3. Enabling/disabling sets of tests in 
`metron-interface/metron-alerts/protractor.conf.js` also allowed me to see that 
all the tests can pass on an intermittent basis.  It seems that earlier test 
failures may cause later tests to fail unnecessarily.

4. Run the e2e tests using the following steps.

```
cd metron-interface/metron-alerts
npm install
./node_modules/.bin/webdriver-manager update
./node_modules/.bin/protractor
```

Thanks for working through this with me @iraghumitra.



---

Re: [DISCUSS] Upcoming Release

[Otto Fowler]

Considering the problems we are having with people building the node stuff
on centos,  would it make sense to wait to take
a potential PR that allows full metron builds in our ansible docker image?



On November 26, 2017 at 21:26:27, Matt Foley (ma...@apache.org) wrote:

Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
Regarding status of the release effort, still pending METRON-1252, so not
making the release branch yet.

Regards,
--Matt

On 11/17/17, 1:32 PM, "Matt Foley"  wrote:

(With release manager hat on)

The community has proposed a release of Metron in the near future, focusing
on Meta-alerts running in Elasticsearch.
Congrats on getting so many of the below already done. At this point, only
METRON-1252, and the discussion of how to handle joint release of the
Metron bro plugin, remain as gating items for the release. I project these
will be resolved next week, so let’s propose the following:

Sometime next week, after the last bits are done, I’ll start the release
process and create the release branch.

The proposed new version will be 0.4.2, unless there are backward
incompatible changes that support making it 0.5.0.
Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
having Docs Text indicating so.
***If anyone knows that some of the commits included since 0.4.1 introduce
backward incompatibility, please say so now on this thread, and mark the
Jira as such.***

The 90 or so jiras/commits already in master branch since 0.4.1 are listed
below.
Thanks,
--Matt

METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some
Records (nickwallen) closes apache/metron#832
METRON-1294 IP addresses are not formatted correctly in facet and group
results (merrimanr) closes apache/metron#827
METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
cluster (merrimanr) closes apache/metron#826
METRON-1290 Only first 10 alerts are update when a MetaAlert status is
changed to inactive (justinleet) closes apache/metron#842
METRON-1311 Service Check Should Check Elasticsearch Index Templates
(nickwallen) closes apache/metron#839
METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr)
closes apache/metron#824
METRON-1309 Change metron-deployment to pull the plugin from
apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
closes apache/metron#838
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes
apache/metron#828
METRON-1307 Force install of java8 since java9 does not appear to work with
the scripts (brianhurley via ottobackwards) closes apache/metron#835
METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
cestella) closes apache/incubator-metron#829
METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes
apache/metron#821
METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen)
closes apache/metron#820
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page
(iraghumitra via merrimanr) closes apache/metron#819
METRON-1283 Install Elasticsearch template as a part of the mpack startup
scripts (anandsubbu via nickwallen) closes apache/metron#817
METRON-1254: Conditionals as map keys do not function in Stellar closes
apache/incubator-metron#801
METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes
apache/metron#805
METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet)
closes apache/metron#818
METRON-1270: fix for warnings missing @return tag argument in
metron-analytics/metron-profiler-common and metron-profiler-client closes
apache/incubator-metron#810
METRON-1272 Hide child alerts from searches and grouping if they belong to
meta alerts (justinleet) closes apache/metron#811
METRON-1224 Add time range selection to search control (iraghumitra via
james-sirota) closes apache/metron#796
METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
justinleet) closes apache/metron#816
METRON-1243: Add a REST endpoint which allows us to get a list of all
indice closes apache/incubator-metron#797
METRON-1196 Increment master version number to 0.4.2 for on-going
development (mattf-horton) closes apache/metron#767
METRON-1278 Strip Build Status widget from root README.md in
site-book build (mattf-horton) closes apache/metron#815
METRON-1274 Master has failure in StormControllerIntegrationTest
(merrimanr) closes apache/metron#813
METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
apache/metron#809
METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes
apache/metron#804
METRON-1251: Typo and formatting fixes for metron-rest README closes
apache/incubator-metron#800
METRON-1241: Enable the REST API to use a cache for the zookeeper config
similar to the Bolts closes apache/incubator-metron#795

Re: [DISCUSS] NPM / Node Problems

[Otto Fowler]

Also, since I changed the profiles to not run the rpm docker if you are in
docker already ( and put the rpm tools into the ansible docker ) a while ago
we may be able to build world in the ansible image, and point folks having
issues to that….


On November 27, 2017 at 10:57:03, Otto Fowler (ottobackwa...@gmail.com)
wrote:

OK,
So I have >mvn clean package working in docker.
I want to try a couple of things and maybe I can throw a pr together.



On November 27, 2017 at 10:03:31, Otto Fowler (ottobackwa...@gmail.com)
wrote:

First issue is that we need c++ 11 on centos 6.8



On November 27, 2017 at 09:53:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Well, that’s good news on that issue. Reproducing the problem is half way
to solving it, right?

I would still say there are some systemic things going on that have
manifested in a variety of ways on both the users and dev list, so it’s
worth us having a good look at a more robust approach to node dependencies
(both npm ones, and the native ones)

Simon

On 27 Nov 2017, at 13:30, Otto Fowler  wrote:

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve 

Re: [DISCUSS] NPM / Node Problems

[Otto Fowler]

OK,
So I have >mvn clean package working in docker.
I want to try a couple of things and maybe I can throw a pr together.



On November 27, 2017 at 10:03:31, Otto Fowler (ottobackwa...@gmail.com)
wrote:

First issue is that we need c++ 11 on centos 6.8



On November 27, 2017 at 09:53:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Well, that’s good news on that issue. Reproducing the problem is half way
to solving it, right?

I would still say there are some systemic things going on that have
manifested in a variety of ways on both the users and dev list, so it’s
worth us having a good look at a more robust approach to node dependencies
(both npm ones, and the native ones)

Simon

On 27 Nov 2017, at 13:30, Otto Fowler  wrote:

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve for the dev community.
>>>
>>> (1) NPM is causing us some major headaches. Which version do we require?

>>> How do I install that version (on Mac, Windows, Linux)? Does YARN help
>>> here at all?
>>>
>>> (2) Can we automate the prerequisite checks that we currently do
manually
>>> with `platform-info.sh`? An automated check 

Re: [DISCUSS] NPM / Node Problems

[zeo...@gmail.com]

Note that I cleaned up the ansible scripts that install C++ 11 in my latest
PR , but it's not super
relevant to this conversation.

Jon

On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com  wrote:

> That was also required for bro 2.5.2, so I did that here
> .
> Feel free to reuse the approach elsewhere
>
> Jon
>
> On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
> wrote:
>
>> First issue is that we need c++ 11 on centos 6.8
>>
>>
>>
>> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Well, that’s good news on that issue. Reproducing the problem is half way
>> to solving it, right?
>>
>> I would still say there are some systemic things going on that have
>> manifested in a variety of ways on both the users and dev list, so it’s
>> worth us having a good look at a more robust approach to node dependencies
>> (both npm ones, and the native ones)
>>
>> Simon
>>
>> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>>
>> I can reproduce the failure in out ansible docker build container, which
>> is
>> also centos.
>> The issue is building our node on centos in all these cases.
>>
>>
>>
>> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Thinking about this, doesn’t our build plugin explicitly install it’s own
>> node? So actually all the node version things may be a red herring, since
>> this is under our control through the pom. Not sure if we actually
>> exercising this control. It seems that some of the errors people report
>> are
>> more to do with compilation failures for native node modules, which is
>> doesn’t pin (i.e. things like system library dependencies). I’m not sure
>> what we have in the dependency tree that requires complex native
>> dependencies, but this might just be one of those node things we could doc
>> around.
>>
>> This scenario would be fixed by standardising the build container.
>>
>> Yarn’s big thing is that it enables faster dependency resolution and local
>> caching, right? It does not seem to address any of the problems we see,
>> but
>> sure, it’s the shiny new dependency system for node modules, which might
>> make npm less horrible to deal with, so worth looking into.
>>
>> The other issue that I’ve seen people run into a lot is flat out download
>> errors. This could be helped by finding our versions, maybe with yarn, but
>> let’s face it, package-lock.json could also do that with npm, albeit with
>> a
>> slightly slower algorithm. However, short of bundling and hosting deps
>> ourselves, I suspect the download errors are beyond our control, and
>> certainly beyond the scope of this project (fix maven, fix npm, fix all
>> the
>> node hosting servers…)
>>
>> Simon
>>
>>
>> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
>> raghumitra@gmail.com>
>> wrote:
>> >
>> > Looking at some of the build failure emails and past experience i
>> > would suggest having a node & npm version check in our build scripts
>> > and moving dependency management to yarn.
>> >
>> > We need not restrict the build to a specific version of node & npm but
>> > we can surely suggest a min version required to build UI successfully.
>> >
>> > -Raghu
>> >
>> >
>> >
>> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>> >  wrote:
>> >> Agreeing with Nick, it seems like the main reason people are building
>> themselves, and hitting all these environmental issues, is that we do not
>> as a project produce binary release artefacts (the rpms which users could
>> just install) and instead leave that for the commercial distributors to
>> do.
>> >>
>> >> Yarn may help with some of the dependency version issues we’re having,
>> but not afaik with the core missing library headers / build tools / node
>> and npm version issue, those would seem to fit a documentation fix and
>> improvements to platform-info to flag the problems, so this can then be a
>> pre-flight check tool as well as a diagnostic tool.
>> >>
>> >> Another option I would put on the table is to standardise our build
>> environment, so that the non-java bits are run in a standard docker image
>> or something fo the sort, that way we can take control of all the
>> environmental and OS dependent pieces, much as we do right now with the
>> rpm
>> build sections of the mpack build.
>> >>
>> >> The challenge here will be adding the relevant maven support. At the
>> moment we’re relying on the maven npm and node build plugins, this would
>> likely need replacing with something custom and a challenge to support to
>> go dow this route.
>> >>
>> >> Perhaps the real answer here is to push people who are just kicking the
>> tyres towards a binary distribution, or at least rpm artefacts as part of
>> the Apache release to give them a 

Re: [DISCUSS] NPM / Node Problems

[zeo...@gmail.com]

That was also required for bro 2.5.2, so I did that here
.
Feel free to reuse the approach elsewhere

Jon

On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
wrote:

> First issue is that we need c++ 11 on centos 6.8
>
>
>
> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Well, that’s good news on that issue. Reproducing the problem is half way
> to solving it, right?
>
> I would still say there are some systemic things going on that have
> manifested in a variety of ways on both the users and dev list, so it’s
> worth us having a good look at a more robust approach to node dependencies
> (both npm ones, and the native ones)
>
> Simon
>
> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>
> I can reproduce the failure in out ansible docker build container, which is
> also centos.
> The issue is building our node on centos in all these cases.
>
>
>
> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Thinking about this, doesn’t our build plugin explicitly install it’s own
> node? So actually all the node version things may be a red herring, since
> this is under our control through the pom. Not sure if we actually
> exercising this control. It seems that some of the errors people report are
> more to do with compilation failures for native node modules, which is
> doesn’t pin (i.e. things like system library dependencies). I’m not sure
> what we have in the dependency tree that requires complex native
> dependencies, but this might just be one of those node things we could doc
> around.
>
> This scenario would be fixed by standardising the build container.
>
> Yarn’s big thing is that it enables faster dependency resolution and local
> caching, right? It does not seem to address any of the problems we see, but
> sure, it’s the shiny new dependency system for node modules, which might
> make npm less horrible to deal with, so worth looking into.
>
> The other issue that I’ve seen people run into a lot is flat out download
> errors. This could be helped by finding our versions, maybe with yarn, but
> let’s face it, package-lock.json could also do that with npm, albeit with a
> slightly slower algorithm. However, short of bundling and hosting deps
> ourselves, I suspect the download errors are beyond our control, and
> certainly beyond the scope of this project (fix maven, fix npm, fix all the
> node hosting servers…)
>
> Simon
>
>
> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
> raghumitra@gmail.com>
> wrote:
> >
> > Looking at some of the build failure emails and past experience i
> > would suggest having a node & npm version check in our build scripts
> > and moving dependency management to yarn.
> >
> > We need not restrict the build to a specific version of node & npm but
> > we can surely suggest a min version required to build UI successfully.
> >
> > -Raghu
> >
> >
> >
> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
> >  wrote:
> >> Agreeing with Nick, it seems like the main reason people are building
> themselves, and hitting all these environmental issues, is that we do not
> as a project produce binary release artefacts (the rpms which users could
> just install) and instead leave that for the commercial distributors to do.
> >>
> >> Yarn may help with some of the dependency version issues we’re having,
> but not afaik with the core missing library headers / build tools / node
> and npm version issue, those would seem to fit a documentation fix and
> improvements to platform-info to flag the problems, so this can then be a
> pre-flight check tool as well as a diagnostic tool.
> >>
> >> Another option I would put on the table is to standardise our build
> environment, so that the non-java bits are run in a standard docker image
> or something fo the sort, that way we can take control of all the
> environmental and OS dependent pieces, much as we do right now with the rpm
> build sections of the mpack build.
> >>
> >> The challenge here will be adding the relevant maven support. At the
> moment we’re relying on the maven npm and node build plugins, this would
> likely need replacing with something custom and a challenge to support to
> go dow this route.
> >>
> >> Perhaps the real answer here is to push people who are just kicking the
> tyres towards a binary distribution, or at least rpm artefacts as part of
> the Apache release to give them a head start for a happy path on a known
> good OS environment.
> >>
> >> Simon
> >>
> >>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
> >>>
> >>> Yes, it is a problem. I think you've identified a couple important
> things
> >>> that we could address in parallel. I see these as challenges we need to
> >>> solve for the dev community.
> >>>
> >>> (1) NPM is causing us some 

Re: [DISCUSS] NPM / Node Problems

[Otto Fowler]

First issue is that we need c++ 11 on centos 6.8



On November 27, 2017 at 09:53:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Well, that’s good news on that issue. Reproducing the problem is half way
to solving it, right?

I would still say there are some systemic things going on that have
manifested in a variety of ways on both the users and dev list, so it’s
worth us having a good look at a more robust approach to node dependencies
(both npm ones, and the native ones)

Simon

On 27 Nov 2017, at 13:30, Otto Fowler  wrote:

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve for the dev community.
>>>
>>> (1) NPM is causing us some major headaches. Which version do we require?

>>> How do I install that version (on Mac, Windows, Linux)? Does YARN help
>>> here at all?
>>>
>>> (2) Can we automate the prerequisite checks that we currently do
manually
>>> with `platform-info.sh`? An automated check could run and fail as part
of
>>> the build or deployment process.
>>>
>>>
>>>
>>> More importantly though is that users should not have to build Metron at

>>> all. They should not have to worry about 

Re: [DISCUSS] NPM / Node Problems

[Simon Elliston Ball]

Well, that’s good news on that issue. Reproducing the problem is half way to 
solving it, right? 

I would still say there are some systemic things going on that have manifested 
in a variety of ways on both the users and dev list, so it’s worth us having a 
good look at a more robust approach to node dependencies (both npm ones, and 
the native ones) 

Simon

> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
> 
> I can reproduce the failure in out ansible docker build container, which is 
> also centos.
> The issue is building our node on centos in all these cases.
> 
> 
> 
> On November 27, 2017 at 07:02:51, Simon Elliston Ball 
> (si...@simonellistonball.com ) wrote:
> 
>> Thinking about this, doesn’t our build plugin explicitly install it’s own 
>> node? So actually all the node version things may be a red herring, since 
>> this is under our control through the pom. Not sure if we actually 
>> exercising this control. It seems that some of the errors people report are 
>> more to do with compilation failures for native node modules, which is 
>> doesn’t pin (i.e. things like system library dependencies). I’m not sure 
>> what we have in the dependency tree that requires complex native 
>> dependencies, but this might just be one of those node things we could doc 
>> around.  
>> 
>> This scenario would be fixed by standardising the build container. 
>> 
>> Yarn’s big thing is that it enables faster dependency resolution and local 
>> caching, right? It does not seem to address any of the problems we see, but 
>> sure, it’s the shiny new dependency system for node modules, which might 
>> make npm less horrible to deal with, so worth looking into.  
>> 
>> The other issue that I’ve seen people run into a lot is flat out download 
>> errors. This could be helped by finding our versions, maybe with yarn, but 
>> let’s face it, package-lock.json could also do that with npm, albeit with a 
>> slightly slower algorithm. However, short of bundling and hosting deps 
>> ourselves, I suspect the download errors are beyond our control, and 
>> certainly beyond the scope of this project (fix maven, fix npm, fix all the 
>> node hosting servers…) 
>> 
>> Simon 
>> 
>> 
>> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda > > > wrote: 
>> >  
>> > Looking at some of the build failure emails and past experience i 
>> > would suggest having a node & npm version check in our build scripts 
>> > and moving dependency management to yarn. 
>> >  
>> > We need not restrict the build to a specific version of node & npm but 
>> > we can surely suggest a min version required to build UI successfully. 
>> >  
>> > -Raghu 
>> >  
>> >  
>> >  
>> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball 
>> > > wrote: 
>> >> Agreeing with Nick, it seems like the main reason people are building 
>> >> themselves, and hitting all these environmental issues, is that we do not 
>> >> as a project produce binary release artefacts (the rpms which users could 
>> >> just install) and instead leave that for the commercial distributors to 
>> >> do. 
>> >>  
>> >> Yarn may help with some of the dependency version issues we’re having, 
>> >> but not afaik with the core missing library headers / build tools / node 
>> >> and npm version issue, those would seem to fit a documentation fix and 
>> >> improvements to platform-info to flag the problems, so this can then be a 
>> >> pre-flight check tool as well as a diagnostic tool. 
>> >>  
>> >> Another option I would put on the table is to standardise our build 
>> >> environment, so that the non-java bits are run in a standard docker image 
>> >> or something fo the sort, that way we can take control of all the 
>> >> environmental and OS dependent pieces, much as we do right now with the 
>> >> rpm build sections of the mpack build. 
>> >>  
>> >> The challenge here will be adding the relevant maven support. At the 
>> >> moment we’re relying on the maven npm and node build plugins, this would 
>> >> likely need replacing with something custom and a challenge to support to 
>> >> go dow this route. 
>> >>  
>> >> Perhaps the real answer here is to push people who are just kicking the 
>> >> tyres towards a binary distribution, or at least rpm artefacts as part of 
>> >> the Apache release to give them a head start for a happy path on a known 
>> >> good OS environment. 
>> >>  
>> >> Simon 
>> >>  
>> >>> On 24 Nov 2017, at 16:01, Nick Allen > >>> > wrote: 
>> >>>  
>> >>> Yes, it is a problem. I think you've identified a couple important 
>> >>> things 
>> >>> that we could address in parallel. I see these as challenges we need to 
>> >>> solve for the dev community. 
>> >>>  
>> >>> (1) NPM is causing us some major headaches. Which version do we require? 

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
@iraghumitra Can you describe how you install Node/NPM on your development 
box?  I want to install using the same mechanism (and versions) and see if I 
can get the e2e tests all working like you.  Thanks


---

Re: [DISCUSS] NPM / Node Problems

[Otto Fowler]

I can reproduce the failure in out ansible docker build container, which is
also centos.
The issue is building our node on centos in all these cases.



On November 27, 2017 at 07:02:51, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Thinking about this, doesn’t our build plugin explicitly install it’s own
node? So actually all the node version things may be a red herring, since
this is under our control through the pom. Not sure if we actually
exercising this control. It seems that some of the errors people report are
more to do with compilation failures for native node modules, which is
doesn’t pin (i.e. things like system library dependencies). I’m not sure
what we have in the dependency tree that requires complex native
dependencies, but this might just be one of those node things we could doc
around.

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local
caching, right? It does not seem to address any of the problems we see, but
sure, it’s the shiny new dependency system for node modules, which might
make npm less horrible to deal with, so worth looking into.

The other issue that I’ve seen people run into a lot is flat out download
errors. This could be helped by finding our versions, maybe with yarn, but
let’s face it, package-lock.json could also do that with npm, albeit with a
slightly slower algorithm. However, short of bundling and hosting deps
ourselves, I suspect the download errors are beyond our control, and
certainly beyond the scope of this project (fix maven, fix npm, fix all the
node hosting servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda 
wrote:
>
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
>
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
>
> -Raghu
>
>
>
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building
themselves, and hitting all these environmental issues, is that we do not
as a project produce binary release artefacts (the rpms which users could
just install) and instead leave that for the commercial distributors to do.
>>
>> Yarn may help with some of the dependency version issues we’re having,
but not afaik with the core missing library headers / build tools / node
and npm version issue, those would seem to fit a documentation fix and
improvements to platform-info to flag the problems, so this can then be a
pre-flight check tool as well as a diagnostic tool.
>>
>> Another option I would put on the table is to standardise our build
environment, so that the non-java bits are run in a standard docker image
or something fo the sort, that way we can take control of all the
environmental and OS dependent pieces, much as we do right now with the rpm
build sections of the mpack build.
>>
>> The challenge here will be adding the relevant maven support. At the
moment we’re relying on the maven npm and node build plugins, this would
likely need replacing with something custom and a challenge to support to
go dow this route.
>>
>> Perhaps the real answer here is to push people who are just kicking the
tyres towards a binary distribution, or at least rpm artefacts as part of
the Apache release to give them a head start for a happy path on a known
good OS environment.
>>
>> Simon
>>
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>>
>>> Yes, it is a problem. I think you've identified a couple important
things
>>> that we could address in parallel. I see these as challenges we need to
>>> solve for the dev community.
>>>
>>> (1) NPM is causing us some major headaches. Which version do we
require?
>>> How do I install that version (on Mac, Windows, Linux)? Does YARN help
>>> here at all?
>>>
>>> (2) Can we automate the prerequisite checks that we currently do
manually
>>> with `platform-info.sh`? An automated check could run and fail as part
of
>>> the build or deployment process.
>>>
>>>
>>>
>>> More importantly though is that users should not have to build Metron
at
>>> all. They should not have to worry about installing NPM and the rest of
>>> the development tooling. Here are some options that are not mutually
>>> exclusive.
>>>
>>>
>>> (1) Create an image in Atlas that has Metron fully installed. A new
user
>>> could run single node Metron on their laptop with a single command and
the
>>> only prereqs would be Vagrant and Virtualbox. We could cut new images
for
>>> each Metron release. Or selectively cut new dev images from master as
we
>>> see fit.
>>>
>>> (2) Distribute the Metron RPMs (and the MPack tarball?) so that users
can
>>> install Metron on a cluster without 

Re: [DISCUSS] NPM / Node Problems

[Simon Elliston Ball]

Thinking about this, doesn’t our build plugin explicitly install it’s own node? 
So actually all the node version things may be a red herring, since this is 
under our control through the pom. Not sure if we actually exercising this 
control. It seems that some of the errors people report are more to do with 
compilation failures for native node modules, which is doesn’t pin (i.e. things 
like system library dependencies). I’m not sure what we have in the dependency 
tree that requires complex native dependencies, but this might just be one of 
those node things we could doc around. 

This scenario would be fixed by standardising the build container.

Yarn’s big thing is that it enables faster dependency resolution and local 
caching, right? It does not seem to address any of the problems we see, but 
sure, it’s the shiny new dependency system for node modules, which might make 
npm less horrible to deal with, so worth looking into. 

The other issue that I’ve seen people run into a lot is flat out download 
errors. This could be helped by finding our versions, maybe with yarn, but 
let’s face it, package-lock.json could also do that with npm, albeit with a 
slightly slower algorithm. However, short of bundling and hosting deps 
ourselves, I suspect the download errors are beyond our control, and certainly 
beyond the scope of this project (fix maven, fix npm, fix all the node hosting 
servers…)

Simon


> On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda  
> wrote:
> 
> Looking at some of the build failure emails and past experience i
> would suggest having a node & npm version check in our build scripts
> and moving dependency management to yarn.
> 
> We need not restrict the build to a specific version of node & npm but
> we can surely suggest a min version required to build UI successfully.
> 
> -Raghu
> 
> 
> 
> On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>  wrote:
>> Agreeing with Nick, it seems like the main reason people are building 
>> themselves, and hitting all these environmental issues, is that we do not as 
>> a project produce binary release artefacts (the rpms which users could just 
>> install) and instead leave that for the commercial distributors to do.
>> 
>> Yarn may help with some of the dependency version issues we’re having, but 
>> not afaik with the core missing library headers / build tools / node and npm 
>> version issue, those would seem to fit a documentation fix and improvements 
>> to platform-info to flag the problems, so this can then be a pre-flight 
>> check tool as well as a diagnostic tool.
>> 
>> Another option I would put on the table is to standardise our build 
>> environment, so that the non-java bits are run in a standard docker image or 
>> something fo the sort, that way we can take control of all the environmental 
>> and OS dependent pieces, much as we do right now with the rpm build sections 
>> of the mpack build.
>> 
>> The challenge here will be adding the relevant maven support. At the moment 
>> we’re relying on the maven npm and node build plugins, this would likely 
>> need replacing with something custom and a challenge to support to go dow 
>> this route.
>> 
>> Perhaps the real answer here is to push people who are just kicking the 
>> tyres towards a binary distribution, or at least rpm artefacts as part of 
>> the Apache release to give them a head start for a happy path on a known 
>> good OS environment.
>> 
>> Simon
>> 
>>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
>>> 
>>> Yes, it is a problem.  I think you've identified a couple important things
>>> that we could address in parallel.  I see these as challenges we need to
>>> solve for the dev community.
>>> 
>>> (1) NPM is causing us some major headaches.  Which version do we require?
>>> How do I install that version (on Mac, Windows, Linux)?  Does YARN help
>>> here at all?
>>> 
>>> (2) Can we automate the prerequisite checks that we currently do manually
>>> with `platform-info.sh`?  An automated check could run and fail as part of
>>> the build or deployment process.
>>> 
>>> 
>>> 
>>> More importantly though is that users should not have to build Metron at
>>> all.  They should not have to worry about installing NPM and the rest of
>>> the development tooling.   Here are some options that are not mutually
>>> exclusive.
>>> 
>>> 
>>> (1) Create an image in Atlas that has Metron fully installed.  A new user
>>> could run single node Metron on their laptop with a single command and the
>>> only prereqs would be Vagrant and Virtualbox.  We could cut new images for
>>> each Metron release.  Or selectively cut new dev images from master as we
>>> see fit.
>>> 
>>> (2) Distribute the Metron RPMs (and the MPack tarball?) so that users can
>>> install Metron on a cluster without having to build it.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Nov 24, 2017 at 10:11 AM, Otto Fowler 
>>> 

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/840
  
Is this the best time to ask for changing field name convention to avoid 
dot or colon? We are externally using Hive external tables on HDFS data, due to 
Hive limitations we need to change the Metron field convention. I heard there 
is a long term plan in future to use ORC files instead of JSON and maybe Hive 
table can be supported directly. If this is right maybe this is the best time 
we can move towards changing field seperators accordingly.


---