Re: [VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

[Ryan Merriman]

+1

> On Nov 16, 2020, at 5:19 PM, David Lyle  wrote:
> 
> +1
> 
>> On Mon, Nov 16, 2020 at 3:10 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>> 
>> +1
>> 
>>> On Mon, Nov 16, 2020 at 7:01 AM Justin Leet  wrote:
>>> 
>>> Hi all,
>>> 
>>> This is a vote thread to retire Metron to the Attic, and dissolve the
>> PMC.
>>> This follows a discussion thread on the dev list ([DISCUSS] Retire Metron
>>> to the Attic
>>> <
>>> 
>> https://lists.apache.org/thread.html/reb31f643fac20d3ad09521fd702b19922412b7a4e8e08062968268c5%40%3Cdev.metron.apache.org%3E
 ).
>>> More details can be found in that discussion, but the most relevant link
>> is
>>> the specific process at Moving a project to the Attic
>>> .
>>> 
>>> As noted in the process page, this is a PMC vote. As usual, feel
>> encouraged
>>> to contribute non-binding votes.
>>> 
>>> The vote will run 72 hours, until Nov 19th at 9:00 am EST.
>>> 
>>> Thank you,
>>> Justin
>>> 
>> 

Re: Discuss: Time to update bundled SOLR support?

[Ryan Merriman]

Here's the Jira in case anyone wants to see the changes involved:
https://issues.apache.org/jira/browse/METRON-2225.

On Wed, Nov 13, 2019 at 8:48 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> That is correct - this was/is handled in the feature branch.
>
> On Wed, Nov 13, 2019 at 7:35 AM Justin Leet  wrote:
>
> > Someone working more on the feature branch can correct me if I'm wrong,
> but
> > I believe that's occurring as part of the general "Upgrade HDP version"
> > branch, since that involves a lot of major upgrades to more supported
> > versions of basically all the components. Specifically, it looks like it
> > occurs here
> > <
> >
> https://github.com/apache/metron/commit/ad71c046977f1b3ab1aa58fea38b846a1e37
> > >
> > .
> >
> > Having said that, especially for feature branches, having more eyes on it
> > is generally pretty helpful if you're interested in hopping in to catch
> any
> > issues or opportunities.  I believe the people most involved are Mike
> > Miklavcic, Nick Allen, and Ryan Merriman, so they may have some more
> > input or have seen some opportunities if you're looking to contribute
> > something there.
> >
> > On Wed, Nov 13, 2019 at 2:46 AM Dale Richardson 
> > wrote:
> >
> > > At the moment, Metron currently ships with Solr 6.6.2 support (I think
> > > that matches HDP Search 3).
> > > Horton Works HDP Search 4 and 5 is based off Apache Solr 7.4
> > > Cloudera search is based off Apache Solr 7.4 as well.
> > > Lucidworks Solr is based on 7.x, soon to be 8.x (if not already) - they
> > > always seem to push the bleeding edge with SOLR in production.
> > >
> > > Does anybody have any issues if I upgrade the bundled SOLR support to
> > Solr
> > > 7.4?  I'm hoping most people that use Metron/Solr in production use a
> > > supported distribution, and thus we should try and keep up with the
> > > supported Solr versions as much as possible.
> > >
> > > Regards,
> > > Dale.
> > >
> >
>

Re: [DISCUSS] Parser Aggregation in Management UI

[Ryan Merriman]

"We planning to add the changes to the latest PR as additional commits to
avoid impacting the PR sequence. We will refer to the source PR in the
commit message of the fix. Also adding a link to the comment section of the
source PR of the change request to the fixing commit to make them
connected."

I don't think this is going to work.  If changes are requested and applied
in a different PR, how would you test the original PR?  What would you do
if there were merge conflicts introduced between the current and final PR?
What's the point of even having any PRs before the final one if they won't
get committed or changed?  I don't see any way to avoid having to propagate
changes through all subsequent PRs.

On Wed, May 29, 2019 at 7:03 AM Tibor Meller  wrote:

> Hi all,
>
> *We still need some volunteer reviewers for this feature.* All individual
> PR is under 1000 line of changes except one and it is due to an
> autogenerated package.lock.json file.
> Just a heads up: parser aggregation by default turned on for bro, snort and
> yarn parser on full dev. Without this changeset, full dev is broken.
>
> https://lists.apache.org/thread.html/beeb4cfddfca7958a22ab926f72f52f46a33c42edce714112df9a2da@%3Cdev.metron.apache.org%3E
>
>
>
> On Fri, May 24, 2019 at 3:20 PM Tibor Meller 
> wrote:
>
> > Please find below the list of the PRs we opened for Parser Aggregation.
> > With Shane, Tamas we tried to provide as much information as possible to
> > make the reviewing process easier.
> > Please keep that in mind these PRs are not against muster but a Parser
> > Aggregation feature branch.
> > If you like to read more about the process we followed with these PRs
> > please read the previous three message in this thread.
> >
> > PR#1 METRON-2114: [UI] Moving components to sensor parser module
> > 
> > PR#2 METRON-2116: [UI] Removing redundant AppConfigService
> > 
> > PR#3 METRON-2117: [UI] Aligning models to grouping feature
> > 
> > PR#4 METRON-2115: [UI] Aligning UI to the parser aggregation AP
> > 
> > PR#5 METRON-2122: [UI] Fixing early app config access issue
> > 
> > PR#6 METRON-2124: [UI] Move status information and start/stop to the
> > Aggregate level 
> > PR#7 METRON-2125: [UI] Making changes visible in the parser list by
> > marking changed items 
> > PR#8 METRON-2131: Add NgRx and related dependencies
> > 
> > PR#9 METRON-2133: Add NgRx effects to communicate with the server
> > 
> > PR#10 METRON-2134: Add NgRx reducers to perform parser and group changes
> > in the store 
> > PR#11 METRON-2135: Add NgRx actions to trigger state changes
> > 
> > PR#12 METRON-2136: Add parser aggregation sidebar
> > 
> > PR#13 METRON-2137: Implement drag and drop mechanism and wire NgRx
> > 
> > PR#14 METRON-2138: Code clean up
> > 
> > PR#15 METRON-2139: Refactoring sensor-parser-config.component and wire
> > NgRx 
> >
> > Thanks,
> > Tibor
> >
> >
> > On Thu, May 23, 2019 at 11:45 AM Tibor Meller 
> > wrote:
> >
> >> Yes, am expecting that some change request will rase due to the review.
> >> We planning to add the changes to the latest PR as additional commits to
> >> avoid impacting the PR sequence. We will refer to the source PR in the
> >> commit message of the fix. Also adding a link to the comment section of
> the
> >> source PR of the change request to the fixing commit to make them
> connected.
> >>
> >> On Wed, May 22, 2019 at 5:49 PM Michael Miklavcic <
> >> michael.miklav...@gmail.com> wrote:
> >>
> >>> Tibor, that sounds reasonable to me. If PR #1 ends up requiring code
> >>> changes, will you guys just percolate those up through the remaining k
> >>> PRs
> >>> in order, or just the final PR? I'm wondering how this works in
> reference
> >>> to your last point in #5 about rebasing.
> >>>
> >>> On Wed, May 22, 2019 at 8:47 AM Tibor Meller 
> >>> wrote:
> >>>
> >>> > I would like to describe quickly *our approach to breaking down
> Parser
> >>> > Aggregation PR for smaller chunks*
> >>> >
> >>> > *1. we squashed the commits in the original development branch*
> >>> > - when we started to open smaller PRs from the commits from the
> >>> original
> >>> > branch, we found ourself opening PRs out of historical states of the
> >>> code
> >>> > instead of the final one
> >>> > - none of those states of development are worth (or make sense) to be
> >>> > reviewed (initial 

Re: [DISCUSS] Shaded jar classifiers

[Ryan Merriman]

Thanks Casey.  We definitely need more testing.  At this point I've just
done some light smoke testing with full dev to ensure nothing obvious is
broken (should cover ES though).  I imagine we'll need to test Solr as you
suggest and also test all our scripts with an actual use case to ensure we
haven't introduced runtime classpath issues.  We have also noticed some
regressions while using Stellar in the enrichment topology so it might be a
good time to create a test suite for that.  I believe Mike Miklavcic is
currently working on that.

On Mon, Jun 3, 2019 at 11:02 AM Casey Stella  wrote:

> This looks good to me, honestly.  Anything to make the build more
> understandable and help find classpath issues easier is a good idea IMO.
>
> Just curious, did you test that PR in both solr and ES (you added an
> exclude in the ES portion of the code) and did you spin it up in full-dev
> (to ensure ambari doesn't have any dependencies on the jar names)?
>
> Other than that, I'm +1 to the effort!
>
> On Mon, Jun 3, 2019 at 8:55 AM Ryan Merriman  wrote:
>
> > I recently opened a PR <https://github.com/apache/metron/pull/1436> that
> > has potential to significantly change (for the better in my opinion) the
> > way our Maven build process works.  I want to highlight this and get any
> > feedback on potential issues that may come with this change.
> >
> > I frequently run into the classpath version issues (especially with the
> > recent module reorganization work) and find them extremely challenging to
> > troubleshoot.  I believe we have found the root cause (from the PR
> > description):
> >
> > "When a module that uses the shaded plugin without a classifier is added
> to
> > another module as a dependency:
> >
> > 1. Any Maven excludes added to that dependency are ignored
> > 2. The Maven dependency:tree tool does not accurately report the
> transitive
> > dependencies pulled in by that dependency"
> >
> > After making this change, a number of classpath version problems popped
> up
> > as expected.  However they are now easy to track down and resolve.
> >
> > Does anyone have any concerns with making this change?  Are there things
> > I'm not thinking of?
> >
>

Re: [DISCUSS] Metron RPM spec file changelog

[Ryan Merriman]

I vote we get rid of them.  It's easy enough to look through the commit
history to see what changed and when.  If there is a need to explain a
change I think an inline comment would be more appropriate.

On Thu, May 30, 2019 at 3:52 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> During a recent PR review, I discovered that we have missed updating the
> changelog for our metron.spec file many, many times -
>
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec#L728
>
> Are we getting any value out of this feature, and do we want to keep it
> around? It seems like it's an extra point of work, as well as potential
> confusion, now that we have a number of missing entries. I started walking
> through the Git commit history and out of 81 changes, we started missing
> changes as early as the 9th change to this file. I logged a Jira
> https://issues.apache.org/jira/browse/METRON-2144 for addressing fixing
> the
> issue, but might it be better to simply jettison this feature?
>
> Mike
>

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

[Ryan Merriman]

I am working on the backend change mentioned above (
https://issues.apache.org/jira/browse/METRON-2034) and should a PR up today.

On Mon, Apr 29, 2019 at 1:16 PM Tamás Fodor  wrote:

> As Justin pointed out, we've already implemented the frontend related part
> of the aggregation in https://github.com/apache/metron/pull/1360. Since
> it's a very big changeset, we would like to double check it again before we
> merge it back to master. Also, we're waiting for a small change in the
> backend code to fully cover everything related to parser aggregation. Once
> we have introduced this small patch and fully tested it manually we can
> solve the issue with the aggregated sensors on the management UI.
>
> On Sun, Apr 28, 2019 at 8:30 PM Nick Allen  wrote:
>
> > I agree with Justin.  My +1 stands.
> >
> > Considering that this is a known gap, we have already released with this
> > gap, and we have a backlog of numerous improvements that should be
> released
> > to the community, I am not in favor of delaying the release.  Metron
> > provides a wide variety of functionality at varying levels of maturity.
> > This is to be expected.  If we expect perfection, we will never get a
> > release out.
> >
> >
> > On Sat, Apr 27, 2019 at 6:12 PM Justin Leet 
> wrote:
> >
> > > Mike is correct, that is because of the combination of full dev
> > > restrictions and the lack of support in the configuration UI for parser
> > > aggregation.  This was introduced in
> > > https://github.com/apache/metron/pull/1207 and also was true of the
> last
> > > release. Currently, parser aggregation is an advanced/manual feature
> > whose
> > > (bare minimum) configuration can be done via Ambari, out of
> convenience.
> > >
> > > I haven't looked into it, but
> https://github.com/apache/metron/pull/1360
> > > is
> > > likely the work for this (and need additional work before merging).
> > >
> > > I'm personally letting my binding +1 stand, although I would support
> > either
> > > ensuring we get that PR cleaned up and in and/or additional
> documentation
> > > regarding the current limitations of this feature.
> > >
> > >
> > > On Sat, Apr 27, 2019 at 2:38 PM Anand Subramanian <
> > > asubraman...@hortonworks.com> wrote:
> > >
> > > > I can confirm that I've seen the Mgmt UI shows the sensor status
> > > correctly
> > > > when they run as single topologies.
> > > >
> > > > -Anand
> > > >
> > > > On 4/27/19, 11:37 PM, "Michael Miklavcic" <
> > michael.miklav...@gmail.com>
> > > > wrote:
> > > >
> > > > I believe that is bc of parser aggregation. The UI does not
> support
> > > it
> > > > currently. IIRC there was a PR to change the bro, snort, and yaf
> > > > sensors to
> > > > aggregated bc full dev didn't have enough resources. The upshot
> is
> > > > that the
> > > > UI still works for single sensors, but the feature for enabling
> > > > aggregated
> > > > sensors has not yet been completed.
> > > >
> > > > On Sat, Apr 27, 2019, 11:33 AM Otto Fowler <
> > ottobackwa...@gmail.com>
> > > > wrote:
> > > >
> > > > > -1
> > > > >
> > > > > Ran the script and ran full dev, all good.
> > > > > In the configuration ui, the status of the sensors is not
> > correct.
> > > > It
> > > > > does not show any running, but they are running in storm and
> the
> > > > data was
> > > > > moved correctly.
> > > > >
> > > > >
> > > > > On April 26, 2019 at 09:58:02, Otto Fowler (
> > > ottobackwa...@gmail.com)
> > > > > wrote:
> > > > >
> > > > > Curious Anand,
> > > > > are your steps for bringing up an open stack cluster something
> we
> > > > could
> > > > > script like the AWS stuff?
> > > > >
> > > > >
> > > > > On April 26, 2019 at 09:35:29, Anand Subramanian (
> > > > > asubraman...@hortonworks.com) wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * Built RPMs and mpacks.
> > > > > * Brought up Metron stack on 12-node CentOS 7 openstack
> cluster.
> > > > > * Ran sensor-stubs and validated events in the Alerts UI for
> the
> > > > default
> > > > > sensors.
> > > > > * Management UI, Alerts UI and Swagger UI sanity check
> > > > >
> > > > > Regards,
> > > > > Anand
> > > > >
> > > > > On 4/26/19, 5:18 AM, "Nick Allen"  wrote:
> > > > >
> > > > > +1 Verified release with all documented steps and ran up Full
> > Dev.
> > > > >
> > > > > On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > Ok cool, just finished the validation and updated the steps
> in
> > > the
> > > > doc to
> > > > > > reflect the current code base.
> > > > > >
> > > > > > On Thu, Apr 25, 2019 at 3:45 PM Nick Allen <
> n...@nickallen.org
> > >
> > > > wrote:
> > > > > >
> > > > > > > No voting required. Those are just docs. Whoever is willing
> > to
> > > > correct
> > > > > > > and has access, should be 

Re: [DISCUSS] Next Release

[Ryan Merriman]

Jon is correct.   I am actively working on this and hope to have it
completed soon.   I realize it will hold up the release so it's a priority
for me.

On Sat, Mar 30, 2019 at 6:09 PM zeo...@gmail.com  wrote:

> Isn't the documentation already in progress?
>
> https://github.com/apache/metron/pull/1330#issuecomment-466453372
>
> If not I would still consider it important to complete prior to a release
> and I agree with Justin's comments in
>
>
> https://lists.apache.org/thread.html/50b89b919bd8bef3f7fcdef167cbd7e489fa74a1e2da3e4fddb08b13@
> 
>
> Jon Zeolla
>
> On Thu, Mar 28, 2019, 2:16 PM Michael Miklavcic <
> michael.miklav...@gmail.com>
> wrote:
>
> > Jon and Ryan - this was a convo/negotiation between you two at the time.
> > Any thoughts?
> >
> > On Thu, Mar 28, 2019 at 9:08 AM Nick Allen  wrote:
> >
> > > Is anyone volunteering to take this on?  Would be nice to get a release
> > > out.
> > >
> > > On Thu, Mar 14, 2019, 4:53 PM zeo...@gmail.com 
> wrote:
> > >
> > > > We should likely get METRON-2014 in, based on
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E
> > > >
> > > > On Thu, Mar 14, 2019 at 4:24 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Ticket is now done and merged. I'm also good on 0.7.1.
> > > > >
> > > > > On Thu, Mar 14, 2019 at 2:18 PM Justin Leet  >
> > > > wrote:
> > > > >
> > > > > > I'm in favor doing a release, pending the ticket Mike pointed out
> > > (and
> > > > > > anything else someone comes up with).
> > > > > >
> > > > > > To the best of my knowledge, I think 0.7.1 is sufficient, but if
> > > > someone
> > > > > > comes up with something, it's not hard to pivot.
> > > > > >
> > > > > > On Wed, Mar 13, 2019, 13:08 Michael Miklavcic <
> > > > > michael.miklav...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I'd like to see this fixed for the next release.
> > > > > > > https://issues.apache.org/jira/browse/METRON-2036. Even though
> > > it's
> > > > a
> > > > > > > non-prod issue, this is a core part of our
> > > infrastructure/development
> > > > > > > lifecycle that is currently broken and fits with our previous
> > > > > agreements
> > > > > > of
> > > > > > > holding a release until all intermittent test failures are
> > > addressed.
> > > > > > >
> > > > > > > On Wed, Mar 13, 2019 at 11:33 AM Nick Allen <
> n...@nickallen.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > I would like to open a discussion in regards to the next
> > release.
> > > > Our
> > > > > > > last
> > > > > > > > 0.7.0 release was on Dec 11th.
> > > > > > > >
> > > > > > > > I believe we have a significant number of bug fixes and
> > > performance
> > > > > > > > improvements that would make a worthy point release; 0.7.1.
> > > > > Although,
> > > > > > we
> > > > > > > > should review the change log and see if there are any
> breaking
> > > > > changes
> > > > > > > that
> > > > > > > > would require a bump to the minor version.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > $ git log --format=%B
> tags/apache-metron_0.7.0-release..HEAD |
> > > > grep
> > > > > > > > METRON-
> > > > > > > > Merge remote-tracking branch 'apache/master' into METRON-2035
> > > > > > > > METRON-2035 Allow User to Configure Role Names for Access
> > Control
> > > > > > > > METRON-2030 SensorParserGroupControllerIntegrationTest
> > > intermittent
> > > > > > > errors
> > > > > > > > (merrimanr via mmiklavc) closes apache/metron#1352
> > > > > > > > METRON-2031 [UI] Turning off initial search request and
> polling
> > > by
> > > > > > > default
> > > > > > > > on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
> > > > > > > > METRON-2012 Unable to Execute Stellar Functions Against HBase
> > in
> > > > the
> > > > > > REPL
> > > > > > > > (nickwallen) closes apache/metron#1345
> > > > > > > > METRON-1971 Short timeout value in Cypress may cause build
> > > failures
> > > > > > > > (sardell) closes apache/metron#1323
> > > > > > > > METRON-1940 Check if not and install Elastic search
> templates /
> > > > Solr
> > > > > > > > collections when indexing server is restarted (MohanDV)
> closes
> > > > > > > > apache/metron#1305
> > > > > > > > METRON-2019 Improve Metron REST Logging (merrimanr) closes
> > > > > > > > apache/metron#1347
> > > > > > > > METRON-2016 Parser aggregate groups should be persisted and
> > > > available
> > > > > > > > through REST (merrimanr) closes apache/metron#1346
> > > > > > > > METRON-1987 Upgrade Alert UI to stable Bootstrap 4 (sardell)
> > > closes
> > > > > > > > apache/metron#1336
> > > > > > > > METRON-1968 Messages are lost when a parser produces multiple
> > > > > messages
> > > > > > > and
> > > > > > > > batch size is greater than 1 (merrimanr) closes
> > > apache/metron#1330
> > > > > > > > METRON-1778 Out-of-order timestamps may delay flush in Storm
> > > > Profiler
> > > > > > > 

[DISCUSS] Upgrading HBase and Kafka support

[Ryan Merriman]

I have been researching the effort involved to upgrade to HDP 3.  Along the
way I've found a couple challenging issues that we will need to solve, both
involving our integration testing strategy.

The first issue is Kafka.  We are moving from 0.10.0 to 2.0.0 and there
have been significant changes to the API.  This creates an issue in the
KafkaComponent class, which we use as an in-memory Kafka server in
integration tests.  Most of the classes that were previously used have gone
away, and to the best of my knowledge, were not supported as public APIs.
I also don't see any publicly documented APIs to replace them.

The second issue is HBase.  We are moving from 1.1.2 to 2.0.2 so another
significant change.  This creates an issue in the MockHTable class
becausethe HTableInterface class has changed to Table, essentially
requiring that MockHTable be rewritten to conform to the new interface.
It's my opinion that this class is complicated and difficult to maintain as
it is anyways.

These 2 issues have the potential to add a significant amount of work to
upgrading Metron to HDP 3.  I want to take a step back and review our
options before we move forward.  Here are some initial thoughts I had on
how to approach this.  For HBase:

   1. Update MockHTable to work with the new HBase API.  We would continue
   using a mock server approach for HBase.
   2. Research replacing MockHTable with an in-memory HBase server.
   3. Replace MockHTable with a Docker container running HBase.

For Kafka:

   1. Replace KafkaComponent with a mock server implementation.
   2. Update KafkaComponent to work with the new API.  We would probably
   need to leverage some internal Kafka classes.  I do not see a testing API
   documented publicly.
   3. Replace KafkaComponent with a Docker container running Kafka.

What other options are there?  Whatever we choose I think we should follow
a similar approach for both (mock servers, in memory servers, Docker, other
options I'm not thinking of).

This will not shock anyone but I would be in favor of Docker containers.
They have the advantage of classpath isolation, easy upgrades, and accurate
integration testing.  The downside is we will have to adjusts our tests and
travis script to incorporate these Docker containers into our build
process.  We have discussed this at length in the past and it has generally
stalled for various reasons.  Maybe if we move a few services at a time it
might be more palatable?  As for the other 2 approaches, I think if either
worked well we wouldn't be having this discussion.  Mock servers are hard
to maintain and I don't see in memory testing classes documented in
javadocs for either service.

Thoughts?

Re: [DISCUSS] Architecture documentation

[Ryan Merriman]

I feel like the code itself is pretty well documented.  I updated existing
javadocs and added javadocs to classes that didn't have them before this
PR.  In my opinion the level of documentation for these classes has
increased significantly.

On Mon, Feb 25, 2019 at 1:52 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Tentatively agreed on further clarification of what we consider in/out of
> scope for documentation re: document something that wasn't documented
> before. Ryan, can you give a quick summary of what you *have* added/updated
> in documentation on this PR vs what you want to leave out?
>
> My initial concern in punting on docs right now is that part of what made
> this PR/task more challenging in the first place was not having
> documentation. We risk losing context and detail again if we don't do this
> immediately. Would it be reasonable to split it up as follows?:
>
>1. Additional overarching documentation feels out of scope - make it a
>follow on (see comments below).
>2. Adding documentation to our existing README's and java code comments
>that describe the new/modified functionality should be in scope because
>it's part of the unit of work. I expect that a developer should be able
> to
>look at the code, tests, comments, and README's and understand how this
>code functions without having to start from scratch.
>
> The way we've handled follow-on work before, at least as far as feature
> branches are concerned, was to create Jiras and link them to the
> appropriate discussions for context. Maybe we can take that one step
> further and do the release manager a favor by also labeling the
> required/requested release on the Jira as a gating factor. This follows our
> pattern for intermittent test failure reporting, e.g.
>
> https://issues.apache.org/jira/browse/METRON-1946?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-failure%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> .
>
> I'm also in favor of continuing to document architecture and technical
> details as part of the code base as Ryan and Jon have suggested. I think we
> should have an "architecture.md" in metron root that replaces this -
>
> https://github.com/apache/metron/blob/d7d4fd9afb19e2bd2e66babb7e1514a19eae07d0/README.md#navigating-the-architecture
> and covers the broad architecture with links to the appropriate modules for
> detail. Minimally, it would be nice if we had a simple diagram showing the
> basic flow of data in Metron. I think we probably want an updated version
> of this wiki entry from back in the day -
> https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture
>
> Best,
> Mike
>
>
> On Mon, Feb 25, 2019 at 7:18 AM Nick Allen  wrote:
>
> > I don't think we should hold up this work to document something that
> wasn't
> > previously documented.  A follow-on is sufficient.
> >
> > On Mon, Feb 25, 2019 at 8:50 AM Ryan Merriman 
> wrote:
> >
> > > Recently I submitted a PR <https://github.com/apache/metron/pull/1330>
> > > that
> > > introduces a large number of changes to a critical part of our code
> base.
> > > Reviewers feel like it is significant enough to document at an
> > > architectural level (and I agree).  There are a couple points I would
> > like
> > > to clarify.
> > >
> > > Generally architectural documentation lives in the README of the
> > > appropriate module.  Do we want to continue documenting architecture
> > here?
> > > I think it makes sense because it will be versioned along with the
> code.
> > > Just wanted to confirm there are no objections to continuing this
> > practice.
> > >
> > > A reviewer suggested we could accept the PR as is and leave the
> > > architectural documentation as a follow on.  I think this makes sense
> > > because it can be tedious to maintain a large PR as other smaller
> commits
> > > are accepted into master.  An important requirement is the
> documentation
> > > follow on must be completed in a timely manner, before the next
> release.
> > > Are there any objections to doing it this way?
> > >
> >
>

[DISCUSS] Architecture documentation

[Ryan Merriman]

Recently I submitted a PR  that
introduces a large number of changes to a critical part of our code base.
Reviewers feel like it is significant enough to document at an
architectural level (and I agree).  There are a couple points I would like
to clarify.

Generally architectural documentation lives in the README of the
appropriate module.  Do we want to continue documenting architecture here?
I think it makes sense because it will be versioned along with the code.
Just wanted to confirm there are no objections to continuing this practice.

A reviewer suggested we could accept the PR as is and leave the
architectural documentation as a follow on.  I think this makes sense
because it can be tedious to maintain a large PR as other smaller commits
are accepted into master.  An important requirement is the documentation
follow on must be completed in a timely manner, before the next release.
Are there any objections to doing it this way?

Re: [DISCUSS] Writer class refactor

[Ryan Merriman]

Thanks Mike, very helpful to have all that context.  I'm in agreement with
everything you've said.  Accepting duplicates may be a tradeoff we accept
to keep performance high.

Your comments are centered around Kafka but how does this apply to other
writers?  Since we're now handling multiple messages that come from a
single tuple, how should we handle partial failures?  The flush() method on
the Kafka writer waits until all messages have been written so we don't
have to worry about partial failures/successes there.  What about the ES
writer?  The way it's implemented now it returns a status of which messages
were successfully written and which messages were not.  Is it possible to
make a bulk write with the ES client an atomic operation?  If not I think
we'll have to accept duplicates (if we're not already).  I think this is an
issue in general for any writer we may implement and we need to be clear
about how messages are acked in this case.

I agree with your suggestion of using Map> but have
a small change I would like to propose.  Instead of Tuple can we use a
transaction id (String type)?  I would prefer to see Storm dependencies be
moved up to the bolts.  Maintaining a relationship of Tuples to transaction
ids there should be trivial.

On Fri, Jan 18, 2019 at 5:25 PM zeo...@gmail.com  wrote:

> Totally on board with everybody's comments above this point.
>
> Jon
>
> On Fri, Jan 18, 2019, 6:07 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Thanks for the write up, Ryan. I had to touch on some of this when
>> refactoring the kafka writer away from the async model so we could
>> guarantee delivery. We had potential to drop messages before that change
>> because of the async producer calls, which would ack the Storm tuple as
>> soon as the writer returned.
>>
>>- https://github.com/apache/metron/pull/1045
>>
>> We'll want to talk about these fixes/updates in context of our message
>> delivery semantics, both in Storm and Kafka. As it currently stands, we do
>> NOT use Storm Trident, which means we have at-least-once message
>> processing
>> in Storm. There is an inherent possibility that we will publish duplicate
>> messages in some instances. From a Kafka perspective, we have the same
>> issue. As of Kafka 0.11.0, they provide a way to get exactly-once
>> semantics, but I'm not sure we've done much to explicitly achieve that.
>>
>>- https://kafka.apache.org/10/documentation.html#semantics
>>
>> From a Kafka delivery guarantee perspective, it appears we're currently
>> setting # required acks to 1 by default. This means we get commit
>> confirmation as soon as the leader has written the message to its local
>> log. In this case should the leader fail immediately after acknowledging
>> the record but before the followers have replicated it then the record
>> will
>> be lost. We could investigate settings acks=all or acks=-1, but this would
>> be a tradeoff in performance for us.
>>
>>-
>>
>> https://github.com/apache/metron/blob/341960b91f8fe742d5cf947633b7edd2275587d5/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L87
>>- https://kafka.apache.org/10/documentation/#producerconfigs
>>
>> Per the KafkaProducer documentation, the flush() command will wait until
>> all messages are batched and sent, and will return with either success
>> (acked) or an error. "A request is considered completed when it is
>> successfully acknowledged according to the acks configuration you have
>> specified or else it results in an error."
>>
>>-
>>
>> https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
>>
>> With this combination of factors, I believe we can continue to guarantee
>> at-least-once semantics in the writer, regardless of batch size. To your
>> point about not passing 2 separate lists, I suggest that we modify the API
>> by passing in something like Map> so that the
>> tuples always get acked with respect to their messages. This way we can
>> avoid the tuple-message batch boundary problem by ensuring we only ack a
>> tuple when all associated messages are successfully written to Kafka.
>>
>> Best,
>> Mike
>>
>>
>> On Fri, Jan 18, 2019 at 1:31 PM Otto Fowler 
>> wrote:
>>
>> > Agreed
>> >
>> >
>> > On January 18, 2019 at 14:52:32, Ryan Merriman (merrim...@gmail.com)
>> > wrote:
>> >
>> > I am on board with that. In that case, I think it's even more important
>> > that we get the Writer interfaces right.
>> >
>> > On Fri, Ja

Re: [DISCUSS] Writer class refactor

[Ryan Merriman]

I am on board with that.  In that case, I think it's even more important
that we get the Writer interfaces right.

On Fri, Jan 18, 2019 at 1:34 PM Otto Fowler  wrote:

> I think that the writers should be loaded as, and act as extension points,
> such that it is possible to have 3rd party writers, and would structure
> them as such.
>
>
>
> On January 18, 2019 at 13:55:00, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> Recently there was a bug reported by a user where a parser that emits
> multiple messages from a single tuple doesn't work correctly:
> https://issues.apache.org/jira/browse/METRON-1968. This has exposed a
> problem with how the writer classes work.
>
> The fundamental issue is this: the writer classes operate under the
> assumption that there is a 1 to 1 mapping between tuples and messages to
> be
> written. A couple of examples:
>
> KafkaWriter
> <
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236>
>
> -
> This class writes messages by iterating through the list of tuples and
> fetching the message with the same index. This is the cause of the Jira
> above. We could iterate through the message list instead but then we don't
> know which tuples have been fully processed. It would be possible for a
> batch to be flushed before all messages from a tuple are passed to the
> writer.
>
> BulkWriterComponent
> <
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java#L250>
>
> - The tuple list size is used to determine when a batch should be flushed.
> While inherently incorrect in my opinion (should be message list size),
> this also causes an issue where only the first message from the last tuple
> in a batch is written.
>
> I do not believe there are easy fixes to these problems. There is no way
> to properly store the relationship between tuples and messages to be
> written with the current BulkMessageWriter interface and
> BulkWriterResponse
> class. If we did have a way, how should we handle partial failures? If
> multiple messages are parsed from a tuple but only half of them are
> written
> successfully, what should happen? Should we replay the tuple? Should we
> just report the failed messages and continue on? I think it may be a good
> time to review our writer classes and consider a refactor. Do others
> agree? Are there easy fixes I'm missing?
>
> Assuming there is interest in refactoring, I will throw out some ideas for
> consideration. For those not as familiar with the writer classes, they are
> organized as follows (in order from lowest to highest level):
>
> Writers - These classes do the actual writing and implement the
> BulkMessageWriter or MessageWriter interfaces. There are 6 implementations
> I can see including KafkaWriter, SolrWriter, ElasticsearchWriter,
> HdfsWriter, etc. There is also an implementation that adapts a
> MessageWriter to a BulkMessageWriter (WriterToBulkWriter). The result of a
> writing operation is a BulkWriterResponse containing a list of either
> successful or failed tuples.
>
> Writer Containers - This includes the BulkWriterComponent and
> WriterHandler
> classes. These are responsible for batching and flushing messages,
> handling errors and acking tuples.
>
> Bolts - This includes ParserBolt, WriterBolt and BulkMessageWriterBolt.
> These classes implement the Storm Bolt interfaces, setup
> writers/components
> and execute tuples.
>
> I think the first step is to reevaluate the separation of concerns for
> these classes. Here is how I would change from what we currently have:
>
> Writers - These classes should only be concerned with writing messages and
> reporting what happened. They would also manage the lifecycle and
> configuration of the underlying client libraries as they do now. Instead
> of accepting 2 separate lists, they should accept a data structure that
> accurately represents the relationship between tuples and messages.
>
> Writer Containers - These classes would continue to handling batching and
> flushing but would only report the results of a flush rather than actually
> doing the acking or error handling.
>
> Bolts - These would now be responsible for acking and error reporting on
> tuples. They would transform a tuple into something the Writer Containers
> can accept as input.
>
> I think working through this and adjusting the contracts between the
> different layers will be necessary to fix the bugs described above. While
> we're at it I think there are other improvements we could also make:
>
> Decouple Storm - It would be beneficial to remove the depen

[DISCUSS] Writer class refactor

[Ryan Merriman]

Recently there was a bug reported by a user where a parser that emits
multiple messages from a single tuple doesn't work correctly:
https://issues.apache.org/jira/browse/METRON-1968.  This has exposed a
problem with how the writer classes work.

The fundamental issue is this:  the writer classes operate under the
assumption that there is a 1 to 1 mapping between tuples and messages to be
written.  A couple of examples:

KafkaWriter

-
This class writes messages by iterating through the list of tuples and
fetching the message with the same index.  This is the cause of the Jira
above.  We could iterate through the message list instead but then we don't
know which tuples have been fully processed.  It would be possible for a
batch to be flushed before all messages from a tuple are passed to the
writer.

BulkWriterComponent

- The tuple list size is used to determine when a batch should be flushed.
While inherently incorrect in my opinion (should be message list size),
this also causes an issue where only the first message from the last tuple
in a batch is written.

I do not believe there are easy fixes to these problems.  There is no way
to properly store the relationship between tuples and messages to be
written with the current BulkMessageWriter interface and BulkWriterResponse
class.  If we did have a way, how should we handle partial failures?  If
multiple messages are parsed from a tuple but only half of them are written
successfully, what should happen?  Should we replay the tuple?  Should we
just report the failed messages and continue on?  I think it may be a good
time to review our writer classes and consider a refactor.  Do others
agree?  Are there easy fixes I'm missing?

Assuming there is interest in refactoring, I will throw out some ideas for
consideration.  For those not as familiar with the writer classes, they are
organized as follows (in order from lowest to highest level):

Writers - These classes do the actual writing and implement the
BulkMessageWriter or MessageWriter interfaces.  There are 6 implementations
I can see including KafkaWriter, SolrWriter, ElasticsearchWriter,
HdfsWriter, etc.  There is also an implementation that adapts a
MessageWriter to a BulkMessageWriter (WriterToBulkWriter).  The result of a
writing operation is a BulkWriterResponse containing a list of either
successful or failed tuples.

Writer Containers - This includes the BulkWriterComponent and WriterHandler
classes.  These are responsible for batching and flushing messages,
handling errors and acking tuples.

Bolts - This includes ParserBolt, WriterBolt and BulkMessageWriterBolt.
These classes implement the Storm Bolt interfaces, setup writers/components
and execute tuples.

I think the first step is to reevaluate the separation of concerns for
these classes.  Here is how I would change from what we currently have:

Writers - These classes should only be concerned with writing messages and
reporting what happened.  They would also manage the lifecycle and
configuration of the underlying client libraries as they do now.  Instead
of accepting 2 separate lists, they should accept a data structure that
accurately represents the relationship between tuples and messages.

Writer Containers - These classes would continue to handling batching and
flushing but would only report the results of a flush rather than actually
doing the acking or error handling.

Bolts - These would now be responsible for acking and error reporting on
tuples.  They would transform a tuple into something the Writer Containers
can accept as input.

I think working through this and adjusting the contracts between the
different layers will be necessary to fix the bugs described above.  While
we're at it I think there are other improvements we could also make:

Decouple Storm - It would be beneficial to remove the dependency on tuples
in our writers and writer containers.  We could replace this with a simple
abstraction (an id would probably work fine).  This will allow us to more
easily port Metron to other streaming platforms.

Remove MessageWriter Interface - This is not being actively used as far as
I can tell.  Is that true?  Removing this will make our code simpler and
easier to follow (WriterHandler and WriterToBulkWriter classes can probably
go away).  I don't see any reason future writers, even those without bulk
writing capabilities, could not fit into the BulkMessageWriter interface.
A writer could either iterate through messages and write one at a time or
throw an exception.  As far as I know, the writer interfaces are not
something we advertise as extension points.  Is that true?

Consolidate our BulkMessageWriterBolt and WriterBolt classes - Is there 

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

I just put up a PR that adds Metron as a Knox service here:
https://github.com/apache/metron/pull/1275.  This should give everyone a
good idea of what is involved.  I added a section on outstanding items that
highlights some of the things we have been discussion here.

On Fri, Nov 16, 2018 at 10:54 AM Ryan Merriman  wrote:

> I would also add that defaulting to Knox being on simplifies things at a
> technical level.
>
> On Fri, Nov 16, 2018 at 10:52 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> That's fantastic, thanks for that detail.
>>
>> Also, I'm in agreement with the recent comments from Otto and Simon.
>>
>> On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman 
>> wrote:
>>
>> > I was still able to spin up the UI locally and debug in my testing.  I
>> am
>> > in complete agreement, we need to ensure the developer experience
>> doesn't
>> > change.
>> >
>> > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic <
>> > michael.miklav...@gmail.com> wrote:
>> >
>> > > Ryan, what's remote debugging look like for UI testing with Knox
>> enabled?
>> > > Anything we lose from a dev testability standpoint? The discussion of
>> > > defaults sounds reasonable to me, and I'd like to understand any other
>> > > tradeoffs there may be for non-prod deployments like full dev.
>> > >
>> > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman 
>> > wrote:
>> > >
>> > > > Most of the research I've done around adding Metron as a Knox
>> service
>> > is
>> > > > based on how other projects do it.  The documentation is not easy to
>> > > follow
>> > > > so I learned by reading other service definition files.  The
>> assumption
>> > > > that we are doing things drastically different is false.
>> > > >
>> > > > I completely agree with Simon.  Why would we want to be dependent on
>> > > Knox's
>> > > > release cycle?  How does that benefit us?  It may reduce some
>> > operational
>> > > > complexity but it makes our install process more complicated
>> because we
>> > > > require a certain version of Knox (who knows when that gets
>> released).
>> > > > What do we do in the meantime?  I would also like to point out that
>> > > Metron
>> > > > is inherently different than other Hadoop stack services.  We are a
>> > > > full-blown application with multiple UIs so the way we expose
>> services
>> > > > through Knox may be a little different.
>> > > >
>> > > > I think this will be easier to discuss when we can all see what is
>> > > actually
>> > > > involved.  I am working on a PR that adds Metron as a Knox service
>> and
>> > > will
>> > > > have that out soon.  That should give everyone more context.
>> > > >
>> > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball <
>> > > > si...@simonellistonball.com> wrote:
>> > > >
>> > > > > You could say the same thing about Ambari, but that provides
>> mpacks.
>> > > Knox
>> > > > > is also designed to be extensible through Knox service stacks
>> since
>> > > they
>> > > > > realized they can’t support every project. The challenge is that
>> the
>> > > docs
>> > > > > have not made it as easy as they could for the ecosystem to plug
>> into
>> > > > Knox,
>> > > > > which has led to some confusion around this being a recommended
>> > pattern
>> > > > > (which it is).
>> > > > >
>> > > > > The danger of trying to get your bits into Knox is that that ties
>> you
>> > > to
>> > > > > their release cycle (a problem Ambari has felt hard, hence their
>> > > > community
>> > > > > is moving away from the everything inside model towards
>> everything is
>> > > an
>> > > > > mpack).
>> > > > >
>> > > > > A number of implementations of Knox also use the approach Ryan is
>> > > > > suggesting for their own organization specific end points, so it’s
>> > not
>> > > > like
>> > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is
>> > > designed
>>

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

I would also add that defaulting to Knox being on simplifies things at a
technical level.

On Fri, Nov 16, 2018 at 10:52 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> That's fantastic, thanks for that detail.
>
> Also, I'm in agreement with the recent comments from Otto and Simon.
>
> On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman  wrote:
>
> > I was still able to spin up the UI locally and debug in my testing.  I am
> > in complete agreement, we need to ensure the developer experience doesn't
> > change.
> >
> > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Ryan, what's remote debugging look like for UI testing with Knox
> enabled?
> > > Anything we lose from a dev testability standpoint? The discussion of
> > > defaults sounds reasonable to me, and I'd like to understand any other
> > > tradeoffs there may be for non-prod deployments like full dev.
> > >
> > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman 
> > wrote:
> > >
> > > > Most of the research I've done around adding Metron as a Knox service
> > is
> > > > based on how other projects do it.  The documentation is not easy to
> > > follow
> > > > so I learned by reading other service definition files.  The
> assumption
> > > > that we are doing things drastically different is false.
> > > >
> > > > I completely agree with Simon.  Why would we want to be dependent on
> > > Knox's
> > > > release cycle?  How does that benefit us?  It may reduce some
> > operational
> > > > complexity but it makes our install process more complicated because
> we
> > > > require a certain version of Knox (who knows when that gets
> released).
> > > > What do we do in the meantime?  I would also like to point out that
> > > Metron
> > > > is inherently different than other Hadoop stack services.  We are a
> > > > full-blown application with multiple UIs so the way we expose
> services
> > > > through Knox may be a little different.
> > > >
> > > > I think this will be easier to discuss when we can all see what is
> > > actually
> > > > involved.  I am working on a PR that adds Metron as a Knox service
> and
> > > will
> > > > have that out soon.  That should give everyone more context.
> > > >
> > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball <
> > > > si...@simonellistonball.com> wrote:
> > > >
> > > > > You could say the same thing about Ambari, but that provides
> mpacks.
> > > Knox
> > > > > is also designed to be extensible through Knox service stacks since
> > > they
> > > > > realized they can’t support every project. The challenge is that
> the
> > > docs
> > > > > have not made it as easy as they could for the ecosystem to plug
> into
> > > > Knox,
> > > > > which has led to some confusion around this being a recommended
> > pattern
> > > > > (which it is).
> > > > >
> > > > > The danger of trying to get your bits into Knox is that that ties
> you
> > > to
> > > > > their release cycle (a problem Ambari has felt hard, hence their
> > > > community
> > > > > is moving away from the everything inside model towards everything
> is
> > > an
> > > > > mpack).
> > > > >
> > > > > A number of implementations of Knox also use the approach Ryan is
> > > > > suggesting for their own organization specific end points, so it’s
> > not
> > > > like
> > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is
> > > designed
> > > > to
> > > > > work in the future, than the legacy of it only being able to
> handle a
> > > > > subset of Hadoop projects.
> > > > >
> > > > > Knox remains optional In our scenario, but we keep control over the
> > > > > shipping of things like rewrite rules, which allows Metron to
> control
> > > its
> > > > > release destiny should things like url patterns in the ui need to
> > > change
> > > > > (with a new release of angular / new module / new rest endpoint
> etc)
> > > > > instead of making a Metron release dependent on a Knox release.
> > > > >
> > > > > Imag

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

I was still able to spin up the UI locally and debug in my testing.  I am
in complete agreement, we need to ensure the developer experience doesn't
change.

On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ryan, what's remote debugging look like for UI testing with Knox enabled?
> Anything we lose from a dev testability standpoint? The discussion of
> defaults sounds reasonable to me, and I'd like to understand any other
> tradeoffs there may be for non-prod deployments like full dev.
>
> On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman  wrote:
>
> > Most of the research I've done around adding Metron as a Knox service is
> > based on how other projects do it.  The documentation is not easy to
> follow
> > so I learned by reading other service definition files.  The assumption
> > that we are doing things drastically different is false.
> >
> > I completely agree with Simon.  Why would we want to be dependent on
> Knox's
> > release cycle?  How does that benefit us?  It may reduce some operational
> > complexity but it makes our install process more complicated because we
> > require a certain version of Knox (who knows when that gets released).
> > What do we do in the meantime?  I would also like to point out that
> Metron
> > is inherently different than other Hadoop stack services.  We are a
> > full-blown application with multiple UIs so the way we expose services
> > through Knox may be a little different.
> >
> > I think this will be easier to discuss when we can all see what is
> actually
> > involved.  I am working on a PR that adds Metron as a Knox service and
> will
> > have that out soon.  That should give everyone more context.
> >
> > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> > > You could say the same thing about Ambari, but that provides mpacks.
> Knox
> > > is also designed to be extensible through Knox service stacks since
> they
> > > realized they can’t support every project. The challenge is that the
> docs
> > > have not made it as easy as they could for the ecosystem to plug into
> > Knox,
> > > which has led to some confusion around this being a recommended pattern
> > > (which it is).
> > >
> > > The danger of trying to get your bits into Knox is that that ties you
> to
> > > their release cycle (a problem Ambari has felt hard, hence their
> > community
> > > is moving away from the everything inside model towards everything is
> an
> > > mpack).
> > >
> > > A number of implementations of Knox also use the approach Ryan is
> > > suggesting for their own organization specific end points, so it’s not
> > like
> > > this is an uncommon, or anti-pattern, it’s more the way Knox is
> designed
> > to
> > > work in the future, than the legacy of it only being able to handle a
> > > subset of Hadoop projects.
> > >
> > > Knox remains optional In our scenario, but we keep control over the
> > > shipping of things like rewrite rules, which allows Metron to control
> its
> > > release destiny should things like url patterns in the ui need to
> change
> > > (with a new release of angular / new module / new rest endpoint etc)
> > > instead of making a Metron release dependent on a Knox release.
> > >
> > > Imagine how we would have done with the Ambari side if we’d had to wait
> > > for them to release every time we needed to change something in the
> > > mpack... we don’t want that happening with Knox.
> > >
> > > Simon
> > >
> > > > On 16 Nov 2018, at 13:22, Otto Fowler 
> wrote:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support
> > > >
> > > > Solr is angular for example.
> > > >
> > > >
> > > > On November 16, 2018 at 08:12:55, Otto Fowler (
> ottobackwa...@gmail.com
> > )
> > > > wrote:
> > > >
> > > > Ok,  here is something I don’t understand, but would like to.
> > > >
> > > > Knox comes configured with build in services for a number of other
> > apache
> > > > products and UI’s.
> > > > It would seem to me, that the best integration with Knox would be to
> do
> > > > what these other products have done.
> > > >
> > > >
> > > > 1. Do whatever you have to do to make 

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

Most of the research I've done around adding Metron as a Knox service is
based on how other projects do it.  The documentation is not easy to follow
so I learned by reading other service definition files.  The assumption
that we are doing things drastically different is false.

I completely agree with Simon.  Why would we want to be dependent on Knox's
release cycle?  How does that benefit us?  It may reduce some operational
complexity but it makes our install process more complicated because we
require a certain version of Knox (who knows when that gets released).
What do we do in the meantime?  I would also like to point out that Metron
is inherently different than other Hadoop stack services.  We are a
full-blown application with multiple UIs so the way we expose services
through Knox may be a little different.

I think this will be easier to discuss when we can all see what is actually
involved.  I am working on a PR that adds Metron as a Knox service and will
have that out soon.  That should give everyone more context.

On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> You could say the same thing about Ambari, but that provides mpacks. Knox
> is also designed to be extensible through Knox service stacks since they
> realized they can’t support every project. The challenge is that the docs
> have not made it as easy as they could for the ecosystem to plug into Knox,
> which has led to some confusion around this being a recommended pattern
> (which it is).
>
> The danger of trying to get your bits into Knox is that that ties you to
> their release cycle (a problem Ambari has felt hard, hence their community
> is moving away from the everything inside model towards everything is an
> mpack).
>
> A number of implementations of Knox also use the approach Ryan is
> suggesting for their own organization specific end points, so it’s not like
> this is an uncommon, or anti-pattern, it’s more the way Knox is designed to
> work in the future, than the legacy of it only being able to handle a
> subset of Hadoop projects.
>
> Knox remains optional In our scenario, but we keep control over the
> shipping of things like rewrite rules, which allows Metron to control its
> release destiny should things like url patterns in the ui need to change
> (with a new release of angular / new module / new rest endpoint etc)
> instead of making a Metron release dependent on a Knox release.
>
> Imagine how we would have done with the Ambari side if we’d had to wait
> for them to release every time we needed to change something in the
> mpack... we don’t want that happening with Knox.
>
> Simon
>
> > On 16 Nov 2018, at 13:22, Otto Fowler  wrote:
> >
> >
> https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support
> >
> > Solr is angular for example.
> >
> >
> > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Ok,  here is something I don’t understand, but would like to.
> >
> > Knox comes configured with build in services for a number of other apache
> > products and UI’s.
> > It would seem to me, that the best integration with Knox would be to do
> > what these other products have done.
> >
> >
> > 1. Do whatever you have to do to make your own stuff compatible.
> > 2. Create a knox service definition and provide it or try to get it into
> > knox itself
> >
> > This would make the knox integration with metron optional and pluggable
> > wouldn’t it?
> >
> > Then knox with metron would just be the same as knox with anything else.
> > Please help me if I am wrong, but we seem to be going our own way here.
> > Why don’t we just do what these other products have done?
> > Why don’t we try to get apache metron services accepted to the knox
> > project?  Why don’t we model our knox integration with how XYZ does it?
> > Have we looked at how others integrate?   Having all the code and being
> > able to track stuff is kind of the point of this whole thing isn’t it?
> >
> > Maybe this is implied and I’m missing it, if so I apologize.
> >
> > I think consistency with the rest of the hadoop stack with knox helps us.
> >
> >
> >
> > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com)
> wrote:
> >
> > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI
> > as far as I know. I put up a PR with a proposed solution here:
> > https://github.com/apache/metron/pull/1266.
> > 2) Yes Knox is a service you can install with Ambari, similar to Ranger
> or
> > Spark. There are some things that are specifically confi

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

1) Sorry I misspoke.  I meant to say this is not possible in the Alerts UI
as far as I know.  I put up a PR with a proposed solution here:
https://github.com/apache/metron/pull/1266.
2) Yes Knox is a service you can install with Ambari, similar to Ranger or
Spark.  There are some things that are specifically configured in Knox and
there are some things specific to Metron.  I will put up a PR with the
changes needed so you can see exactly what is involved.
3) I don't understand what you mean here.  Is this a question?
4) I think it's a little early to predict the Ambari changes required.
This will depend on how tasks 1-3 go.  I imagine it's similar to other
mpack work:  expose some parameters in ambari and bind those to config
files.  My understanding from this thread so far is that we should focus on
a manual, documented approach to start.

On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Thanks Ryan,
>
> 1) Can you clarify "not a good way to do this?" Are you saying we don't
> have a way to set this and need to add the config option, or that a
> solution is not obvious and it's unclear what to do? It seems to me you're
> saying the former, but I'd like to be sure.
> 2) Is Knox not a service made available by Ambari similar to Ranger or
> Spark? I'm assuming that similar to Kerberos, there are some things that
> are specifically configured in Knox and others that are app-specific. Some
> explanation of what this looks like would be helpful.
> 3) Sounds like this follows pretty naturally from #1
> 4) Relates to #2. I think we need some guidance on what a manual vs
> MPack/automated install would look like.
>
> Cheers,
> Mike
>
>
> On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman  wrote:
>
> > Wanted to give an update on the context path issue.  I investigated
> > rewriting url links in the outgoing static assets with Knox and it was
> not
> > trivial.  Fortunately I found a simple solution that works with or
> without
> > Knox.  I changed the base tag in index.html from  to  > href="./">, or in other words made the base href relative.
> >
> > I believe I am at the point where I can task this out and provide a high
> > level overview of the changes needed.  I think that each task will be a
> > manageable size and can stand alone so I don't think we need a feature
> > branch.
> >
> > The first task involves a general change to the UI code.  We need a way
> to
> > set the path to the REST service with a configuration setting because it
> is
> > different with and without Knox.  Currently there is not a good way to do
> > this in the UI.  We can use the environment files but that is a build
> time
> > setting and is not flexible.  I can see this capability being useful for
> > other use cases in the future.  I think we could even split this up into
> 2
> > separate tasks, one for the alerts UI and one for the management UI.
> >
> > The second task involves adding Knox to our stack either by default as a
> > dependency in the mpack or with a documented approach.  We would add our
> > REST service, Alerts UI, and Management UI as services in Knox.
> Everything
> > would continue to function as it currently does but with all
> communication
> > going through Knox.  LDAP authentication would be required when using
> Knox
> > and Knox will authenticate with the REST service by passing along an
> > Authorization header.  Enabling Knox would be a manual process that
> > involves deploying assets (Knox descriptor files) and changing
> > configuration.  There would be no change to how the UI functions by
> default
> > (without Knox) and either LDAP or JDBC authentication could still be
> used..
> >
> > The third task involves enabling SSO with Knox.  We would update the REST
> > service so that it can authenticate with a Knox SSO token.  We would
> > provide documentation on how to update the Knox settings in Ambari to
> > enable SSO.  The Alerts/Management UI would need to expose configuration
> > properties for the REST url and login url since these would be different
> > when Knox is enabled.  We would also need to provide documentation on how
> > to make these UI configuration changes, based on the work done in task 1.
> >
> > An optional forth task would be exposing configuration settings and
> > enabling Knox with Ambari.  We would eliminate the manual steps necessary
> > for enabling Knox and instead automate those steps with an Ambari input
> > control, similar to how LDAP is enabled.
> >
> > Thoughts on this plan?
> >
> > On Thu, Nov 15, 2018 at 10:22 AM James Sirota 
> wro

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

Wanted to give an update on the context path issue.  I investigated
rewriting url links in the outgoing static assets with Knox and it was not
trivial.  Fortunately I found a simple solution that works with or without
Knox.  I changed the base tag in index.html from  to , or in other words made the base href relative.

I believe I am at the point where I can task this out and provide a high
level overview of the changes needed.  I think that each task will be a
manageable size and can stand alone so I don't think we need a feature
branch.

The first task involves a general change to the UI code.  We need a way to
set the path to the REST service with a configuration setting because it is
different with and without Knox.  Currently there is not a good way to do
this in the UI.  We can use the environment files but that is a build time
setting and is not flexible.  I can see this capability being useful for
other use cases in the future.  I think we could even split this up into 2
separate tasks, one for the alerts UI and one for the management UI.

The second task involves adding Knox to our stack either by default as a
dependency in the mpack or with a documented approach.  We would add our
REST service, Alerts UI, and Management UI as services in Knox.  Everything
would continue to function as it currently does but with all communication
going through Knox.  LDAP authentication would be required when using Knox
and Knox will authenticate with the REST service by passing along an
Authorization header.  Enabling Knox would be a manual process that
involves deploying assets (Knox descriptor files) and changing
configuration.  There would be no change to how the UI functions by default
(without Knox) and either LDAP or JDBC authentication could still be used..

The third task involves enabling SSO with Knox.  We would update the REST
service so that it can authenticate with a Knox SSO token.  We would
provide documentation on how to update the Knox settings in Ambari to
enable SSO.  The Alerts/Management UI would need to expose configuration
properties for the REST url and login url since these would be different
when Knox is enabled.  We would also need to provide documentation on how
to make these UI configuration changes, based on the work done in task 1.

An optional forth task would be exposing configuration settings and
enabling Knox with Ambari.  We would eliminate the manual steps necessary
for enabling Knox and instead automate those steps with an Ambari input
control, similar to how LDAP is enabled.

Thoughts on this plan?

On Thu, Nov 15, 2018 at 10:22 AM James Sirota  wrote:

> In my view Knox SSO is such a minor feature when it comes to Metron's
> capabilities than it's not worth supporting multiple scenarios where it
> works with Knox or without Knox.  Where we should be configurable (and are
> configurable) is on the analytics and stream processing.  But this?  As
> long as the UI authenticates securely I don't think anyone is going to care
> what proxy it's using.  The code itself should be written in a way that
> it's pluggable so if we ever wanted to use another proxy or disable it all
> together we could.  But this should not be a configuration we pass on to
> the user.  The added complexity is simply not worth it here.  We have to
> start being opinionated about making sensible choices on behalf of the
> user.  A sensible choice here is to run with Knox and LDAP.  The JDBC
> component should exist for another release to allow the community to
> migrate over to LDAP and then be deprecated.  The code should still be
> pluggable and if anyone wanted to extend it to work with JDBC they could,
> or if people wanted to plug in another proxy they could, but this is not
> something we would officially support.
>
> Thanks,
> James
>
> 12.11.2018, 07:36, "Ryan Merriman" :
> > Let me clarify on exposing both legacy and Knox URLs at the same time.
> The
> > base urls will look something like this:
> >
> > Legacy REST - http://node1:8082/api/v1
> > Legacy Alerts UI - http://node1:4201:/alerts-list
> >
> > Knox REST - https://node1:8443/gateway/default/metron/api/v1
> > Knox Alerts UI -
> > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list
> >
> > If Knox were turned on and the alerts UI deployed as is, it would not
> > work. This is because static assets are referenced with
> > http://node1:4201/assets/some-asset.js which does not include the
> correct
> > context path to the alerts UI in knox. To make it work, you have to set
> > the base ref to "/gateway/default/metron-alerts-ui" so that static assets
> > are referenced at
> > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js
> .
> > When you do that, the legacy alerts UI will no longer work. I guess the
>

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

I'm just coming up to speed on Knox so maybe rewriting assets links are
trivial.  If anyone has a good example of how to do that or can point to
some documentation, please share.

On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Doing the Knox proxy work first certainly does make a lot of sense vs the
> SSO first approach, so I'm in favour of this. It bypasses all the anti-CORS
> proxying stuff the other solution needed by being on the same URL space.
>
> Is there are reason we're not re-writing the asset link URLs in Knox? We
> should have a reverse content rewrite rule to avoid that problem and make
> it entirely transparent whether there is Knox or not. We shouldn't be
> changing anything about the UI services themselves. If the rewrite service
> is complete, there is no change to base ref in the UI code, Knox would
> effectively apply it by content filtering. Note also that the gateway URL
> is configurable and likely to vary from Knox to Knox, so baking it into the
> ng build will break non-full-dev builds. (e.g. gateway/default could well
> be gateway/xyz).
>
> I would also like to discuss removing the JDBC auth, because it's a set of
> plaintext passwords in a mysql DB... it introduces a problematic dependency
> (mysql) a ton of java dependencies we could cut out (JPA, eclipselink) and
> opens up a massive security hole. I personally know of several
> organisations who are blocked from using Metron by the presence of the JDBC
> authentication method in its current form.
>
> Simon
>
> On Mon, 12 Nov 2018 at 14:36, Ryan Merriman  wrote:
>
> > Let me clarify on exposing both legacy and Knox URLs at the same time.
> The
> > base urls will look something like this:
> >
> > Legacy REST - http://node1:8082/api/v1
> > Legacy Alerts UI - http://node1:4201:/alerts-list
> >
> > Knox REST - https://node1:8443/gateway/default/metron/api/v1
> > Knox Alerts UI -
> > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list
> >
> > If Knox were turned on and the alerts UI deployed as is, it would not
> > work.  This is because static assets are referenced with
> > http://node1:4201/assets/some-asset.js which does not include the
> correct
> > context path to the alerts UI in knox.  To make it work, you have to set
> > the base ref to "/gateway/default/metron-alerts-ui" so that static assets
> > are referenced at
> > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js
> .
> > When you do that, the legacy alerts UI will no longer work.  I guess the
> > point I'm trying to make is that we would have to switch between them or
> > have 2 separate application running.  I imagine most users only need one
> or
> > the other running so probably not an issue.
> >
> > Jon, the primary upgrade consideration I see is with authentication.  To
> be
> > able to use Knox, you would have to upgrade to LDAP-based authentication
> if
> > you were still using JDBC-based authentication in REST.  The urls would
> > also change obviously.
> >
> > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com 
> wrote:
> >
> > > Phew, that was quite the thread to catch up on.
> > >
> > > I agree that this should be optional/pluggable to start, and I'm
> > interested
> > > to hear the issues as they relate to upgrading an existing cluster
> (given
> > > the suggested approach) and exposing both legacy and knox URLs at the
> > same
> > > time.
> > >
> > > Jon
> > >
> > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com>
> > > wrote:
> > >
> > > > A couple more things, and I think this goes without saying - whatever
> > we
> > > do
> > > > with Knox should NOT
> > > >
> > > >1. Require unit and integration tests to use Knox
> > > >2. Break fulldev
> > > >
> > > > Also, I don't know that I saw you mention this, but I'm unsure how we
> > > > should leverage Knox as a core piece of the platform. i.e. should we
> > make
> > > > this required or optional? I'm open to hearing opinions on this, but
> > I'm
> > > > inclined to keep this a pluggable option.
> > > >
> > > > Mike
> > > >
> > > >
> > > > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Thanks for the update Ryan. Per my earlier comments, I thought it
> >

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

curity.
> > > > >>
> > > > >>  The version of Knox used is the default from HDP. The link
> version
> > > you
> > > > >>  mention is a docs link. I'll update it to be the older version,
> > which
> > > > is
> > > > >>  the same and we can decide if we want to maintain the freshness
> of
> > it
> > > > when
> > > > >>  we look to upgrade underlying patterns. Either way, the content
> is
> > > the
> > > > >>  same.
> > > > >>
> > > > >>  I did consider other hosting mechanisms, including Undertow a
> > > > >>
> > > > >>  If you have a different suggestion to using the Spring default
> ways
> > > of
> > > > >>  doing things, or we want to use a framework other than Spring for
> > > this,
> > > > >>  then maybe we could change to that, but the route chosen here is
> > > > definitely
> > > > >>  the easy path in the context of the decision made to use Spring
> in
> > > > metron
> > > > >>  rest, and if anything opens up our choices while minimising, in
> > fact
> > > > >>  reducing, our dependency management overhead.
> > > > >>
> > > > >>  I hope that explains some of the thinking behind the choices
> made,
> > > but
> > > > the
> > > > >>  guiding principals I followed were:
> > > > >>  * Don't fight the framework if you don't have to
> > > > >>  * Reduce the need for additional installation pieces and third
> > party
> > > > repos
> > > > >>  * Minimize dependencies we would have to manage
> > > > >>  * Avoid excessive change of the architecture, or forcing users to
> > > adopt
> > > > >>  Knox if they didn't want the SSL overhead.
> > > > >>
> > > > >>  Simon
> > > > >>
> > > > >>  On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic <
> > > > >>  michael.miklav...@gmail.com> wrote:
> > > > >>
> > > > >>>  Thanks for the write-up Ryan, this is a great start. I have some
> > > > further
> > > > >>>  questions based on your feedback and in addition to my initial
> > > thread.
> > > > >>>
> > > > >>>  Just for clarification, what version of Knox are we using? HDP
> > > 2.6.5,
> > > > >>>  which
> > > > >>>  is what we currently run full dev against, supports 0.12.0.
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
> > > > >>>  .
> > > > >>>  I see references to Knox 1.1.0 (latest) in this committed PR -
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> https://github.com/apache/metron/pull//files#diff-70b412194819f3cb829566f05d77c1a6R122
> > > > >>>  .
> > > > >>>  This is probably just a super small mismatch, and it probably
> goes
> > > > without
> > > > >>>  saying, but I just want to be doubly sure that we're installing
> > the
> > > > >>>  default
> > > > >>>  via the standard install mechanism as opposed to something
> > separate
> > > > and
> > > > >>>  manual.
> > > > >>>
> > > > >>>  On the subject of Zuul wrt Nodejs filters. I'd like to hear some
> > > more
> > > > >>>  detail on:
> > > > >>>
> > > > >>> 1. Why do we need filtering via Zuul? For instance, is
> > filtering
> > > > >>>  routing
> > > > >>> not handled by Knox? From the beginner docs: "The gateway
> > itself
> > > > is a
> > > > >>>  layer
> > > > >>> over an embedded Jetty JEE server. At the very highest level
> > the
> > > > >>>  gateway
> > > > >>> processes requests by using request URLs to lookup specific
> JEE
> > > > Servlet
> > > > >>> Filter chain that is used to process the request. The gateway
> > > > framework
>

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

[Ryan Merriman]

+1

On Thu, Nov 1, 2018 at 5:38 PM Casey Stella  wrote:

> +1
> On Thu, Nov 1, 2018 at 18:34 Nick Allen  wrote:
>
> > +1
> >
> > On Thu, Nov 1, 2018, 6:27 PM Justin Leet  wrote:
> >
> > > +1, I haven't seen any case where the split-join topology isn't made
> > > obsolete by the unified topology.
> > >
> > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Fellow Metronians,
> > > >
> > > > We've had the unified enrichment topology around for a number of
> months
> > > > now, it has proved itself stable, and there is yet to be a time that
> I
> > > have
> > > > seen the split-join topology outperform the unified one. Here are
> some
> > > > simple reasons to deprecate the split-join topology.
> > > >
> > > >1. Unified topology performs better.
> > > >2. The configuration, especially for performance tuning is much,
> > much
> > > >simpler in the unified model.
> > > >3. The footprint within the cluster is smaller.
> > > >4. One of the first activities for any install is that we spend
> time
> > > >instructing users to switch to the unified topology.
> > > >5. One less moving part to maintain.
> > > >
> > > > I'd like to recommend that we deprecate the split-join topology and
> > make
> > > > the unified enrichment topology the new default.
> > > >
> > > > Best,
> > > > Mike
> > > >
> > >
> >
>

Re: [DISCUSS] Stellar REST client

[Ryan Merriman]

Just FYI, I put a PR up for this feature:
https://github.com/apache/metron/pull/1250.  I believe I've addressed
everyone's questions either in the PR description or the code itself.  I'm
think we can continue the discussion there.

On Fri, Oct 19, 2018 at 12:02 PM Otto Fowler 
wrote:

> I believe the issue of introducing and supporting higher latency
> enrichments is a systemic one, and should be solved as such,
> with the rest and other higher latency enrichments build on top of that
> framework.
>
>
>
>
> On October 19, 2018 at 12:22:28, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> Thanks Casey, good questions.
>
> As far as the verbs go, just thinking we might want to support calls other
> than GET at some point. For the use case stated (enriching messages from
> 3rd party services) GET is all we need. Probably a moot point anyways
> since every http library will support the different HTTP verbs.
>
> Agreed on the caching. I will defer to those that are more familiar with
> the Stellar internals on what the correct approach is.
>
> I was thinking the same thing with regards to the client libraries. Apache
> HttpComponents is probably the safest choice but OkHttp looks nice and
> could reduce effort and complexity as long as it meets our requirements.
>
> On Fri, Oct 19, 2018 at 10:58 AM Casey Stella  wrote:
>
> > I think it makes a lot of sense. A couple of questions:
> >
> > - What actions do you see the REST verbs corresponding to? I would
> > understand GET (which is in effect "evaluate an expression", right?),
> > but
> > I'm not sure about the others.
> > - We should probably be careful about caching stellar expressions. Not
> > all stellar expressions are deterministic (e.g. PROFILE_GET may not be
> > as
> > the lookback window is bound to current time). Ultimately, I think we
> > should probably bake whether a function is deterministic into stellar so
> > that *stellar* can cache where appropriate (e.g. if every part of an
> > expression is deterministic, then pull from cache otherwise recompute).
> > All of this to say, if you're going to make it configurable, IMO we
> > should
> > make it a configuration that the user passes in at request time so they
> > have the control over whether the expression is safe to cache or
> > otherwise.
> >
> > Without more compelling reasons to not do so, I'd suggest we use HTTP
> > Components as it's another apache project and under active
> > development/support. I'd also be ok with OkHttp if it's actively
> > maintained.
> >
> > On Fri, Oct 19, 2018 at 11:46 AM Ryan Merriman 
> > wrote:
> >
> > > I want to open up discussion around adding a Stellar REST client
> > function.
> > > There are services available to enrich security telemetry and they are
> > > commonly exposed through a REST interface. The primary purpose of this
> > > discuss thread to collect requirements from the community and agree on
> a
> > > general architectural approach.
> > >
> > > At a minimum I see a Stellar REST client supporting:
> > >
> > > - Common HTTP verbs including GET, POST, DELETE, etc
> > > - Option to provide headers and request parameters as needed
> > > - Support for basic authentication
> > > - Proper request and error handling (we can discuss further how this
> > > should work)
> > > - SSL support
> > > - Option to use a proxy server (including authentication)
> > > - JSON format
> > >
> > > In addition to these functional requirements, I would also propose we
> > > include these performance requirements:
> > >
> > > - Provide a configurable caching layer
> > > - Provide a mechanism for pooling connections
> > > - Provide clear documentation and guidance on how to properly use this
> > > feature since there is a significant risk of introducing latency
> > issues
> > >
> > > What else would you like to see included?
> > >
> > > I think the primary architectural decision we need to make (based on
> the
> > > agreed upon requirements of course) is an appropriate Java HTTP/REST
> > client
> > > library. Ideally we choose a library that supports everything we need
> > > OOTB. I think the majority of the work for this feature will involve
> > > wrapping this library in a Stellar function and exposing the
> > configuration
> > > knobs through Metron's configuration interface (Ambari, Zookeeper,
> > etc). I
> > > have done some very light research and here is my initial list:
>

Re: [DISCUSS] Stellar REST client

[Ryan Merriman]

Thanks Casey, good questions.

As far as the verbs go, just thinking we might want to support calls other
than GET at some point.  For the use case stated (enriching messages from
3rd party services) GET is all we need.  Probably a moot point anyways
since every http library will support the different HTTP verbs.

Agreed on the caching.  I will defer to those that are more familiar with
the Stellar internals on what the correct approach is.

I was thinking the same thing with regards to the client libraries.  Apache
HttpComponents is probably the safest choice but OkHttp looks nice and
could reduce effort and complexity as long as it meets our requirements.

On Fri, Oct 19, 2018 at 10:58 AM Casey Stella  wrote:

> I think it makes a lot of sense.  A couple of questions:
>
>- What actions do you see the REST verbs corresponding to?  I would
>understand GET (which is in effect "evaluate an expression", right?),
> but
>I'm not sure about the others.
>- We should probably be careful about caching stellar expressions.  Not
>all stellar expressions are deterministic (e.g. PROFILE_GET may not be
> as
>the lookback window is bound to current time).  Ultimately, I think we
>should probably bake whether a function is deterministic into stellar so
>that *stellar* can cache where appropriate (e.g. if every part of an
>expression is deterministic, then pull from cache otherwise recompute).
>All of this to say, if you're going to make it configurable, IMO we
> should
>make it a configuration that the user passes in at request time so they
>have the control over whether the expression is safe to cache or
> otherwise.
>
> Without more compelling reasons to not do so, I'd suggest we use HTTP
> Components as it's another apache project and under active
> development/support.  I'd also be ok with OkHttp if it's actively
> maintained.
>
> On Fri, Oct 19, 2018 at 11:46 AM Ryan Merriman 
> wrote:
>
> > I want to open up discussion around adding a Stellar REST client
> function.
> > There are services available to enrich security telemetry and they are
> > commonly exposed through a REST interface.  The primary purpose of this
> > discuss thread to collect requirements from the community and agree on a
> > general architectural approach.
> >
> > At a minimum I see a Stellar REST client supporting:
> >
> >- Common HTTP verbs including GET, POST, DELETE, etc
> >- Option to provide headers and request parameters as needed
> >- Support for basic authentication
> >- Proper request and error handling (we can discuss further how this
> >should work)
> >- SSL support
> >- Option to use a proxy server (including authentication)
> >- JSON format
> >
> > In addition to these functional requirements, I would also propose we
> > include these performance requirements:
> >
> >- Provide a configurable caching layer
> >- Provide a mechanism for pooling connections
> >- Provide clear documentation and guidance on how to properly use this
> >feature since there is a significant risk of introducing latency
> issues
> >
> > What else would you like to see included?
> >
> > I think the primary architectural decision we need to make (based on the
> > agreed upon requirements of course) is an appropriate Java HTTP/REST
> client
> > library.  Ideally we choose a library that supports everything we need
> > OOTB.  I think the majority of the work for this feature will involve
> > wrapping this library in a Stellar function and exposing the
> configuration
> > knobs through Metron's configuration interface (Ambari, Zookeeper,
> etc).  I
> > have done some very light research and here is my initial list:
> >
> >- Apache HttpComponents - https://hc.apache.org/
> >- Has support for all of the features listed above as far as I can
> tell
> >   - Doesn't introduce a large number of new dependencies (am I wrong
> >   here?)
> >   - Is sort of included already (we will need to upgrade from
> >   httpclient)
> >   - Lower level
> >- Google HTTP Client Library for Java -
> >
> >
> https://developers.google.com/api-client-library/java/google-http-java-client/
> >- Higher level API with pluggable components
> >   - Introduces dependencies (we've had issues with Guava in the past)
> >- Netflix Ribbon - https://github.com/Netflix/ribbon
> >   - Has a lot of nice features that may be useful in the future
> >   - Introduces dependencies (including guava)
> >   - Hasn't been committed to in the

[DISCUSS] Stellar REST client

[Ryan Merriman]

I want to open up discussion around adding a Stellar REST client function.
There are services available to enrich security telemetry and they are
commonly exposed through a REST interface.  The primary purpose of this
discuss thread to collect requirements from the community and agree on a
general architectural approach.

At a minimum I see a Stellar REST client supporting:

   - Common HTTP verbs including GET, POST, DELETE, etc
   - Option to provide headers and request parameters as needed
   - Support for basic authentication
   - Proper request and error handling (we can discuss further how this
   should work)
   - SSL support
   - Option to use a proxy server (including authentication)
   - JSON format

In addition to these functional requirements, I would also propose we
include these performance requirements:

   - Provide a configurable caching layer
   - Provide a mechanism for pooling connections
   - Provide clear documentation and guidance on how to properly use this
   feature since there is a significant risk of introducing latency issues

What else would you like to see included?

I think the primary architectural decision we need to make (based on the
agreed upon requirements of course) is an appropriate Java HTTP/REST client
library.  Ideally we choose a library that supports everything we need
OOTB.  I think the majority of the work for this feature will involve
wrapping this library in a Stellar function and exposing the configuration
knobs through Metron's configuration interface (Ambari, Zookeeper, etc).  I
have done some very light research and here is my initial list:

   - Apache HttpComponents - https://hc.apache.org/
   - Has support for all of the features listed above as far as I can tell
  - Doesn't introduce a large number of new dependencies (am I wrong
  here?)
  - Is sort of included already (we will need to upgrade from
  httpclient)
  - Lower level
   - Google HTTP Client Library for Java -
   
https://developers.google.com/api-client-library/java/google-http-java-client/
   - Higher level API with pluggable components
  - Introduces dependencies (we've had issues with Guava in the past)
   - Netflix Ribbon - https://github.com/Netflix/ribbon
  - Has a lot of nice features that may be useful in the future
  - Introduces dependencies (including guava)
  - Hasn't been committed to in the last 5-6 months
   - Unirest - https://github.com/Kong/unirest-java
  - Lightweight API built on top of HttpComponents
  - Pluggable serialization library (jackson is an issue for us so this
  is nice)
  - Also has not received a commit in a while
   - OkHttp - http://square.github.io/okhttp/
   - Good documentation and looks easy to use
  - Actively maintained

Obviously we have a lot of choices.  I think it comes down to balancing the
tradeoff between ease of use (HttpComponents will likely require the most
work since it is lower level) and capability.  Introducing additional
dependencies is something we should also be mindful of because our shading
practices.

This should get us started.  Let me know what you think!

Re: [DISCUSS] Batch Profiler Feature Branch

[Ryan Merriman]

+1 from me.  Great work.

On Thu, Sep 27, 2018 at 12:41 PM Justin Leet  wrote:

> I'm +1 on merging the feature branch into master. There's a lot of good
> work here, and it's definitely been nice to see the couple remaining
> improvements make it in.
>
> Thanks a lot for the contribution, this is great stuff!
>
> On Wed, Sep 26, 2018 at 6:26 PM Nick Allen  wrote:
>
> > Or support to be offered for merging this feature branch into master?
> >
> > On Wed, Sep 26, 2018 at 6:20 PM Nick Allen  wrote:
> >
> > > Thanks for the review.  With
> https://github.com/apache/metron/pull/1209
> > complete,
> > > I think the feature branch is ready to be merged.  Sounds like I have
> > > Mike's support.  Anyone else have comments, concerns, questions?
> > >
> > > On Tue, Sep 25, 2018 at 10:33 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> I just made a couple minor comments on that PR, and I am in agreement
> > >> about
> > >> the readiness for merging with master. Good stuff Nick.
> > >>
> > >> On Fri, Sep 21, 2018 at 12:37 PM Nick Allen 
> wrote:
> > >>
> > >> > Here is a PR that adds the input time constraints to the Batch
> > Profiler
> > >> > (METRON-1787);  https://github.com/apache/metron/pull/1209.
> > >> >
> > >> > It seems that the consensus is that this is probably the last
> feature
> > we
> > >> > need before merging the FB into master.  The other two can wait
> until
> > >> after
> > >> > the feature branch has been merged.  Let me know if you disagree.
> > >> >
> > >> > Thanks
> > >> >
> > >> >
> > >> > On Thu, Sep 20, 2018 at 1:55 PM Nick Allen 
> > wrote:
> > >> >
> > >> > > Yeah, agreed.  Per use case 3, when deploying to production there
> > >> really
> > >> > > wouldn't be a huge overlap like 3 months of already profiled data.
> > >> Its
> > >> > day
> > >> > > 1, the profile was just deployed around the same time as you are
> > >> running
> > >> > > the Batch Profiler, so the overlap is in minutes, maybe hours.
> But
> > I
> > >> can
> > >> > > definitely see the usefulness of the feature for re-runs, etc as
> you
> > >> have
> > >> > > described.
> > >> > >
> > >> > > Based on this discussion, I created a few JIRAs.  Thanks all for
> the
> > >> > great
> > >> > > feedback and keep it coming.
> > >> > >
> > >> > > [1] METRON-1787 - Input Time Constraints for Batch Profiler
> > >> > > [2] METRON-1788 - Fetch Profile Definitions from Zk for Batch
> > Profiler
> > >> > > [3] METRON-1789 - MPack Should Define Default Input Path for Batch
> > >> > > Profiler
> > >> > >
> > >> > >
> > >> > > --
> > >> > > [1] https://issues.apache.org/jira/browse/METRON-1787
> > >> > > [2] https://issues.apache.org/jira/browse/METRON-1788
> > >> > > [3] https://issues.apache.org/jira/browse/METRON-1789
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Thu, Sep 20, 2018 at 1:34 PM Michael Miklavcic <
> > >> > > michael.miklav...@gmail.com> wrote:
> > >> > >
> > >> > >> I think we might want to allow the flexibility to choose the date
> > >> range
> > >> > >> then. I don't yet feel like I have a good enough understanding of
> > all
> > >> > the
> > >> > >> ways in which users would want to seed to force them to run the
> > batch
> > >> > job
> > >> > >> over all the data. It might also make it easier to deal with
> > >> > remediation,
> > >> > >> ie an error doesn't force you to re-run over the entire history.
> > Same
> > >> > goes
> > >> > >> for testing out the profile seeing batch job in the first place.
> > >> > >>
> > >> > >> On Thu, Sep 20, 2018 at 11:23 AM Nick Allen 
> > >> wrote:
> > >> > >>
> > >> > >> > Assuming you have 9 months of data archived, yes.
> > >> > >> >
> > >> > >> > On Thu, Sep 20, 2018 at 1:22 PM Michael Miklavcic <
> > >> > >> > michael.miklav...@gmail.com> wrote:
> > >> > >> >
> > >> > >> > > So in the case of 3 - if you had 6 months of data that hadn't
> > >> been
> > >> > >> > profiled
> > >> > >> > > and another 3 that had been profiled (9 months total data),
> in
> > >> its
> > >> > >> > current
> > >> > >> > > form the batch job runs over all 9 months?
> > >> > >> > >
> > >> > >> > > On Thu, Sep 20, 2018 at 11:13 AM Nick Allen <
> > n...@nickallen.org>
> > >> > >> wrote:
> > >> > >> > >
> > >> > >> > > > > How do we establish "tm" from 1.1 above? Any concerns
> about
> > >> > >> overlap
> > >> > >> > or
> > >> > >> > > > gaps after the seeding is performed?
> > >> > >> > > >
> > >> > >> > > > Good point.  Right now, if the Streaming and Batch Profiler
> > >> > overlap
> > >> > >> the
> > >> > >> > > > last write wins.  And presumably the output of the
> Streaming
> > >> and
> > >> > >> Batch
> > >> > >> > > > Profiler are the same, so no worries, right? :)
> > >> > >> > > >
> > >> > >> > > > So it kind of works, but it is definitely not ideal for use
> > >> case
> > >> > >> 3.  I
> > >> > >> > > > could add --begin and --end args to constrain the time
> frame
> > >> over
> > >> > >> which
> > >> > >> > > the
> > >> > >> > > > 

Re: [DISCUSS] Knox SSO feature branch review and features

[Ryan Merriman]

I have reviewed a couple different PRs so I'll add some context where I
can.  Obviously Simon would be the most qualified to answer but I'll add my
thoughts.

For question 1, while they may not all be necessary I think it does make
sense to include them in this feature branch if our primary goal is
integrating Knox SSO.  We could push off removing JDBC authentication for
reasons I'll get to in my response to question 2.  If we want to do one at
a time (switch to spring boot, add Zuul as a dependency, then add Knox SSO)
then that's ok but I do think there are dependencies and should be done in
order.  For example, adding Knox SSO requires some work around request
filtering.  If we were to do this before moving to Spring Boot we would
need to implement the filters in Nodejs which would be throwaway once we
get around to migrating away from that.  For Zuul, I believe it's purpose
is to facilitate the filtering (although it does a lot more) so it doesn't
make sense to add that separate from the Knox SSO work.

For question 2, I think you bring up a good point.  We probably don't want
to just rip our current authentication method out.  We might want to
consider deprecating it instead and making Knox SSO and LDAP authentication
optional.

For question 3, this is a bigger shift than just a component upgrade.  It's
more like shifting platforms, from Elasticsearch to Solr for example.  Like
I alluded to in my response to question 1, I don't think we should require
throwaway work just because we want to review these parts separately.

For question 4, I will defer to Simon.  I don't believe we necessarily
require Zuul so I will let him elaborate on why he choose that library and
what the potential impact is of adding it to our project.

For question 5 and 6, I will also defer to Simon on this.  The focus of
this feature as I understand it is a consistent authentication mechanism
and support for SSO.  I will let him lay out his vision for micro services.

Knox SSO would be a great improvement and is what I think we should focus
on in this feature branch.  Micro services is something we should certainly
discuss but it might be a bit of a distraction and I wouldn't want to hold
up the other useful parts.

On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey all,
>
> I started looking through the Knox SSO feature branch (see here
> https://issues.apache.org/jira/browse/METRON-1663). This is some great new
> security functionality work and it looks like it will bring some important
> new features to the Metron platform. I'm coming at this pretty green, so I
> do have some questions regarding the proposed changes from a high level
> architectural perspective. There are a few changes within the current FB
> PR's that I think could use some further explanation. At first glance, it
> seems we could potentially simplify this branch a great deal and get it
> completed much sooner if we narrowed the focus a bit. But I could certainly
> be wrong here and happy for other opinions. I searched through the mailing
> list history to see if there is any additional background and the main
> DISCUSS thread I could find was regarding initially setting up the feature
> branch, which talked about adding Knox and LDAP.
>
> https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E
> .
> If I've missed any follow-up, please let me know.
>
> Looking at the broader set of Jiras associated with 1663 and the first PR
> 1665, it looks like there are 4 main thrusts of this branch right now:
>
>1.  Knox/SSO
>2.  Node migrated to Spring Boot
>3.  JDBC removed completely in favor of LDAP
>4.  Introduction of Zuul, also microservices?
>
> I strongly urge for the purpose of reviewing this feature branch that we
> base much of the discussion off of
> https://issues.apache.org/jira/browse/METRON-1755, the architecture
> diagram. Minimally, an explanation of the current architecture along with
> discussion around the additional proposed changes and rationale would be
> useful for evaluation. I don't have a solid enough understanding yet of the
> full scope of changes and how they differ from the existing architecture
> just from looking at the PR's alone.
>
>1. The first question is a general one regarding the necessity of the 3
>additional features alongside Knox - migrating Node to Spring Boot,
>removing JDBC altogether, adding dependencies on Netflix's Zuul
> framework.
>Are these necessary for adding Knox/SSO? They seem like potentially
>separate features, imo.
>2. It looks like LDAP will be a required component for interacting with
>Metron via the UI's. I see this PR
>https://github.com/apache/metron/pull/1186 which removes JDBC
>authentication. Are we ready to remove it completely or would it be
> better
>to leave it as a minimal installation option? What is the proposed
>migration path 

Re: [DISCUSS] Internal Metron fields

[Ryan Merriman]

Internal means it’s not configurable, doesn’t contain our default separator 
(dots) and is namespaced with metron.  We can definitely improve on DRY but 
there’s more to it than that.  For example, having 2 different versions of this 
field name (ES and Solr) adds a significant amount of complexity for no real 
benefit.

> On Sep 7, 2018, at 5:12 PM, Michael Miklavcic  
> wrote:
> 
> Can you elaborate on what you mean by "convert to internal?" From your
> description, it looks like the challenge is from our violations of DRY when
> it comes to constants referencing those keys, which would be eliminated by
> refactoring.
> 
>> On Fri, Sep 7, 2018, 3:50 PM Ryan Merriman  wrote:
>> 
>> I recently worked on a PR that involved changing the default behavior of
>> the ElasticsearchWriter to store data using field names with the default
>> Metron separator, dots.  One of the unfortunate consequences of this is
>> that although dots are allowed in more recent versions of ES, it changes
>> how these fields are stored.  Having a dot in a field name causes ES to
>> treat it as an object field type.  We're not quite comfortable with this
>> because it could introduce unforeseen side effects that may not be
>> obvious.  Here's the PR:  https://github.com/apache/metron/pull/1181
>> 
>> As I worked through it I noticed there are a couple fields that include
>> separators where it's not actually necessary.  They are not nested by
>> nature and are internal to Metron.  The fact that they are internal means
>> they show up in constants and are hardcoded in several different places.
>> That made the work in the PR above much harder and tedious than it should
>> have been.  There are 2 in particular that I had to deal with:  source:type
>> and threat:triage:score in metaalerts.
>> 
>> Is it worth considering converting these to internal Metron fields so that
>> they stay constant and this isn't a problem in the future?  I could see
>> these fields following the same pattern as 'metron_alert'.  However this
>> would cause pain when upgrading because existing data would need to be
>> updated with these new fields.
>> 
>> Just an idea.  Curious if other have an opinion on the subject.
>> 

[DISCUSS] Internal Metron fields

[Ryan Merriman]

I recently worked on a PR that involved changing the default behavior of
the ElasticsearchWriter to store data using field names with the default
Metron separator, dots.  One of the unfortunate consequences of this is
that although dots are allowed in more recent versions of ES, it changes
how these fields are stored.  Having a dot in a field name causes ES to
treat it as an object field type.  We're not quite comfortable with this
because it could introduce unforeseen side effects that may not be
obvious.  Here's the PR:  https://github.com/apache/metron/pull/1181

As I worked through it I noticed there are a couple fields that include
separators where it's not actually necessary.  They are not nested by
nature and are internal to Metron.  The fact that they are internal means
they show up in constants and are hardcoded in several different places.
That made the work in the PR above much harder and tedious than it should
have been.  There are 2 in particular that I had to deal with:  source:type
and threat:triage:score in metaalerts.

Is it worth considering converting these to internal Metron fields so that
they stay constant and this isn't a problem in the future?  I could see
these fields following the same pattern as 'metron_alert'.  However this
would cause pain when upgrading because existing data would need to be
updated with these new fields.

Just an idea.  Curious if other have an opinion on the subject.

Re: [DISCUSS] Pcap query branch completion

[Ryan Merriman]

The feature branch has been merged into master.

On Thu, Aug 16, 2018 at 5:53 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'm +1, thanks for adding that fix, Ryan. (Note, for purposes of vote, I
> was a contributor in the feature branch).
>
> Mike
>
> On Thu, Aug 16, 2018, 4:17 PM Ryan Merriman  wrote:
>
> > We discovered a bug in our testing and felt it should be fixed before we
> > merge.  There is a PR up for review that already has a +1:
> > https://github.com/apache/metron/pull/1168.  I don't anticipate this
> > changing anyone's vote but wanted to be clear about the state of the
> > branch.  If anyone is concerned with this and would like more discussion
> > before we merge, let me know.
> >
> > On Thu, Aug 16, 2018 at 8:25 AM, James Sirota 
> wrote:
> >
> > > +1 on the merge as well
> > >
> > > 16.08.2018, 05:46, "Casey Stella" :
> > > > I'm +1 on the merge. This is great work and congrats to those who
> > > > contributed to it!
> > > >
> > > > On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler  >
> > > wrote:
> > > >
> > > >>  Looks good, thanks!
> > > >>
> > > >>  On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com
> )
> > > wrote:
> > > >>
> > > >>  Otto, I believe the items you requested are in the feature branch
> > now.
> > > Is
> > > >>  there anything outstanding that we missed? The Jiras for the Pcap
> > > feature
> > > >>  branch should be up to date:
> > > >>  https://issues.apache.org/jira/browse/METRON-1554
> > > >>
> > > >>  On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <
> merrim...@gmail.com>
> > > >>  wrote:
> > > >>
> > > >>  > - Date range limits on queries
> > > >>  >
> > > >>  > I will add a warning in the Job cleanup PR. That seems like an
> > > >>  > appropriate place for it (ie. make sure you don't cause health
> > > issues in
> > > >>  > your cluster).
> > > >>  >
> > > >>  > - UI should manage a queue/history of jobs
> > > >>  >
> > > >>  > I can add some documentation around killing jobs manually with
> the
> > > YARN
> > > >>  > CLI. However if they haven't set up a YARN queue, I'm not sure
> how
> > > you
> > > >>  > would view only Pcap jobs. I'm also not sure how you would get
> the
> > > >>  > application id for the job to kill because it's not displayed
> > > anywhere in
> > > >>  > the UI. However, I believe we are wired for a job name but REST
> > > doesn't
> > > >>  > set this. Maybe we could get a proper job name associated with
> pcap
> > > >>  > queries and then this would be possible to document?
> > > >>  >
> > > >>  > - Documentation/blueprint for YARN configuration
> > > >>  >
> > > >>  > You make a good point. A YARN tuning guide for Metron does sound
> > > useful.
> > > >>  > I will add a follow on Jira.
> > > >>  >
> > > >>  > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <
> > > ottobackwa...@gmail.com>
> > > >>  > wrote:
> > > >>  >
> > > >>  >>
> > > >>  >> - Date range limits on queries
> > > >>  >>
> > > >>  >> I took the point the wrong way apparently, sorry, I withdraw. I
> > > thought
> > > >>  >> you meant allow specifying a limit on the query, not the system
> > > imposing
> > > >>  a
> > > >>  >> limit.
> > > >>  >> This should be documented with a warning or something
> > > >>  >>
> > > >>  >> - UI should manage a queue/history of jobs
> > > >>  >>
> > > >>  >> I was thinking that if there where multiple users/jobs, there
> > should
> > > >>  >> be some thought or documentation + script on how to manage them.
> > > >>  >> “To see all the jobs still running on your cluster, across users
> > > and ui
> > > >>  >> instances do X”
> > > >>  >> “If there is an issue with the jobs you can’t resolve in the UI
> > for
> &g

Re: [DISCUSS] Pcap query branch completion

[Ryan Merriman]

We discovered a bug in our testing and felt it should be fixed before we
merge.  There is a PR up for review that already has a +1:
https://github.com/apache/metron/pull/1168.  I don't anticipate this
changing anyone's vote but wanted to be clear about the state of the
branch.  If anyone is concerned with this and would like more discussion
before we merge, let me know.

On Thu, Aug 16, 2018 at 8:25 AM, James Sirota  wrote:

> +1 on the merge as well
>
> 16.08.2018, 05:46, "Casey Stella" :
> > I'm +1 on the merge. This is great work and congrats to those who
> > contributed to it!
> >
> > On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler 
> wrote:
> >
> >>  Looks good, thanks!
> >>
> >>  On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com)
> wrote:
> >>
> >>  Otto, I believe the items you requested are in the feature branch now.
> Is
> >>  there anything outstanding that we missed? The Jiras for the Pcap
> feature
> >>  branch should be up to date:
> >>  https://issues.apache.org/jira/browse/METRON-1554
> >>
> >>  On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman 
> >>  wrote:
> >>
> >>  > - Date range limits on queries
> >>  >
> >>  > I will add a warning in the Job cleanup PR. That seems like an
> >>  > appropriate place for it (ie. make sure you don't cause health
> issues in
> >>  > your cluster).
> >>  >
> >>  > - UI should manage a queue/history of jobs
> >>  >
> >>  > I can add some documentation around killing jobs manually with the
> YARN
> >>  > CLI. However if they haven't set up a YARN queue, I'm not sure how
> you
> >>  > would view only Pcap jobs. I'm also not sure how you would get the
> >>  > application id for the job to kill because it's not displayed
> anywhere in
> >>  > the UI. However, I believe we are wired for a job name but REST
> doesn't
> >>  > set this. Maybe we could get a proper job name associated with pcap
> >>  > queries and then this would be possible to document?
> >>  >
> >>  > - Documentation/blueprint for YARN configuration
> >>  >
> >>  > You make a good point. A YARN tuning guide for Metron does sound
> useful.
> >>  > I will add a follow on Jira.
> >>  >
> >>  > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <
> ottobackwa...@gmail.com>
> >>  > wrote:
> >>  >
> >>  >>
> >>  >> - Date range limits on queries
> >>  >>
> >>  >> I took the point the wrong way apparently, sorry, I withdraw. I
> thought
> >>  >> you meant allow specifying a limit on the query, not the system
> imposing
> >>  a
> >>  >> limit.
> >>  >> This should be documented with a warning or something
> >>  >>
> >>  >> - UI should manage a queue/history of jobs
> >>  >>
> >>  >> I was thinking that if there where multiple users/jobs, there should
> >>  >> be some thought or documentation + script on how to manage them.
> >>  >> “To see all the jobs still running on your cluster, across users
> and ui
> >>  >> instances do X”
> >>  >> “If there is an issue with the jobs you can’t resolve in the UI for
> that
> >>  >> user, or you are an admin and want to do something then X"
> >>  >>
> >>  >> - Documentation/blueprint for YARN configuration
> >>  >>
> >>  >> I agree with what you are saying. Although, we offer guidance on
> storm
> >>  >> tuning, and that is conceptually the same isn’t it? That is why it
> comes
> >>  >> to mind.
> >>  >> Maybe this can be a follow on, in the tuning guide?
> >>  >>
> >>  >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com)
> >>  >> wrote:
> >>  >>
> >>  >> - Date range limits on queries
> >>  >>
> >>  >> Can you describe what you think is needed here? Each Metron user
> could
> >>  >> have different volumes of pcap data spread out over different time
> >>  >> periods. Are you saying we should limit the data range to something
> >>  either
> >>  >>
> >>  >> constant or configurable? Are we sure all users would want this? Am
> I
> >>  >> misinterpreting this requirement?
> >>  >>
> >>  >> - UI should manage a queue/history of jobs
> >>  >>
> >>  >> What should we document here? Reading that bullet point again, it's
> sort
> >>  >> of vague and not very description. What I am referring to is a
> design
> >>  that
> >>  >>
> >>  >> provides users a way to view and manage jobs in the UI. Currently
> jobs
> >>  can
> >>  >>
> >>  >> only be run 1 at a time and progress is shown with a status bar, so
> it's
> >>  >> somewhat interactive.
> >>  >>
> >>  >> - Documentation/blueprint for YARN configuration
> >>  >>
> >>  >>
> >>  >
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: [DISCUSS] Pcap query branch completion

[Ryan Merriman]

Otto, I believe the items you requested are in the feature branch now.  Is
there anything outstanding that we missed?  The Jiras for the Pcap feature
branch should be up to date:
https://issues.apache.org/jira/browse/METRON-1554

On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman  wrote:

> - Date range limits on queries
>
> I will add a warning in the Job cleanup PR.  That seems like an
> appropriate place for it (ie. make sure you don't cause health issues in
> your cluster).
>
> - UI should manage a queue/history of jobs
>
> I can add some documentation around killing jobs manually with the YARN
> CLI.  However if they haven't set up a YARN queue, I'm not sure how you
> would view only Pcap jobs.  I'm also not sure how you would get the
> application id for the job to kill because it's not displayed anywhere in
> the UI.  However, I believe we are wired for a job name but REST doesn't
> set this.  Maybe we could get a proper job name associated with pcap
> queries and then this would be possible to document?
>
> - Documentation/blueprint for YARN configuration
>
> You make a good point.  A YARN tuning guide for Metron does sound useful.
> I will add a follow on Jira.
>
> On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler 
> wrote:
>
>>
>> - Date range limits on queries
>>
>> I took the point the wrong way apparently, sorry, I withdraw.  I thought
>> you meant allow specifying a limit on the query, not the system imposing a
>> limit.
>> This should be documented with a warning or something
>>
>> - UI should manage a queue/history of jobs
>>
>> I was thinking that if there where multiple users/jobs, there should
>> be some thought or documentation + script on how to manage them.
>> “To see all the jobs still running on your cluster, across users and ui
>> instances do X”
>> “If there is an issue with the jobs you can’t resolve in the UI for that
>> user, or you are an admin and want to do something then X"
>>
>> - Documentation/blueprint for YARN configuration
>>
>> I agree with what you are saying.  Although, we offer guidance on storm
>> tuning, and that is conceptually the same isn’t it?  That is why it comes
>> to mind.
>> Maybe this can be a follow on, in the tuning guide?
>>
>> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com)
>> wrote:
>>
>> - Date range limits on queries
>>
>> Can you describe what you think is needed here? Each Metron user could
>> have different volumes of pcap data spread out over different time
>> periods. Are you saying we should limit the data range to something either
>>
>> constant or configurable? Are we sure all users would want this? Am I
>> misinterpreting this requirement?
>>
>> - UI should manage a queue/history of jobs
>>
>> What should we document here? Reading that bullet point again, it's sort
>> of vague and not very description. What I am referring to is a design that
>>
>> provides users a way to view and manage jobs in the UI. Currently jobs can
>>
>> only be run 1 at a time and progress is shown with a status bar, so it's
>> somewhat interactive.
>>
>> - Documentation/blueprint for YARN configuration
>>
>>
>

Re: [DISCUSS] Pcap query branch completion

[Ryan Merriman]

- Date range limits on queries

I will add a warning in the Job cleanup PR.  That seems like an appropriate
place for it (ie. make sure you don't cause health issues in your cluster).

- UI should manage a queue/history of jobs

I can add some documentation around killing jobs manually with the YARN
CLI.  However if they haven't set up a YARN queue, I'm not sure how you
would view only Pcap jobs.  I'm also not sure how you would get the
application id for the job to kill because it's not displayed anywhere in
the UI.  However, I believe we are wired for a job name but REST doesn't
set this.  Maybe we could get a proper job name associated with pcap
queries and then this would be possible to document?

- Documentation/blueprint for YARN configuration

You make a good point.  A YARN tuning guide for Metron does sound useful.
I will add a follow on Jira.

On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler 
wrote:

>
> - Date range limits on queries
>
> I took the point the wrong way apparently, sorry, I withdraw.  I thought
> you meant allow specifying a limit on the query, not the system imposing a
> limit.
> This should be documented with a warning or something
>
> - UI should manage a queue/history of jobs
>
> I was thinking that if there where multiple users/jobs, there should
> be some thought or documentation + script on how to manage them.
> “To see all the jobs still running on your cluster, across users and ui
> instances do X”
> “If there is an issue with the jobs you can’t resolve in the UI for that
> user, or you are an admin and want to do something then X"
>
> - Documentation/blueprint for YARN configuration
>
> I agree with what you are saying.  Although, we offer guidance on storm
> tuning, and that is conceptually the same isn’t it?  That is why it comes
> to mind.
> Maybe this can be a follow on, in the tuning guide?
>
> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) wrote:
>
> - Date range limits on queries
>
> Can you describe what you think is needed here? Each Metron user could
> have different volumes of pcap data spread out over different time
> periods. Are you saying we should limit the data range to something either
>
> constant or configurable? Are we sure all users would want this? Am I
> misinterpreting this requirement?
>
> - UI should manage a queue/history of jobs
>
> What should we document here? Reading that bullet point again, it's sort
> of vague and not very description. What I am referring to is a design that
>
> provides users a way to view and manage jobs in the UI. Currently jobs can
>
> only be run 1 at a time and progress is shown with a status bar, so it's
> somewhat interactive.
>
> - Documentation/blueprint for YARN configuration
>
>

Re: [DISCUSS] Pcap query branch completion

[Ryan Merriman]

Thanks for the feedback Otto.  I have created a sub task for documenting
the Job cleanup documentation:
https://issues.apache.org/jira/browse/METRON-1737.  I completely agree with
you there, this needs to be documented.  For the others you marked "Follow
on" I will create follow on tasks in Jira.

I have a few questions about a couple others you commented on

- Date range limits on queries

Can you describe what you think is needed here?  Each Metron user could
have different volumes of pcap data spread out over different time
periods.  Are you saying we should limit the data range to something either
constant or configurable?  Are we sure all users would want this?  Am I
misinterpreting this requirement?

- UI should manage a queue/history of jobs

What should we document here?  Reading that bullet point again, it's sort
of vague and not very description.  What I am referring to is a design that
provides users a way to view and manage jobs in the UI.  Currently jobs can
only be run 1 at a time and progress is shown with a status bar, so it's
somewhat interactive.

- Documentation/blueprint for YARN configuration

We are setup for YARN scheduling in that we offer a configuration setting
to submit a Pcap query to a specified YARN queue (this part is
documented).  Any YARN setup or tuning would be out of scope since
everyone's YARN settings will be different and potentially expand beyond
the Metron use case.  I think a Hadoop admin is likely to have this
knowledge and to have already set up YARN queues.  Do you disagree?



On Mon, Aug 13, 2018 at 8:21 AM, Otto Fowler 
wrote:

> - Job cleanup/TTL
>
> Documented at least, or a helper script to help yourself if you are in a
> situation
>
>
> - Expose the Query filter (vs Fixed) in the UI
>
> Follow on
>
>
> - Date range limits on queries
>
> I don’t see how this won’t be immediately required. I would do this for
> minimum viable.
>
>
> - Pcap query as a separate UI
>
> Follow on
>
>
> - UI should manage a queue/history of jobs
>
> Follow on, but maybe we need documentation
>
>
> - BPF filtering
>
> This is going to be a PITA, follow on
>
>
> - Sharing PCA jobs with other users
>
> Follow on
>
>
> - Provide a way in the UI to populate a pcap query from an alert/metaalert
>
> Follow on
>
>
> - Documentation/blueprint for YARN configuration
>
> Should have
>
>
>

[DISCUSS] Pcap query branch completion

[Ryan Merriman]

We are nearing a fully functional Pcap query feature branch.  I want to
take a moment before we merge to review the original discussion threads and
make sure the community is happy with the state of this feature branch
before we accept it into master.

The original discuss threads are located at:
- Back end architecture thread:
https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E
- UI requirements:
https://lists.apache.org/thread.html/e62e361971092e49446e2012550319f06c8c31944224bcd6326718d9@%3Cdev.metron.apache.org%3E

The JIRA epic can be found here:
https://issues.apache.org/jira/browse/METRON-1554.  The state of each task
should be accurate.  We expect all tasks to be finished within the next
couple days (except for https://issues.apache.org/jira/browse/METRON-1561).

I reviewed the original discuss threads and overall I think we accomplished
a lot.  We were able to create abstractions around managing and submitting
jobs.  We were able to configure the YARN queue for Pcap queries so we are
set up for multi-tenancy in the future.  We have basic guards in place to
keep users from overwhelming the cluster.  We were able to expose results
in the UI and as a binary download.  We have basic authorization in place
that can be expanded later.

We expect the outstanding Jira mentioned above to be converted to a follow
on Jira.  There are several other ideas that were brought up in the discuss
threads but not done in the feature branch.  They do not currently have
Jiras :

- Job cleanup/TTL
- Expose the Query filter (vs Fixed) in the UI
- Date range limits on queries
- Pcap query as a separate UI
- UI should manage a queue/history of jobs
- BPF filtering
- Sharing PCA jobs with other users
- Provide a way in the UI to populate a pcap query from an alert/metaalert
- Documentation/blueprint for YARN configuration

I'm sure I missed some so please chime in with any you want to add.  Which
of these do we still feel should be done?  Are there any features or
changes you feel need to be done before this feature branch is merged?  I
will create the appropriate Jiras as needed.

Ryan

Re: [DISCUSS] Deprecating metron-api

[Ryan Merriman]

Adding user list.  Is anyone out there currently using the metron-api
module to query pcap data?

On Fri, Jun 29, 2018 at 4:35 PM, Casey Stella  wrote:

> I have no objection and would consider it to be a prerequisite to bringing
> in the PR unless there's someone depending on it out there.  You might want
> to cc user@ as well, to get a broader set of input for the "are people
> using it?" question.
>
> On Fri, Jun 29, 2018 at 5:21 PM Ryan Merriman  wrote:
>
> > We are currently working on adding pcap query capabilities to the Alerts
> UI
> > as part of https://issues.apache.org/jira/browse/METRON-1554.  This
> > involves exposing pcap endpoints in our REST application which will make
> > metron-api obsolete.
> >
> > Is anyone currently using this module?  Are there any objections to
> > deprecating it and removing it from our codebase once this feature branch
> > is complete?
> >
>

[DISCUSS] Deprecating metron-api

[Ryan Merriman]

We are currently working on adding pcap query capabilities to the Alerts UI
as part of https://issues.apache.org/jira/browse/METRON-1554.  This
involves exposing pcap endpoints in our REST application which will make
metron-api obsolete.

Is anyone currently using this module?  Are there any objections to
deprecating it and removing it from our codebase once this feature branch
is complete?

Re: [DISCUSS] Field conversions

[Ryan Merriman]

I agree completely.  I will leave this thread open for a day or two to give
others a chance to weigh in.  If no one opposes, I will creates Jiras for
removing field transformations and transforming existing data.

On Tue, Jun 5, 2018 at 8:21 AM, Casey Stella  wrote:

> Well, on write it is a transformation, on read it's a translation.  This is
> to say that you're providing a mapping on read to translate field names
> given the index you're using.  The other approach that I was considering
> last night is a field transformation REST call which translates field names
> that the UI could call.  So, the UI would pass 'source.type' to the field
> translation service and in Solr it'd return source.type and in ES it'd
> return source:type.  Underneath the hood the service would use the same
> transformation as the writer uses.  That's another way to skin this cat.
>
> Ultimately, I think we should just ditch this field transformation
> business, as Laurens said, as long as we have a utility to transform
> existing data.
>
> On Tue, Jun 5, 2018 at 8:54 AM Ryan Merriman  wrote:
>
> > Having 2 different patterns for configuring field name transformations on
> > read vs write is confusing to me.  I agree with both of you that
> > normalizing on '.' and not having to do the translation at all would be
> > ideal.  Like you both suggested, we would need some utility or script to
> > convert preexisting data to match this format.  There could also be some
> > adjustments a user would need to make in the UI but I feel like we could
> > document around that.  Are there any objections to doing it this way?
> >
> >
> >
> > On Mon, Jun 4, 2018 at 4:30 PM, Laurens Vets  wrote:
> >
> > > ES 2.x support officially ended 4 months ago (
> > > https://www.elastic.co/support/eol), so why still support ':' at all?
> :)
> > > Additionally, 2.x isn't even supported at all on the last 2 Ubuntu LTS
> > > releases (16.04 & 18.05).
> > >
> > > Therefor, move everything to use '.' and provide a conversion/upgrade
> > > script to change '.' to ':'?
> > >
> > >
> > > On 2018-06-04 13:55, Ryan Merriman wrote:
> > >
> > >> We've been dealing with a reoccurring challenge in Metron.  It is
> common
> > >> for various fields to contain '.' characters for the purpose of making
> > >> them
> > >> more readable, namespacing, etc.  At one point we only supported
> > >> Elasticsearch 2.3 which did not allow dots and forced us to use ':'
> > >> instead.  This limitation does not exist in later versions of
> > >> Elasticsearch
> > >> or Solr.
> > >>
> > >> Now we're in a situation where we need to allow a user to use either
> one
> > >> because they may still be using ES 2.3 or have data with ':'
> characters
> > in
> > >> field names.  We've attempted to make this configurable in a couple
> > >> different PRs:
> > >>
> > >> https://github.com/apache/metron/pull/1022
> > >> https://github.com/apache/metron/pull/1010
> > >> https://github.com/apache/metron/pull/1038
> > >>
> > >> The approaches taken in these are not consistent and fall short in
> > >> different ways.  The first (METRON-1569 Allow user to change field
> name
> > >> conversion when indexing) only applies to indexing and not querying.
> > The
> > >> others only apply to a single field which does not scale well.  Now we
> > >> have
> > >> an issue with another field in
> > >> https://issues.apache.org/jira/browse/METRON-1600.  Rather than
> > >> continuing
> > >> with a patchwork of different fixes I want to attempt to design a
> > >> system-wide solution.
> > >>
> > >> My first thought is to expand
> > https://github.com/apache/metron/pull/1022
> > >> to
> > >> apply globally.  However this is not trivial and would require
> > significant
> > >> changes.  It would also make https://github.com/apache/
> metron/pull/1010
> > >> obsolete and we might end up having to revert all of it.
> > >>
> > >> Does anyone have any ideas or opinions?  I am still researching
> > solutions
> > >> but would love some guidance from the community.
> > >>
> > >
> >
>

Re: [DISCUSS] Field conversions

[Ryan Merriman]

Having 2 different patterns for configuring field name transformations on
read vs write is confusing to me.  I agree with both of you that
normalizing on '.' and not having to do the translation at all would be
ideal.  Like you both suggested, we would need some utility or script to
convert preexisting data to match this format.  There could also be some
adjustments a user would need to make in the UI but I feel like we could
document around that.  Are there any objections to doing it this way?



On Mon, Jun 4, 2018 at 4:30 PM, Laurens Vets  wrote:

> ES 2.x support officially ended 4 months ago (
> https://www.elastic.co/support/eol), so why still support ':' at all? :)
> Additionally, 2.x isn't even supported at all on the last 2 Ubuntu LTS
> releases (16.04 & 18.05).
>
> Therefor, move everything to use '.' and provide a conversion/upgrade
> script to change '.' to ':'?
>
>
> On 2018-06-04 13:55, Ryan Merriman wrote:
>
>> We've been dealing with a reoccurring challenge in Metron.  It is common
>> for various fields to contain '.' characters for the purpose of making
>> them
>> more readable, namespacing, etc.  At one point we only supported
>> Elasticsearch 2.3 which did not allow dots and forced us to use ':'
>> instead.  This limitation does not exist in later versions of
>> Elasticsearch
>> or Solr.
>>
>> Now we're in a situation where we need to allow a user to use either one
>> because they may still be using ES 2.3 or have data with ':' characters in
>> field names.  We've attempted to make this configurable in a couple
>> different PRs:
>>
>> https://github.com/apache/metron/pull/1022
>> https://github.com/apache/metron/pull/1010
>> https://github.com/apache/metron/pull/1038
>>
>> The approaches taken in these are not consistent and fall short in
>> different ways.  The first (METRON-1569 Allow user to change field name
>> conversion when indexing) only applies to indexing and not querying.  The
>> others only apply to a single field which does not scale well.  Now we
>> have
>> an issue with another field in
>> https://issues.apache.org/jira/browse/METRON-1600.  Rather than
>> continuing
>> with a patchwork of different fixes I want to attempt to design a
>> system-wide solution.
>>
>> My first thought is to expand https://github.com/apache/metron/pull/1022
>> to
>> apply globally.  However this is not trivial and would require significant
>> changes.  It would also make https://github.com/apache/metron/pull/1010
>> obsolete and we might end up having to revert all of it.
>>
>> Does anyone have any ideas or opinions?  I am still researching solutions
>> but would love some guidance from the community.
>>
>

[DISCUSS] Field conversions

[Ryan Merriman]

We've been dealing with a reoccurring challenge in Metron.  It is common
for various fields to contain '.' characters for the purpose of making them
more readable, namespacing, etc.  At one point we only supported
Elasticsearch 2.3 which did not allow dots and forced us to use ':'
instead.  This limitation does not exist in later versions of Elasticsearch
or Solr.

Now we're in a situation where we need to allow a user to use either one
because they may still be using ES 2.3 or have data with ':' characters in
field names.  We've attempted to make this configurable in a couple
different PRs:

https://github.com/apache/metron/pull/1022
https://github.com/apache/metron/pull/1010
https://github.com/apache/metron/pull/1038

The approaches taken in these are not consistent and fall short in
different ways.  The first (METRON-1569 Allow user to change field name
conversion when indexing) only applies to indexing and not querying.  The
others only apply to a single field which does not scale well.  Now we have
an issue with another field in
https://issues.apache.org/jira/browse/METRON-1600.  Rather than continuing
with a patchwork of different fixes I want to attempt to design a
system-wide solution.

My first thought is to expand https://github.com/apache/metron/pull/1022 to
apply globally.  However this is not trivial and would require significant
changes.  It would also make https://github.com/apache/metron/pull/1010
obsolete and we might end up having to revert all of it.

Does anyone have any ideas or opinions?  I am still researching solutions
but would love some guidance from the community.

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Yes there will be an admin role that can read and delete all.

On Fri, May 11, 2018 at 4:11 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Do we at least require a admin/super user?  See’s all the queues and jobs?
>
>
> On May 11, 2018 at 17:03:34, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Thanks everyone for the input and feedback. I will attempt to summarize so
> we can come to a consensus and get this tasked out.
>
> The following endpoints will be included:
>
> - GET /api/v1/pcap/metadata?basePath - This endpoint will return
> metadata of pcap data stored in HDFS. This would include pcap size, date
> ranges (how far back can I go), etc. It would accept an optional HDFS
> basePath parameter for cases where pcap data is stored in multiple places
> and/or different from the default location.
> - POST /api/v1/pcap/fixed - This endpoint would accept a fixed pcap
> request, submit a pcap job, and return a job id. The request would be an
> object containing the options documented here for the fixed filter:
> https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-
> backend#query-filter-utility
> <https://github.com/apache/metron/tree/master/metron-
> platform/metron-pcap-backend#query-filter-utility>.
> A job will be associated with a user that submits it. An exception will be
> returned for violating constraints like too many queries submitted, query
> parameters out of limits, etc. A record of the user and job id will be
> persisted to a data store so a list of a user's jobs can later be
> retrieved.
> - POST /api/v1/pcap/query - This endpoint would accept a query pcap
> request, submit a pcap job, and return a job id. The request would be
> an object containing the options documented here for the query filter:
> https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-
> backend#query-filter-utility
> <https://github.com/apache/metron/tree/master/metron-
> platform/metron-pcap-backend#query-filter-utility>.
> A job will be associated with a user that submits it. An exception will be
> returned for violating constraints like too many queries submitted, query
> parameters out of limits, etc. A record of the user and job id will be
> persisted to a data store so a list of a user's jobs can later be
> retrieved.
> - GET /api/v1/pcap/status/ - This endpoint will return the YARN
> status of a running/completed job.
> - GET /api/v1/pcap/stop/ - This endpoint would kill a running
> pcap job. If the job has already completed this is a noop.
> - GET /api/v1/pcap/list - This endpoint will list a user's submitted
> pcap queries. Items in the list would contain job id, status (is it
> finished?), start/end time, and number of pages.
> - GET /api/v1/pcap/pdml// - This endpoint will return
> pcap results for the given page in pdml format (
> https://wiki.wireshark.org/PDML <https://wiki.wireshark.org/PDML>). Are
> there other formats we want to support?
> - GET /api/v1/pcap/raw// - This endpoint will allow a
> user to download raw pcap results for the given page.
> - DELETE /api/v1/pcap/ - This endpoint will delete pcap query
> results.
>
> With respect to security, users will only be able to see their list of
> jobs
> and query results. We will also include an admin role that will be able to
> read and delete all. Jobs will be submitted as the metron service user and
> user to job relationships will be managed and persisted by the REST
> application.
>
> This is a substantial feature and compromises should be made to get an
> initial version out (baby steps). Here are some areas we will compromise
> on and enhance in the future:
>
> - Security - Initially we will rely on Spring Security for authorization
> and authentication. Eventually this feature will fit into a broader
> security strategy. This could mean an authentication strategy that is
> consistent with the rest of Metron, integration with Ranger for
> authorization, and submitting jobs as individual users rather than a
> service user. We can also explore sharing access between users and more
> fine-grained ACL-based security.
> - Priority and scheduling - Eventually we should form a strategy for
> prioritizing jobs, imposing limits, etc with YARN scheduling. Submitting
> jobs with individual users will give us even more flexibility in this
> area.
> - Job cleanup - Initially cleanup will be a manual process either
> through exposed endpoints. Later we can explore automated cleanup
> strategies and introduce data retention policies.
> - Filter options - Initially we will expose the 2 filter options that
> currently exist in Metron: fixed and pcap. Eventually we can add more
> filters like bpf.
> - Data directory - This could be a TOC of differ

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

I will task it out in Jira and we can get started.


On Fri, May 11, 2018 at 11:52 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> On the sharing and securing points, it seems like having the result of a
> query run hang around in HDFS on a given URL would work, we can then use
> Ranger policies (RBAC, or even ABAC later) to control access, this also
> solves the page storage problem, and gives us a kind of two step approach,
> both of which could (maybe, possibly, but probably not) be large enough to
> need distribution, i.e. the initial search everything and the subsequent
> sort / page through the results. Does anyone imaging sorting? Maybe
> sub-filtering, but PCAP is heavily time based, so timestamp sort ok?
>
> I also suspect it's worth considering the lifecycle of our stored result
> sets as being meta-data driven. If I'm doing a speculative search I don't
> really care if an admin cleans that up after at the end of day / week /
> disk space nervousness limit. However, if I find something good, I might
> want to mark the result set as immune from automatic deletion.
>
> The other issue I would raise, which has implications for our PCAP capture,
> and also impacts Otto's suggestion of 'self-uploaded' PCAPs is how we
> namespace PCAP collection and retrieval. The problem here is that I might
> have PCAPs from multiple locations which have conflicting private IP
> ranges, so I can't logically dump them all in the same repository. Solving
> the collection end of that is probably a separate unit of effort, but this
> retrieval architecture should support multiple file system locations.
>
> If we wanted to get fancy about it, we should look at using the stored
> result sets as a kind of cache, for other queries, as people refine and
> narrow down queries, it may make sense to be more sophisticated about where
> our query jobs pull from (i.e. filter the subset from a previous resultset,
> rather than scanning petabytes of source data). This may imply some kind of
> TOC for the cache. The underlying immutability of the PCAP store should
> make this fairly tractable.
>
> FYI, I've been doing a lot of thinking around data security, API and
> configuration security and auditing recently, but I suspect that is a
> different discuss thread. I'll kick something off shortly with a few
> thoughts.
>
> I see a lot of this as long term goals to be honest, so as Jon says, we can
> definitely take a few baby steps to start.
>
> Simon
>
> On 11 May 2018 at 15:40, Otto Fowler <ottobackwa...@gmail.com> wrote:
>
> > Don’t lose the use case for manually uploading PCAPS for analysis Jon.
> >
> >
> > On May 11, 2018 at 10:14:02, zeo...@gmail.com (zeo...@gmail.com) wrote:
> >
> > I think baby steps are fine - admin gets access to all, otherwise you
> only
> > see your own pcaps, but we file a jira for a future add of API security,
> > which more mature SOCs that align with the Metron personas will need.
> >
> > Jon
> >
> > On Fri, May 11, 2018, 09:27 Ryan Merriman <merrim...@gmail.com> wrote:
> >
> > > That's a good point Jon. There are different levels of effort
> associated
> > > with different options. If we want to allow pcaps to be shared with
> > > specific users, we will need to introduce ACL security in our REST
> > > application using something like the ACL capability that comes with
> > Spring
> > > Security or Ranger. This would be more complex to design and implement.
> > > If we want something more broad like admin roles that can see all or
> > > allowing pcap files to become public, this would be less work. Do you
> > > think ACL security is required or would the other options be
> acceptable?
> > >
> > > On Thu, May 10, 2018 at 2:47 PM, zeo...@gmail.com <zeo...@gmail.com>
> > > wrote:
> > >
> > > > At the very least there needs to be the ability to share downloaded
> > PCAPs
> > > > with other users and/or have roles that can see all pcaps. A platform
> > > > engineer may want to clean up old pcaps after x time, or a manger may
> > ask
> > > > an analyst to find all of the traffic that exhibits xyz behavior,
> dump
> > a
> > > > pcap, and then point him to it so the manager can review. Since the
> > > > pcap may be huge, we wouldn't want to try to push people to sending
> it
> > > via
> > > > email, uploading to a file server, finding an external hard drive,
> etc.
> > > >
> > > > Jon
> > > >
> > > > On Thu, May 10, 2018 at 10:16 AM Ryan Merriman <merrim...@gmail.com>
> >

Re: [DISCUSS] Release Manager

[Ryan Merriman]

Yes +1 to Justin being RM.  Thank you for taking that on.

On Thu, May 10, 2018 at 11:08 AM, Casey Stella <ceste...@gmail.com> wrote:

> I'm +1 to Justin being RM; he's going to have big shoes to fill with Matt
> gone. ;) Also, if it wasn't obvious, deep and hearty thanks to Matt again
> for being our RM.
>
> On Thu, May 10, 2018 at 12:06 PM Ryan Merriman <merrim...@gmail.com>
> wrote:
>
> > Thanks for all your help Matt.
> >
> > On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Thanks Matt for doing this for the community.
> > >
> > > Justin Leet as new lord commander of the Night's Watch? Aye, dilly,
> > dilly.
> > >
> > > On Thu, May 10, 2018 at 9:07 AM, Justin Leet <justinjl...@gmail.com>
> > > wrote:
> > >
> > > > I'd be happy to to volunteer to take over for a while.
> > > >
> > > > Thanks to Matt for all the help through the last couple releases!
> > > >
> > > > Justin
> > > >
> > > > On Thu, May 10, 2018 at 11:06 AM, Casey Stella <ceste...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Matt Foley, our esteemed Release manager for the last couple
> > releases,
> > > > has
> > > > > asked to be relieved.  So, I'm calling on volunteers for the next
> > > release
> > > > > manager.  It should be a committer and there are a few things that
> > > > require
> > > > > a PMC member, I believe, but the release manager can ask for help
> > from
> > > a
> > > > > PMC member.
> > > > >
> > > > > So, Matt's watch has ended, who wants to volunteer?
> > > > >
> > > > > Casey
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Release Manager

[Ryan Merriman]

Thanks for all your help Matt.

On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Thanks Matt for doing this for the community.
>
> Justin Leet as new lord commander of the Night's Watch? Aye, dilly, dilly.
>
> On Thu, May 10, 2018 at 9:07 AM, Justin Leet 
> wrote:
>
> > I'd be happy to to volunteer to take over for a while.
> >
> > Thanks to Matt for all the help through the last couple releases!
> >
> > Justin
> >
> > On Thu, May 10, 2018 at 11:06 AM, Casey Stella 
> wrote:
> >
> > > Hi All,
> > >
> > > Matt Foley, our esteemed Release manager for the last couple releases,
> > has
> > > asked to be relieved.  So, I'm calling on volunteers for the next
> release
> > > manager.  It should be a committer and there are a few things that
> > require
> > > a PMC member, I believe, but the release manager can ask for help from
> a
> > > PMC member.
> > >
> > > So, Matt's watch has ended, who wants to volunteer?
> > >
> > > Casey
> > >
> >
>

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the
fixed query option which we have covered.  I agree with you that
deprecating the metron-api module should be a goal of this feature.

On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> This looks like a pretty good start Ryan. Does the metadata endpoint cover
> this https://github.com/apache/metron/tree/master/
> metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint
> from the original metron-api? If so, then we would be able to deprecate the
> existing metron-api project. If we later go to micro-services, a pcap
> module would spin back into the fold, but it would probably look different
> from metron-api.
>
> I commented on the UI thread, but to reiterate for the purpose of backend
> functionality here I don't believe there is a way to "PAUSE" or "SUSPEND"
> jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient for
> the job management operations.
>
> On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman <merrim...@gmail.com>
> wrote:
>
> > Now that we are confident we can run submit a MR job from our current
> REST
> > application, is this the desired approach?  Just want to confirm.
> >
> > Next I think we should map out what the REST interface will look like.
> > Here are the endpoints I'm thinking about:
> >
> > GET /api/v1/pcap/metadata?basePath
> >
> > This endpoint will return metadata of pcap data stored in HDFS.  This
> would
> > include pcap size, date ranges (how far back can I go), etc.  It would
> > accept an optional HDFS basePath parameter for cases where pcap data is
> > stored in multiple places and/or different from the default location.
> >
> > POST /api/v1/pcap/query
> >
> > This endpoint would accept a pcap request, submit a pcap query job, and
> > return a job id.  The request would be an object containing the
> parameters
> > documented here:  https://github.com/apache/metron/tree/master/
> > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> > would be associated with a user that submits it.  An exception will be
> > returned for violating constraints like too many queries submitted, query
> > parameters out of limits, etc.
> >
> > GET /api/v1/pcap/status/
> >
> > This endpoint will return the status of a running job.  I imagine this is
> > just a proxy to the YARN REST api.  We can discuss the implementation
> > behind these endpoints later.
> >
> > GET /api/v1/pcap/stop/
> >
> > This endpoint would kill a running pcap job.  If the job has already
> > completed this is a noop.
> >
> > GET /api/v1/pcap/list
> >
> > This endpoint will list a user's submitted pcap queries.  Items in the
> list
> > would contain job id, status (is it finished?), start/end time, and
> number
> > of pages.  Maybe there is some overlap with the status endpoint above and
> > the status endpoint is not needed?
> >
> > GET /api/v1/pcap/pdml//
> >
> > This endpoint will return pcap results for the given page in pdml format
> (
> > https://wiki.wireshark.org/PDML).  Are there other formats we want to
> > support?
> >
> > GET /api/v1/pcap/raw//
> >
> > This endpoint will allow a user to download raw pcap results for the
> given
> > page.
> >
> > DELETE /api/v1/pcap/
> >
> > This endpoint will delete pcap query results.  Not sure yet how this fits
> > in with our broader cleanup strategy.
> >
> > This should get us started.  What did I miss and what would you change
> > about these?  I did not include much detail related to security, cleanup
> > strategy, or underlying implementation details but these are items we
> > should discuss at some point.
> >
> > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Sweet! That's great news. The pom changes are a lot simpler than I
> > > expected. Very nice.
> > >
> > > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman <merrim...@gmail.com>
> > wrote:
> > >
> > > > Finally figured it out.  Commit is here:
> > > > https://github.com/merrimanr/incubator-metron/commit/
> > > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > > >
> > > > It came down to figuring out the right combination of maven
> > dependencies
> > > > and passing in the HDP version to REST as a Java system property.  I
> > also
> > > > included some HDFS setup tasks.

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Security is another important topic related to our pcap architecture.  This
may spill over into a more general, system-wide discussion and we can start
a separate thread for that if necessary.

I'm assuming we want to manage pcap queries by user.  One important
question is which user do we use to submit MR jobs?  Right now they are
submitted with the "metron" service user.  If we continue with this
approach (all jobs run as the metron service user) it will require less
Kerberos work and configuration but we will need to manage user to job
relationships in the REST layer.  It would also give us less flexibility
since all assets are permissioned for a single user.  If we decided to have
REST impersonate users and submit jobs that way there would be substantial
work (maybe?) to get to that point since we don't do it now.  We would need
to add LDAP authentication to REST, sync OS users with LDAP, and all the
other stuff that goes along with setting that up.  Maybe we want to do this
anyways in the future.  Has anyone done this before and has a clear
understanding of what's involved?  If this were the ideal approach we need
to think about how we get to that point.  Managing user to job
relationships in REST would be throwaway in that case since that
information would now be stored in YARN.

For authorization I'm assuming that each user should only have access to
information about their queries and query results.  Any actions like
downloading results or cleaning up queries (deleting results) would also be
limited to that user.  Does this sound reasonable?  Do we want to add an
admin role that can do everything?  Is there anything anyone else wants to
discuss with regards to authorization or security in general?

On Wed, May 9, 2018 at 1:22 PM, Ryan Merriman <merrim...@gmail.com> wrote:

> Thanks for the feedback Jon.  I'm am not as familiar with BPF filtering as
> you probably are.  Do you have an idea of much effort would be involved in
> implementing this?  I suspect this would be another PcapFilter (
> https://github.com/apache/metron/blob/master/metron-
> platform/metron-pcap/src/main/java/org/apache/metron/pcap/
> filter/PcapFilter.java) similar to fixed and query so regardless of when
> we decide to add it our endpoint strategy should support 1 to n filters.
> Maybe we have an endpoint for each type of filter:
>
> POST /api/v1/pcap/fixed
> POST /api/v1/pcap/query
> POST /api/v1/pcap/bpf
>
> This would allow us to accept requests that are structured differently and
> specific to the type of filter.
>
> On Wed, May 9, 2018 at 12:32 PM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
>> This looks really great and gets me excited to maybe revisit some old
>> conversations about PCAP capture in Metron.  The only thing that I think
>> it's missing is the ability to filter using bpf.  I think the same thing
>> can technically be accomplished by using packet_filter and I wouldn't
>> throw
>> a fit if that's considered a follow-on, but bpf is the standard language
>> that people who do packet munging for a living know.
>>
>> Jon
>>
>> On Wed, May 9, 2018 at 1:00 PM Ryan Merriman <merrim...@gmail.com> wrote:
>>
>> > Now that we are confident we can run submit a MR job from our current
>> REST
>> > application, is this the desired approach?  Just want to confirm.
>> >
>> > Next I think we should map out what the REST interface will look like.
>> > Here are the endpoints I'm thinking about:
>> >
>> > GET /api/v1/pcap/metadata?basePath
>> >
>> > This endpoint will return metadata of pcap data stored in HDFS.  This
>> would
>> > include pcap size, date ranges (how far back can I go), etc.  It would
>> > accept an optional HDFS basePath parameter for cases where pcap data is
>> > stored in multiple places and/or different from the default location.
>> >
>> > POST /api/v1/pcap/query
>> >
>> > This endpoint would accept a pcap request, submit a pcap query job, and
>> > return a job id.  The request would be an object containing the
>> parameters
>> > documented here:  https://github.com/apache/metron/tree/master/
>> > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
>> > would be associated with a user that submits it.  An exception will be
>> > returned for violating constraints like too many queries submitted,
>> query
>> > parameters out of limits, etc.
>> >
>> > GET /api/v1/pcap/status/
>> >
>> > This endpoint will return the status of a running job.  I imagine this
>> is
>> > just a proxy to the YARN REST api.  We can discuss the implementation
>> > behind these

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Thanks for the feedback Jon.  I'm am not as familiar with BPF filtering as
you probably are.  Do you have an idea of much effort would be involved in
implementing this?  I suspect this would be another PcapFilter (
https://github.com/apache/metron/blob/master/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilter.java)
similar to fixed and query so regardless of when we decide to add it our
endpoint strategy should support 1 to n filters.  Maybe we have an endpoint
for each type of filter:

POST /api/v1/pcap/fixed
POST /api/v1/pcap/query
POST /api/v1/pcap/bpf

This would allow us to accept requests that are structured differently and
specific to the type of filter.

On Wed, May 9, 2018 at 12:32 PM, zeo...@gmail.com <zeo...@gmail.com> wrote:

> This looks really great and gets me excited to maybe revisit some old
> conversations about PCAP capture in Metron.  The only thing that I think
> it's missing is the ability to filter using bpf.  I think the same thing
> can technically be accomplished by using packet_filter and I wouldn't throw
> a fit if that's considered a follow-on, but bpf is the standard language
> that people who do packet munging for a living know.
>
> Jon
>
> On Wed, May 9, 2018 at 1:00 PM Ryan Merriman <merrim...@gmail.com> wrote:
>
> > Now that we are confident we can run submit a MR job from our current
> REST
> > application, is this the desired approach?  Just want to confirm.
> >
> > Next I think we should map out what the REST interface will look like.
> > Here are the endpoints I'm thinking about:
> >
> > GET /api/v1/pcap/metadata?basePath
> >
> > This endpoint will return metadata of pcap data stored in HDFS.  This
> would
> > include pcap size, date ranges (how far back can I go), etc.  It would
> > accept an optional HDFS basePath parameter for cases where pcap data is
> > stored in multiple places and/or different from the default location.
> >
> > POST /api/v1/pcap/query
> >
> > This endpoint would accept a pcap request, submit a pcap query job, and
> > return a job id.  The request would be an object containing the
> parameters
> > documented here:  https://github.com/apache/metron/tree/master/
> > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> > would be associated with a user that submits it.  An exception will be
> > returned for violating constraints like too many queries submitted, query
> > parameters out of limits, etc.
> >
> > GET /api/v1/pcap/status/
> >
> > This endpoint will return the status of a running job.  I imagine this is
> > just a proxy to the YARN REST api.  We can discuss the implementation
> > behind these endpoints later.
> >
> > GET /api/v1/pcap/stop/
> >
> > This endpoint would kill a running pcap job.  If the job has already
> > completed this is a noop.
> >
> > GET /api/v1/pcap/list
> >
> > This endpoint will list a user's submitted pcap queries.  Items in the
> list
> > would contain job id, status (is it finished?), start/end time, and
> number
> > of pages.  Maybe there is some overlap with the status endpoint above and
> > the status endpoint is not needed?
> >
> > GET /api/v1/pcap/pdml//
> >
> > This endpoint will return pcap results for the given page in pdml format
> (
> > https://wiki.wireshark.org/PDML).  Are there other formats we want to
> > support?
> >
> > GET /api/v1/pcap/raw//
> >
> > This endpoint will allow a user to download raw pcap results for the
> given
> > page.
> >
> > DELETE /api/v1/pcap/
> >
> > This endpoint will delete pcap query results.  Not sure yet how this fits
> > in with our broader cleanup strategy.
> >
> > This should get us started.  What did I miss and what would you change
> > about these?  I did not include much detail related to security, cleanup
> > strategy, or underlying implementation details but these are items we
> > should discuss at some point.
> >
> > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Sweet! That's great news. The pom changes are a lot simpler than I
> > > expected. Very nice.
> > >
> > > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman <merrim...@gmail.com>
> > wrote:
> > >
> > > > Finally figured it out.  Commit is here:
> > > > https://github.com/merrimanr/incubator-metron/commit/
> > > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > > >
> > > > It came down to figuring out the right combination of maven
&g

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Now that we are confident we can run submit a MR job from our current REST
application, is this the desired approach?  Just want to confirm.

Next I think we should map out what the REST interface will look like.
Here are the endpoints I'm thinking about:

GET /api/v1/pcap/metadata?basePath

This endpoint will return metadata of pcap data stored in HDFS.  This would
include pcap size, date ranges (how far back can I go), etc.  It would
accept an optional HDFS basePath parameter for cases where pcap data is
stored in multiple places and/or different from the default location.

POST /api/v1/pcap/query

This endpoint would accept a pcap request, submit a pcap query job, and
return a job id.  The request would be an object containing the parameters
documented here:  https://github.com/apache/metron/tree/master/
metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
would be associated with a user that submits it.  An exception will be
returned for violating constraints like too many queries submitted, query
parameters out of limits, etc.

GET /api/v1/pcap/status/

This endpoint will return the status of a running job.  I imagine this is
just a proxy to the YARN REST api.  We can discuss the implementation
behind these endpoints later.

GET /api/v1/pcap/stop/

This endpoint would kill a running pcap job.  If the job has already
completed this is a noop.

GET /api/v1/pcap/list

This endpoint will list a user's submitted pcap queries.  Items in the list
would contain job id, status (is it finished?), start/end time, and number
of pages.  Maybe there is some overlap with the status endpoint above and
the status endpoint is not needed?

GET /api/v1/pcap/pdml//

This endpoint will return pcap results for the given page in pdml format (
https://wiki.wireshark.org/PDML).  Are there other formats we want to
support?

GET /api/v1/pcap/raw//

This endpoint will allow a user to download raw pcap results for the given
page.

DELETE /api/v1/pcap/

This endpoint will delete pcap query results.  Not sure yet how this fits
in with our broader cleanup strategy.

This should get us started.  What did I miss and what would you change
about these?  I did not include much detail related to security, cleanup
strategy, or underlying implementation details but these are items we
should discuss at some point.

On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Sweet! That's great news. The pom changes are a lot simpler than I
> expected. Very nice.
>
> On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman <merrim...@gmail.com> wrote:
>
> > Finally figured it out.  Commit is here:
> > https://github.com/merrimanr/incubator-metron/commit/
> > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> >
> > It came down to figuring out the right combination of maven dependencies
> > and passing in the HDP version to REST as a Java system property.  I also
> > included some HDFS setup tasks.  I tested this in full dev and can now
> > successfully run a pcap query and get results.  All you should have to do
> > is generate some pcap data first.
> >
> > On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > @Ryan - pulled your branch and experimented with a few things. In doing
> > so,
> > > it dawned on me that by adding the yarn and hadoop classpath, you
> > probably
> > > didn't introduce a new classpath issue, rather you probably just moved
> > onto
> > > the next classpath issue, ie hbase per your exception about hbase jaxb.
> > > Anyhow, I put up a branch with some pom changes worth trying in
> > conjunction
> > > with invoking the rest app startup via "/usr/bin/yarn jar"
> > >
> > > https://github.com/mmiklavc/metron/tree/ryan-rest-test
> > >
> > > https://github.com/mmiklavc/metron/commit/
> 5ca23580fc6e043fafae2327c80b65
> > > b20ca1c0c9
> > >
> > > Mike
> > >
> > >
> > > On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
> > > si...@simonellistonball.com> wrote:
> > >
> > > > That would be a step closer to something more like a micro-service
> > > > architecture. However, I would want to make sure we think about the
> > > > operational complexity, and mpack implications of having another
> server
> > > > installed and running somewhere on the cluster (also, ssl, kerberos,
> > etc
> > > > etc requirements for that service).
> > > >
> > > > On 8 May 2018 at 14:27, Ryan Merriman <merrim...@gmail.com> wrote:
> > > >
> > > > > +1 to having metron-api as it's own service and using a gateway
> type
> > > > &g

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Finally figured it out.  Commit is here:
https://github.com/merrimanr/incubator-metron/commit/22fe5e9ff3c167b42ebeb7a9f1000753a409aff1

It came down to figuring out the right combination of maven dependencies
and passing in the HDP version to REST as a Java system property.  I also
included some HDFS setup tasks.  I tested this in full dev and can now
successfully run a pcap query and get results.  All you should have to do
is generate some pcap data first.

On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> @Ryan - pulled your branch and experimented with a few things. In doing so,
> it dawned on me that by adding the yarn and hadoop classpath, you probably
> didn't introduce a new classpath issue, rather you probably just moved onto
> the next classpath issue, ie hbase per your exception about hbase jaxb.
> Anyhow, I put up a branch with some pom changes worth trying in conjunction
> with invoking the rest app startup via "/usr/bin/yarn jar"
>
> https://github.com/mmiklavc/metron/tree/ryan-rest-test
>
> https://github.com/mmiklavc/metron/commit/5ca23580fc6e043fafae2327c80b65
> b20ca1c0c9
>
> Mike
>
>
> On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > That would be a step closer to something more like a micro-service
> > architecture. However, I would want to make sure we think about the
> > operational complexity, and mpack implications of having another server
> > installed and running somewhere on the cluster (also, ssl, kerberos, etc
> > etc requirements for that service).
> >
> > On 8 May 2018 at 14:27, Ryan Merriman <merrim...@gmail.com> wrote:
> >
> > > +1 to having metron-api as it's own service and using a gateway type
> > > pattern.
> > >
> > > On Tue, May 8, 2018 at 8:13 AM, Otto Fowler <ottobackwa...@gmail.com>
> > > wrote:
> > >
> > > > Why not have metron-api as it’s own service and use a ‘gateway’ type
> > > > pattern in rest?
> > > >
> > > >
> > > > On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com)
> wrote:
> > > >
> > > > Moving the yarn classpath command earlier in the classpath now gives
> > this
> > > > error:
> > > >
> > > > Caused by: java.lang.NoSuchMethodError:
> > > > javax.servlet.ServletContext.getVirtualServerName()Ljava/
> lang/String;
> > > >
> > > > I will experiment with other combinations, I suspect we will need
> > > > finer-grain control over the order.
> > > >
> > > > The grep matches class names inside jar files. I use this all the
> time
> > > and
> > > > it's really useful.
> > > >
> > > > The metron-rest jar is already shaded.
> > > >
> > > > Reverse engineering the yarn jar command was the next thing I was
> going
> > > to
> > > > try. Will let you know how it goes.
> > > >
> > > > On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > What order did you add the hadoop or yarn classpath? The "shaded"
> > > > package
> > > > > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > > > > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try
> > adding
> > > > > those packages earlier on the classpath.
> > > > >
> > > > > I think that find command needs a "jar tvf", otherwise you're
> looking
> > > > for a
> > > > > class name in jar file names.
> > > > >
> > > > > Have you tried shading the rest jar?
> > > > >
> > > > > I'd also look at the classpath you get when running "yarn jar" to
> > start
> > > > the
> > > > > existing pcap service, per the instructions in
> metron-api/README.md.
> > > > >
> > > > >
> > > > > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman <merrim...@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > To explore the idea of merging metron-api into metron-rest and
> > > running
> > > > > pcap
> > > > > > queries inside our REST application, I created a simple test
> here:
> > > > > > https://github.com/merrimanr/incubator-metron/tree/pcap-
> rest-test.
> > A
> > > > > > summar

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

+1 to having metron-api as it's own service and using a gateway type
pattern.

On Tue, May 8, 2018 at 8:13 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> Why not have metron-api as it’s own service and use a ‘gateway’ type
> pattern in rest?
>
>
> On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Moving the yarn classpath command earlier in the classpath now gives this
> error:
>
> Caused by: java.lang.NoSuchMethodError:
> javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
>
> I will experiment with other combinations, I suspect we will need
> finer-grain control over the order.
>
> The grep matches class names inside jar files. I use this all the time and
> it's really useful.
>
> The metron-rest jar is already shaded.
>
> Reverse engineering the yarn jar command was the next thing I was going to
> try. Will let you know how it goes.
>
> On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > What order did you add the hadoop or yarn classpath? The "shaded"
> package
> > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
> > those packages earlier on the classpath.
> >
> > I think that find command needs a "jar tvf", otherwise you're looking
> for a
> > class name in jar file names.
> >
> > Have you tried shading the rest jar?
> >
> > I'd also look at the classpath you get when running "yarn jar" to start
> the
> > existing pcap service, per the instructions in metron-api/README.md.
> >
> >
> > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman <merrim...@gmail.com>
> wrote:
> >
> > > To explore the idea of merging metron-api into metron-rest and running
> > pcap
> > > queries inside our REST application, I created a simple test here:
> > > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A
> > > summary of what's included:
> > >
> > > - Added pcap as a dependency in the metron-rest pom.xml
> > > - Added a pcap query controller endpoint at
> > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > queryUsingGET
> > > - Added a pcap query service that runs a simple, hardcoded query
> > >
> > > Generate some pcap data using pycapa (
> > > https://github.com/apache/metron/tree/master/metron-sensors/pycapa)
> and
> > > the
> > > pcap topology (
> > > https://github.com/apache/metron/tree/master/metron-
> > > platform/metron-pcap-backend#starting-the-topology).
> > > After this initial setup there should be data in HDFS at
> > > "/apps/metron/pcap". I believe this should be enough to exercise the
> > > issue. Just hit the endpoint referenced above. I tested this in an
> > > already running full dev by building and deploying the metron-rest
> jar.
> > I
> > > did not rebuild full dev with this change but I would still expect it
> to
> > > work. Let me know if it doesn't.
> > >
> > > The first error I see when I hit this endpoint is:
> > >
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> > >
> > > Here are the things I've tried so far:
> > >
> > > - Run the REST application with the YARN jar command since this is how
> > > all our other YARN/MR-related applications are started (metron-api,
> > > MAAS,
> > > pcap query, etc). I wouldn't expect this to work since we have
> > runtime
> > > dependencies on our shaded elasticsearch and parser jars and I'm not
> > > aware
> > > of a way to add additional jars to the classpath with the YARN jar
> > > command
> > > (is there a way?). Either way I get this error:
> > >
> > > 18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir
> > using
> > > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> > skipping.
> > > java.lang.NullPointerException
> > >
> > >
> > > - I tried adding `yarn classpath` and `hadoop classpath` to the
> > > classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > > script). I
> > > get this error:
> > >
> > > java.lang.ClassNotFoundException:
> > > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > > jaxrs.JacksonJaxbJsonProvider
> > >
> > >
> > > - I sea

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Moving the yarn classpath command earlier in the classpath now gives this
error:

Caused by: java.lang.NoSuchMethodError:
javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;

I will experiment with other combinations, I suspect we will need
finer-grain control over the order.

The grep matches class names inside jar files.  I use this all the time and
it's really useful.

The metron-rest jar is already shaded.

Reverse engineering the yarn jar command was the next thing I was going to
try.  Will let you know how it goes.

On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> What order did you add the hadoop or yarn classpath? The "shaded" package
> stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
> those packages earlier on the classpath.
>
> I think that find command needs a "jar tvf", otherwise you're looking for a
> class name in jar file names.
>
> Have you tried shading the rest jar?
>
> I'd also look at the classpath you get when running "yarn jar" to start the
> existing pcap service, per the instructions in metron-api/README.md.
>
>
> On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman <merrim...@gmail.com> wrote:
>
> > To explore the idea of merging metron-api into metron-rest and running
> pcap
> > queries inside our REST application, I created a simple test here:
> > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test.  A
> > summary of what's included:
> >
> >- Added pcap as a dependency in the metron-rest pom.xml
> >- Added a pcap query controller endpoint at
> >http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> queryUsingGET
> >- Added a pcap query service that runs a simple, hardcoded query
> >
> > Generate some pcap data using pycapa (
> > https://github.com/apache/metron/tree/master/metron-sensors/pycapa) and
> > the
> > pcap topology (
> > https://github.com/apache/metron/tree/master/metron-
> > platform/metron-pcap-backend#starting-the-topology).
> > After this initial setup there should be data in HDFS at
> > "/apps/metron/pcap".  I believe this should be enough to exercise the
> > issue.  Just hit the endpoint referenced above.  I tested this in an
> > already running full dev by building and deploying the metron-rest jar.
> I
> > did not rebuild full dev with this change but I would still expect it to
> > work.  Let me know if it doesn't.
> >
> > The first error I see when I hit this endpoint is:
> >
> > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> >
> > Here are the things I've tried so far:
> >
> >- Run the REST application with the YARN jar command since this is how
> >all our other YARN/MR-related applications are started (metron-api,
> > MAAS,
> >pcap query, etc).  I wouldn't expect this to work since we have
> runtime
> >dependencies on our shaded elasticsearch and parser jars and I'm not
> > aware
> >of a way to add additional jars to the classpath with the YARN jar
> > command
> >(is there a way?).  Either way I get this error:
> >
> > 18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir
> using
> > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> skipping.
> > java.lang.NullPointerException
> >
> >
> >- I tried adding `yarn classpath` and `hadoop classpath` to the
> >classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > script).  I
> >get this error:
> >
> > java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > jaxrs.JacksonJaxbJsonProvider
> >
> >
> >- I searched for the class in the previous attempt but could not find
> it
> >in full dev:
> >
> > find / -name "*.jar" 2>/dev/null | xargs grep
> > org/apache/hadoop/hbase/shaded/org/codehaus/jackson/
> > jaxrs/JacksonJaxbJsonProvider
> > 2>/dev/null
> >
> >
> >- Further up in the stack trace I see the error happens when
> initiating
> >the org.apache.hadoop.yarn.util.timeline.TimelineUtils class.  I
> tried
> >setting "yarn.timeline-service.enabled" in Ambari to false and then I
> > get
> >this error:
> >
> > Unable to parse
> > '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> > URI, check the setting for mapreduce.application.framework.

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

To explore the idea of merging metron-api into metron-rest and running pcap
queries inside our REST application, I created a simple test here:
https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test.  A
summary of what's included:

   - Added pcap as a dependency in the metron-rest pom.xml
   - Added a pcap query controller endpoint at
   http://node1:8082/swagger-ui.html#!/pcap-query-controller/queryUsingGET
   - Added a pcap query service that runs a simple, hardcoded query

Generate some pcap data using pycapa (
https://github.com/apache/metron/tree/master/metron-sensors/pycapa) and the
pcap topology (
https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#starting-the-topology).
After this initial setup there should be data in HDFS at
"/apps/metron/pcap".  I believe this should be enough to exercise the
issue.  Just hit the endpoint referenced above.  I tested this in an
already running full dev by building and deploying the metron-rest jar.  I
did not rebuild full dev with this change but I would still expect it to
work.  Let me know if it doesn't.

The first error I see when I hit this endpoint is:

java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.

Here are the things I've tried so far:

   - Run the REST application with the YARN jar command since this is how
   all our other YARN/MR-related applications are started (metron-api, MAAS,
   pcap query, etc).  I wouldn't expect this to work since we have runtime
   dependencies on our shaded elasticsearch and parser jars and I'm not aware
   of a way to add additional jars to the classpath with the YARN jar command
   (is there a way?).  Either way I get this error:

18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir using
jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar. skipping.
java.lang.NullPointerException


   - I tried adding `yarn classpath` and `hadoop classpath` to the
   classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start script).  I
   get this error:

java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.shaded.org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider


   - I searched for the class in the previous attempt but could not find it
   in full dev:

find / -name "*.jar" 2>/dev/null | xargs grep
org/apache/hadoop/hbase/shaded/org/codehaus/jackson/jaxrs/JacksonJaxbJsonProvider
2>/dev/null


   - Further up in the stack trace I see the error happens when initiating
   the org.apache.hadoop.yarn.util.timeline.TimelineUtils class.  I tried
   setting "yarn.timeline-service.enabled" in Ambari to false and then I get
   this error:

Unable to parse
'/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
URI, check the setting for mapreduce.application.framework.path


   - I've tried adding different hadoop, hbase, yarn and mapreduce Maven
   dependencies without any success
  - hadoop-yarn-client
  - hadoop-yarn-common
  - hadoop-mapreduce-client-core
  - hadoop-yarn-server-common
  - hadoop-yarn-api
  - hbase-server

I will keep exploring other possible solutions.  Let me know if anyone has
any ideas.

On Mon, May 7, 2018 at 9:02 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> I can imagine a new generic service(s) capability whose job ( pun intended
> ) is to
> abstract the submittal, tracking, and storage of results to yarn.
>
> It would be extended with storage providers, queue provider, possibly some
> set of policies or rather strategies.
>
> The pcap ‘report’ would be a client to that service, the specializes the
> service operation for the way we want pcap to work.
>
> We can then re-use the generic service for other long running yarn
> things…..
>
>
> On May 7, 2018 at 09:56:51, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> RE: Tracking v. users
>
> The submittal and tracking can associate the submitter with the yarn job
> and track that,
> regardless of the yarn credentials.
>
> IE> if all submittals and monitoring are by the same yarn user ( Metron )
> from a single or
> co-operative set of services, that service can maintain the mapping.
>
>
>
> On May 7, 2018 at 09:39:52, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Otto, your use case makes sense to me. We'll have to think about how to
> manage the user to job relationships. I'm assuming YARN jobs will be
> submitted as the metron service user so YARN won't keep track of this for
> us. Is that assumption correct? Do you have any ideas for doing that?
>
> Mike, I can start a feature branch and experiment with merging metron-api
> into metron-rest. That should allow us to collaborate on any issues or
> challenges. Also, can you expand on your idea to manage external
> dependencies as a special module? That seems like a very attractive option
>

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Otto, your use case makes sense to me.  We'll have to think about how to
manage the user to job relationships.  I'm assuming YARN jobs will be
submitted as the metron service user so YARN won't keep track of this for
us.  Is that assumption correct?  Do you have any ideas for doing that?

Mike, I can start a feature branch and experiment with merging metron-api
into metron-rest.  That should allow us to collaborate on any issues or
challenges.   Also, can you expand on your idea to manage external
dependencies as a special module?  That seems like a very attractive option
to me.

On Fri, May 4, 2018 at 8:39 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> From my response on the other thread, but applicable to the backend stuff:
>
> "The PCAP Query seems more like PCAP Report to me.  You are generating a
> report based on parameters.
> That report is something that takes some time and external process to
> generate… ie you have to wait for it.
>
> I can almost imagine a flow where you:
>
> * Are in the AlertUI
> * Ask to generate a PCAP report based on some selected alerts/meta-alert,
> possibly picking from on or more report ‘templates’
> that have query options etc
> * The report request is ‘queued’, that is dispatched to be be
> executed/generated
> * You as a user have a ‘queue’ of your report results, and when the report
> is done it is queued there
> * We ‘monitor’ the report/queue press through the yarn rest ( report
> info/meta has the yarn details )
> * You can select the report from your queue and view it either in a new UI
> or custom component
> * You can then apply a different ‘view’ to the report or work with the
> report data
> * You can print / save etc
> * You can associate the report with the alerts ( again in the report info
> ) with…. a ‘case’ or ‘ticket’ or investigation something or other
>
>
> We can introduce extensibility into the report templates, report views (
> thinks that work with the json data of the report )
>
> Something like that.”
>
> Maybe we can do :
>
> template -> query parameters -> script => yarn info
> yarn info + query info + alert context + yarn status => report info ->
> stored in a user’s ‘report queue’
> report persistence added to report info
> metron-rest -> api to monitor the queue, read results ( page ), etc etc
>
>
> On May 4, 2018 at 09:23:39, Ryan Merriman (merrim...@gmail.com) wrote:
>
> I started a separate thread on Pcap UI considerations and user
> requirements
> at Otto's request. This should help us keep these two related but separate
> discussions focused.
>
> On Fri, May 4, 2018 at 7:19 AM, Michel Sumbul <michelsum...@gmail.com>
> wrote:
>
> > Hello,
> >
> >
> >
> > (Youhouuu my first reply on this kind of mail chain^^)
> >
> >
> >
> > If I may, I would like to share my view on the following 3 points.
> >
> > - Backend:
> >
> > The current metron-api is totally seperate, it will be logic for me to
> have
> > it at the same place as the others rest api. Especially when more
> security
> > will be added, it will not be needed to do the job twice.
> > The current implementation send back a pcap object which still need to
> be
> > decoded. In the opensoc, the decoding was done with tshard on the
> frontend.
> > It will be good to have this decoding happening directly on the backend
> to
> > not create a load on frontend. An option will be to install tshark on
> the
> > rest server and to use to convert the pcap to xml and then to a json
> that
> > will be send to the frontend.
> >
> > I tried to start directly the map/reduce job to search over all the pcap
> > data from the rest server and as Ryan mention it, we had trouble. I will
> > try to find back the error.
> >
> > Then in the POC, what we tried is to use the pcap_query script and this
> > work fine. I just modified it that he sends back directly the job_id of
> > yarn and not waiting that the job is finished. Then it will allow the UI
> > and the rest server to know what the status of the research by querying
> the
> > yarn rest api. This will allow the UI and the rest server to be async
> > without any blocking phase. What do you think about that?
> >
> >
> >
> > Having the job submitted directly from the code of the rest server will
> be
> > perfect, but it will need a lot of investigation I think (but I'm not
> the
> > expert so I might be completely wrong ^^).
> >
> > We know that the pcap_query scritp work fine so why not calling it? Is
> it
> > that bad? (maybe stupid question, but I really don’t see a lot of

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

I started a separate thread on Pcap UI considerations and user requirements
at Otto's request.  This should help us keep these two related but separate
discussions focused.

On Fri, May 4, 2018 at 7:19 AM, Michel Sumbul 
wrote:

> Hello,
>
>
>
> (Youhouuu my first reply on this kind of mail chain^^)
>
>
>
> If I may, I would like to share my view on the following 3 points.
>
> - Backend:
>
> The current metron-api is totally seperate, it will be logic for me to have
> it at the same place as the others rest api. Especially when more security
> will be added, it will not be needed to do the job twice.
> The current implementation send back a pcap object which still need to be
> decoded. In the opensoc, the decoding was done with tshard on the frontend.
> It will be good to have this decoding happening directly on the backend to
> not create a load on frontend. An option will be to install tshark on the
> rest server and to use to convert the pcap to xml and then to a json that
> will be send to the frontend.
>
> I tried to start directly the map/reduce job to search over all the pcap
> data from the rest server and as Ryan mention it, we had trouble. I will
> try to find back the error.
>
> Then in the POC, what we tried is to use the pcap_query script and this
> work fine. I just modified it that he sends back directly the job_id of
> yarn and not waiting that the job is finished. Then it will allow the UI
> and the rest server to know what the status of the research by querying the
> yarn rest api. This will allow the UI and the rest server to be async
> without any blocking phase. What do you think about that?
>
>
>
> Having the job submitted directly from the code of the rest server will be
> perfect, but it will need a lot of investigation I think (but I'm not the
> expert so I might be completely wrong ^^).
>
> We know that the pcap_query scritp work fine so why not calling it? Is it
> that bad? (maybe stupid question, but I really don’t see a lot of drawback)
>
>
>
> - Front end:
>
> Adding the the pcap search to the alert UI is, I think, the easiest way to
> move forward. But indeed, it will then be the “Alert UI and pcapquery”.
> Maybe the name of the UI should just change to something like “Monitoring &
> Investigation UI” ?
>
>
>
> Is there any roadmap or plan for the different UI? I mean did you already
> had discussion on how you see the ui evolving with the new feature that
> will come in the future?
>
>
>
> - Microservices:
>
>
>
> What do you mean exactly by microservices? Is it to separate all the
> features in different projects? Or something like having the different
> components in container like kubernet? (again maybe stupid question, but I
> don’t clearly understand what you mean J )
>
>
>
> Michel
>

[DISCUSS] Pcap UI user requirements

[Ryan Merriman]

Continuing a discussion that started in a discuss thread about exposing
Pcap query capabilities in the back end.  How should we expose this feature
to users?  Should it be integrated into the Alerts UI or be separate
standalone UI?

To summarize the general points made in the other thread:

   - Adding this capability to the Alerts UI will make it more of a
   composite app.  Is that really what we want since we have separate UIs for
   Alerts and management?
   - Would it be better to bring it in on it's own so it can be released
   with qualifiers and tested with the right expectations without affecting
   the Alerts UI?
   - There are some use cases that begin with an infosec analyst doing a
   search on alerts
   followed by them going to query pcap data corresponding to the
   threats they're investigating.  Would having these features in the same
   UI streamline this process?

There was also mention of some features we should consider:

   - Pcap queries should be made asynchronous via the UI
   - Take care that a user doesn't hit refresh or POST multiple times and kick
   off 50 mapreduce jobs
   - Options for managing the YARN queue that is used
   - Provide a "cancel" option that kills the MR job, or tell the user to
   go to the CLI to kill their job
   - Managing data if multiple users run queries
   - Strategy for cleaning up jobs and implementing a TTL (I think this one
   will be tricky and definitely needs discussion)
   - Date range or other query limits

A couple other features I would add:

   - Ability to paginate through results
   - Ability to download results through the UI
   - Realtime status of a running job in the UI

Let me know if I missed any points or did not correctly capture them
here.  What
other points do we need to consider?  What other features should be
required?  Nice to have?

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

I know, I was running with it :)

> On May 3, 2018, at 10:21 PM, Michael Miklavcic <michael.miklav...@gmail.com> 
> wrote:
> 
> Tabs vs spaces was a Silicon Valley joke, man :-)
> 
>> On Thu, May 3, 2018, 8:42 PM Ryan Merriman <merrim...@gmail.com> wrote:
>> 
>> Mike,
>> 
>> I never said there was anything problematic in metron-api, just that is was
>> inconsistent with the rest of Metron.  There is work involved in making it
>> consistent which is why I listed it as a downside.  I'm less concerned with
>> whether we use tabs or spaces but that we use one or the other.
>> 
>> I apologize for not making this clearer in my original message, but I did
>> not lead the POC development.  My involvement was helping troubleshoot
>> issues they ran into and answering questions about Metron in general.  I've
>> shared with you the information that I have which is my observations about
>> the types of issues they ran into.  I don't have a branch or pom file you
>> can experiment with.  I will reach out to that person and see if they are
>> able to share the exact errors they hit.  Also, the "trade-offs that you
>> seem to have already decided on" is not based on a specific issue or
>> challenge they faced in the POC.  It's based off of the past couple years
>> of working on our REST module and the reoccurring challenges and patterns I
>> see over a period of time.
>> 
>> Otto,
>> 
>> Makes sense to me.  I will start the other threads.
>> 
>> On Thu, May 3, 2018 at 8:50 PM, Otto Fowler <ottobackwa...@gmail.com>
>> wrote:
>> 
>>> I think my point is that maybe we should have a discuss about:
>>> 
>>> * PCAP UI, goals etc
>>> * Where it would live and why, what that would mean etc
>>> * Backend ( this original mail )
>>> 
>>> 
>>> 
>>> On May 3, 2018 at 18:34:00, Michael Miklavcic (
>> michael.miklav...@gmail.com)
>>> wrote:
>>> 
>>> Otto, what are you and your customers finding useful and/or difficult
>> from
>>> a split management/alerts UI perspective? It might help us to restate the
>>> original scope and intent around maintaining separate management and
>> alert
>>> UI's, to your point about "contrary to previous direction." I personally
>>> don't have a strong position on this other than 1) management is a
>>> different feature set from drilling into threat intel, yet many apps
>> still
>>> have their management UI combined with the end user experience and 2) we
>>> should probably consider pcap in context of a workflow with alerts.
>>> 
>>> On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwa...@gmail.com>
>>> wrote:
>>> 
>>>> If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t
>> the
>>>> alerts ui anymore.
>>>> 
>>>> It is becoming more of a “composite” app, with multiple feature ui’s
>>>> together. I didn’t think that
>>>> was what we were going for, thus the config ui and the alert ui.
>>>> 
>>>> Just adding disparate thing as ‘new tabs’ to a ui may be expedient but
>>> it
>>>> seems contrary to
>>>> our previous direction.
>>>> 
>>>> There are a few things to consider if we are going to start moving
>>>> everything into Alerts Ui aren’t there?
>>>> 
>>>> It may be a better road to bring it in on it’s own like the alerts ui
>>>> effort, so it can be released with ‘qualifiers’ and tested with
>>>> the right expectations without effecting the Alerts UI.
>>>> 
>>>> 
>>>> 
>>>> On May 3, 2018 at 17:25:54, Ryan Merriman (merrim...@gmail.com) wrote:
>>>> 
>>>> Otto,
>>>> 
>>>> I'm assuming just adding it to the Alerts UI is less work but I
>> wouldn't
>>> be
>>>> strongly opposed to it being it's own UI. What are the reasons for
>> doing
>>>> that?
>>>> 
>>>> Mike,
>>>> 
>>>> On using metron-api:
>>>> 
>>>> 1. I'm making an assumption about it not being used much. Maybe it
>>>> still works without issue. I agree, we'll have to test anything we
>> build
>>>> so this is a minor issue.
>>>> 2. Updating metron-api to be asynchronous is a requirement in my
>> opinion
>>>> 3. The MPack work is the major drawback for me. We're essentially
>>>> creatin

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Mike,

I never said there was anything problematic in metron-api, just that is was
inconsistent with the rest of Metron.  There is work involved in making it
consistent which is why I listed it as a downside.  I'm less concerned with
whether we use tabs or spaces but that we use one or the other.

I apologize for not making this clearer in my original message, but I did
not lead the POC development.  My involvement was helping troubleshoot
issues they ran into and answering questions about Metron in general.  I've
shared with you the information that I have which is my observations about
the types of issues they ran into.  I don't have a branch or pom file you
can experiment with.  I will reach out to that person and see if they are
able to share the exact errors they hit.  Also, the "trade-offs that you
seem to have already decided on" is not based on a specific issue or
challenge they faced in the POC.  It's based off of the past couple years
of working on our REST module and the reoccurring challenges and patterns I
see over a period of time.

Otto,

Makes sense to me.  I will start the other threads.

On Thu, May 3, 2018 at 8:50 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> I think my point is that maybe we should have a discuss about:
>
> * PCAP UI, goals etc
> * Where it would live and why, what that would mean etc
> * Backend ( this original mail )
>
>
>
> On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklav...@gmail.com)
> wrote:
>
> Otto, what are you and your customers finding useful and/or difficult from
> a split management/alerts UI perspective? It might help us to restate the
> original scope and intent around maintaining separate management and alert
> UI's, to your point about "contrary to previous direction." I personally
> don't have a strong position on this other than 1) management is a
> different feature set from drilling into threat intel, yet many apps still
> have their management UI combined with the end user experience and 2) we
> should probably consider pcap in context of a workflow with alerts.
>
> On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the
> > alerts ui anymore.
> >
> > It is becoming more of a “composite” app, with multiple feature ui’s
> > together. I didn’t think that
> > was what we were going for, thus the config ui and the alert ui.
> >
> > Just adding disparate thing as ‘new tabs’ to a ui may be expedient but
> it
> > seems contrary to
> > our previous direction.
> >
> > There are a few things to consider if we are going to start moving
> > everything into Alerts Ui aren’t there?
> >
> > It may be a better road to bring it in on it’s own like the alerts ui
> > effort, so it can be released with ‘qualifiers’ and tested with
> > the right expectations without effecting the Alerts UI.
> >
> >
> >
> > On May 3, 2018 at 17:25:54, Ryan Merriman (merrim...@gmail.com) wrote:
> >
> > Otto,
> >
> > I'm assuming just adding it to the Alerts UI is less work but I wouldn't
> be
> > strongly opposed to it being it's own UI. What are the reasons for doing
> > that?
> >
> > Mike,
> >
> > On using metron-api:
> >
> > 1. I'm making an assumption about it not being used much. Maybe it
> > still works without issue. I agree, we'll have to test anything we build
> > so this is a minor issue.
> > 2. Updating metron-api to be asynchronous is a requirement in my opinion
> > 3. The MPack work is the major drawback for me. We're essentially
> > creating a brand new Metron component. There are a lot of examples we
> can
> > draw from but it's going to be a large chunk of new MPack code to
> maintain
> > and MPack development has been painful in the past. I think it will
> > include:
> > 1. Creating a start script
> > 2. Creating master.py and commands.py scripts for managing the
> > application lifecycle, service checks, etc
> > 3. Creating an -env.xml file for exposing properties in Ambari
> > 4. Adding the component to the various MPack files
> > (metron_theme.json, metainfo.xml, service_advisor.py, etc.)
> > 4. Our Storm topologies are completely different use cases and much more
> > complex so I don't understand the comparison. But if you prefer this
> > coding style then I think this is a minor issue as well.
> >
> > On micro-services:
> >
> > 1. Our REST service already includes a lot of dependencies and is
> > difficult to manage in it's current state. I just went through this on
> > https://github.com/apache/metron/pul

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Otto,

I'm assuming just adding it to the Alerts UI is less work but I wouldn't be
strongly opposed to it being it's own UI.  What are the reasons for doing
that?

Mike,

On using metron-api:

   1. I'm making an assumption about it not being used much.  Maybe it
   still works without issue.  I agree, we'll have to test anything we build
   so this is a minor issue.
   2. Updating metron-api to be asynchronous is a requirement in my opinion
   3. The MPack work is the major drawback for me.  We're essentially
   creating a brand new Metron component.  There are a lot of examples we can
   draw from but it's going to be a large chunk of new MPack code to maintain
   and MPack development has been painful in the past.  I think it will
   include:
  1. Creating a start script
  2. Creating master.py and commands.py scripts for managing the
  application lifecycle, service checks, etc
  3. Creating an -env.xml file for exposing properties in Ambari
  4. Adding the component to the various MPack files
  (metron_theme.json, metainfo.xml, service_advisor.py, etc.)
   4. Our Storm topologies are completely different use cases and much more
   complex so I don't understand the comparison.  But if you prefer this
   coding style then I think this is a minor issue as well.

On micro-services:

   1. Our REST service already includes a lot of dependencies and is
   difficult to manage in it's current state.  I just went through this on
   https://github.com/apache/metron/pull/1008.  It was painful.  When we
   tried to include mapreduce and yarn dependencies it became what seemed like
   an endless NoSuchMethod, NoClassDef and similar errors.  Even if we can get
   it to work it's going to make managing our REST service that much harder
   than it already is.  I think the shaded jars are the source of all this
   trouble and I agree it would be nice to improve our architecture in this
   area.  However I don't think it's a simple fix and now we're getting into
   the "will likely take a long time to plan and implement" concern.  If
   anyone has ideas on how to solve our shaded jar challenge I would be all
   for it.
   2. All the MPack work listed above would also be required here.  A
   micro-services pattern is a significant shift and can't even give you
   concrete examples of what exactly we would have to do.  We would need to go
   through extensive design and planning to even get to that point.
   3. It would be a branch new component.  See above plus any new
   infrastructure we would need (web server/proxy, service discovery, etc)

On pcap-query:

   1. I don't recall any users or customers directly using metron-api but
   if you say so I believe you :)
   2. As I understand it the pcap topology and pcap query are somewhat
   decoupled.  Maybe location of pcap files would be shared?  MPack work here
   is likely to include adding a couple properties and moving some around so
   they can be shared.  Deciding between Ambari and global config would be
   similar to properties we add to any component.

I think you may be underestimating how difficult it's going to be to solve
our dependency problem.  Or maybe it's me that is overestimating it :)  It
could be something we experiment with before we start on the pcap work.
There is major upside and it would benefit the whole project.  But until
then we can't fit anymore more screwdrivers in the toolbox.  For me the
only reasonable options are to use the existing metron-api as it's own
separate service or call out to the pcap_query.sh script from our existing
REST app.  I could go either way really.  I'm just not excited about all
the MPack code we have to write for a new component.  Maybe it won't be
that bad.

On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> First thought is why the Alerts-UI and Not a dedicated  Query UI?
>
>
> On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) wrote:
>
> We are planning on adding the pcap query feature to the Alerts UI. Before
> we start this work, I think it is important to get community buy in on the
> architectural approach. There are a couple different options.
>
> One option is to leverage the existing metron-api module that exposes pcap
> queries through a REST service. The upsides are:
>
> - some work has already been done
> - it's part of our build so we know unit and integration tests pass
>
> The downsides are:
>
> - It hasn't been used in a while and will need some end to end testing
> to make sure it still functions properly
> - It is synchronous and will block the UI, using up the limited number
> of concurrent connections available in a browser
> - It will require significant MPack work to properly set it up on install
> - It is a legacy module from OpenSOC and coding style is significantly
> different
>
> Another option would be moving to a micro-s

[DISCUSS] Pcap panel architecture

[Ryan Merriman]

We are planning on adding the pcap query feature to the Alerts UI.  Before
we start this work, I think it is important to get community buy in on the
architectural approach.  There are a couple different options.

One option is to leverage the existing metron-api module that exposes pcap
queries through a REST service.  The upsides are:

   - some work has already been done
   - it's part of our build so we know unit and integration tests pass

The downsides are:

   - It hasn't been used in a while and will need some end to end testing
   to make sure it still functions properly
   - It is synchronous and will block the UI, using up the limited number
   of concurrent connections available in a browser
   - It will require significant MPack work to properly set it up on install
   - It is a legacy module from OpenSOC and coding style is significantly
   different

Another option would be moving to a micro-services architecture.  We have
experimented with a proof of concept and found it was too hard to add this
feature into our existing REST services because of all the dependencies
that must coexist in the same application.  The upsides are:

   - Would provide a platform for future Batch/MR/YARN type features
   - There would be fewer technical compromises since we are building it
   from the ground up

The downsides are:

   - Will require the most effort and will likely take a long time to plan
   and implement
   - Like the previous option, will require significant MPack work

A third option would be to add an endpoint to our existing REST service
that delegates to the pcap_query.sh script through the Java Process class.
The upsides to this approach are:

   - We know the pcap_query.sh script works and would require minimal
   changes
   - Minimal MPack work is required since our REST service is already
   included

The downsides are:

   - Does not set us up to easily add other batch-oriented features in the
   future
   - OS-level security becomes a concern since we are delegating to a
   script in a separate process

I feel like ultimately we want to transition to a micro-services
architecture because it will provide more flexibility and make it easier to
grow our set of features.  But in the meantime, wrapping the pcap_query.sh
script would allow us to add this feature with less work and fewer lines of
code.  If and when we decide to deploy a separate REST application for
batch features, the UI portion would require minimal changes.

What does everyone think?

[DISCUSS] Development environment for UI work

[Ryan Merriman]

In anticipation of more UI work starting (Alerts UI specifically), I want
to kick off a discussion about how we can best provide a backend for UI
developers to code against.  I believe our full dev environment will not
work for this because:

   1. It has become more resource intensive over time and requires constant
   tuning to make it stable.
   2. Takes a long time to build.  I think this will eventually lead to UI
   developers building less often and not being current with master.
   3. UI developers will likely not have experience with big data projects
   and should not be required to manage a full metron VM install locally.

I think we need something more lightweight that only includes components
needed for UI development.  Here are what I think are the requirements for
this:

   1. Must include the REST component as this is how the UIs interact with
   Metron
   2. Should also include Search (ES or Solr), Kafka, HBase and Zookeeper
   since REST depends on these (am I missing any?).
   3. Should be portable such that it can be run locally or in a shared
   environment like AWS
   4. The environment should always contain the latest version of master

There is currently work being done to externalize our integration test
infrastructure (https://issues.apache.org/jira/browse/METRON-1352) that I
believe can also be leveraged here.  Are there other options or approaches
people can think of?  What's the best way to provide an environment that's
easy to spin up and always current with master?

Re: [DISCUSS] Persistence store for user profile settings

[Ryan Merriman]

t don't need HA/DR, just use the DB that gets spun-up with Ambari.
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Feb 2, 2018 at 7:17 AM Simon Elliston Ball <
> > > si...@simonellistonball.com> wrote:
> > >
> > >> Introducing a RDBMS to the stack seems unnecessary for this.
> > >>
> > >> If we consider the data access patterns for user profiles, we are
> > unlikely
> > >> to query into them, or indeed do anything other than look them up, or
> > write
> > >> them out by a username key. To that end, using an ORM to translate a a
> > >> nested config object into a load of tables seems to introduce
> complexity
> > >> and brittleness we then have to take away through relying on
> relational
> > >> consistency models. We would also end up with, as Mike points out, a
> > whole
> > >> new disk deployment patterns and a bunch of additional DBA ops process
> > >> requirements for every install.
> > >>
> > >> Since the access pattern is almost entirely key => value, hbase seems
> a
> > >> good option (because we already have it there, it would be kinda crazy
> > at
> > >> this scale if we didn’t already have it) or arguably zookeeper, but
> that
> > >> might be at the other end of the scale argument. I’d even go as far as
> > to
> > >> suggest files on HDFS to keep it simple.
> > >>
> > >> Simon
> > >>
> > >>> On 1 Feb 2018, at 23:24, Michael Miklavcic <
> > michael.miklav...@gmail.com>
> > >> wrote:
> > >>>
> > >>> Personally, I'd be in favor of something like Maria DB as an open
> > source
> > >>> repo. Or any other ansi sql store. On the positive side, it should
> mesh
> > >>> seamlessly with ORM tools. And the schema for this should be pretty
> > >>> vanilla, I'd imagine. I might even consider skipping ORM for straight
> > >> JDBC
> > >>> and simple command scripts in Java for something this small. I'm not
> > >>> worried so much about migrations of this sort. Large scale DBs can
> get
> > >>> involved with major schema changes, but thats usually when the
> > datastore
> > >> is
> > >>> a massive set of tables with complex relationships, at least in my
> > >>> experience.
> > >>>
> > >>> We could also use hbase, which probably wouldn't be that hard either,
> > but
> > >>> there may be more boilerplate to write for the client as compared to
> > >>> standard SQL. But I'm assuming we could reuse a fair amount of
> existing
> > >>> code from our enrichments. One additional reason in favor of hbase
> > might
> > >> be
> > >>> data replication. For a SQL instance we'd probably recommend a RAID
> > store
> > >>> or backup procedure, but we get that pretty easy with hbase too.
> > >>>
> > >>> On Feb 1, 2018 2:45 PM, "Casey Stella" <ceste...@gmail.com> wrote:
> > >>>
> > >>>> So, I'll answer your question with some questions:
> > >>>>
> > >>>>  - No matter the data store we use upgrading will take some care,
> > >> right?
> > >>>>  - Do we currently depend on a RDBMS anywhere?  I want to say that
> we
> > >> do
> > >>>>  in the REST layer already, right?
> > >>>>  - If we don't use a RDBMs, what's the other option?  What are the
> > pros
> > >>>>  and cons?
> > >>>>  - Have we considered non-server offline persistent solutions (e.g.
> > >>>>  https://www.html5rocks.com/en/features/storage)?
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Thu, Feb 1, 2018 at 9:11 AM, Ryan Merriman <merrim...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> There is currently a PR up for review that allows a user to
> configure
> > >> and
> > >>>>> save the list of facet fields that appear in the left column of the
> > >>>> Alerts
> > >>>>> UI:  https://github.com/apache/metron/pull/853.  The REST layer
> has
> > >> ORM
> > >>>>> support which means we can store those in a relational database.
> > >>>>>
> > >>>>> However I'm not 100% sure this is the best place to keep this.  As
> we
> > >> add
> > >>>>> more use cases like this the backing tables in the RDBMS will need
> to
> > >> be
> > >>>>> managed.  This could make upgrading more tedious and error-prone.
> Is
> > >>>> there
> > >>>>> are a better way to store this, assuming we can leverage a
> component
> > >>>> that's
> > >>>>> already included in our stack?
> > >>>>>
> > >>>>> Ryan
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

[DISCUSS] Persistence store for user profile settings

[Ryan Merriman]

There is currently a PR up for review that allows a user to configure and
save the list of facet fields that appear in the left column of the Alerts
UI:  https://github.com/apache/metron/pull/853.  The REST layer has ORM
support which means we can store those in a relational database.

However I'm not 100% sure this is the best place to keep this.  As we add
more use cases like this the backing tables in the RDBMS will need to be
managed.  This could make upgrading more tedious and error-prone.  Is there
are a better way to store this, assuming we can leverage a component that's
already included in our stack?

Ryan

[DISCUSS] Overcoming developer inertia when spinning up new environments

[Ryan Merriman]

I want to revisit the idea of providing an alternative container-based
approach (Docker, Kubernetes, etc) to spinning up Metron that is faster and
uses less resources (a "Metron light").  This would provide a way for
reviewers to more quickly review and test out changes.  Full dev with
ansible will still serve it's purpose, this would just be another tool for
cases where full dev is not the best fit.

This would be a new non-trivial module that will need to be maintained.
There have been discussions in the past that resulted in the community not
wanting to maintain another installation path.  However it has been a while
since we had those discussions and Metron is now more mature.  We would
also be able to leverage the work already being done in
https://issues.apache.org/jira/browse/METRON-1352 to unify the integration
testing infrastructure.

There are other potential use cases for this too.  It could be expanded to
provide a demo environment for exploring the UIs and Metron API.  Providing
container support for Metron could also be the beginning of a broader cloud
deployment strategy.

Is this something we want to explore?  What would the requirements be?

Re: [DISCUSS] Integration/e2e test infrastructure requirements

[Ryan Merriman]

I took a first pass at adding tasks and will continue adding more as I
think of them.  I will wait for feedback on which modules to include before
I add all those (only added metron-elasticsearch for now).  I left all but
a couple unassigned so that anyone can pick up a task if they want.

On Wed, Dec 13, 2017 at 4:41 PM, Ryan Merriman <merrim...@gmail.com> wrote:

> Jira is here:  https://issues.apache.org/jira/browse/METRON-1352.  I am
> starting to create sub-tasks based on the requirements outlined above and
> included in that Jira description.
>
> I am compiling a list of modules that we'll need to convert to the testing
> infrastructure.  Based on imports of ComponentRunner, I get these modules:
>
>- metron-elasticsearch
>- metron-enrichment
>- metron-indexing
>- metron-integration-test
>- metron-maas-service
>- metron-management
>- metron-pcap-backend
>- metron-profiler
>- metron-rest
>- metron-solr
>
> I am planning on creating sub-tasks for each of these.  I know that
> metron-common should also be converted because it uses the Zookeeper in
> memory server but doesn't use ComponentRunner to manage it.  Are there
> other modules like this that you know of?
>
> On Wed, Dec 13, 2017 at 2:44 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> Same as the feature branch name?  I just want to find it and set a watch
>> on it ;)
>>
>>
>> On December 13, 2017 at 15:29:00, Ryan Merriman (merrim...@gmail.com)
>> wrote:
>>
>> I'm open to ideas. What do you think the title should be?
>>
>> On Wed, Dec 13, 2017 at 2:13 PM, Otto Fowler <ottobackwa...@gmail.com>
>> wrote:
>>
>> > What is the Master Jira going to be?
>> >
>> >
>> >
>> > On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com)
>> > wrote:
>> >
>> > I am going to start the process of creating Jiras out of these initial
>> > requirements. I agree with them and think they are a good starting
>> point.
>> > Feel free to join in at anytime and add/change/remove requirements as
>> > needed. I will update the thread once I have the initial Jiras created
>> and
>> > we can go from there.
>> >
>> > On Mon, Dec 11, 2017 at 4:10 PM, Ryan Merriman <merrim...@gmail.com>
>> > wrote:
>> >
>> > > The purpose of this discussion is map out what is required to get the
>> > POC
>> > > started with https://github.com/apache/metron/pull/858 into master.
>> > >
>> > > The following features were added in the previously mentioned PR:
>> > >
>> > > - Dockerfile for Metron REST
>> > > - Dockerfile for Metron UIs
>> > > - Docker Compose application including Metron images, Elasticsearch,
>> > > Kafka, Zookeeper
>> > > - Modified travis file that manages the Docker environment and runs
>> > > the e2e tests as part of the build
>> > > - Maven pom.xml that installs all the required assets into the Docker
>> > > e2e module
>> > > - Modified metron-alerts pom.xml that allows e2e tests to be run
>> > > through Maven
>> > > - An example integration test that has been converted to use the new
>> > > infrastructure
>> > >
>> > > Here are the initial features proposed for acceptance into master:
>> > >
>> > > - All e2e and integration tests run on common infrastructure.
>> > > - All e2e and integration tests are run automatically in the Travis
>> > > build.
>> > > - All e2e and integration tests run repeatably and reliably in the
>> > > Travis build.
>> > > - Debugging options are available and documented.
>> > > - The new infra and how to interact with it is documented.
>> > > - Old infrastructure removed (anything unused or commented out is
>> > > deleted, instead of staying).
>> > >
>> > > Are there other requirements people want to add to this list?
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>>
>>
>

Re: [DISCUSS] Integration/e2e test infrastructure requirements

[Ryan Merriman]

I'm open to ideas.  What do you think the title should be?

On Wed, Dec 13, 2017 at 2:13 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> What is the Master Jira going to be?
>
>
>
> On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> I am going to start the process of creating Jiras out of these initial
> requirements. I agree with them and think they are a good starting point.
> Feel free to join in at anytime and add/change/remove requirements as
> needed. I will update the thread once I have the initial Jiras created and
> we can go from there.
>
> On Mon, Dec 11, 2017 at 4:10 PM, Ryan Merriman <merrim...@gmail.com>
> wrote:
>
> > The purpose of this discussion is map out what is required to get the
> POC
> > started with https://github.com/apache/metron/pull/858 into master.
> >
> > The following features were added in the previously mentioned PR:
> >
> > - Dockerfile for Metron REST
> > - Dockerfile for Metron UIs
> > - Docker Compose application including Metron images, Elasticsearch,
> > Kafka, Zookeeper
> > - Modified travis file that manages the Docker environment and runs
> > the e2e tests as part of the build
> > - Maven pom.xml that installs all the required assets into the Docker
> > e2e module
> > - Modified metron-alerts pom.xml that allows e2e tests to be run
> > through Maven
> > - An example integration test that has been converted to use the new
> > infrastructure
> >
> > Here are the initial features proposed for acceptance into master:
> >
> > - All e2e and integration tests run on common infrastructure.
> > - All e2e and integration tests are run automatically in the Travis
> > build.
> > - All e2e and integration tests run repeatably and reliably in the
> > Travis build.
> > - Debugging options are available and documented.
> > - The new infra and how to interact with it is documented.
> > - Old infrastructure removed (anything unused or commented out is
> > deleted, instead of staying).
> >
> > Are there other requirements people want to add to this list?
> >
> >
> >
> >
>
>

Re: [DISCUSS] Integration/e2e test infrastructure requirements

[Ryan Merriman]

I am going to start the process of creating Jiras out of these initial
requirements.  I agree with them and think they are a good starting point.
Feel free to join in at anytime and add/change/remove requirements as
needed.  I will update the thread once I have the initial Jiras created and
we can go from there.

On Mon, Dec 11, 2017 at 4:10 PM, Ryan Merriman <merrim...@gmail.com> wrote:

> The purpose of this discussion is map out what is required to get the POC
> started with https://github.com/apache/metron/pull/858 into master.
>
> The following features were added in the previously mentioned PR:
>
>- Dockerfile for Metron REST
>- Dockerfile for Metron UIs
>- Docker Compose application including Metron images, Elasticsearch,
>Kafka, Zookeeper
>- Modified travis file that manages the Docker environment and runs
>the e2e tests as part of the build
>- Maven pom.xml that installs all the required assets into the Docker
>e2e module
>- Modified metron-alerts pom.xml that allows e2e tests to be run
>through Maven
>- An example integration test that has been converted to use the new
>infrastructure
>
> Here are the initial features proposed for acceptance into master:
>
>- All e2e and integration tests run on common infrastructure.
>- All e2e and integration tests are run automatically in the Travis
>build.
>- All e2e and integration tests run repeatably and reliably in the
>Travis build.
>- Debugging options are available and documented.
>- The new infra and how to interact with it is documented.
>- Old infrastructure removed (anything unused or commented out is
>deleted, instead of staying).
>
> Are there other requirements people want to add to this list?
>
>
>
>

[DISCUSS] Integration/e2e test infrastructure requirements

[Ryan Merriman]

The purpose of this discussion is map out what is required to get the POC
started with https://github.com/apache/metron/pull/858 into master.

The following features were added in the previously mentioned PR:

   - Dockerfile for Metron REST
   - Dockerfile for Metron UIs
   - Docker Compose application including Metron images, Elasticsearch,
   Kafka, Zookeeper
   - Modified travis file that manages the Docker environment and runs the
   e2e tests as part of the build
   - Maven pom.xml that installs all the required assets into the Docker
   e2e module
   - Modified metron-alerts pom.xml that allows e2e tests to be run through
   Maven
   - An example integration test that has been converted to use the new
   infrastructure

Here are the initial features proposed for acceptance into master:

   - All e2e and integration tests run on common infrastructure.
   - All e2e and integration tests are run automatically in the Travis
   build.
   - All e2e and integration tests run repeatably and reliably in the
   Travis build.
   - Debugging options are available and documented.
   - The new infra and how to interact with it is documented.
   - Old infrastructure removed (anything unused or commented out is
   deleted, instead of staying).

Are there other requirements people want to add to this list?

Re: [DISCUSS] e2e test infrastructure

[Ryan Merriman]

We will also need an ES and Metron REST container for the e2e tests, but
yeah you get the idea.  I think having the tests be responsible for setup
will work.

Maybe the next step is to build a simple example and let everyone try it
out.  If we don't like it, we can throw it away.

On Wed, Nov 29, 2017 at 1:28 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> So we will just have a :
>
> ZK container
> Kafka Container
> HDFS Container
>
> and not deploy any metron stuff to them in the docker setup, the test
> itself will deploy what it needs and cleanup?
>
>
> On November 29, 2017 at 11:53:46, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> “I would feel better using docker if each docker container only had the
> base services, and did not require a separate but parallel deployment path
> to ambari”
>
> This exactly how it works. There is a container for each base service,
> just like we now have an in-memory component for each base service. There
> is also no deployment path to Ambari. Ambari is not involved at all.
>
> From a client perspective (our e2e/integration tests in this case) there
> really is not much of a difference. At the end of the day services are up
> and running and available on various ports.
>
> Also there is going to be maintenance required no matter what approach we
> decide on. If we add another ES template that needs to be loaded by the
> MPack, our e2e/integration test infrastructure will also have to load that
> template. I have had to do this with our current integration tests.
>
> > On Nov 29, 2017, at 9:38 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
> >
> > So the issue with metron-docker is that it is all custom setup for
> metron components, and understanding how to maintain it when you make
> changes to the system is difficult for the developers.
> > This is a particular issue to me, because I would have to re-write a big
> chunk of it to accommodate 777.
> >
> > I would feel better using docker if each docker container only had the
> base services, and did not require a separate but parallel deployment path
> to ambari. That is to say if the docker components
> > were functional equivalent and limited to the in memory components
> functionality and usage. I apologize if that is in fact what you are
> getting at.
> >
> > Then we could move the integrations and e2e to them.
> >
> >
> >
> >> On November 29, 2017 at 10:00:20, Ryan Merriman (merrim...@gmail.com)
> wrote:
> >>
> >> Thanks for the feedback so far everyone. All good points.
> >>
> >> Otto, if we did decide to go down the Docker route, we could
> >> use /master/metron-contrib/metron-docker as a starting point. The
> reason I
> >> initially create that module was to support Management UI testing
> because
> >> full dev was unusable for that purpose at that time. This is the same
> use
> >> case. A lot of the work has already been done but we would need to
> review
> >> it and bring it up to date with the current state of master. Once we
> get
> >> it to a point where we can manually spin up the Docker environment and
> get
> >> the e2e tests to pass, we would then need to add it into our Travis
> >> workflow.
> >>
> >> Mike, yes this is one of the options I listed at the start of the
> discuss
> >> thread although I'm not sure I agree with the Docker disadvantages you
> >> list. We could use a similar approach for HDFS in Docker by setting it
> to
> >> local FS and creating a shared volume that all the containers have
> access
> >> to. I've also found that Docker Compose makes the networking part much
> >> easier. What other advantages would in-memory components in separate
> >> process offer us that you can think of? Are there other disadvantages
> with
> >> using Docker?
> >>
> >> Justin, I think that's a really good point and I would be on board with
> >> it. I see this use case (e2e testing infrastructure) as a good way to
> >> evaluate our options without making major changes across our codebase.
> I
> >> would agree that standardizing on an approach would be ideal and
> something
> >> we should work towards. The debugging request is also something that
> would
> >> be extremely helpful. The only issue I see is debugging a Storm
> topology,
> >> this would still need to be run locally using LocalCluster because
> remote
> >> debugging does not work well in Storm (per previous comments from Storm
> >> committers). At one point I was able to get this to work with Docker
> &

Re: [DISCUSS] e2e test infrastructure

[Ryan Merriman]

“I would feel better using docker if each docker container only had the base 
services, and did not require a separate but parallel deployment path to ambari”

This exactly how it works.  There is a container for each base service, just 
like we now have an in-memory component for each base service.  There is also 
no deployment path to Ambari.  Ambari is not involved at all.

From a client perspective (our e2e/integration tests in this case) there really 
is not much of a difference.  At the end of the day services are up and running 
and available on various ports.

Also there is going to be maintenance required no matter what approach we 
decide on.  If we add another ES template that needs to be loaded by the MPack, 
our e2e/integration test infrastructure will also have to load that template.  
I have had to do this with our current integration tests.

> On Nov 29, 2017, at 9:38 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:
> 
> So the issue with metron-docker is that it is all custom setup for metron 
> components, and understanding how to maintain it when you make changes to the 
> system is difficult for the developers.
> This is a particular issue to me, because I would have to re-write a big 
> chunk of it to accommodate 777.
> 
> I would feel better using docker if each docker container only had the base 
> services, and did not require a separate but parallel deployment path to 
> ambari.  That is to say if the docker components
> were functional equivalent and limited to the in memory components 
> functionality and usage.  I apologize if that is in fact what you are getting 
> at.
> 
> Then we could move the integrations and e2e to them.
>  
> 
> 
>> On November 29, 2017 at 10:00:20, Ryan Merriman (merrim...@gmail.com) wrote:
>> 
>> Thanks for the feedback so far everyone. All good points. 
>> 
>> Otto, if we did decide to go down the Docker route, we could 
>> use /master/metron-contrib/metron-docker as a starting point. The reason I 
>> initially create that module was to support Management UI testing because 
>> full dev was unusable for that purpose at that time. This is the same use 
>> case. A lot of the work has already been done but we would need to review 
>> it and bring it up to date with the current state of master. Once we get 
>> it to a point where we can manually spin up the Docker environment and get 
>> the e2e tests to pass, we would then need to add it into our Travis 
>> workflow. 
>> 
>> Mike, yes this is one of the options I listed at the start of the discuss 
>> thread although I'm not sure I agree with the Docker disadvantages you 
>> list. We could use a similar approach for HDFS in Docker by setting it to 
>> local FS and creating a shared volume that all the containers have access 
>> to. I've also found that Docker Compose makes the networking part much 
>> easier. What other advantages would in-memory components in separate 
>> process offer us that you can think of? Are there other disadvantages with 
>> using Docker? 
>> 
>> Justin, I think that's a really good point and I would be on board with 
>> it. I see this use case (e2e testing infrastructure) as a good way to 
>> evaluate our options without making major changes across our codebase. I 
>> would agree that standardizing on an approach would be ideal and something 
>> we should work towards. The debugging request is also something that would 
>> be extremely helpful. The only issue I see is debugging a Storm topology, 
>> this would still need to be run locally using LocalCluster because remote 
>> debugging does not work well in Storm (per previous comments from Storm 
>> committers). At one point I was able to get this to work with Docker 
>> containers but we would definitely need to revisit it and create tooling 
>> around it. 
>> 
>> So in summary, I think we agree on these points so far: 
>> 
>> - no one seems to be in favor of mocking our backend so I'll take that 
>> option off the table 
>> - everyone seems to be in favor of moving to a strategy where we spin up 
>> backend services at the beginning of all tests and spin down at the end, 
>> rather than spinning up/down for each class or suite of tests 
>> - the ability to debug our code locally is important and something to 
>> keep in mind as we evaluate our options 
>> 
>> I think the next step is to decide whether we pursue in-memory/separate 
>> process vs Docker. Having used both, there are a couple disadvantages I 
>> see with the in-memory approach: 
>> 
>> - The in-memory components are different than real installations and 
>> come with separate issues. There hav

Re: [DISCUSS] e2e test infrastructure

[Ryan Merriman]

Mike, I think we are in agreement:  any solution involving in-memory components 
would have them running in separate processes vs. a single process like they do 
now.

> On Nov 29, 2017, at 9:14 AM, Michael Miklavcic <michael.miklav...@gmail.com> 
> wrote:
> 
> I understood the item on "in-memory components" to be similar to what we're
> currently doing in the integration tests, which we cannot and should not
> do. They are spun up in a single jvm process which causes major problems
> with classpath isolation. My main point here is to be sure each component
> is separate from one another, and that they can be utilized for both the
> e2e and integration tests.
> 
>> On Nov 29, 2017 8:00 AM, "Ryan Merriman" <merrim...@gmail.com> wrote:
>> 
>> Thanks for the feedback so far everyone.  All good points.
>> 
>> Otto, if we did decide to go down the Docker route, we could
>> use /master/metron-contrib/metron-docker as a starting point.  The reason
>> I
>> initially create that module was to support Management UI testing because
>> full dev was unusable for that purpose at that time.  This is the same use
>> case.  A lot of the work has already been done but we would need to review
>> it and bring it up to date with the current state of master.  Once we get
>> it to a point where we can manually spin up the Docker environment and get
>> the e2e tests to pass, we would then need to add it into our Travis
>> workflow.
>> 
>> Mike, yes this is one of the options I listed at the start of the discuss
>> thread although I'm not sure I agree with the Docker disadvantages you
>> list.  We could use a similar approach for HDFS in Docker by setting it to
>> local FS and creating a shared volume that all the containers have access
>> to.  I've also found that Docker Compose makes the networking part much
>> easier.  What other advantages would in-memory components in separate
>> process offer us that you can think of?  Are there other disadvantages with
>> using Docker?
>> 
>> Justin, I think that's a really good point and I would be on board with
>> it.  I see this use case (e2e testing infrastructure) as a good way to
>> evaluate our options without making major changes across our codebase.  I
>> would agree that standardizing on an approach would be ideal and something
>> we should work towards.  The debugging request is also something that would
>> be extremely helpful.  The only issue I see is debugging a Storm topology,
>> this would still need to be run locally using LocalCluster because remote
>> debugging does not work well in Storm (per previous comments from Storm
>> committers).  At one point I was able to get this to work with Docker
>> containers but we would definitely need to revisit it and create tooling
>> around it.
>> 
>> So in summary, I think we agree on these points so far:
>> 
>>   - no one seems to be in favor of mocking our backend so I'll take that
>>   option off the table
>>   - everyone seems to be in favor of moving to a strategy where we spin up
>>   backend services at the beginning of all tests and spin down at the end,
>>   rather than spinning up/down for each class or suite of tests
>>   - the ability to debug our code locally is important and something to
>>   keep in mind as we evaluate our options
>> 
>> I think the next step is to decide whether we pursue in-memory/separate
>> process vs Docker.  Having used both, there are a couple disadvantages I
>> see with the in-memory approach:
>> 
>>   - The in-memory components are different than real installations and
>>   come with separate issues.  There have been cases where an in-memory
>>   component had a bug (looking at you Kafka) that a normal installation
>>   wouldn't have and required effort to put workarounds in place.
>>   - Spinning up the in-memory components in separate processes and
>>   managing their life cycles is not a trivial task.  In Otto's words, I
>>   believe this will inevitably become a "large chuck of custom development
>>   that has to be maintained".  Docker Compose exposes a declarative
>> interface
>>   that is much simpler in my opinion (check out
>>   https://github.com/apache/metron/blob/master/metron-
>> contrib/metron-docker/compose/docker-compose.yml
>>   as an example).  I also think our testing infrastructure will be more
>>   accessible to outside contributors because Docker is a common skill in
>> the
>>   industry.  Otherwise a contributor would have to come up to speed with
>> our
>>   custom in-memory process module before

Re: [DISCUSS] e2e test infrastructure

[Ryan Merriman]

 against
> them). I'm worried that's quick vs full dev all over again.  But without us
> being able to easily kill one because half of tests depend on one and half
> on the other.
>
> On Wed, Nov 29, 2017 at 1:22 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > What about just spinning up each of the components in their own process?
> > It's even lighter weight, doesn't have the complications for HDFS (you
> can
> > use the local FS easily, for example), and doesn't have any issues around
> > ports and port mapping with the containers.
> >
> > On Tue, Nov 28, 2017 at 3:48 PM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > As long as there is not a large chuck of custom deployment that has to
> be
> > > maintained docker sounds ideal.
> > > I would like to understand what it would take to create the docker e2e
> > env.
> > >
> > >
> > >
> > > On November 28, 2017 at 17:27:13, Ryan Merriman (merrim...@gmail.com)
> > > wrote:
> > >
> > > Currently the e2e tests for our Alerts UI depends on full dev being up
> > and
> > > running. This is not a good long term solution because it forces a
> > > contributor/reviewer to run the tests manually with full dev running.
> It
> > > would be better if the backend services could be made available to the
> > e2e
> > > tests while running in Travis. This would allow us to add the e2e tests
> > to
> > > our automated build process.
> > >
> > > What is the right approach? Here are some options I can think of:
> > >
> > > - Use the in-memory components we use for the backend integration tests
> > > - Use a Docker approach
> > > - Use mock components designed for the e2e tests
> > >
> > > Mocking the backend would be my least favorite option because it would
> > > introduce a complex module of code that we have to maintain.
> > >
> > > The in-memory approach has some shortcomings but we may be able to
> solve
> > > some of those by moving components to their own process and spinning
> them
> > > up/down at the beginning/end of tests. Plus we are already using them.
> > >
> > > My preference would be Docker because it most closely mimics a real
> > > installation and gives you isolation, networking and dependency
> > management
> > > features OOTB. In many cases Dockerfiles are maintained and published
> by
> > a
> > > third party and require no work other than some setup like loading data
> > or
> > > templates/schemas. Elasticsearch is a good example.
> > >
> > > I believe we could make any of these approaches work in Travis. What
> does
> > > everyone think?
> > >
> > > Ryan
> > >
> >
>

Re: [GitHub] metron issue #852: METRON-1239 Drop extra dev environments

[Ryan Merriman]

I wrote the ReadMeUtils class a long time ago as a way to make documenting
the REST endpoints easier.  The Controller class methods are annotated so
that endpoint documentation is displayed in Swagger but it is also
duplicated in the README.   It seemed like a good idea at the time to
provide a utility to make this easier so that you only had to document in
one place.  It was actually helpful (to me anyways) when we first
introduced a large number of REST endpoints and saved some tedious
copy/pasting.

In hindsight, there was no way of enforcing that we use the utility along
with the `README.vm` template. People intuitively edit the README.md
instead and the template quickly became stale.  Eventually I got tired of
keeping the template in sync so I stopped using it as well.  This class can
(and should) be safely removed.

On Wed, Nov 29, 2017 at 7:16 AM, justinleet  wrote:

> Github user justinleet commented on the issue:
>
> https://github.com/apache/metron/pull/852
>
> Glancing briefly, it looks like `ReadMeUtils` uses it as a template
> for the metron-rest README.md.  Just running the main in there overwrites
> the metron-rest README.md.  Which seems very odd, given that `ReadMeUtils`
> is in the test package.
>
> There seems to be no documentation of this class, or its purpose, and
> I didn't dig enough into the code to figure it out. Even not knowing the
> details and assuming I'm not misreading what's happening, I don't like that
> there's an expectation of editing a `README.vm` file, then running a
> program to produce the final output `README.md`.  `README.md` can vary
> independently of `README.vm`.  And it already has.
>
> It's outside the scope of this ticket, but at minimum, that class
> needs to be moved out of test, it needs to be actually documented what the
> purpose of it is, the steps to use it, etc. Right now, though, unless
> someone comes up with a compelling reason not to, I'm in favor of killing
> it entirely. I don't ever see that being properly managed, even if it does
> have some utility built in.
>
>
> ---
>

[DISCUSS] e2e test infrastructure

[Ryan Merriman]

Currently the e2e tests for our Alerts UI depends on full dev being up and
running.  This is not a good long term solution because it forces a
contributor/reviewer to run the tests manually with full dev running.  It
would be better if the backend services could be made available to the e2e
tests while running in Travis.  This would allow us to add the e2e tests to
our automated build process.

What is the right approach?  Here are some options I can think of:

   - Use the in-memory components we use for the backend integration tests
   - Use a Docker approach
   - Use mock components designed for the e2e tests

Mocking the backend would be my least favorite option because it would
introduce a complex module of code that we have to maintain.

The in-memory approach has some shortcomings but we may be able to solve
some of those by moving components to their own process and spinning them
up/down at the beginning/end of tests.  Plus we are already using them.

My preference would be Docker because it most closely mimics a real
installation and gives you isolation, networking and dependency management
features OOTB.  In many cases Dockerfiles are maintained and published by a
third party and require no work other than some setup like loading data or
templates/schemas.  Elasticsearch is a good example.

I believe we could make any of these approaches work in Travis.  What does
everyone think?

Ryan

Re: [DISCUSS] Upcoming Release

[Ryan Merriman]

Makes sense now.  Thanks Matt.

> On Nov 17, 2017, at 4:25 PM, Matt Foley <mfo...@hortonworks.com> wrote:
> 
> Hi Ryan,
> Yes and no.  The last release (see 
> https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced on 
> 9/19.
> Immediately after that we bumped the  of builds from master branch, 
> per https://issues.apache.org/jira/browse/METRON-1196 .  This is consistent 
> with the Release Process “Clean up” phase: “It is good practice to increment 
> the build version in master immediately after a Feature Release, so that dev 
> builds with new stuff from master cannot be mistaken for builds of the 
> release version. So, immediately after a release, increment the MINOR version 
> number (eg, with the 0.4.0 just released, set the new version number to 
> 0.4.1)” 
> (https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup
>  )
> 
> So you’re correct that the version of development builds is currently set to 
> 0.4.2.  After we actually make a release of 0.4.2, we’ll change dev build 
>  to 0.4.3 (regardless of whether we expect the following release to 
> be 0.4.3 or 0.5.0).
> 
> Hope this clarifies,
> --Matt
> 
> On 11/17/17, 1:59 PM, "Ryan Merriman" <merrim...@gmail.com> wrote:
> 
>Matt,
> 
>I think we are currently on version 0.4.2.  If that is the case would the
>next version be 0.4.3?
> 
>Ryan
> 
>>On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <ma...@apache.org> wrote:
>> 
>> (With release manager hat on)
>> 
>> The community has proposed a release of Metron in the near future,
>> focusing on Meta-alerts running in Elasticsearch.
>> Congrats on getting so many of the below already done.  At this point,
>> only METRON-1252, and the discussion of how to handle joint release of the
>> Metron bro plugin, remain as gating items for the release.  I project these
>> will be resolved next week, so let’s propose the following:
>> 
>> Sometime next week, after the last bits are done, I’ll start the release
>> process and create the release branch.
>> 
>> The proposed new version will be 0.4.2, unless there are backward
>> incompatible changes that support making it 0.5.0.
>> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
>> having Docs Text indicating so.
>> ***If anyone knows that some of the commits included since 0.4.1 introduce
>> backward incompatibility, please say so now on this thread, and mark the
>> Jira as such.***
>> 
>> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
>> below.
>> Thanks,
>> --Matt
>> 
>>METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
>> Some Records (nickwallen) closes apache/metron#832
>>METRON-1294 IP addresses are not formatted correctly in facet and
>> group results (merrimanr) closes apache/metron#827
>>METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
>> cluster (merrimanr) closes apache/metron#826
>>METRON-1290 Only first 10 alerts are update when a MetaAlert status is
>> changed to inactive (justinleet) closes apache/metron#842
>>METRON-1311 Service Check Should Check Elasticsearch Index Templates
>> (nickwallen) closes apache/metron#839
>>METRON-1289 Alert fields are lost when a MetaAlert is created
>> (merrimanr) closes apache/metron#824
>>METRON-1309 Change metron-deployment to pull the plugin from
>> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
>>METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
>> closes apache/metron#838
>>METRON-1275: Fix Metron Documentation closes
>> apache/incubator-metron#833
>>METRON-1295 Unable to Configure Logging for REST API (nickwallen)
>> closes apache/metron#828
>>METRON-1307 Force install of java8 since java9 does not appear to work
>> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
>>METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
>> cestella) closes apache/incubator-metron#829
>>METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
>> closes apache/metron#821
>>METRON-1287 Full Dev Fails When Installing EPEL Repository
>> (nickwallen) closes apache/metron#820
>>METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
>> page (iraghumitra via merrimanr) closes apache/metron#819
>>METRON-1283 Install Elasticsearch template as a part of the mpack
>> startup scripts (anandsubbu via nickwallen) closes apache

Re: [DISCUSS] Upcoming Release

[Ryan Merriman]

Matt,

I think we are currently on version 0.4.2.  If that is the case would the
next version be 0.4.3?

Ryan

On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley  wrote:

> (With release manager hat on)
>
> The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
> Congrats on getting so many of the below already done.  At this point,
> only METRON-1252, and the discussion of how to handle joint release of the
> Metron bro plugin, remain as gating items for the release.  I project these
> will be resolved next week, so let’s propose the following:
>
> Sometime next week, after the last bits are done, I’ll start the release
> process and create the release branch.
>
> The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
> having Docs Text indicating so.
> ***If anyone knows that some of the commits included since 0.4.1 introduce
> backward incompatibility, please say so now on this thread, and mark the
> Jira as such.***
>
> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
> below.
> Thanks,
> --Matt
>
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> closes apache/metron#821
> METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (iraghumitra via merrimanr) closes apache/metron#819
> METRON-1283 Install Elasticsearch template as a part of the mpack
> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> METRON-1254: Conditionals as map keys do not function in Stellar
> closes apache/incubator-metron#801
> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> closes apache/metron#805
> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
> METRON-1270: fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client closes
> apache/incubator-metron#810
> METRON-1272 Hide child alerts from searches and grouping if they
> belong to meta alerts (justinleet) closes apache/metron#811
> METRON-1224 Add time range selection to search control (iraghumitra
> via james-sirota) closes apache/metron#796
> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> justinleet) closes apache/metron#816
> METRON-1243: Add a REST endpoint which allows us to get a list of all
> indice closes apache/incubator-metron#797
> METRON-1196 Increment master version number to 0.4.2 for on-going
> development (mattf-horton) closes apache/metron#767
> METRON-1278 Strip Build Status widget from root README.md
> in site-book build (mattf-horton) closes apache/metron#815
> METRON-1274 Master has failure in StormControllerIntegrationTest
> (merrimanr) closes apache/metron#813
> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
> apache/metron#809
> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
> closes apache/metron#804
> METRON-1251: Typo and formatting fixes for metron-rest README closes
> apache/incubator-metron#800
> METRON-1241: Enable the REST API to use a cache for the zookeeper
> config similar to the Bolts closes apache/incubator-metron#795
> METRON-1267 Alerts UI returns a 404 when 

Re: [GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions

[Ryan Merriman]

Which 2 files are in /home/travis/build/apache/
metron/metron-stellar/stellar-common/target/rat.txt?  Do you get this same
error when you run mvn apache-rat:check locally?  You will need to either
add licenses to those 2 files or add exclusions in /metron/pom.xml if they
should in fact be excluded (test data for example).

Ryan

On Wed, Nov 1, 2017 at 4:56 PM, jasper-k  wrote:

> Github user jasper-k commented on the issue:
>
> https://github.com/apache/metron/pull/823
>
> The Travis build failed with an error relating to licensing and Rat. I
> don't know how to tackle this problem. The error points to the new files I
> added in this PR. Locally I could suppress the error with the extra MVN
> switch "-Drat.numUnapprovedLicenses=100"
>
> Error:
> Too many files with unapproved license: 2 See RAT report in:
> /home/travis/build/apache/metron/metron-stellar/stellar-
> common/target/rat.txt
>
> Other then this the build and the tests are fine.
> How can I solve this?
>
>
> ---
>

Re: [DISCUSS] Build broken due to transitive dependencies

[Ryan Merriman]

Haha you're too kind Laurens.  Glad that was it.

On Thu, Oct 19, 2017 at 8:30 AM, Nick Allen <n...@nickallen.org> wrote:

> Thanks for testing the suggested fix, Laurens.  I created METRON-1265 so we
> can get this dependency doc'd.
>
> https://issues.apache.org/jira/browse/METRON-1265
>
> On Wed, Oct 18, 2017 at 11:55 AM, Laurens Vets <laur...@daemon.be> wrote:
>
> > I was hesitant to believe Ryan that this was a compiler issue, but I
> > upgraded my compiler on CentOS 6 to 4.9.2 and the build worked on the
> first
> > try... Lesson learned: Never question Ryan again!
> >
> > How to upgrade compiler on CentOS 6:
> >
> > $ sudo yum install centos-release-scl
> > $ sudo yum install devtoolset-3-toolchain
> > $ scl enable devtoolset-3 bash
> > $ 
> >
> > On 2017-10-13 11:12, Ryan Merriman wrote:
> >
> >> We recently ran into this and the cause was an old C++ compiler version.
> >> It wants a compiler that has support for C++11:
> >> https://gcc.gnu.org/projects/cxx-status.html#cxx11.
> >>
> >> On Fri, Oct 13, 2017 at 1:00 PM, Laurens Vets <laur...@daemon.be>
> wrote:
> >>
> >> ...
> >>> [INFO] --- frontend-maven-plugin:1.3:npm (ng build) @ metron-config ---
> >>> [DEBUG] Configuring mojo com.github.eirslett:frontend-m
> >>> aven-plugin:1.3:npm
> >>> from plugin realm ClassRealm[plugin>com.github.e
> >>> irslett:frontend-maven-plugin:1.3, parent:
> >>> sun.misc.Launcher$AppClassLoad
> >>> er@70dea4e]
> >>> [DEBUG] Configuring mojo 'com.github.eirslett:frontend-
> >>> maven-plugin:1.3:npm'
> >>> with basic configurator -->
> >>> [DEBUG]   (f) arguments = run build
> >>> [DEBUG]   (f) npmInheritsProxyConfigFromMaven = false
> >>> [DEBUG]   (f) project = MavenProject: org.apache.metron:metron-confi
> >>> g:0.4.1
> >>> @ /root/metron/metron-interface/metron-config/pom.xml
> >>> [DEBUG]   (f) repositorySystemSession = org.eclipse.aether.DefaultRepo
> >>> sitorySystemSession@e883a51
> >>> [DEBUG]   (f) session = org.apache.maven.execution.
> MavenSession@2aaefbd
> >>> [DEBUG]   (f) skip = false
> >>> [DEBUG]   (f) skipTests = true
> >>> [DEBUG]   (f) workingDirectory = /root/metron/metron-interface/
> >>> metron-config
> >>> [DEBUG]   (f) execution = com.github.eirslett:frontend-m
> >>> aven-plugin:1.3:npm
> >>> {execution: ng build}
> >>> [DEBUG] -- end configuration --
> >>> [INFO] npm not inheriting proxy config from Maven
> >>> [INFO] Running 'npm run build' in /root/metron/metron-interface/
> >>> metron-config
> >>> [INFO]
> >>> [INFO] > metron-management-ui@0.4.1 build
> /root/metron/metron-interface/
> >>> metron-config
> >>> [INFO] > ./node_modules/angular-cli/bin/ng build -prod
> >>> [INFO]
> >>> [INFO] Cannot find module 'tough-cookie'
> >>> [INFO] Error: Cannot find module 'tough-cookie'
> >>> [INFO] at Function.Module._resolveFilename (module.js:440:15)
> >>> [INFO] at Function.Module._load (module.js:388:25)
> >>> [INFO] at Module.require (module.js:468:17)
> >>> [INFO] at require (internal/module.js:20:19)
> >>> [INFO] at Object. (/root/metron/metron-interface
> >>> /metron-config/node_modules/request/lib/cookies.js:3:13)
> >>> [INFO] at Module._compile (module.js:541:32)
> >>> [INFO] at Object.Module._extensions..js (module.js:550:10)
> >>> [INFO] at Module.load (module.js:458:32)
> >>> [INFO] at tryModuleLoad (module.js:417:12)
> >>> [INFO] at Function.Module._load (module.js:409:3)
> >>> [INFO] at Module.require (module.js:468:17)
> >>> [INFO] at require (internal/module.js:20:19)
> >>> [INFO] at Object. (/root/metron/metron-interface
> >>> /metron-config/node_modules/request/index.js:18:15)
> >>> [INFO] at Module._compile (module.js:541:32)
> >>> [INFO] at Object.Module._extensions..js (module.js:550:10)
> >>> [INFO] at Module.load (module.js:458:32)
> >>> [INFO] at tryModuleLoad (module.js:417:12)
> >>> [INFO] at Function.Module._load (module.js:409:3)
> >>> [INFO] at Module.require (module.js:468:17)
> >>> [INFO] at require (internal/module.js:20:19)
> >>> [INFO] at Leek._enqueue (/root/metron/metron-interface
&g

Re: [DISCUSS] Build broken due to transitive dependencies

[Ryan Merriman]

We recently ran into this and the cause was an old C++ compiler version.
It wants a compiler that has support for C++11:
https://gcc.gnu.org/projects/cxx-status.html#cxx11.

On Fri, Oct 13, 2017 at 1:00 PM, Laurens Vets  wrote:

> ...
> [INFO] --- frontend-maven-plugin:1.3:npm (ng build) @ metron-config ---
> [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.3:npm
> from plugin realm ClassRealm[plugin>com.github.e
> irslett:frontend-maven-plugin:1.3, parent: sun.misc.Launcher$AppClassLoad
> er@70dea4e]
> [DEBUG] Configuring mojo 'com.github.eirslett:frontend-maven-plugin:1.3:npm'
> with basic configurator -->
> [DEBUG]   (f) arguments = run build
> [DEBUG]   (f) npmInheritsProxyConfigFromMaven = false
> [DEBUG]   (f) project = MavenProject: org.apache.metron:metron-config:0.4.1
> @ /root/metron/metron-interface/metron-config/pom.xml
> [DEBUG]   (f) repositorySystemSession = org.eclipse.aether.DefaultRepo
> sitorySystemSession@e883a51
> [DEBUG]   (f) session = org.apache.maven.execution.MavenSession@2aaefbd
> [DEBUG]   (f) skip = false
> [DEBUG]   (f) skipTests = true
> [DEBUG]   (f) workingDirectory = /root/metron/metron-interface/
> metron-config
> [DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.3:npm
> {execution: ng build}
> [DEBUG] -- end configuration --
> [INFO] npm not inheriting proxy config from Maven
> [INFO] Running 'npm run build' in /root/metron/metron-interface/
> metron-config
> [INFO]
> [INFO] > metron-management-ui@0.4.1 build /root/metron/metron-interface/
> metron-config
> [INFO] > ./node_modules/angular-cli/bin/ng build -prod
> [INFO]
> [INFO] Cannot find module 'tough-cookie'
> [INFO] Error: Cannot find module 'tough-cookie'
> [INFO] at Function.Module._resolveFilename (module.js:440:15)
> [INFO] at Function.Module._load (module.js:388:25)
> [INFO] at Module.require (module.js:468:17)
> [INFO] at require (internal/module.js:20:19)
> [INFO] at Object. (/root/metron/metron-interface
> /metron-config/node_modules/request/lib/cookies.js:3:13)
> [INFO] at Module._compile (module.js:541:32)
> [INFO] at Object.Module._extensions..js (module.js:550:10)
> [INFO] at Module.load (module.js:458:32)
> [INFO] at tryModuleLoad (module.js:417:12)
> [INFO] at Function.Module._load (module.js:409:3)
> [INFO] at Module.require (module.js:468:17)
> [INFO] at require (internal/module.js:20:19)
> [INFO] at Object. (/root/metron/metron-interface
> /metron-config/node_modules/request/index.js:18:15)
> [INFO] at Module._compile (module.js:541:32)
> [INFO] at Object.Module._extensions..js (module.js:550:10)
> [INFO] at Module.load (module.js:458:32)
> [INFO] at tryModuleLoad (module.js:417:12)
> [INFO] at Function.Module._load (module.js:409:3)
> [INFO] at Module.require (module.js:468:17)
> [INFO] at require (internal/module.js:20:19)
> [INFO] at Leek._enqueue (/root/metron/metron-interface
> /metron-config/node_modules/leek/lib/leek.js:60:30)
> [INFO] at Leek.track (/root/metron/metron-interface
> /metron-config/node_modules/leek/lib/leek.js:87:15)
> [INFO] at Class.Command.validateAndRun (/root/metron/metron-interface
> /metron-config/node_modules/angular-cli/lib/models/command.js:119:18)
> [INFO] at /root/metron/metron-interface/metron-config/node_modules/ang
> ular-cli/lib/cli/cli.js:86:22
> [INFO] at tryCatch (/root/metron/metron-interface
> /metron-config/node_modules/rsvp/dist/lib/rsvp/-internal.js:198:12)
> [INFO] at invokeCallback (/root/metron/metron-interface
> /metron-config/node_modules/rsvp/dist/lib/rsvp/-internal.js:211:13)
> [INFO] at /root/metron/metron-interface/metron-config/node_modules/rsv
> p/dist/lib/rsvp/then.js:26:14
> [INFO] at flush (/root/metron/metron-interface
> /metron-config/node_modules/rsvp/dist/lib/rsvp/asap.js:80:5)
> [INFO] at _combinedTickCallback (internal/process/next_tick.js:67:7)
> [INFO] at process._tickCallback (internal/process/next_tick.js:98:9)
> [ERROR]
> [ERROR] npm ERR! Linux 2.6.32-696.13.2.el6.x86_64
> [ERROR] npm ERR! argv "/root/metron/metron-interface/metron-config/node/node"
> "/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
> "run" "build"
> [ERROR] npm ERR! node v6.2.0
> [ERROR] npm ERR! npm  v3.8.9
> [ERROR] npm ERR! code ELIFECYCLE
> [ERROR] npm ERR! metron-management-ui@0.4.1 build:
> `./node_modules/angular-cli/bin/ng build -prod`
> [ERROR] npm ERR! Exit status 1
> [ERROR] npm ERR!
> [ERROR] npm ERR! Failed at the metron-management-ui@0.4.1 build script
> './node_modules/angular-cli/bin/ng build -prod'.
> [ERROR] npm ERR! Make sure you have the latest version of node.js and npm
> installed.
> [ERROR] npm ERR! If you do, this is most likely a problem with the
> metron-management-ui package,
> [ERROR] npm ERR! not with npm itself.
> [ERROR] npm ERR! Tell the author that this fails on your system:
> [ERROR] npm ERR! 

[DISCUSS] How should Management UI save changes?

[Ryan Merriman]

Recently @nickwallen brought up some good points about the usability of the
Management UI here:
https://github.com/apache/metron/pull/737#issuecomment-330632113.  The
issues he brings up apply to all child panels so I think it makes sense to
agree on a common approach and apply it to all of them.

Most child panels have a save button that saves changes to the local
(browser) copy of the config.  The save button on the primary panel
persists the changes to zookeeper and closes all panels.  Should we change
the buttons or button text?  What should the different buttons do?  One
idea could be to just skip saving to a local copy, meaning hitting the save
button persists changes in that panel to zookeeper.  Another idea could be
to get rid of the save buttons on child panels and changes to the form
would immediately update the local copy.  In this case we would likely need
an indicator that there are changes to be saved (or should we have that no
matter what?).  Other ideas?

There is also the issue of being able to discard changes and go back to
what they were before.  Now you can close a child or primary panel but you
discard all changes in that panel and all changes period in the case of the
primary panel.  We could be to expose a revert link or button for each form
input (a lot of work probably).  Other ideas?

Ryan

Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions

[Ryan Merriman]

I will attempt to clarify parsers vs sensors.  Parsers refer to concrete
parser classes and sensors refer to configuration + one of the parser
classes (with parser class being defined in the configuration).  The
architecture was designed so that a parser class can be made dynamic and
behave differently depending on configuration.  This allows multiple use
cases to be handled by a single parser class with different
configurations.  A good example of this (and the primary reason this
pattern was chosen) is the GrokParser.  This parser can handle multiple
sensors even though it is the same parser class in each instance.

What are some reasons we would want to use the bundle system to deploy a
configuration only sensor?  From what I understand creating a bundle is not
a simple process and requires manually editing configuration files (correct
me if I am wrong).  The whole reason we created the management UI was to
allow people to create sensors without having to go through this difficult,
error-prone process that initially made Metron hard to use.  You can create
sensors right now with the management UI as long as the necessary parser
class is available.  You mention test coverage but I would argue the unit
and integration tests for that parser class should cover the edge cases of
configuration.  What happens if I change the configuration?  Do I have to
also go update the unit/integration tests for that parser?  Do I have to
rebuild and reinstall/update the parser?

This bundle system is absolutely necessary for parser classes but for me it
feels a little too tightly coupled to configuration.  Not saying
configuration shouldn't be involved at all, there is definitely value in
providing a default/bootstrap configuration for a parser class for someone
to start with.  Creating a parser class or a bundle is a complex task
suited for an engineer (with the help of the community).  A SOC analyst
should only have to understand the various configuration options to create
and maintain a sensor and be able to make these changes through a user
interface.

This feature branch moves pretty fast so it's entirely possible my
knowledge is incorrect or outdated.  Don't hesitate to correct anything I'm
missing or have incorrect.  My main concern is that adding this feature
should make Metron easier to use and not harder.  Is it possible these 2
options could coexist?

Ryan



On Wed, Sep 20, 2017 at 9:21 AM, Otto Fowler 
wrote:

> Simon, I’m sorry, I may not have answered your question.
> I use parser and sensors as the same thing, but from what you say I think I
> mean parser.
>
>
> On September 20, 2017 at 10:08:25, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So,
> The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if
> it is defined somewhere and I have missed it, please point me in the right
> direction.
> While I don’t want to fork the discussion, the question is where you find
> it so to speak, so about bundles and configurations.
>
> With the extension system you have two options for ‘config’ only
> parser/sensor installs.
>
> 1.  The previous mechanism of creating the configurations and manually
> pushing them zookeeper and possibly patterns still works, although they
> will not be managed as extensions.  This I believe is a feature gap that
> already exists and I did not address it as part of this effort.
> 2. Create a new parser from the archetype, but remove all the src/main code
> and make it configuration only.  This I think should be the recommended way
> to create configuration only parsers, since such parsers will still have
> the unit and integration tests.  Flows where you start with the archetype
> and refine through tests or manually deploy, refine and then build from the
> archetype can be imagined.
>
>
> The main question for this thread however, is should metron parsers and
>  3rd party parsers be treated the same -> they are all extensions,
> manageable through the extensions ui.
>
> I can demo for you whenever you are free if you don’t want to wait for the
> community demo.
>
>
> On September 20, 2017 at 09:52:56, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Otto,
>
> Can you just clarify what you mean by parsers in this instance. To my mind
> parsers in metron are be classes, and do not have any configuration
> settings. Instances of parsers are referred to in the ui as sensors, and
> are essentially concrete instances of parsers and as such do have config.
>
> The parser vs sensor distinction feels like a valuable one to make here.
> I'd really like a clearer understanding of the role of bundles in config,
> and how we maintain the ability to control config without the need for
> files in the bundle.
>
> Maybe some samples / a demo would really clear this up, but to be honest
> right now I'm a little confused about how this would work for an average
> SOC team who do not have maven available.
>
> Thanks,
> Simon
>
> Sent from my 

Re: [DISCUSS] Metron release 0.4.1

[Ryan Merriman]

Matt,

This was committed a few hours. I think you saw it but just wanted to make sure.

Ryan

> On Sep 7, 2017, at 11:26 AM, Matt Foley <ma...@apache.org> wrote:
> 
> Okay.  Please ping when committed.
> Also, any input on https://issues.apache.org/jira/browse/METRON-1163 ?
> 
> On 9/7/17, 7:39 AM, "Ryan Merriman" <merrim...@gmail.com> wrote:
> 
>Matt,
> 
>We recently found a bug that's breaking certain features in the management
>UI.  The fixes are still in review (
>https://github.com/apache/metron/pull/729 and
>https://github.com/apache/metron/pull/730) and should make it in soon.  It
>would be good to include these if possible.
> 
>Ryan
> 
>>On Thu, Sep 7, 2017 at 12:23 AM, Matt Foley <ma...@apache.org> wrote:
>> 
>> I’ve got a blocker bug, https://issues.apache.org/jira/browse/METRON-1163
>> , RAT failures for metron-interface/metron-alerts.  A comment in the jira
>> suggests a way to address it, but someone familiar with the code should
>> look at it.
>> 
>> 
>> 
>> Raghu, would you be able to take a look?
>> 
>> 
>> 
>> Thanks,
>> 
>> --Matt
>> 
>> 
>> 
>> From: Matt Foley <mfo...@hortonworks.com> on behalf of Matt Foley <
>> ma...@apache.org>
>> Date: Tuesday, September 5, 2017 at 10:01 AM
>> To: "dev@metron.apache.org" <dev@metron.apache.org>
>> Subject: Re: [DISCUSS] Metron release 0.4.1
>> 
>> 
>> 
>> Great, working on it!
>> 
>> 
>> 
>> From: Nick Allen <n...@nickallen.org>
>> Date: Tuesday, September 5, 2017 at 8:00 AM
>> To: Casey Stella <ceste...@gmail.com>, "zeo...@gmail.com" <
>> zeo...@gmail.com>
>> Cc: Anand Subramanian <asubraman...@hortonworks.com>, Matt Foley <
>> mfo...@hortonworks.com>, "dev@metron.apache.org" <dev@metron.apache.org>
>> Subject: Re: [DISCUSS] Metron release 0.4.1
>> 
>> 
>> 
>> All set here.  Let's get this shipped!
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Tue, Sep 5, 2017 at 10:44 AM Casey Stella <ceste...@gmail.com> wrote:
>> 
>> me too
>> 
>> 
>> 
>> On Tue, Sep 5, 2017 at 10:43 AM, zeo...@gmail.com <zeo...@gmail.com>
>> wrote:
>> 
>> All set from my perspective.
>> 
>> 
>> 
>> Jon
>> 
>> 
>> 
>> On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
>> asubraman...@hortonworks.com> wrote:
>> 
>> Hello Matt,
>> 
>> Simon's pull request supersedes mine (METRON-1139 /
>> https://github.com/apache/metron/pull/722) and is already merged into
>> master.
>> 
>> Thanks,
>> Anand
>> 
>> 
>> 
>>> On 9/1/17, 12:41 AM, "Matt Foley" <mfo...@hortonworks.com> wrote:
>>> 
>>> Please mark them 0.4.1, as that’s what the community says we want to call
>> the upcoming release, and everything that’s there when I throw the switch
>> will be included.
>>> 
>>> Jon and Anand, will they be in by end/day Friday?
>>> Thanks,
>>> --Matt
>>> 
>>> On 8/31/17, 7:45 AM, "Nick Allen" <n...@nickallen.org> wrote:
>>> 
>>>   Matt, et al - For JIRAs that are going into master, should we be
>> marking
>>>   these as "Next + 1" or "0.4.1" ?
>>> 
>>>>   On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com <zeo...@gmail.com>
>>> wrote:
>>> 
>>>> Can I advocate to get METRON-1129 in the RC, and throw in a second
>> vote for
>>>> METRON-1134?  Both in an attempt to better support of prod/offline
>> use.
>>>> Happy to provide testing cycles for the former.
>>>> 
>>>> Jon
>>>> 
>>>> On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
>>>> asubraman...@hortonworks.com> wrote:
>>>> 
>>>>> Hi Matt,
>>>>> 
>>>>> This defect needs to be included as a follow-on to METRON-1122:
>>>>> * METRON-1141 (https://github.com/apache/metron/pull/723)
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Anand
>>>>> 
>>>>> 
>>>>> 
>>>>> On 8/30/17, 8:57 PM, "Michael Miklavcic" <
>> michael.miklav...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> I have some work around fixing how we handle config

Re: [DISCUSS] Metron release 0.4.1

[Ryan Merriman]

Matt,

We recently found a bug that's breaking certain features in the management
UI.  The fixes are still in review (
https://github.com/apache/metron/pull/729 and
https://github.com/apache/metron/pull/730) and should make it in soon.  It
would be good to include these if possible.

Ryan

On Thu, Sep 7, 2017 at 12:23 AM, Matt Foley  wrote:

> I’ve got a blocker bug, https://issues.apache.org/jira/browse/METRON-1163
> , RAT failures for metron-interface/metron-alerts.  A comment in the jira
> suggests a way to address it, but someone familiar with the code should
> look at it.
>
>
>
> Raghu, would you be able to take a look?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Matt Foley  on behalf of Matt Foley <
> ma...@apache.org>
> Date: Tuesday, September 5, 2017 at 10:01 AM
> To: "dev@metron.apache.org" 
> Subject: Re: [DISCUSS] Metron release 0.4.1
>
>
>
> Great, working on it!
>
>
>
> From: Nick Allen 
> Date: Tuesday, September 5, 2017 at 8:00 AM
> To: Casey Stella , "zeo...@gmail.com" <
> zeo...@gmail.com>
> Cc: Anand Subramanian , Matt Foley <
> mfo...@hortonworks.com>, "dev@metron.apache.org" 
> Subject: Re: [DISCUSS] Metron release 0.4.1
>
>
>
> All set here.  Let's get this shipped!
>
>
>
>
>
>
>
> On Tue, Sep 5, 2017 at 10:44 AM Casey Stella  wrote:
>
> me too
>
>
>
> On Tue, Sep 5, 2017 at 10:43 AM, zeo...@gmail.com 
> wrote:
>
> All set from my perspective.
>
>
>
> Jon
>
>
>
> On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> Hello Matt,
>
> Simon's pull request supersedes mine (METRON-1139 /
> https://github.com/apache/metron/pull/722) and is already merged into
> master.
>
> Thanks,
> Anand
>
>
>
> On 9/1/17, 12:41 AM, "Matt Foley"  wrote:
>
> >Please mark them 0.4.1, as that’s what the community says we want to call
> the upcoming release, and everything that’s there when I throw the switch
> will be included.
> >
> >Jon and Anand, will they be in by end/day Friday?
> >Thanks,
> >--Matt
> >
> >On 8/31/17, 7:45 AM, "Nick Allen"  wrote:
> >
> >Matt, et al - For JIRAs that are going into master, should we be
> marking
> >these as "Next + 1" or "0.4.1" ?
> >
> >On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com 
> wrote:
> >
> >> Can I advocate to get METRON-1129 in the RC, and throw in a second
> vote for
> >> METRON-1134?  Both in an attempt to better support of prod/offline
> use.
> >> Happy to provide testing cycles for the former.
> >>
> >> Jon
> >>
> >> On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
> >> asubraman...@hortonworks.com> wrote:
> >>
> >> > Hi Matt,
> >> >
> >> > This defect needs to be included as a follow-on to METRON-1122:
> >> > * METRON-1141 (https://github.com/apache/metron/pull/723)
> >> >
> >> >
> >> > Thanks,
> >> > Anand
> >> >
> >> >
> >> >
> >> > On 8/30/17, 8:57 PM, "Michael Miklavcic" <
> michael.miklav...@gmail.com>
> >> > wrote:
> >> >
> >> > >I have some work around fixing how we handle config with Ambari
> that I'd
> >> > >like to see go in. No PR yet, but coming soon. I expect to have
> this by
> >> > the
> >> > >RC deadline.
> >> > >
> >> > >Mike
> >> > >
> >> > >On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen 
> wrote:
> >> > >
> >> > >> The following PRs are usability enhancements for the
> Profiler.  They
> >> are
> >> > >> fairly simple and I think are very helpful for
> troubleshooting.  I
> >> don't
> >> > >> want to hold up the release, but it would be a "nice to have"
> to get
> >> > these
> >> > >> in.
> >> > >>
> >> > >> If anyone has cycles, I would appreciate some reviews of these
> PRs.
> >> > >> https://github.com/apache/metron/pull/721
> >> > >> https://github.com/apache/metron/pull/716
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Tue, Aug 29, 2017 at 8:59 PM Matt Foley 
> wrote:
> >> > >>
> >> > >> > Okay, just to be clear, you’re requesting we wait until
> Friday, if
> >> > >> > necessary, to cut an RC with 717 in it?
> >> > >> > Thanks,
> >> > >> > --Matt
> >> > >> >
> >> > >> > On 8/29/17, 11:45 AM, "Casey Stella" 
> wrote:
> >> > >> >
> >> > >> > 709 is in and 717 is under concerted review by Otto.
> I'd like
> >> to
> >> > see
> >> > >> > it in
> >> > >> > by Friday.
> >> > >> >
> >> > >> > On Tue, Aug 29, 2017 at 1:35 PM, Matt Foley <
> ma...@apache.org>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Hi all,
> >> > >> > > Thanks for your inputs.  The three PRs Nick mentioned
> have
> >> been
> >> 

Re: REST and ManagementUI : Grok

[Ryan Merriman]

Check out line 336 in
https://github.com/apache/metron/blob/master/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts.
That's where the check is done to determine if a grok parser has been
selected.  I would also check the browser console for errors.

On Thu, Aug 17, 2017 at 10:51 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> That is the “I have a branch where I have broken that” part comes in ;)
>
> Actually I’m working on making HDFS actually work, because it doesn’t from
> the
> rule loading perspective.  We ( mmiklavcic and I ) thing everything has
> been loading
> from class path because the grokpath doesn’t resolve on hdfs, it is
> missing the /apps/metron part.
>
> I have fix all of that, and made it so the parsers load grok from hdfs
> correctly now, but the REST
> and thus the GUI are not working now.
>
> When I create a new parser config in the ui (+) the Grok pattern
> label/button disappear the moment i select a parser type, even grok.
>
>
>
>
> On August 17, 2017 at 11:36:24, Ryan Merriman (merrim...@gmail.com) wrote:
>
> The UI shows the Grok editor when you select the GrokParser as the parser
> type. Did the package of the GrokParser change by chance? Sounds like
> this is a bug to me.
>
> You can view, edit, test and save grok patterns so I would say you can do
> everything in the UI . The fact that a pattern can be retrieved from the
> classpath makes it tricky though. For example, if you are looking at a
> parser that stores it's pattern in the classpath (Yaf for example) the
> path
> would be "/patterns/yaf". Once you go to save it you have to change the
> path to a path that exists in HDFS (/apps/metron/patterns/yaf) because you
> obviously can't update the pattern in the jar classpath. I regret that we
> added this (I think it was me actually) because it's caused nothing but
> confusion and doesn't add any value. I would be in favor of keeping
> patterns exclusively in HDFS.
>
> Ryan
>
> On Thu, Aug 17, 2017 at 10:25 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > Question,
> >
> > How does the UI decide, when you edit a parser, to show the grok pattern
> > field? I have branch where I have broken that, for example if I add a
> new
> > configuration, I see the grok pattern field, until I actually select
> GROK
> > as the parser type, then it goes away.
> >
> > Also, just to confirm, we don’t *do* anything with the grok patterns in
> the
> > ui, we only enter and test them or test data against them?
> >
>
>

Re: [DISCUSS] Using Yarn package manager for metron-alerts

[Ryan Merriman]

Thanks for this Raghu.  You make a pretty compelling argument.  I'm +1 on
moving to yarn.

Ryan

On Wed, Aug 16, 2017 at 3:51 PM, Nick Allen  wrote:

> It is also my understanding that
> ​there is no hard cut-over to yarn
> .
> ​After we
> introduce the yarn.lock
> ​
> ​,​
> as a developer you can choose to continue to use npm or switch to yarn.
>
> Other developers on the project can keep using npm, so you don’t need to
> > get everyone on your project to convert at the same time. The developers
> > using yarn will all get exactly the same configuration as each other, and
> > the developers using npm may get slightly different configurations, which
> > is the intended behavior of npm.
>
>
> https://yarnpkg.com/lang/en/docs/migrating-from-npm/
>
>
> ​Oh, and I just switched metron-alerts projects to yarn (as a test) and
> performed an offline install.  It was stupid simple.​
>
>
>
>
> On Wed, Aug 16, 2017 at 4:12 PM Nick Allen  wrote:
>
> > Thanks for laying this all out for us, Raghu.  Based on the built-in
> > support for offline installs and version locking, I think this is a great
> > suggestion. (However unfortunate the namespace collision might be.)
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Aug 16, 2017 at 8:51 AM RaghuMitra Kandikonda <
> > raghumitra@gmail.com> wrote:
> >
> >> I would like to start a discussion around using 'yarn' for managing
> >> dependencies for metron-alerts instead of 'npm'.
> >>
> >> This article beautifully summarizes the need of yarn and npm.
> >> (https://code.facebook.com/posts/1840075619545360)
> >>
> >> If you have read the above article you can skip the next two sections
> >> and jump to 'Additional advantages of Yarn'
> >>
> >> 
> >> 
> >> ===
> >> Why do we need a new package manager ?.
> >>
> >> While 'npm' does a good job for downloading all the required
> >> dependencies. npm always tries to download the latest and greatest
> >> versions of all these dependencies. This would create a problem in
> >> replicating the same build every time we build. Having hard coded
> >> versions in the package.json seems like a possible solution but this
> >> will prevent us from knowing that a library has been updated. In JS
> >> world the version updates are very frequent and we might be missing on
> >> some of the latest updates and some of these updates might be related
> >> to security or a cool feature we would like to have in our code base.
> >> Ex: Angular made 10 releases in last two months, bootstrap made 2
> >> releases in last two months.
> >>
> >> 
> >> 
> >> ===
> >> What is Yarn  ?.
> >>
> >> Yarn is a new age package manager that can (needs to) be installed
> >> over npm (or bower). Yarn resolves issues around versioning and
> >> non-determinism of JS dependencies by using lock files and an install
> >> algorithm that is deterministic and reliable. These lock files lock
> >> the installed dependencies to a specific version and ensure that every
> >> install results in the exact same file structure in node_modules
> >> across all machines. This kind of a locking mechanism is not available
> >> with vanilla node.
> >>
> >> 
> >> 
> >> ===
> >> Additional advantages of Yarn ?.
> >>
> >> 1.Yarn helps us to check licenses of all the frameworks we are using.
> >> (This feature is built in)
> >> 2.It will reduce the build time of UI for dev as well as in Travis as
> >> all the dependencies are cached inside '~/.config/yarn/global'
> >> 3.We can do an offline install of UI as we can zip the dependencies
> >> and supply it to Yarn instead of downloading from the internet
> >> 4.Yarn is already integrated with Travis
> >> (https://blog.travis-ci.com/2016-11-21-travis-ci-now-supports-yarn)
> >>
> >> 
> >> 
> >> ===
> >> How to migrate ?.
> >>
> >> A yarn.lock file can be created from existing package.json file and
> >> this file would be checked in.
> >>
> >> 
> >> 
> >> ===
> >> How does the process change ?.
> >>
> >> 1.All the developers would use 'npm install' so that they can get the
> >> latest versions of the dependencies.
> >> 2.The build would use 'yarn install'. ( This change would be made in
> >> metron-alerts pom.xml file )
> >> 3.When the dev notices that a new version of the library is available
> >> we can test it thoroughly and update yarn.lock file
> >>
> >> 

Re: [DISCUSS] Persisting user data

[Ryan Merriman]

Spring is JDBC-generic so I think we're good there.  Improving our docs on
this topic is being discussed in https://github.com/apache/metron/pull/646
so hopefully this will be clear once that's worked out.

Simon is correct, I found out the hard way that Hibernate is not an option
because of it's license.  I think EclipseLink would be a good alternative.
I've seen it used in other open source projects (Ambari for example) and I
was able to get it working in a POC without much effort.

On Thu, Aug 3, 2017 at 5:26 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Anything spring based is likely multi-db by definition as long as a we
> pick a good friendly ORM (not hibernate because licensing problems with
> apache, eclipselink?) But I suspect we should pick a good default and that
> that default should be postgres.
>
> > On 3 Aug 2017, at 10:24, Casey Stella <ceste...@gmail.com> wrote:
> >
> > I'd vote for a DB-based solution, but I'd argue that any solution
> shouldn't
> > be database specific (i.e. postgres), but JDBC-generic.  People and
> > organizations have very strong views regarding databases and I'd prefer
> to
> > side-step those holy wars by being agnostic.
> >
> > On Wed, Aug 2, 2017 at 9:36 PM, Ryan Merriman <merrim...@gmail.com>
> wrote:
> >
> >> Spring supports a variety of databases including Postgres.  I have no
> >> problem with using Postgres instead of MySQL.
> >>
> >> On Wed, Aug 2, 2017 at 3:32 PM, Simon Elliston Ball <
> >> si...@simonellistonball.com> wrote:
> >>
> >>> Agreed on Postgres. It's a lot easier to work with license-wise in
> apache
> >>> projects, and has a lot of the capability we need here, especially if
> we
> >>> can find a sensible ORM. Anyone got any thoughts on what would work
> >> there?
> >>>
> >>> Simon
> >>>
> >>>> On 2 Aug 2017, at 21:21, Matt Foley <ma...@apache.org> wrote:
> >>>>
> >>>> Hi Ryan,
> >>>> Zookeeper has a default (and seldom changed) max znode size of 1MB,
> but
> >>> it is “designed to store data on the order of kilobytes in size.”[1]
> And
> >>> it’s not really intended for frequently-changing data, which is okay
> >> here.
> >>> But I just included it for completeness, I’m not advocating for its use
> >>> here.
> >>>>
> >>>> I agree with you that the problem, especially because it includes
> >> shared
> >>> config, would fit well in a db.  I’d suggest you consider PostgreSQL
> >> rather
> >>> than MySQL, as postgres is built into Redhat 6 and 7, and Ambari now
> uses
> >>> it by default, so an available server might be conveniently at hand in
> >> most
> >>> deployments.  Definitely assume the user will want to use an external
> db
> >>> instance, rather than one dedicated to this use.  Conveniently Postgres
> >>> also has a native REST interface, with the usual authorization options.
> >>>>
> >>>> Never mind about Ambari Views for now.  It’s just a way to get GUI
> >>> dashboards without writing all the infrastructure for it, which as you
> >> say
> >>> is somewhat water under the bridge.
> >>>> Cheers,
> >>>> --Matt
> >>>>
> >>>> [1] https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html
> >>>>
> >>>>
> >>>>
> >>>> On 8/2/17, 12:34 PM, "Ryan Merriman" <merrim...@gmail.com> wrote:
> >>>>
> >>>>   Matt,
> >>>>
> >>>>   Thank you for the suggestions.  I forgot to include Zookeeper.  Are
> >>> there
> >>>>   any tradeoffs we should be aware of if we decide to use Zookeeper?
> >>> Are
> >>>>   there guidelines for how much data can be stored in Zookeeper?
> >>>>
> >>>>   To answer your questions:
> >>>>
> >>>>   1.  I think both use cases make sense so a combination of shared and
> >>>>   personal.
> >>>>   2.  I was planning on managing authorization in the REST layer.  For
> >>> now
> >>>>   viewer login auth (which is really REST auth) will suffice but we
> >>> might
> >>>>   consider other methods since authentication is pluggable here.
> >>>>   3.  I had not considered Ambari Views since this will support an
> >>> existing
> >>>>   

Re: [DISCUSS] Persisting user data

[Ryan Merriman]

Spring supports a variety of databases including Postgres.  I have no
problem with using Postgres instead of MySQL.

On Wed, Aug 2, 2017 at 3:32 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Agreed on Postgres. It's a lot easier to work with license-wise in apache
> projects, and has a lot of the capability we need here, especially if we
> can find a sensible ORM. Anyone got any thoughts on what would work there?
>
> Simon
>
> > On 2 Aug 2017, at 21:21, Matt Foley <ma...@apache.org> wrote:
> >
> > Hi Ryan,
> > Zookeeper has a default (and seldom changed) max znode size of 1MB, but
> it is “designed to store data on the order of kilobytes in size.”[1]  And
> it’s not really intended for frequently-changing data, which is okay here.
> But I just included it for completeness, I’m not advocating for its use
> here.
> >
> > I agree with you that the problem, especially because it includes shared
> config, would fit well in a db.  I’d suggest you consider PostgreSQL rather
> than MySQL, as postgres is built into Redhat 6 and 7, and Ambari now uses
> it by default, so an available server might be conveniently at hand in most
> deployments.  Definitely assume the user will want to use an external db
> instance, rather than one dedicated to this use.  Conveniently Postgres
> also has a native REST interface, with the usual authorization options.
> >
> > Never mind about Ambari Views for now.  It’s just a way to get GUI
> dashboards without writing all the infrastructure for it, which as you say
> is somewhat water under the bridge.
> > Cheers,
> > --Matt
> >
> > [1] https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html
> >
> >
> >
> > On 8/2/17, 12:34 PM, "Ryan Merriman" <merrim...@gmail.com> wrote:
> >
> >Matt,
> >
> >Thank you for the suggestions.  I forgot to include Zookeeper.  Are
> there
> >any tradeoffs we should be aware of if we decide to use Zookeeper?
> Are
> >there guidelines for how much data can be stored in Zookeeper?
> >
> >To answer your questions:
> >
> >1.  I think both use cases make sense so a combination of shared and
> >personal.
> >2.  I was planning on managing authorization in the REST layer.  For
> now
> >viewer login auth (which is really REST auth) will suffice but we
> might
> >consider other methods since authentication is pluggable here.
> >3.  I had not considered Ambari Views since this will support an
> existing
> >UI.  How would Ambari Views help us here?
> >
> >I will proceed initially with a saved search POC using a relational
> >database unless you think that is a bad idea or there are other better
> >options.  Hopefully an example will further the discussion.
> >
> >Ryan
> >
> >>On Wed, Jul 26, 2017 at 6:31 PM, Matt Foley <ma...@apache.org>
> wrote:
> >>
> >> There’s a couple other places you could put config info (but maybe not
> >> saved searches):
> >> -  Zookeeper
> >> -  metron-alerts-ui/config.xml or config.json  file
> >> -  the Ambari database, whichever it happens to be
> >>
> >> Questions that influence the decision include:
> >> 1. Should there be one configuration shared among users, or strictly
> >> per-user config?  Or a combination of shared and personal?
> >> 2. What security do you wish to maintain on changing those settings,
> both
> >> shared and personal?  What authentication/authorization scheme will you
> >> use?  Is viewer login auth sufficient for this?
> >> 3. Will you assume Ambari exists?  Did you consider using Ambari Views
> as
> >> the basis? (https://cwiki.apache.org/confluence/display/AMBARI/Views )
> >>
> >> On 7/26/17, 2:54 PM, "Ryan Merriman" <merrim...@gmail.com> wrote:
> >>
> >>In anticipation of METRON-988 being merged into master, there will
> be a
> >>need to persist user preferences such as UI layout, saved searches,
> >> search
> >>history, etc.  I think where and how we persist this data should be
> >>discussed in order to facilitate a design.  This data won't be large
> in
> >>scale and may or may not be relational.  The initial features I am
> >> aware of
> >>don't require a relational model but I'm sure there will be some that
> >> do in
> >>the future.  I'm also assuming this code will live in the REST
> >> application
> >>but someone correct me if there is a reason to

Re: [DISCUSS] Persisting user data

[Ryan Merriman]

Matt,

Thank you for the suggestions.  I forgot to include Zookeeper.  Are there
any tradeoffs we should be aware of if we decide to use Zookeeper?  Are
there guidelines for how much data can be stored in Zookeeper?

To answer your questions:

1.  I think both use cases make sense so a combination of shared and
personal.
2.  I was planning on managing authorization in the REST layer.  For now
viewer login auth (which is really REST auth) will suffice but we might
consider other methods since authentication is pluggable here.
3.  I had not considered Ambari Views since this will support an existing
UI.  How would Ambari Views help us here?

I will proceed initially with a saved search POC using a relational
database unless you think that is a bad idea or there are other better
options.  Hopefully an example will further the discussion.

Ryan

On Wed, Jul 26, 2017 at 6:31 PM, Matt Foley <ma...@apache.org> wrote:

> There’s a couple other places you could put config info (but maybe not
> saved searches):
> -  Zookeeper
> -  metron-alerts-ui/config.xml or config.json  file
> -  the Ambari database, whichever it happens to be
>
> Questions that influence the decision include:
> 1. Should there be one configuration shared among users, or strictly
> per-user config?  Or a combination of shared and personal?
> 2. What security do you wish to maintain on changing those settings, both
> shared and personal?  What authentication/authorization scheme will you
> use?  Is viewer login auth sufficient for this?
> 3. Will you assume Ambari exists?  Did you consider using Ambari Views as
> the basis? (https://cwiki.apache.org/confluence/display/AMBARI/Views )
>
> On 7/26/17, 2:54 PM, "Ryan Merriman" <merrim...@gmail.com> wrote:
>
> In anticipation of METRON-988 being merged into master, there will be a
> need to persist user preferences such as UI layout, saved searches,
> search
> history, etc.  I think where and how we persist this data should be
> discussed in order to facilitate a design.  This data won't be large in
> scale and may or may not be relational.  The initial features I am
> aware of
> don't require a relational model but I'm sure there will be some that
> do in
> the future.  I'm also assuming this code will live in the REST
> application
> but someone correct me if there is a reason to keep it somewhere else.
>
> I think it would be preferable to leverage something that is already
> in our
> stack and available as a dependency.  However I would not be against
> adding
> something if it really were the right tool for the job.  Assuming
> others
> agree we should stick with out current stack, I see these options:
>
>- MySQL (or other relational database)
>   - good fit for the size of data
>   - relational capabilities
>   - an ORM framework will be necessary which will increase our
>   dependencies and complexity
>- HBase
>   - client setup and code will likely be simpler and less complex
>   - limited data model
>- Elasticsearch
>   - json is a convenient data model
>   - we already store user preferences here (Kibana dashboards)
>   - we have abstracted our search engine interactions in several
> places
>   and would have to here too
>
> Elasticsearch is out for me because we view search engines as
> pluggable.  I
> think HBase would be the easiest to implement and get working but I'm
> worried we'll have similar use cases that won't be a good fit for
> HBase.
> In that case we would need to come up with an alternative persistence
> solution anyways.  I think MySQL is a good fit long term but I'm
> concerned
> about adding a heavy ORM framework.  Also, we can't use Hibernate
> because
> it is not license friendly.
>
> Does anyone have any thoughts on these options or other ideas?
>
> This requirement also brings up another topic that is outside of this
> discussion.  Should we reevaluate our authentication strategy?
> Currently
> the REST application uses JDBC for this but if we decide a different
> mechanism is better then we no longer need a relational database.  This
> might affect our decision to use MySQL for this kind of data
> persistence.
>
> Ryan
>
>
>
>

Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

[Ryan Merriman]

A couple things I would like to point out.  You can test Stellar statements
without having to send data through parser/enrichment topologies.  There is
a REST endpoint that allows you to pass in a sample message and parser
config and returns a message with Stellar statements applied.  This could
easily be expanded to enrichment configs or testing generic stellar
statements against test messages.

Moving statements to a separate file is going to require a lot of work and
will make our mechanism for managing configuration in bolts more complex.
We would have to also listen for changes in these files and reconcile which
parser/enrichment configs are affected.



On Fri, Jul 14, 2017 at 12:42 PM, Otto Fowler 
wrote:

> I think the ‘files’ should be stored in zk, and updated with the same
> mechanism.
>
> On July 14, 2017 at 13:27:36, Matt Foley (mfo...@hortonworks.com) wrote:
>
> In the abstract, this is a good idea. I see it as related to METRON-987,
> which was the first step in allowing sequences of Stellar statements (aka
> "programs" :-) ) instead of just unrelated groups of single statements.
> Your proposal lets us really work with programs as first-class entities.
>
> However, some concerns need to be resolved:
>
> 1. Syntax.
>
> Currently Stellar syntax and JSON fit neatly together. Where would be the
> cut line for file substitutions? Referencing METRON-987, would you only
> allow a file substitution where we currently allow square-bracketed Stellar
> string sequences? What about Profile config syntax, where several chunks of
> code are intimately related (hence want to be located in the same file),
> but don't all get executed at the same time? (This is not a showstopper
> question because Profile configs are usually simple and don't really need
> file substitution. The need is much greater in Enrichment.)
>
> 2. Config Updates.
>
> Currently Metron configuration is stored in ZK, but managed through Curator
> libraries. In return for considerable complexity, this gives instant update
> whenever a config changes, without effort in the BI part of the
> application. This differs sharply from file-based configuration, where
> updates in response to config changes require either a restart, an explicit
> reload command from the user, or frequent state-checking in the
> application.
>
> So currently people trying to develop a new enrichment can update the
> config, and immediately test the result, without restarting and without any
> explicit reload command. We probably want to not lose this.
>
> Rather than roll our own file pointer model, can we use JSON Pointers? Will
> they work with Curator? Both of those get into some fairly obscure
> features, that would need to be studied. It also actually relates to the
> syntax question presented above.
>
>
> On 7/14/17, 6:17 AM, "Otto Fowler"  wrote:
>
> https://issues.apache.org/jira/browse/METRON-1046
>
> I was thinking this morning that managing stellar statements in the config
> json could become, and maybe is kind of unwieldy.
> To that end, if in say a parser configuration I can refer to a ‘file’ in
> zookeeper as an alternative, we would add the capability to execute and
> manage more complex statements, and even chain multiple statements
> together.
>
> These files could be shared as well.
>
> This could be a Bad Idea™, so I thought I’d throw it out to the list.
>
> Please check out the jira, give some thought, and comment there or on the
> list or both.
>
> O
>

Re: Metron REST - Logging Config

[Ryan Merriman]

The only way I know of is to change log4j.properites.  Did you every figure
out a better way?

On Tue, Jul 11, 2017 at 2:10 PM, Nick Allen  wrote:

> How do I configure logging for Metron REST on a deployed host?
>
> Right now a log4j.properties file gets packaged into the metron-rest JAR
> itself.  Is there is an easy way that I am missing?
>

[DISCUSS] Search in REST

[Ryan Merriman]

This discussion is an attempt to clarify some questions and discuss design
decisions related to METRON-1022.

The primary purpose of METRON-1022 is to provide a foundation for building
Metron-specific Elasticsearch (or other search engine implementations)
functions in our REST application.  This translates into 3 features
provided by METRON-1022:  a common approach to setting up a
TransportClient, a search abstraction layer and a simple Elasticsearch
implementation consisting of a single function.  I believe the setup part
is fairly straightforward and doesn't require a detailed discussion.
Please chime in if I'm wrong there.

The first order of business is to all agree on an architectural approach.
How and where should we query Elasticsearch?  METRON-1022 duplicates some
functionality in METRON-990 but is architecturally different.  Instead of
the client-side code interacting directly with Elasticsearch through it's
REST api, this PR interacts with Elasticsearch through the Java api in a
Metron REST service.  I believe there are a couple of advantages to doing
it this way:

   - A Metron-specific search service in REST can be reused by other UIs
   and clients.  It would be possible to make an angular service reusable but
   that would take some work and it would only be reusable in javascript as an
   imported library and not as flexible as a service available over http.
   - Metron provides an integration testing framework for Java-based
   classes.  METRON-1022 leverages this without much additional effort.  It
   would take a lot more work to enable javascript modules to use this.
   - In my experience, the Metron community is much more comfortable with
   developing and reviewing features written in Java as opposed to
   javascript.  I think that is important for foundational pieces like this.

Some arguments to consider for keeping Elasticsearch functions in a
javascript service:

   - More efficient since there is no proxy in the middle (Metron REST
   being the proxy)
   - Eliminates the task of resolving version conflicts that comes with
   adding the Elasticsearch dependency to a Maven module although there are
   ways to make this easier

The second topic to discuss is the search abstraction.  This has been
requested several times and I think there is consensus that we need it.
METRON-1022 attempts to do this by:

   - creating model classes that represent search requests/responses
   - creating a search interface that accepts these model classes as input
   and return parameters
   - creating a controller that exposes this interface over REST
   - using Spring's IOC framework to select the correct implementation

An implementation of a search function was included in METRON-1022 as an
example.  ElasticsearchServiceImpl implements SearchService and is selected
as the implementation by default.  This could have been a separate PR but I
felt having it in context would help reviewers understand the design
pattern.

How does this relate to METRON-990?  Currently they overlap with
METRON-1022 offering a subset of the functionality in the
METRON-990 Elasticsearch service.  The idea is to first ensure METRON-990
and METRON-1022 both conform to the same search abstraction (which has been
discussed in METRON-990 feedback).  The next step would be to replace the
search service in METRON-990 to one that queries the Metron REST service
instead.  Ideally this only involves changing one class since the
abstraction is used in all the other components of METRON-990 and is
trivial since the complexity is now in Metron REST and not javascript.
Next other services (getting index metadata for example) would be converted
using the same process in incremental PRs.  Then, moving forward, all
Elasticsearch interactions would instead be developed as Metron REST
endpoints using the foundation established in METRON-1022.

This is a lot to digest so I'm happy to more detail as needed.  Interested
to hear others' thoughts and reactions.

Ryan

Re: [VOTE] Apache Metron 0.4.0 release

[Ryan Merriman]

+1 (binding)

   - Verified Keys
   - Verified mvn clean install
   - Ran full dev and performed several smoke tests including:
  - tested several REST endpoints
  - verified data in ES
  - tested Management UI
  - verified Storm topologies in Storm UI
  - verified data in Kibana


On Fri, Jun 30, 2017 at 12:24 PM, Matt Foley  wrote:

> Hey all, we need more votes!
>
> So far we have 6 +1’s (including mine) and no 0’s or -1’s.
> BUT, only two are binding, ie from PMC members.
> Rules require 3 or more PMC member +1’s.  Could one more
> member please step up and vote?
>
> Thanks!
> --Matt
>
> On 6/29/17, 3:00 PM, "James Sirota"  wrote:
>
> +1 (Binding)
>
> * Verified Keys
> * Verified mvn clean install completed successfully
> * Verified AWS install of core via Mpack
>
>
> 29.06.2017, 09:14, "Justin Leet" :
> > +1 (Non-binding)
> >
> > * Verified Keys
> > * Verified mvn clean install completed successfully
> > * Ran full dev: saw data flow through, ran a couple of the REST
> APIs, and
> > opened up and clicked through a bit of the Management API.
> > * Examined site-book and didn't see any issues
> >
> > On Thu, Jun 29, 2017 at 11:46 AM, Casey Stella 
> wrote:
> >
> >>  +1 (binding)
> >>  * Verified keys
> >>  * Verified mvn build
> >>  * Verified unit and integration tests run
> >>  * Verified license check runs
> >>  * Verified fulldev spun up with smoketest
> >>
> >>  On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian <
> >>  asubraman...@hortonworks.com> wrote:
> >>
> >>  > +1 (non-binding)
> >>  >
> >>  > * Brought up Metron stack on 12-node CentOS7 openstack cluster
> >>  > * Verify all services come up fine [PASS]
> >>  > * Bro, YAF and snort - ingest into respective kafka topics and
> write
> >>  > indices [PASS]
> >>  > * Add squid telemetry, ingest into kafka topic and write indices
> [PASS]
> >>  > * Metron YAF Zeppelin dashboard with sample ingested YAF data
> [PASS]
> >>  > * Management UI and REST Swagger UI sanity check [PASS]
> >>  >
> >>  >
> >>  > -Anand
> >>  >
> >>  >
> >>  >
> >>  >
> >>  >
> >>  > On 6/28/17, 12:06 AM, "Matt Foley"  wrote:
> >>  >
> >>  > >This is a call to vote on releasing this rc4 as “Apache Metron
> 0.4.0”.
> >>  > >(Note: this is rc4 because the release candidate needed to be
> modified
> >>  > with another commit after the rc3 tag was pushed to public.)
> >>  > >
> >>  > >Full list of changes in this release:
> >>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-
> RC4/RELEASE_NOTES
> >>  > >
> >>  > >The tag/commit to be voted upon is:
> >>  > >d52f574f8294e453ecad3871526858a0c3c2033d (tag
> apache-metron-0.4.0-rc4)
> >>  > >
> >>  > >The source archive being voted upon can be found here:
> >>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-
> >>  > RC4/apache-metron-0.4.0-rc4.tar.gz
> >>  > >and in github at:
> >>  > >https://github.com/apache/metron/tree/Metron_0.4.0
> >>  > >
> >>  > >Other release files, signatures and digests can be found here:
> >>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> >>  > >
> >>  > >The release artifacts are signed with the following key:
> >>  > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> >>  > >pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
> >>  > >Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3
> 1663
> >>  > >uid = Matthew Foley (CODE SIGNING KEY) 
> >>  > >
> >>  > >Please vote on releasing this package as Apache Metron 0.4.0.
> >>  > >When voting, please list the actions taken to verify the
> release.
> >>  > >
> >>  > >Recommended build validation and verification instructions are
> posted
> >>  > here:
> >>  > >https://cwiki.apache.org/confluence/display/METRON/
> Verifying+Builds
> >>  > >
> >>  > >This vote will be open for at least 72 hours. Please vote one
> of the
> >>  > following responses:
> >>  > >+1 Release this package as Apache Metron 0.4.0-RC4
> >>  > >0 No opinion
> >>  > >-1 Do not release this package because...
> >>  > >
> >>  > >Thank you,
> >>  > >--Matt
> >>  > >(your friendly release manager)
> >>  > >
> >>  > >
> >>  > >
> >>  >
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
>
>
>

Re: Build failures

[Ryan Merriman]

Can you confirm you're on the master branch?  I see "metron-streaming" in
your path to RestTestingUtil and that was changed a LONG time ago.  You're
likely on a really old branch.

Ryan

On Wed, Jun 28, 2017 at 3:27 PM, Vasco Yordanov 
wrote:

> Hello , I just forked from github and it seems that " Metron-Pcap_Service"
> is failing with following errors:
>
>
> [ERROR] /home/vasko/metron2/incubator-metron-fork/metron-streaming/
> Metron-Pcap_Service/src/main/java/org/apache/metron/
> pcapservice/RestTestingUtil.java:[212,5] cannot find symbol
> [ERROR] symbol:   class ResponseEntity
> [ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
> [ERROR] /home/vasko/metron2/incubator-metron-fork/metron-streaming/
> Metron-Pcap_Service/src/main/java/org/apache/metron/
> pcapservice/RestTestingUtil.java:[212,63] cannot find symbol
> [ERROR] symbol:   variable HttpMethod
> [ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
>
> Please advise ? Before I start changing pom files ,I 'd like to run this
> through you in case this is known issue. Thank you
>   From: merrimanr 
>  To: dev@metron.apache.org
>  Sent: Wednesday, June 28, 2017 4:05 PM
>  Subject: [GitHub] metron issue #620: Metron-988: UI for viewing alerts
> generated by Metron
>
> Github user merrimanr commented on the issue:
>
> https://github.com/apache/metron/pull/620
>
> Just tested again and I am able to now remove the first filter and
> properly filter on values with special characters (referrer field for
> example).  I did another pass and found some trivial issues as well as a
> few non-trivial issues and have made comments.
>
> I think more thought needs to be put into the AlertService.search and
> AlertService.pollSearch functions.  The AlertService.getAlert function is
> very clear to me:  it requires a couple of clearly named parameters and I
> expect to get an 'Alert' type object back.  The other functions in this
> service are not as clear.  The search function for example takes in a
> QueryBuilder object which provides a generic javascript object as the body
> for the post request.  Then in return the post returns an Observable with a
> generic javascript object.  So essentially Typescript isn't being used here
> when it should because it would make the search interface clearer.
>
> For example, I would prefer this function signature:
> `public search(searchRequest: SearchRequest):
> Observable`
>
> where SearchRequest and SearchResponse are model objects.  The way it
> is now it's not easy to understand what is being sent and what is expected
> back unless you've spent time tracing the search calls to where requests
> are built/response are processed and know all the source code well OR
> already has a lot of experience with the ES query syntax.
>
> The result of all this is that not having a clear contract between the
> search client/server will make developing a middle-tier more tedious.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
>
>

Re: Trying to spin up Metron in EC2: Failed

[Ryan Merriman]

That happens when you don't have the zookeeper url configured correctly.
Can you check the contents of the /etc/sysconfig/metron file on the Metron
host?

On Wed, May 17, 2017 at 1:36 PM, Laurens Vets  wrote:

> For testing purposes, I decided to spin up the default Metron AWS config.
> This resulted in a hang from ansible here:
>
> TASK [librdkafka : include] **
> **
> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
> librdkafka/tasks/main.yml:18
> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
> ompute.amazonaws.com
>
> TASK [librdkafka : Install prerequisites] **
> 
> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
> librdkafka/tasks/dependencies.yml:18
>  ESTABLISH CONNECTION
> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
> ompute.amazonaws.com
> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
> FutureWarning: CTR mode needs counter parameter, not IV
>   self._cipher = factory.new(key, *args, **kwargs)
>  EXEC ( umask 22 &&
> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
> )" && echo "$( echo 
> $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
> )" )
>  PUT /tmp/tmpwnH61y
> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum
>  EXEC /bin/sh -c
> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo 
> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 /usr/bin/python
> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum;
> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/"
> > /dev/null 2>&1'"'"''
>
> Looking in the machine logs, I see the following for Kafka and Metron REST:
>
> Kafka:
> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
> advertised.host.name = null
> metric.reporters = []
> quota.producer.default = 9223372036854775807
> offsets.topic.num.partitions = 50
> log.flush.interval.messages = 9223372036854775807
> auto.create.topics.enable = true
> controller.socket.timeout.ms = 3
> log.flush.interval.ms = null
> principal.builder.class = class org.apache.kafka.common.securi
> ty.auth.DefaultPrincipalBuilder
> replica.socket.receive.buffer.bytes = 65536
> min.insync.replicas = 1
> replica.fetch.wait.max.ms = 500
> num.recovery.threads.per.data.dir = 1
> ssl.keystore.type = JKS
> sasl.mechanism.inter.broker.protocol = GSSAPI
> default.replication.factor = 1
> ssl.truststore.password = null
> log.preallocate = false
> sasl.kerberos.principal.to.local.rules = [DEFAULT]
> fetch.purgatory.purge.interval.requests = 1
> ssl.endpoint.identification.algorithm = null
> replica.socket.timeout.ms = 3
> message.max.bytes = 100
> num.io.threads = 8
> offsets.commit.required.acks = -1
> log.flush.offset.checkpoint.interval.ms = 6
> delete.topic.enable = false
> quota.window.size.seconds = 1
> ssl.truststore.type = JKS
> offsets.commit.timeout.ms = 5000
> quota.window.num = 11
> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
> ompute.amazonaws.com:2181
> authorizer.class.name =
> num.replica.fetchers = 1
> log.retention.ms = null
> log.roll.jitter.hours = 0
> log.cleaner.enable = true
> offsets.load.buffer.size = 5242880
> log.cleaner.delete.retention.ms = 8640
> ssl.client.auth = none
> controlled.shutdown.max.retries = 3
> queued.max.requests = 500
> offsets.topic.replication.factor = 3
> log.cleaner.threads = 1
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> socket.request.max.bytes = 104857600
> ssl.trustmanager.algorithm = PKIX
> zookeeper.session.timeout.ms = 3
> log.retention.bytes = -1
> log.message.timestamp.type = CreateTime
> sasl.kerberos.min.time.before.relogin = 6
> zookeeper.set.acl = false
> connections.max.idle.ms = 60
> offsets.retention.minutes = 8640
> replica.fetch.backoff.ms = 1000
> inter.broker.protocol.version = 0.10.0-IV1
> log.retention.hours = 168
> num.partitions = 1
> broker.id.generation.enable = true
> listeners = PLAINTEXT://ec2-34-209-53-166.
> us-west-2.compute.amazonaws.com:6667
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> log.roll.ms = null
> log.flush.scheduler.interval.ms = 9223372036854775807
> ssl.cipher.suites = null
>  

Re: mvn building errors with 0.3.1

[Ryan Merriman]

Kevin, I think your error is related to npm.  Can you attach the full log
file?

On Mon, May 15, 2017 at 5:23 PM, Kevin Waterson 
wrote:

> I am getting something similar..
>
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 08:08 min
> [INFO] Finished at: 2017-05-15T13:04:00+10:00
> [INFO] Final Memory: 220M/3873M
> [INFO]
> 
> [ERROR] Failed to execute goal
> com.github.eirslett:frontend-maven-plugin:1.3:npm (ng build) on project
> metron-config: Failed to run task: 'npm run build' failed. (error code 1)
> -> [Help 1]
>
> Looking back through the code I can see
> [ERROR] at
> /home/kevin/metron/docs/incubator-metron/metron-
> interface/metron-config/node_modules/resolve/lib/async.js:24:24
> [ERROR] at FSReqWrap.oncomplete (fs.js:117:15)
> [ERROR]
> [ERROR] npm ERR! Linux 4.4.0-75-generic
> [ERROR] npm ERR! argv
> "/home/kevin/metron/docs/incubator-metron/metron-
> interface/metron-config/node/node"
> "/home/kevin/metron/docs/incubator-metron/metron-
> interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
> "run" "build"
> [ERROR] npm ERR! node v6.2.0
> [ERROR] npm ERR! npm  v3.8.9
> [ERROR] npm ERR! code ELIFECYCLE
> [ERROR] npm ERR! metron-management-ui@0.4.0 build:
> `./node_modules/angular-cli/bin/ng build -prod`
> [ERROR] npm ERR! Exit status 1
> [ERROR] npm ERR!
> [ERROR] npm ERR! Failed at the metron-management-ui@0.4.0 build script
> './node_modules/angular-cli/bin/ng build -prod'.
> [ERROR] npm ERR! Make sure you have the latest version of node.js and npm
> installed.
> [ERROR] npm ERR! If you do, this is most likely a problem with the
> metron-management-ui package,
> [ERROR] npm ERR! not with npm itself.
> [ERROR] npm ERR! Tell the author that this fails on your system:
> [ERROR] npm ERR! ./node_modules/angular-cli/bin/ng build -prod
> [ERROR] npm ERR! You can get information on how to open an issue for this
> project with:
> [ERROR] npm ERR! npm bugs metron-management-ui
> [ERROR] npm ERR! Or if that isn't available, you can get their info via:
> [ERROR] npm ERR! npm owner ls metron-management-ui
> [ERROR] npm ERR! There is likely additional logging output above.
> [ERROR]
>
>
> Kevin
>
> On Tue, May 16, 2017 at 4:32 AM, Dima Kovalyov 
> wrote:
>
> > Hello Metron devs,
> >
> > I am trying to merge Apache Metron branch Metron_0.3.1 into our Metron
> > fork. I resolved few conflicts in regards to pom and travis xml files.
> > When I build it, I receive following error:
> > [INFO] Compiling 55 source files to
> > /home/redacted/sst-metron/metron-platform/metron-
> enrichment/target/classes
> > /home/redacted/sst-metron/metron-platform/metron-
> > enrichment/src/main/java/org/apache/metron/enrichment/
> > adapters/simplehbase/SimpleHBaseAdapter.java:90:
> > warning: [unchecked] unchecked call to put(K,V) as a member of the raw
> > type HashMap
> >   enriched.put(kv.getKey().type + "." + values.getKey(),
> > values.getValue());
> >   ^
> >   where K,V are type-variables:
> > K extends Object declared in class HashMap
> > V extends Object declared in class HashMap
> > ...
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
> > (default-compile) on project metron-enrichment: Compilation failure ->
> > [Help 1]
> >
> > Attached full file for check.
> >
> > Both repos building fine separately.
> >
> > Can you please advise where I can find file that causes this?
> > Thank you.
> >
> > - Dima
> >
>

Re: [MASTER BROKEN] Possible Issue in METRON.SPEC in master

[Ryan Merriman]

You are correct.  I just finished fixing this to get the RPMs to build.
Should be an easy fix.

On Tue, May 9, 2017 at 3:51 PM, Otto Fowler  wrote:

> I just tried to merge in master, and got conflicts in metron.spec.  Which
> is not unusual since spec files are awesome.
> But I was sing a head>>  commit>> set of comments, that was weird.
>
> When I look at the actual code file in github on the web, the statement is
> actually in github.  Like we did a commit of the conflict file
>
> <<< HEAD
> %{metron_home}/config/zeppelin/metron/metron-connection-report.json
> ===
> %{metron_home}/config/zeppelin/metron/metron-ip-report.json
> >>> 1cace9ff29f31301d74fa6a7b2630d471452e985
>
> So  I don’t think the rpms are going to build off of this.
>

Re: [DISCUSS] REST + ambari

[Ryan Merriman]

It already works the way Simon describes.

On Mon, May 8, 2017 at 8:08 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> My proposal would be that the REST api use it’s application.yml for all
> the parameters and have settings it needs included in that. E.g. the metron
> directory you need is set as a property of that yml, which is then
> accessible through the spring config auto-wiring -
> @Value(“${metron.directory}”) private String path; in your service endpoint.
>
> The application.yml file is managed (generated) by Ambari on start of the
> rest api service (also managed through ambari now).
>
> All the parameters are built from a combination of user specified settings
> in Ambari, or service locations. "Where is storm?” is not something a user
> should have to specify; Ambari already knows. However "where would you like
> the metron directory?" is an input box somewhere in Ambari. Both are
> exposed to the rest api through the application.yml file.
>
> It’s more Ambari pushes the config to the REST layer than pull from
> Ambari. This way we’re using Ambari the way it is intended rather than
> trying to make it be something it is not (e.g. etcd)
>
> Simon
>
>
>
> > On 8 May 2017, at 14:00, Otto Fowler  wrote:
> >
> > I’m not sure I understand.  Could you go on a bit?
> >
> > Right now, we have env parameters in ambari, that rest should honor.  I
> don’t understand how moving rest config into ambari get’s my rest service
> access to those parameters
> >
> >
> >
> > On May 8, 2017 at 08:56:21, Simon Elliston Ball (
> si...@simonellistonball.com ) wrote:
> >
> >> Perhaps a better way of doing this would be to push the configuration
> of the REST api (the application.yml) into ambari control and expose
> parameters like that. That means the REST api doesn’t have to couple
> directly to Ambari, and doesn’t have to reinvent the Ambari paradigm
> (Ambari is not a configuration server, it’s a configuration file manager at
> heart). If the relevant parameters changed in Ambari, a restart of the REST
> service (and consequently re-write of it’s application.yml) would be
> forced. The REST API itself would then just use a simple spring config
> expression to pick up the value.
> >>
> >> Simon
> >>
> >> > On 8 May 2017, at 13:52, Otto Fowler  wrote:
> >> >
> >> > The issue is that we have services that need to access the ambari
> rest api from
> >> > inside the rest server.
> >> >
> >> > Technically, if someone changes the defaults for the metron directory
> etc, the
> >> > rest won’t pick it up.
> >> >
> >> > I have a PR coming as a follow on to METRON-777 for installing
> extensions,
> >> > and I write to hdfs, and hard coded to directory, but we allow the
> configuration.
> >> >
> >> > Other rest services write to hdfs as well.
> >> >
> >> >
> >> > On May 8, 2017 at 08:45:32, Nick Allen (n...@nickallen.org) wrote:
> >> >
> >> > As opposed to using the Ambari REST API to get this information?
> >> >
> >> > On Mon, May 8, 2017 at 8:06 AM, Otto Fowler 
> wrote:
> >> >
> >> >> I was thinking about have an ambari ‘service’ in the rest api.
> >> >> The initial purpose would be to be able to retrieve ambari
> configuration
> >> >> variables for the metron service and components.
> >> >>
> >> >> The api would be
> >> >>
> >> >> PUT: login to ambari -> ambari credentials for use with ambari,
> session
> >> >> variable
> >> >> GET: /component | service if login provided, returns the
> configuration
> >> >> GET:/component | service | property name if login provided get the
> >> >> property value for property name
> >> >>
> >> >> So, the caller would put the credentials, and after that point the
> caller
> >> >> or other services can use ambari service configurations,
> >> >> for example to get the configured metron hdfs root directory.
> >> >>
> >> >> This is opposed to configuring credentials in the rest configuration.
> >> >>
> >> >> Thoughts?
> >> >>
>
>

Re: Error building in Travis after taking master

[Ryan Merriman]

You might want to try clearing the mvn cache.  I've had travis get into a
bad state before because of corrupt maven artifacts.

On Wed, May 3, 2017 at 10:26 AM, Otto Fowler 
wrote:

> https://travis-ci.org/ottobackwards/incubator-metron/builds/228364894?utm_
> source=email_medium=notification
>
> On May 3, 2017 at 11:12:15, Justin Leet (justinjl...@gmail.com) wrote:
>
> > Jacoco is introduced from
> > https://github.com/apache/incubator-metron/pull/459.
> >
> > I'm not sure why you'd be getting a plugin error for it though, because
> > it's available in the standard repos and I was able to build off a
> > completely clean maven cache (and Travis has been able to build it as
> > well).
> >
> > What exactly you were running?
> >
> > On Wed, May 3, 2017 at 11:03 AM, Otto Fowler 
> > wrote:
> >
> > Anyone know what this is?
> >
> > [ERROR] No plugin found for prefix 'jacoco' in the current project and in
> > the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available
> > from the repositories [local (/home/travis/.m2/repository), central (
> > https://repo.maven.apache.org/maven2)] -> [Help 1]
> >
> >
>

Re: [GitHub] incubator-metron issue #562: METRON-915 add node and npm to platform_info.sh

[Ryan Merriman]

We are using a mvn plugin that automatically installs the correct version of 
node and npm locally, at least for the management UI.  

Are there other parts of the project that depend on these tools?   Is it 
desirable to include this even if they aren't a prerequisite for building?  I 
don't think it's a bad idea to include it either way, just want people to be 
aware of the plugin.

> On May 3, 2017, at 7:19 AM, nickwallen  wrote:
> 
> Github user nickwallen commented on the issue:
> 
>https://github.com/apache/incubator-metron/pull/562
> 
>+1 Thanks, Otto!  Tested on OSX and CentOS.  Worked as expected.
> 
> 
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---

Re: [Discussion] Access to Ambari Variables

[Ryan Merriman]

You would supply Ambari credentials when you make the Ambari (not ours)
REST call.

On Mon, May 1, 2017 at 3:14 PM, Otto Fowler  wrote:

> But we don’t have SSO, so that would required the ambari credentials be
> available to the rest api
>
>
> On May 1, 2017 at 16:10:03, Simon Elliston Ball (
> si...@simonellistonball.com)
> wrote:
>
> Ambari provides rest APIs to pull the values as long as you know (or can
> write a chain of calls to find) things like the name of the cluster and
> variable. Of course you’re writing an Ambari service, in which case there
> are various ways. A bad, though arguably viable way would be to grab it
> from the config files ambari writes and hope the restart had happened since
> the last change.
>
> Simon
>
> > On 1 May 2017, at 21:05, Otto Fowler  wrote:
> >
> > To be more specific, from our REST Service.
> >
> > On May 1, 2017 at 16:03:07, Otto Fowler (ottobackwa...@gmail.com) wrote:
> >
> > The question is: How would one correctly access the values of the
> > variables for metron that are set in Ambari?
> > Say, to get the patterns dir in hdfs?
> >
> > Asking for a friend ;)
>

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Ryan Merriman]

I think leveraging the REST application would work for this use case.
Services already exist for most of the functions listed in your pseudocode
(HDFS read/write, Zookeeper read/write).  Asynchronous functions are also
supported so no issues there.

On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler 
wrote:

> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So,  assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out, including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers.  This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this.  We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way?  Should we do an ambari view instead?
> * Is this too much to do in a rest call?  Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
>