Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Michael Miklavcic]

Migrating the ES client, refactoring parser bolt. There are some interface
changes in flight right now that I think would be beneficial to see in the
next release.

On Tue, Oct 16, 2018 at 2:01 PM Nick Allen  wrote:

> I am in favor of a release for both.
>
> There are a lot of really useful bug fixes, management of pcap through
> Ambari, more flexibility for configuring JAAS in Ambari, increased
> Elasticsearch performance, the Syslog parser, and the Batch Profiler, among
> others. I would be happy with calling it a 0.6.1 point release.
>
> Mike - What is outstanding that you would like to see in the release?
>
> On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'd be +1 on going with just the metron-bro-kafka-plugin release. It
> seems
> > like it's ready to go, and I think there are a few more things I'd like
> to
> > see get into our next Metron release so I'm good with holding off there.
> >
> > Mike
> >
> > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet 
> > wrote:
> >
> > > Hi all,
> > >
> > > As you might recall from a prior discussion about release cadence, we
> > were
> > > interested in initiating release threads near our board reports to see
> if
> > > we wanted to do releases or not. Additionally, the work is done to do
> two
> > > separate releases, so our options are releasing both, a single one, or
> > > neither.
> > >
> > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on
> this
> > > thread
> > > <
> > >
> >
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> > > >.
> > > In particular, the prospect of a release came up in the context of
> > having a
> > > version with better (and working) testing.
> > >
> > > Version Number
> > > If we choose to do a core Metron release, I propose 0.6.1. For
> > > metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning
> > for
> > > the plugin is a bit different in that we need x.y for bro-pkg, instead
> of
> > > x.y.z, so we wouldn't release 0.2.1.
> > >
> > > I would personally be in favor of doing a plugin release, but I'm less
> > > inclined to do a core Metron release.
> > >
> > > For Metron, I believe the main feature is the batch profiler work, but
> > > there's a fair amount of fixes and improvements.
> > >
> > > For the plugin, I believe the main improvements would be around the
> > fixing
> > > our testing and more maintenance type work.
> > >
> > > JIRA status
> > > *metron*
> > > There are 24 opens PRs at https://github.com/apache/metron/pulls.  If
> we
> > > do
> > > decide to release, we should work on getting anything meriting
> inclusion
> > > closed out.
> > >
> > > There have been 49 commits since the 0.6.0 release (listed at the end
> of
> > > the message). This assumes a release from master.
> > >
> > > *metron-bro-kafka-plugin*
> > > There are 6 open PRs at
> > > https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do
> > decide
> > > to release, we should get anything we need closed out.
> > >
> > > There have been 2 commits since the 0.2.0 release (listed at the end of
> > the
> > > message).  This assumes a release from master.
> > >
> > > *Metron changelog*
> > > METRON-1801 Allow Customization of Elasticsearch Document ID
> > > (nickwallen) closes apache/metron#1218
> > > METRON-1799 Remove outdated bylaws from site. (justinleet) closes
> > > apache/metron#1216
> > > METRON-1769 Script creation of a release candidate (justinleet)
> > closes
> > > apache/metron#1188
> > > METRON-1761 Allow a grok statement to be applied to each line in a
> > > file. (ottobackwards) closes apache/metron#1184
> > > METRON-1813 Stellar REPL Not Initialized with Client JAAS
> > (nickwallen)
> > > closes apache/metron#1232
> > > METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
> > > closes apache/metron#1230
> > > METRON-1811 Alert Search Fails When Sorting by Alert Status
> > (merrimanr)
> > > closes apache/metron#1231
> > > METRON-1809 Support Column Oriented Input with Batch Profiler
> > > (nickwallen) closes apache/metron#1229
> > > METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via
> > mmiklavc)
> > > closes apache/metron#1224
> > > METRON-1792 Simplify Profile Definitions in Integration Tests
> > > (nickwallen) closes apache/metron#1211
> > > METRON-1807 Auto populate the recommended values to some of the
> > metron
> > > config parameters  (MohanDV via merrimanr) closes apache/metron#1227
> > > METRON-1808 Add Ansible created pyc to gitignore (justinleet)
> closes
> > > apache/metron#1228
> > > METRON-1695 Expose pcap properties through Ambari (anandsubbu)
> closes
> > > apache/metron#1207
> > > METRON-1771 Update REST endpoints to support eventually consistent
> UI
> > > updates (merrimanr) closes apache/metron#1190
> > > METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
> > > 

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Nick Allen]

I am in favor of a release for both.

There are a lot of really useful bug fixes, management of pcap through
Ambari, more flexibility for configuring JAAS in Ambari, increased
Elasticsearch performance, the Syslog parser, and the Batch Profiler, among
others. I would be happy with calling it a 0.6.1 point release.

Mike - What is outstanding that you would like to see in the release?

On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'd be +1 on going with just the metron-bro-kafka-plugin release. It seems
> like it's ready to go, and I think there are a few more things I'd like to
> see get into our next Metron release so I'm good with holding off there.
>
> Mike
>
> On Tue, Oct 16, 2018 at 10:26 AM Justin Leet 
> wrote:
>
> > Hi all,
> >
> > As you might recall from a prior discussion about release cadence, we
> were
> > interested in initiating release threads near our board reports to see if
> > we wanted to do releases or not. Additionally, the work is done to do two
> > separate releases, so our options are releasing both, a single one, or
> > neither.
> >
> > Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
> > thread
> > <
> >
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> > >.
> > In particular, the prospect of a release came up in the context of
> having a
> > version with better (and working) testing.
> >
> > Version Number
> > If we choose to do a core Metron release, I propose 0.6.1. For
> > metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning
> for
> > the plugin is a bit different in that we need x.y for bro-pkg, instead of
> > x.y.z, so we wouldn't release 0.2.1.
> >
> > I would personally be in favor of doing a plugin release, but I'm less
> > inclined to do a core Metron release.
> >
> > For Metron, I believe the main feature is the batch profiler work, but
> > there's a fair amount of fixes and improvements.
> >
> > For the plugin, I believe the main improvements would be around the
> fixing
> > our testing and more maintenance type work.
> >
> > JIRA status
> > *metron*
> > There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we
> > do
> > decide to release, we should work on getting anything meriting inclusion
> > closed out.
> >
> > There have been 49 commits since the 0.6.0 release (listed at the end of
> > the message). This assumes a release from master.
> >
> > *metron-bro-kafka-plugin*
> > There are 6 open PRs at
> > https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do
> decide
> > to release, we should get anything we need closed out.
> >
> > There have been 2 commits since the 0.2.0 release (listed at the end of
> the
> > message).  This assumes a release from master.
> >
> > *Metron changelog*
> > METRON-1801 Allow Customization of Elasticsearch Document ID
> > (nickwallen) closes apache/metron#1218
> > METRON-1799 Remove outdated bylaws from site. (justinleet) closes
> > apache/metron#1216
> > METRON-1769 Script creation of a release candidate (justinleet)
> closes
> > apache/metron#1188
> > METRON-1761 Allow a grok statement to be applied to each line in a
> > file. (ottobackwards) closes apache/metron#1184
> > METRON-1813 Stellar REPL Not Initialized with Client JAAS
> (nickwallen)
> > closes apache/metron#1232
> > METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
> > closes apache/metron#1230
> > METRON-1811 Alert Search Fails When Sorting by Alert Status
> (merrimanr)
> > closes apache/metron#1231
> > METRON-1809 Support Column Oriented Input with Batch Profiler
> > (nickwallen) closes apache/metron#1229
> > METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via
> mmiklavc)
> > closes apache/metron#1224
> > METRON-1792 Simplify Profile Definitions in Integration Tests
> > (nickwallen) closes apache/metron#1211
> > METRON-1807 Auto populate the recommended values to some of the
> metron
> > config parameters  (MohanDV via merrimanr) closes apache/metron#1227
> > METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
> > apache/metron#1228
> > METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
> > apache/metron#1207
> > METRON-1771 Update REST endpoints to support eventually consistent UI
> > updates (merrimanr) closes apache/metron#1190
> > METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
> > closes apache/metron#1210
> > METRON-1804 Update version to 0.6.1 (justinleet) closes
> > apache/metron#1220
> > METRON-1798 Add mpack support for parser aggregation (anandsubbu)
> > closes apache/metron#1215
> > METRON-1750 Create Parser for Syslog RFC 5424 Messages
> (ottobackwards)
> > closes apache/metron#1175
> > METRON-1794 Include User Details When Escalating Alerts (nickwallen)
> > closes apache/metron#1212
> > METRON-1782 Add Kafka 

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I agree with a metron-bro-plugin-kafka release of 0.3.0 (0.3 in bro-pkg),
assuming we can get apache/metron-bro-plugin-kafka#2 in.  I'm working on
adding travis to the metron-bro-plugin-kafka repo, but I'm not sure when I
will have enough time to finish my work there and wouldn't want to hold up
a release just for that.

Jon

On Tue, Oct 16, 2018 at 12:26 PM Justin Leet  wrote:

> Hi all,
>
> As you might recall from a prior discussion about release cadence, we were
> interested in initiating release threads near our board reports to see if
> we wanted to do releases or not. Additionally, the work is done to do two
> separate releases, so our options are releasing both, a single one, or
> neither.
>
> Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
> thread
> <
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> >.
> In particular, the prospect of a release came up in the context of having a
> version with better (and working) testing.
>
> Version Number
> If we choose to do a core Metron release, I propose 0.6.1. For
> metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning for
> the plugin is a bit different in that we need x.y for bro-pkg, instead of
> x.y.z, so we wouldn't release 0.2.1.
>
> I would personally be in favor of doing a plugin release, but I'm less
> inclined to do a core Metron release.
>
> For Metron, I believe the main feature is the batch profiler work, but
> there's a fair amount of fixes and improvements.
>
> For the plugin, I believe the main improvements would be around the fixing
> our testing and more maintenance type work.
>
> JIRA status
> *metron*
> There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we
> do
> decide to release, we should work on getting anything meriting inclusion
> closed out.
>
> There have been 49 commits since the 0.6.0 release (listed at the end of
> the message). This assumes a release from master.
>
> *metron-bro-kafka-plugin*
> There are 6 open PRs at
> https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do decide
> to release, we should get anything we need closed out.
>
> There have been 2 commits since the 0.2.0 release (listed at the end of the
> message).  This assumes a release from master.
>
> *Metron changelog*
> METRON-1801 Allow Customization of Elasticsearch Document ID
> (nickwallen) closes apache/metron#1218
> METRON-1799 Remove outdated bylaws from site. (justinleet) closes
> apache/metron#1216
> METRON-1769 Script creation of a release candidate (justinleet) closes
> apache/metron#1188
> METRON-1761 Allow a grok statement to be applied to each line in a
> file. (ottobackwards) closes apache/metron#1184
> METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen)
> closes apache/metron#1232
> METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
> closes apache/metron#1230
> METRON-1811 Alert Search Fails When Sorting by Alert Status (merrimanr)
> closes apache/metron#1231
> METRON-1809 Support Column Oriented Input with Batch Profiler
> (nickwallen) closes apache/metron#1229
> METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc)
> closes apache/metron#1224
> METRON-1792 Simplify Profile Definitions in Integration Tests
> (nickwallen) closes apache/metron#1211
> METRON-1807 Auto populate the recommended values to some of the metron
> config parameters  (MohanDV via merrimanr) closes apache/metron#1227
> METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
> apache/metron#1228
> METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
> apache/metron#1207
> METRON-1771 Update REST endpoints to support eventually consistent UI
> updates (merrimanr) closes apache/metron#1190
> METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
> closes apache/metron#1210
> METRON-1804 Update version to 0.6.1 (justinleet) closes
> apache/metron#1220
> METRON-1798 Add mpack support for parser aggregation (anandsubbu)
> closes apache/metron#1215
> METRON-1750 Create Parser for Syslog RFC 5424 Messages (ottobackwards)
> closes apache/metron#1175
> METRON-1794 Include User Details When Escalating Alerts (nickwallen)
> closes apache/metron#1212
> METRON-1782 Add Kafka Partition and Offset to Profiler Debug Logs
> (nickwallen) closes apache/metron#1202
> METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
> jonzeolla) closes apache/metron#1179
> METRON-1784: Re-allow remote ssh and scp in Centos full dev (mmiklavc
> via mmiklavc) closes apache/metron#1204
> METRON-1787 Input Time Constraints for Batch Profiler (nickwallen)
> closes apache/metron#1209
> METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
> (nickwallen) closes apache/metron#1185
> METRON-1786 Pcap Topology Status Incorrect (MohanDV via 

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Michael Miklavcic]

I'd be +1 on going with just the metron-bro-kafka-plugin release. It seems
like it's ready to go, and I think there are a few more things I'd like to
see get into our next Metron release so I'm good with holding off there.

Mike

On Tue, Oct 16, 2018 at 10:26 AM Justin Leet  wrote:

> Hi all,
>
> As you might recall from a prior discussion about release cadence, we were
> interested in initiating release threads near our board reports to see if
> we wanted to do releases or not. Additionally, the work is done to do two
> separate releases, so our options are releasing both, a single one, or
> neither.
>
> Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
> thread
> <
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> >.
> In particular, the prospect of a release came up in the context of having a
> version with better (and working) testing.
>
> Version Number
> If we choose to do a core Metron release, I propose 0.6.1. For
> metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning for
> the plugin is a bit different in that we need x.y for bro-pkg, instead of
> x.y.z, so we wouldn't release 0.2.1.
>
> I would personally be in favor of doing a plugin release, but I'm less
> inclined to do a core Metron release.
>
> For Metron, I believe the main feature is the batch profiler work, but
> there's a fair amount of fixes and improvements.
>
> For the plugin, I believe the main improvements would be around the fixing
> our testing and more maintenance type work.
>
> JIRA status
> *metron*
> There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we
> do
> decide to release, we should work on getting anything meriting inclusion
> closed out.
>
> There have been 49 commits since the 0.6.0 release (listed at the end of
> the message). This assumes a release from master.
>
> *metron-bro-kafka-plugin*
> There are 6 open PRs at
> https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do decide
> to release, we should get anything we need closed out.
>
> There have been 2 commits since the 0.2.0 release (listed at the end of the
> message).  This assumes a release from master.
>
> *Metron changelog*
> METRON-1801 Allow Customization of Elasticsearch Document ID
> (nickwallen) closes apache/metron#1218
> METRON-1799 Remove outdated bylaws from site. (justinleet) closes
> apache/metron#1216
> METRON-1769 Script creation of a release candidate (justinleet) closes
> apache/metron#1188
> METRON-1761 Allow a grok statement to be applied to each line in a
> file. (ottobackwards) closes apache/metron#1184
> METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen)
> closes apache/metron#1232
> METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
> closes apache/metron#1230
> METRON-1811 Alert Search Fails When Sorting by Alert Status (merrimanr)
> closes apache/metron#1231
> METRON-1809 Support Column Oriented Input with Batch Profiler
> (nickwallen) closes apache/metron#1229
> METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc)
> closes apache/metron#1224
> METRON-1792 Simplify Profile Definitions in Integration Tests
> (nickwallen) closes apache/metron#1211
> METRON-1807 Auto populate the recommended values to some of the metron
> config parameters  (MohanDV via merrimanr) closes apache/metron#1227
> METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
> apache/metron#1228
> METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
> apache/metron#1207
> METRON-1771 Update REST endpoints to support eventually consistent UI
> updates (merrimanr) closes apache/metron#1190
> METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
> closes apache/metron#1210
> METRON-1804 Update version to 0.6.1 (justinleet) closes
> apache/metron#1220
> METRON-1798 Add mpack support for parser aggregation (anandsubbu)
> closes apache/metron#1215
> METRON-1750 Create Parser for Syslog RFC 5424 Messages (ottobackwards)
> closes apache/metron#1175
> METRON-1794 Include User Details When Escalating Alerts (nickwallen)
> closes apache/metron#1212
> METRON-1782 Add Kafka Partition and Offset to Profiler Debug Logs
> (nickwallen) closes apache/metron#1202
> METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
> jonzeolla) closes apache/metron#1179
> METRON-1784: Re-allow remote ssh and scp in Centos full dev (mmiklavc
> via mmiklavc) closes apache/metron#1204
> METRON-1787 Input Time Constraints for Batch Profiler (nickwallen)
> closes apache/metron#1209
> METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
> (nickwallen) closes apache/metron#1185
> METRON-1786 Pcap Topology Status Incorrect (MohanDV via nickwallen)
> closes apache/metron#1206
> METRON-1709 Add controls to start / stop the PCAP topology 

Re: [DISCUSS] Switching to a better alternative of Pikaday.js

[Michael Miklavcic]

I'm still +1 to this approach. Thanks guys!

To Tibor's other point - the management UI is currently a completely
separate service, as I understand it. If there is infrastructure you think
is shareable while still allowing them to be independently deployed, then I
think it would be desirable to make improvements there also. My only
caution would be that there are instances where early architectural
duplication seems wasteful, but is really just a short-term illusion
because of code maturity. ie, there are instances where you absolutely
don't want to tie architectural components together even if it seems like
you're duplicating code. I don't believe this fits that concern, but I
thought it was worth mentioning. To that end, I wouldn't have your progress
on the alerts UI improvements be hampered by the management UI. I think we
can take the lessons learned in the Alerts UI and apply them to the
management UI when it makes sense to.

Best,
Mike


On Tue, Oct 16, 2018 at 2:21 AM Tibor Meller  wrote:

> Thanks, Tamas! The plan you described seems a good approach to me.
> Let's wait another day before we create Jira tickets from these steps to
> make sure no one else has other concerns.
>
> One more question: how much of these changes could be applicable to the
> Management UI?
> It would be great plus to see those two UI getting closer to each other
> with the underlying technologies instead of shifting away.
>
> On Tue, Oct 16, 2018 at 9:37 AM Tamás Fodor  wrote:
>
> > I'm agreeing with moving forward with Ng Bootstrap. In order to do that,
> I
> > think it would be a good start to refactor those components which use the
> > obsolete jquery based vesrion of bootstrap. Shane has already started it
> by
> > refactoring the Modal dialog in the management UI (We also have this
> > component in the alerts UI). Once we've succeeded that, we can remove
> > (hopefully) jQuery form the dependency list.
> > As the second step, we should migrate to Ng bootstrap. I'm assuming that
> if
> > the jquery based Boostrap is replaced with Ng bootstrap, we're still good
> > since we're using only classes from the bootstrap CSS which is shared
> > across jQuery bootstrap and Ng bootstrap.
> > If everything is good and nothing is broken we can start working on the
> new
> > date/time picker component based on Ng bootstrap and then we can get rid
> of
> > Pikaday.
> > Once Pikaday is completely eliminated from the system, we can be sure
> that
> > the PR about eliminating moment.js is ready to go.
> >
> > Cheers,
> > Tamas
> >
> > On Mon, Oct 15, 2018 at 8:36 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I like point #3 from Tibor - you can pick and choose how you compose
> the
> > > date/time pickers. It would be nice to not have times required as a
> > > dropdown at some point or another, or at the very least something we
> can
> > > customize differently to our liking :)
> > >
> > > Per the comments from Tamas, Shane, and Tibor, is there any reason we
> > > wouldn't want to move forward with the Angular port of Bootstrap, NG
> > > Bootstrap? Per your arguments for it, this sounds like the right path
> > > forward to me. I'm +1 on this approach provided someone doesn't come
> back
> > > with a "well, there's only this problem..."
> > >
> > > Best,
> > > Mike
> > >
> > > On Mon, Oct 15, 2018 at 7:39 AM Shane Ardell  >
> > > wrote:
> > >
> > > > We are definitely on the same page.
> > > >
> > > > On Mon, Oct 15, 2018 at 2:06 PM Tibor Meller  >
> > > > wrote:
> > > >
> > > > > @Shane It seems like we're agreed on this. :D
> > > > >
> > > > > On Mon, Oct 15, 2018 at 1:18 PM Tibor Meller <
> tibor.mel...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Guys,
> > > > > >
> > > > > > I think we could consider moving to ng bootstrap. It solves most
> of
> > > our
> > > > > > problems and makes our code base cleaner easier to maintain.
> > > > > >
> > > > > > Here are some benefits I see:
> > > > > > 1. we could eliminate jQuery from the code base
> > > > > > Currently, we're using bootstrap but an implementation based on
> > > jQuery.
> > > > > > Angular and jQuery doesn't build to live together.
> > > > > > 2. NgBootstrap made to being used with Angular
> > > > > > It uses Angular instead of hacking the rendering/dom manipulation
> > > with
> > > > > > jQuery.
> > > > > > 3. It contains a date and a time picker.
> > > > > > It's easy to combine to a datetime picker.
> > > > > > 4. No dependencies.
> > > > > > By changing currently used bootstrap to NgBootstrap we could
> clear
> > > > > jquery,
> > > > > > moment, pickaday, pickaday-time libraries from our dependencies.
> > > > > >
> > > > > > Another huge advantage of NgBootstrap is that we don't have to
> > > rewrite
> > > > > > anything we don't want to. Our UI already uses Bootstrap.
> > > > > >
> > > > > > What do you guys think?
> > > > > >
> > > > > > On Wed, Oct 10, 2018 at 4:19 PM Nick Allen 
> > > wrote:
> > > > > >
> > > > > >> > Before 

Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

Hi all,

As you might recall from a prior discussion about release cadence, we were
interested in initiating release threads near our board reports to see if
we wanted to do releases or not. Additionally, the work is done to do two
separate releases, so our options are releasing both, a single one, or
neither.

Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
thread
.
In particular, the prospect of a release came up in the context of having a
version with better (and working) testing.

Version Number
If we choose to do a core Metron release, I propose 0.6.1. For
metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning for
the plugin is a bit different in that we need x.y for bro-pkg, instead of
x.y.z, so we wouldn't release 0.2.1.

I would personally be in favor of doing a plugin release, but I'm less
inclined to do a core Metron release.

For Metron, I believe the main feature is the batch profiler work, but
there's a fair amount of fixes and improvements.

For the plugin, I believe the main improvements would be around the fixing
our testing and more maintenance type work.

JIRA status
*metron*
There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we do
decide to release, we should work on getting anything meriting inclusion
closed out.

There have been 49 commits since the 0.6.0 release (listed at the end of
the message). This assumes a release from master.

*metron-bro-kafka-plugin*
There are 6 open PRs at
https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do decide
to release, we should get anything we need closed out.

There have been 2 commits since the 0.2.0 release (listed at the end of the
message).  This assumes a release from master.

*Metron changelog*
METRON-1801 Allow Customization of Elasticsearch Document ID
(nickwallen) closes apache/metron#1218
METRON-1799 Remove outdated bylaws from site. (justinleet) closes
apache/metron#1216
METRON-1769 Script creation of a release candidate (justinleet) closes
apache/metron#1188
METRON-1761 Allow a grok statement to be applied to each line in a
file. (ottobackwards) closes apache/metron#1184
METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen)
closes apache/metron#1232
METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
closes apache/metron#1230
METRON-1811 Alert Search Fails When Sorting by Alert Status (merrimanr)
closes apache/metron#1231
METRON-1809 Support Column Oriented Input with Batch Profiler
(nickwallen) closes apache/metron#1229
METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc)
closes apache/metron#1224
METRON-1792 Simplify Profile Definitions in Integration Tests
(nickwallen) closes apache/metron#1211
METRON-1807 Auto populate the recommended values to some of the metron
config parameters  (MohanDV via merrimanr) closes apache/metron#1227
METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
apache/metron#1228
METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
apache/metron#1207
METRON-1771 Update REST endpoints to support eventually consistent UI
updates (merrimanr) closes apache/metron#1190
METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
closes apache/metron#1210
METRON-1804 Update version to 0.6.1 (justinleet) closes
apache/metron#1220
METRON-1798 Add mpack support for parser aggregation (anandsubbu)
closes apache/metron#1215
METRON-1750 Create Parser for Syslog RFC 5424 Messages (ottobackwards)
closes apache/metron#1175
METRON-1794 Include User Details When Escalating Alerts (nickwallen)
closes apache/metron#1212
METRON-1782 Add Kafka Partition and Offset to Profiler Debug Logs
(nickwallen) closes apache/metron#1202
METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
jonzeolla) closes apache/metron#1179
METRON-1784: Re-allow remote ssh and scp in Centos full dev (mmiklavc
via mmiklavc) closes apache/metron#1204
METRON-1787 Input Time Constraints for Batch Profiler (nickwallen)
closes apache/metron#1209
METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
(nickwallen) closes apache/metron#1185
METRON-1786 Pcap Topology Status Incorrect (MohanDV via nickwallen)
closes apache/metron#1206
METRON-1709 Add controls to start / stop the PCAP topology from Ambari.
(MohanDV via nickwallen) closes apache/metron#1201
METRON-1759 PCAP UI: Removing wrong Input annotations from pcap panel
component (tiborm via nickwallen) closes apache/metron#1180
METRON-1772 Support alternative input formats in the Batch Profiler
(nickwallen) closes apache/metron#1191
METRON-1770 Add Docs for Running the Profiler with Spark on YARN
(nickwallen) closes apache/metron#1189
METRON-1774 Allow user to configure JAAS client in 

Re: HBaseDao and IndexDao abstraction

[Michael Miklavcic]

Hi Muhammed,

I think you probably want to start with our parser infrastructure rather
than the DAO's for what you're doing. This series of blog posts gives a use
case driven walkthrough that should help shed some light on things:
Part 1 (start here) -
https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
TOC of the 7-part series -
https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana

Here's some details about our parser infrastructure -
https://github.com/apache/metron/tree/master/metron-platform/metron-parsers
...which feeds into the data enrichment topology -
https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment
...which feeds into the indexing topology, which you've already found

Hope this helps for a start!

Best,
Mike Miklavcic


On Tue, Oct 16, 2018 at 12:05 AM Muhammed Irshad 
wrote:

> Hi all,
>
> What is the actual use of HBaseDao documented in metron indexing
> documentation
> <
> https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
> >
> under section 'The IndexDao Abstraction' ? From my reading I understand it
> as a HBase indexing implementation which can be clubbed to hdfs for updated
> data. What is the use of it as we cannot chose to index in HBase / hdfs
> dynamically ? Can some one explain an example about how to configure and
> use it ( More documentation link or reference is fine) ? I have a use case
> where I need to maintain an Active Directory inventory, Using AD event logs
> being indexed via metron. Is HBaseDao can be used for this use case ?
>
> --
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t
>

Re: [DISCUSS] Switching to a better alternative of Pikaday.js

[Tibor Meller]

Thanks, Tamas! The plan you described seems a good approach to me.
Let's wait another day before we create Jira tickets from these steps to
make sure no one else has other concerns.

One more question: how much of these changes could be applicable to the
Management UI?
It would be great plus to see those two UI getting closer to each other
with the underlying technologies instead of shifting away.

On Tue, Oct 16, 2018 at 9:37 AM Tamás Fodor  wrote:

> I'm agreeing with moving forward with Ng Bootstrap. In order to do that, I
> think it would be a good start to refactor those components which use the
> obsolete jquery based vesrion of bootstrap. Shane has already started it by
> refactoring the Modal dialog in the management UI (We also have this
> component in the alerts UI). Once we've succeeded that, we can remove
> (hopefully) jQuery form the dependency list.
> As the second step, we should migrate to Ng bootstrap. I'm assuming that if
> the jquery based Boostrap is replaced with Ng bootstrap, we're still good
> since we're using only classes from the bootstrap CSS which is shared
> across jQuery bootstrap and Ng bootstrap.
> If everything is good and nothing is broken we can start working on the new
> date/time picker component based on Ng bootstrap and then we can get rid of
> Pikaday.
> Once Pikaday is completely eliminated from the system, we can be sure that
> the PR about eliminating moment.js is ready to go.
>
> Cheers,
> Tamas
>
> On Mon, Oct 15, 2018 at 8:36 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I like point #3 from Tibor - you can pick and choose how you compose the
> > date/time pickers. It would be nice to not have times required as a
> > dropdown at some point or another, or at the very least something we can
> > customize differently to our liking :)
> >
> > Per the comments from Tamas, Shane, and Tibor, is there any reason we
> > wouldn't want to move forward with the Angular port of Bootstrap, NG
> > Bootstrap? Per your arguments for it, this sounds like the right path
> > forward to me. I'm +1 on this approach provided someone doesn't come back
> > with a "well, there's only this problem..."
> >
> > Best,
> > Mike
> >
> > On Mon, Oct 15, 2018 at 7:39 AM Shane Ardell 
> > wrote:
> >
> > > We are definitely on the same page.
> > >
> > > On Mon, Oct 15, 2018 at 2:06 PM Tibor Meller 
> > > wrote:
> > >
> > > > @Shane It seems like we're agreed on this. :D
> > > >
> > > > On Mon, Oct 15, 2018 at 1:18 PM Tibor Meller  >
> > > > wrote:
> > > >
> > > > > Hi Guys,
> > > > >
> > > > > I think we could consider moving to ng bootstrap. It solves most of
> > our
> > > > > problems and makes our code base cleaner easier to maintain.
> > > > >
> > > > > Here are some benefits I see:
> > > > > 1. we could eliminate jQuery from the code base
> > > > > Currently, we're using bootstrap but an implementation based on
> > jQuery.
> > > > > Angular and jQuery doesn't build to live together.
> > > > > 2. NgBootstrap made to being used with Angular
> > > > > It uses Angular instead of hacking the rendering/dom manipulation
> > with
> > > > > jQuery.
> > > > > 3. It contains a date and a time picker.
> > > > > It's easy to combine to a datetime picker.
> > > > > 4. No dependencies.
> > > > > By changing currently used bootstrap to NgBootstrap we could clear
> > > > jquery,
> > > > > moment, pickaday, pickaday-time libraries from our dependencies.
> > > > >
> > > > > Another huge advantage of NgBootstrap is that we don't have to
> > rewrite
> > > > > anything we don't want to. Our UI already uses Bootstrap.
> > > > >
> > > > > What do you guys think?
> > > > >
> > > > > On Wed, Oct 10, 2018 at 4:19 PM Nick Allen 
> > wrote:
> > > > >
> > > > >> > Before making a decision on what's next, I'd to ask you a
> > question.
> > > Is
> > > > >> it really
> > > > >> a priority and is it really worth the effort to touch our
> currently
> > > used
> > > > >> date picker component to get ~15% reduction in the bundle size by
> > > > removing
> > > > >> moment?
> > > > >>
> > > > >> As an aside, I think there is a greater benefit here too.  We need
> > to
> > > > make
> > > > >> a conscious effort to identify libraries that we are using that
> are
> > > > >> deprecated, lack community support, and are unlikely to be
> > maintained
> > > > and
> > > > >> updated for security vulnerabilities.  We need to actively
> identify
> > > and
> > > > >> replace those.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Oct 10, 2018 at 9:33 AM Tamás Fodor <
> ftamas.m...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I'd like to open a discussion about switching to a new date
> picker
> > > > >> library
> > > > >> > in the Metron Alerts UI regarding to the following:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E
> > > > >> >
> > > > >> 

HBaseDao and IndexDao abstraction

[Muhammed Irshad]

Hi all,

What is the actual use of HBaseDao documented in metron indexing
documentation

under section 'The IndexDao Abstraction' ? From my reading I understand it
as a HBase indexing implementation which can be clubbed to hdfs for updated
data. What is the use of it as we cannot chose to index in HBase / hdfs
dynamically ? Can some one explain an example about how to configure and
use it ( More documentation link or reference is fine) ? I have a use case
where I need to maintain an Active Directory inventory, Using AD event logs
being indexed via metron. Is HBaseDao can be used for this use case ?

-- 
Muhammed Irshad K T
Senior Software Engineer
+919447946359
irshadkt@gmail.com
Skype : muhammed.irshad.k.t