Re: Revert PR #1218

2018-10-23 Thread Nick Allen 
I wouldn't call this complex.  It is much easier to roll it back, so I can
work on a proper fix without impacting the ongoing work of others.

The existing Elasticsearch DAOs do not distinguish between document ID and
Metron GUID as there was no need to before.  So I need to disambiguate
those concepts a bit, which is rather subtle.  In addition, none of the
integration or e2e tests caught the problem because there is a disconnect
between the reader and writer side of the house for Elasticsearch.  I want
to update the tests to ensure this sort of problem is caught.


On Tue, Oct 23, 2018 at 3:11 PM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Would it not make more sense to fix the bug on the DAO side, and roll
> forward? I suspect what we need to do is add a stage in the update
> capability to configure the key field used for update, or worst case have a
> pre-query to lookup the internal ID in the relatively rare scenario where
> we escalate / modify indexed docs. Seems like a simple new ticket, rather
> than a complex roll back and roll forward later. As long as we get the
> follow on in before an Apache release we should be fine, no?
>
> Simon
>
> On Tue, 23 Oct 2018 at 19:58, Nick Allen  wrote:
>
> > Hi Guys -
> >
> > @rmerriman tracked down some problems that were introduced with my PR
> > #1218.  Thanks to him for finding this.  The change was intended to
> improve
> > Elasticsearch write performance by allowing Elasticsearch to set its own
> > document ID.
> >
> > The problem is that if you then go to the Alerts UI and escalate an
> alert,
> > it will create a duplicate alert in the index, rather than updating the
> > existing alert. I've been looking at how to fix the problem and the scope
> > of the fix is larger than I'd like to handle as a follow-on.  There are
> > some prerequisites I'd like to tackle before introducing this change.
> >
> > I am going to revert the change on master, which will introduce an
> > additional commit that is an "undo" of the original commit.  I will then
> > open a separate PR that introduces this new functionality.
> >
> > https://github.com/apache/metron/pull/1218
> >
> > Thanks
> >
>
>
> --
> --
> simon elliston ball
> @sireb
>

Re: Revert PR #1218

2018-10-23 Thread Simon Elliston Ball 
Would it not make more sense to fix the bug on the DAO side, and roll
forward? I suspect what we need to do is add a stage in the update
capability to configure the key field used for update, or worst case have a
pre-query to lookup the internal ID in the relatively rare scenario where
we escalate / modify indexed docs. Seems like a simple new ticket, rather
than a complex roll back and roll forward later. As long as we get the
follow on in before an Apache release we should be fine, no?

Simon

On Tue, 23 Oct 2018 at 19:58, Nick Allen  wrote:

> Hi Guys -
>
> @rmerriman tracked down some problems that were introduced with my PR
> #1218.  Thanks to him for finding this.  The change was intended to improve
> Elasticsearch write performance by allowing Elasticsearch to set its own
> document ID.
>
> The problem is that if you then go to the Alerts UI and escalate an alert,
> it will create a duplicate alert in the index, rather than updating the
> existing alert. I've been looking at how to fix the problem and the scope
> of the fix is larger than I'd like to handle as a follow-on.  There are
> some prerequisites I'd like to tackle before introducing this change.
>
> I am going to revert the change on master, which will introduce an
> additional commit that is an "undo" of the original commit.  I will then
> open a separate PR that introduces this new functionality.
>
> https://github.com/apache/metron/pull/1218
>
> Thanks
>


-- 
--
simon elliston ball
@sireb

Revert PR #1218

2018-10-23 Thread Nick Allen 
Hi Guys -

@rmerriman tracked down some problems that were introduced with my PR
#1218.  Thanks to him for finding this.  The change was intended to improve
Elasticsearch write performance by allowing Elasticsearch to set its own
document ID.

The problem is that if you then go to the Alerts UI and escalate an alert,
it will create a duplicate alert in the index, rather than updating the
existing alert. I've been looking at how to fix the problem and the scope
of the fix is larger than I'd like to handle as a follow-on.  There are
some prerequisites I'd like to tackle before introducing this change.

I am going to revert the change on master, which will introduce an
additional commit that is an "undo" of the original commit.  I will then
open a separate PR that introduces this new functionality.

https://github.com/apache/metron/pull/1218

Thanks

Re: Invite to Slack Channel

2018-10-23 Thread Otto Fowler 
Done


On October 23, 2018 at 02:19:54, Mustafa Akmal (mustafa.ak...@abcdata.org)
wrote:

Hi,
Please send me an invitation link to the slack channel aswell.
Thanks!

Mustafa Akmal
Big Data Consultant
mustafa.ak...@abcdata.org (mailto:mustafa.ak...@abcdata.org)
+923365257705 (tel:+923365257705)
NUST (Technology Incubation Centre), Office No. 208, H-12, Islamabad,
Pakistan (
https://maps.google.com/?q=NUST%20(Technology%20Incubation%20Centre)%2C%20Office%20No.%20208%2C%20H-12%2C%20Islamabad%2C%20Pakistan)

http://www.abcdata.org/

On Oct 23 2018, at 1:01 am, Michael Miklavcic 
wrote:
>
> Sent
> On Mon, Oct 22, 2018 at 1:00 PM vpiserc...@gmail.com 

> wrote:
>
> > Hi,
> > can anyone invite to the metron slack channel?
> > Thanks,
> > vito piserchia
> > On 10/22/18 3:31 PM, zeo...@gmail.com wrote:
> > > Invite sent
> > >
> > > On Mon, Oct 22, 2018 at 9:26 AM Muhammed Irshad 
> > wrote:
> > >
> > > > Some one get me also the slack channel link ?
> > > > Thanks,
> > > > Muhammed Irshad
> > > > Q*Burst*
> > > > www.qburst.com
> > > >
> > > >
> > > > On Wed, Oct 17, 2018 at 7:33 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Sent
> > > > > On Wed, Oct 17, 2018 at 7:23 AM Tibor Meller <
tibor.mel...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Guys,
> > > > > > Can you add me to the apache metron slack chanel?
> > > > > >
> > > > > > Thanks,
> > > > > > On Thu, Oct 4, 2018 at 1:14 PM Otto Fowler <
ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Done
> > > > > > >
> > > > > > > On October 4, 2018 at 05:35:06, Tamás Fodor (
ftamas.m...@gmail.com)
> > > > > > wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > > Michael, can you add me as well?
> > > > > > > Thank you in advance!
> > > > > > > Tamas
> > > > > > > On Wed, Oct 3, 2018 at 4:27 PM Michael Miklavcic <
> > > > > > > michael.miklav...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Sent
> > > > > > > > On Wed, Oct 3, 2018 at 8:17 AM Shane Ardell <
> > > > > shane.m.ard...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > > Is it possible for someone to send me an invite to the
Metron
> > > > Slack
> > > > > > > > > channel?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Shane
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
> > --
> > Vito Piserchia
> > Security and Software Engineer
> >
> > : vito.piserchia[at]dreamlab.net
> > : 4915 8835 2C18 9CAE F14F 2314 613D 51C5 106B 83EA
> > : https://dreamlab.net
> > : +41 31 398 66 66
> > : +41 31 398 66 69
> > -
> >
> > DreamLab Technologies AG
> > Monbijoustrasse 36
> > 3011 Bern, Switzerland
> >
> > -
> > This e-mail may contain confidential and/or privileged information.
> > If you are not the intended recipient (or have received this e-mail
> > in error) please notify the sender immediately and destroy this
> > e-mail. Any unauthorised copying, disclosure or distribution of the
> > material in this e-mail is strictly forbidden.
> >
> > -

Re: HBaseDao and IndexDao abstraction

2018-10-23 Thread Muhammed Irshad 
Hi All,

I have got a solution for this using SHEW ( Simple HBase Enrichment Writer
) which is documented in confluence

but not in metron current book documentation
. I am going to give
this a try and see how it goes. Thanks a lot for Simon Elliston Ball
 & Metron slack channel :)

On Thu, Oct 18, 2018 at 10:51 AM Muhammed Irshad 
wrote:

> Mike,
>
> Thanks for replying. I had gone through it already and we are indexing our
> Active Directory logs to hdfs by streaming from Splunk. But I have a
> requirement of maintaining Active Directory asset inventory ( Just list of
> asset and their status not historic data) along with AD event indexing. So
> I thought of using HBase and was thinking the best place to put this logic
> ( Enrichment by writing a custom stellar which populate HBase column family
> for assets or In indexing layer ) . Then I saw the HBaseDao in
> documentation and wanted to understand what it is and weather it can be
> used to meet my use case.
>
> On Tue, Oct 16, 2018 at 7:41 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Hi Muhammed,
>>
>> I think you probably want to start with our parser infrastructure rather
>> than the DAO's for what you're doing. This series of blog posts gives a
>> use
>> case driven walkthrough that should help shed some light on things:
>> Part 1 (start here) -
>>
>> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
>> TOC of the 7-part series -
>>
>> https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana
>>
>> Here's some details about our parser infrastructure -
>>
>> https://github.com/apache/metron/tree/master/metron-platform/metron-parsers
>> ...which feeds into the data enrichment topology -
>>
>> https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment
>> ...which feeds into the indexing topology, which you've already found
>>
>> Hope this helps for a start!
>>
>> Best,
>> Mike Miklavcic
>>
>>
>> On Tue, Oct 16, 2018 at 12:05 AM Muhammed Irshad 
>> wrote:
>>
>> > Hi all,
>> >
>> > What is the actual use of HBaseDao documented in metron indexing
>> > documentation
>> > <
>> >
>> https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
>> > >
>> > under section 'The IndexDao Abstraction' ? From my reading I understand
>> it
>> > as a HBase indexing implementation which can be clubbed to hdfs for
>> updated
>> > data. What is the use of it as we cannot chose to index in HBase / hdfs
>> > dynamically ? Can some one explain an example about how to configure and
>> > use it ( More documentation link or reference is fine) ? I have a use
>> case
>> > where I need to maintain an Active Directory inventory, Using AD event
>> logs
>> > being indexed via metron. Is HBaseDao can be used for this use case ?
>> >
>> > --
>> > Muhammed Irshad K T
>> > Senior Software Engineer
>> > +919447946359
>> > irshadkt@gmail.com
>> > Skype : muhammed.irshad.k.t
>> >
>>
>
>
> --
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t
>


-- 
Muhammed Irshad K T
Senior Software Engineer
+919447946359
irshadkt@gmail.com
Skype : muhammed.irshad.k.t

Re: Invite to Slack Channel

2018-10-23 Thread Mustafa Akmal 
Hi,
Please send me an invitation link to the slack channel aswell.
Thanks!

Mustafa Akmal
Big Data Consultant
mustafa.ak...@abcdata.org (mailto:mustafa.ak...@abcdata.org)
+923365257705 (tel:+923365257705)
NUST (Technology Incubation Centre), Office No. 208, H-12, Islamabad, Pakistan 
(https://maps.google.com/?q=NUST%20(Technology%20Incubation%20Centre)%2C%20Office%20No.%20208%2C%20H-12%2C%20Islamabad%2C%20Pakistan)
http://www.abcdata.org/

On Oct 23 2018, at 1:01 am, Michael Miklavcic  
wrote:
>
> Sent
> On Mon, Oct 22, 2018 at 1:00 PM vpiserc...@gmail.com 
> wrote:
>
> > Hi,
> > can anyone invite to the metron slack channel?
> > Thanks,
> > vito piserchia
> > On 10/22/18 3:31 PM, zeo...@gmail.com wrote:
> > > Invite sent
> > >
> > > On Mon, Oct 22, 2018 at 9:26 AM Muhammed Irshad 
> > wrote:
> > >
> > > > Some one get me also the slack channel link ?
> > > > Thanks,
> > > > Muhammed Irshad
> > > > Q*Burst*
> > > > www.qburst.com
> > > >
> > > >
> > > > On Wed, Oct 17, 2018 at 7:33 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Sent
> > > > > On Wed, Oct 17, 2018 at 7:23 AM Tibor Meller 
> > > > > wrote:
> > > > >
> > > > > > Hi Guys,
> > > > > > Can you add me to the apache metron slack chanel?
> > > > > >
> > > > > > Thanks,
> > > > > > On Thu, Oct 4, 2018 at 1:14 PM Otto Fowler 
> > > > > > wrote:
> > > > > >
> > > > > > > Done
> > > > > > >
> > > > > > > On October 4, 2018 at 05:35:06, Tamás Fodor 
> > > > > > > (ftamas.m...@gmail.com)
> > > > > > wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > > Michael, can you add me as well?
> > > > > > > Thank you in advance!
> > > > > > > Tamas
> > > > > > > On Wed, Oct 3, 2018 at 4:27 PM Michael Miklavcic <
> > > > > > > michael.miklav...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Sent
> > > > > > > > On Wed, Oct 3, 2018 at 8:17 AM Shane Ardell <
> > > > > shane.m.ard...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > > Is it possible for someone to send me an invite to the Metron
> > > > Slack
> > > > > > > > > channel?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Shane
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
> > --
> > Vito Piserchia
> > Security and Software Engineer
> >
> > : vito.piserchia[at]dreamlab.net
> > : 4915 8835 2C18 9CAE F14F 2314 613D 51C5 106B 83EA
> > : https://dreamlab.net
> > : +41 31 398 66 66
> > : +41 31 398 66 69
> > -
> >
> > DreamLab Technologies AG
> > Monbijoustrasse 36
> > 3011 Bern, Switzerland
> >
> > -
> > This e-mail may contain confidential and/or privileged information.
> > If you are not the intended recipient (or have received this e-mail
> > in error) please notify the sender immediately and destroy this
> > e-mail. Any unauthorised copying, disclosure or distribution of the
> > material in this e-mail is strictly forbidden.
> >
> > -