Re: HBaseDao and IndexDao abstraction

2018-10-23 Thread Muhammed Irshad 
Hi All,

I have got a solution for this using SHEW ( Simple HBase Enrichment Writer
) which is documented in confluence

but not in metron current book documentation
. I am going to give
this a try and see how it goes. Thanks a lot for Simon Elliston Ball
 & Metron slack channel :)

On Thu, Oct 18, 2018 at 10:51 AM Muhammed Irshad 
wrote:

> Mike,
>
> Thanks for replying. I had gone through it already and we are indexing our
> Active Directory logs to hdfs by streaming from Splunk. But I have a
> requirement of maintaining Active Directory asset inventory ( Just list of
> asset and their status not historic data) along with AD event indexing. So
> I thought of using HBase and was thinking the best place to put this logic
> ( Enrichment by writing a custom stellar which populate HBase column family
> for assets or In indexing layer ) . Then I saw the HBaseDao in
> documentation and wanted to understand what it is and weather it can be
> used to meet my use case.
>
> On Tue, Oct 16, 2018 at 7:41 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Hi Muhammed,
>>
>> I think you probably want to start with our parser infrastructure rather
>> than the DAO's for what you're doing. This series of blog posts gives a
>> use
>> case driven walkthrough that should help shed some light on things:
>> Part 1 (start here) -
>>
>> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
>> TOC of the 7-part series -
>>
>> https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana
>>
>> Here's some details about our parser infrastructure -
>>
>> https://github.com/apache/metron/tree/master/metron-platform/metron-parsers
>> ...which feeds into the data enrichment topology -
>>
>> https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment
>> ...which feeds into the indexing topology, which you've already found
>>
>> Hope this helps for a start!
>>
>> Best,
>> Mike Miklavcic
>>
>>
>> On Tue, Oct 16, 2018 at 12:05 AM Muhammed Irshad 
>> wrote:
>>
>> > Hi all,
>> >
>> > What is the actual use of HBaseDao documented in metron indexing
>> > documentation
>> > <
>> >
>> https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
>> > >
>> > under section 'The IndexDao Abstraction' ? From my reading I understand
>> it
>> > as a HBase indexing implementation which can be clubbed to hdfs for
>> updated
>> > data. What is the use of it as we cannot chose to index in HBase / hdfs
>> > dynamically ? Can some one explain an example about how to configure and
>> > use it ( More documentation link or reference is fine) ? I have a use
>> case
>> > where I need to maintain an Active Directory inventory, Using AD event
>> logs
>> > being indexed via metron. Is HBaseDao can be used for this use case ?
>> >
>> > --
>> > Muhammed Irshad K T
>> > Senior Software Engineer
>> > +919447946359
>> > irshadkt@gmail.com
>> > Skype : muhammed.irshad.k.t
>> >
>>
>
>
> --
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t
>


-- 
Muhammed Irshad K T
Senior Software Engineer
+919447946359
irshadkt@gmail.com
Skype : muhammed.irshad.k.t

Re: HBaseDao and IndexDao abstraction

2018-10-17 Thread Muhammed Irshad 
Mike,

Thanks for replying. I had gone through it already and we are indexing our
Active Directory logs to hdfs by streaming from Splunk. But I have a
requirement of maintaining Active Directory asset inventory ( Just list of
asset and their status not historic data) along with AD event indexing. So
I thought of using HBase and was thinking the best place to put this logic
( Enrichment by writing a custom stellar which populate HBase column family
for assets or In indexing layer ) . Then I saw the HBaseDao in
documentation and wanted to understand what it is and weather it can be
used to meet my use case.

On Tue, Oct 16, 2018 at 7:41 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hi Muhammed,
>
> I think you probably want to start with our parser infrastructure rather
> than the DAO's for what you're doing. This series of blog posts gives a use
> case driven walkthrough that should help shed some light on things:
> Part 1 (start here) -
>
> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
> TOC of the 7-part series -
>
> https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana
>
> Here's some details about our parser infrastructure -
> https://github.com/apache/metron/tree/master/metron-platform/metron-parsers
> ...which feeds into the data enrichment topology -
>
> https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment
> ...which feeds into the indexing topology, which you've already found
>
> Hope this helps for a start!
>
> Best,
> Mike Miklavcic
>
>
> On Tue, Oct 16, 2018 at 12:05 AM Muhammed Irshad 
> wrote:
>
> > Hi all,
> >
> > What is the actual use of HBaseDao documented in metron indexing
> > documentation
> > <
> >
> https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
> > >
> > under section 'The IndexDao Abstraction' ? From my reading I understand
> it
> > as a HBase indexing implementation which can be clubbed to hdfs for
> updated
> > data. What is the use of it as we cannot chose to index in HBase / hdfs
> > dynamically ? Can some one explain an example about how to configure and
> > use it ( More documentation link or reference is fine) ? I have a use
> case
> > where I need to maintain an Active Directory inventory, Using AD event
> logs
> > being indexed via metron. Is HBaseDao can be used for this use case ?
> >
> > --
> > Muhammed Irshad K T
> > Senior Software Engineer
> > +919447946359
> > irshadkt@gmail.com
> > Skype : muhammed.irshad.k.t
> >
>


-- 
Muhammed Irshad K T
Senior Software Engineer
+919447946359
irshadkt@gmail.com
Skype : muhammed.irshad.k.t

Re: HBaseDao and IndexDao abstraction

2018-10-16 Thread Michael Miklavcic 
Hi Muhammed,

I think you probably want to start with our parser infrastructure rather
than the DAO's for what you're doing. This series of blog posts gives a use
case driven walkthrough that should help shed some light on things:
Part 1 (start here) -
https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
TOC of the 7-part series -
https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana

Here's some details about our parser infrastructure -
https://github.com/apache/metron/tree/master/metron-platform/metron-parsers
...which feeds into the data enrichment topology -
https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment
...which feeds into the indexing topology, which you've already found

Hope this helps for a start!

Best,
Mike Miklavcic


On Tue, Oct 16, 2018 at 12:05 AM Muhammed Irshad 
wrote:

> Hi all,
>
> What is the actual use of HBaseDao documented in metron indexing
> documentation
> <
> https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
> >
> under section 'The IndexDao Abstraction' ? From my reading I understand it
> as a HBase indexing implementation which can be clubbed to hdfs for updated
> data. What is the use of it as we cannot chose to index in HBase / hdfs
> dynamically ? Can some one explain an example about how to configure and
> use it ( More documentation link or reference is fine) ? I have a use case
> where I need to maintain an Active Directory inventory, Using AD event logs
> being indexed via metron. Is HBaseDao can be used for this use case ?
>
> --
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t
>