Re: How to query profiler data between custom dates

[Mohan Venkateshaiah]

Hi ,


You have to use profiler-spark if you have to profile based on telemetry data 
that was captured in the past . You should set the below properties at 
batch-profiler.properties before running your batch profiler ..


profiler.batch.input.begin="2019-01-01T00:00:00Z"


profiler.batch.input.end="2019-01-30T00:00:00Z"


Thanks
Mohan DV

From: Anil Donthireddy 
Reply-To: "u...@metron.apache.org" 
Date: Tuesday, February 12, 2019 at 8:10 PM
To: "u...@metron.apache.org" , "dev@metron.apache.org" 

Subject: How to query profiler data between custom dates

Hi,

I will have two custom dates (i.e 1st Jan 2019, 30 Jan 2019) picked and I need 
to query profiler data from a profile between these days (i.e. the profiler 
data between 1st Jan 2019 to 30th Jan 2019).

I have gone through the profiler client documentation and unable to figure out 
a way to query profiler data as per the above requirement.

I would like know if there is any way to query profiler data as per above 
requirement if I am missing any documentation.

Looking forward to hear suggestions on implementing above use case.

Thanking you.

Thanks,
Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

[Mohan Venkateshaiah]

Hi Anil, 

I had done similar to this in the past . In the stellar to get the hbase 
configuration we call HBaseConfiguration.create() , in that call hbase adds the 
hbase-site and core-site as resources to the config we probably SHOULD let 
people specify a base config. 

What I had done was in the global config,  I set a property called 
hbase.provider.impl,  it's the fully qualified class name for a class that 
implements the TableProvider interface which has one method: 

public HTableInterface getTable(Configuration config, String tableName) 
throws IOException

if you implement your own where you ignore the config argument and resolve the 
hbase table with your own injected config that will work

Thanks 
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler"  wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthire...@sstech.us) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1)  Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2)  Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath


/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf



One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: [ANNOUNCE] Shane Ardell is a committer

[Mohan Venkateshaiah]

Congrats Shane !!

Thanks 
Mohan DV

On 11/19/18, 10:26 PM, "Michael Miklavcic"  wrote:

Congrats!

On Mon, Nov 19, 2018 at 8:56 AM Shane Ardell 
wrote:

> I want to extend a huge thank you to everyone part of Apache Metron's PMC
> for offering me this opportunity!
>
> Cheers,
> Shane
>
> On Mon, Nov 19, 2018 at 4:53 PM zeo...@gmail.com  wrote:
>
> > Congrats Shane!
> >
> > Jon
> >
> > On Mon, Nov 19, 2018 at 10:43 AM Anand Subramanian <
> > asubraman...@hortonworks.com> wrote:
> >
> > > Many congratulations, Shane!
> > >
> > > Cheers,
> > > Anand
> > >
> > > On 11/19/18, 8:36 PM, "James Sirota"  wrote:
> > >
> > >
> > > The Project Management Committee (PMC) for Apache Metron has
> invited
> > > Shane Ardell to become a committer and we are pleased to announce that
> he
> > > has accepted.  I wanted to congratulate Shane on this achievement.
> > >
> > >
> > > Being a committer enables easier contribution to the project since
> > > there is no need to go via the patch submission process. This should
> > enable
> > > better productivity. Being a PMC member enables assistance with the
> > > management and to guide the direction of the project.
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PMC- Apache Metron
> > > jsirota AT apache DOT org
> > >
> > >
> > >
> > > --
> >
> > Jon Zeolla
> >
>

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

[Mohan Venkateshaiah]

+1 (non-binding)

Thanks
Mohan DV

On 11/2/18, 3:29 PM, "zeo...@gmail.com"  wrote:

+1 totally agree.

Jon

On Fri, Nov 2, 2018, 1:31 AM Anand Subramanian 

wrote:

> Piling on my +1 (non-binding) as well.
>
> On 11/2/18, 4:41 AM, "Ryan Merriman"  wrote:
>
> +1
>
> On Thu, Nov 1, 2018 at 5:38 PM Casey Stella 
> wrote:
>
> > +1
> > On Thu, Nov 1, 2018 at 18:34 Nick Allen  wrote:
> >
> > > +1
> > >
> > > On Thu, Nov 1, 2018, 6:27 PM Justin Leet 
> wrote:
> > >
> > > > +1, I haven't seen any case where the split-join topology isn't
> made
> > > > obsolete by the unified topology.
> > > >
> > > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Fellow Metronians,
> > > > >
> > > > > We've had the unified enrichment topology around for a number
> of
> > months
> > > > > now, it has proved itself stable, and there is yet to be a
> time that
> > I
> > > > have
> > > > > seen the split-join topology outperform the unified one. Here
> are
> > some
> > > > > simple reasons to deprecate the split-join topology.
> > > > >
> > > > >1. Unified topology performs better.
> > > > >2. The configuration, especially for performance tuning is
> much,
> > > much
> > > > >simpler in the unified model.
> > > > >3. The footprint within the cluster is smaller.
> > > > >4. One of the first activities for any install is that we
> spend
> > time
> > > > >instructing users to switch to the unified topology.
> > > > >5. One less moving part to maintain.
> > > > >
> > > > > I'd like to recommend that we deprecate the split-join
> topology and
> > > make
> > > > > the unified enrichment topology the new default.
> > > > >
> > > > > Best,
> > > > > Mike
> > > > >
> > > >
> > >
> >
>
>
> --

Jon Zeolla

Re: Anand is a new Committer!

[Mohan Venkateshaiah]

Congrats Anand !

On 1/11/18, 7:59 PM, "Casey Stella"  wrote:

The Project Management Committee (PMC) for Apache Metron has invited Anand
Subramanian to become a committer and we are pleased to announce that they
have accepted.

Congratulations and welcome, Anand!

Re: [VOTE] Metron 0.4.0 release (RC2)

[Mohan Venkateshaiah]

+1 (non-binding)

completed the below testing on the build 

1. Built metron 0.4.0
2. Deployed metron vagrant full-dev-platform
3. Verified metron is working by checking Ambari, Storm , indexing and data 
written to HDFS.
4. Added new squid telemetry 
5. Created new ‘whois’ enrichment and validated.  

Thanks
Mohan DV
On 4/28/17, 9:55 AM, "Matt Foley"  wrote:

Colleagues,

This is a call to vote on the second release candidate for Apache Metron 
0.4.0 (Note: not incubating any more!)
Full list of changes in this release, compared to 0.3.1:
  
https://dist.apache.org/repos/dist/dev/incubator/metron/0.4.0-RC2/RELEASE_NOTES

The tag/commit to be voted upon is:
  apache-metron-0.4.0-rc2
  This is identical to “master” as of the evening of 27 April.

The source archive being voted upon can be found here:

https://dist.apache.org/repos/dist/dev/incubator/metron/0.4.0-RC2/apache-metron-0.4.0-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/incubator/metron/0.4.0-RC2/

The release artifacts are signed with the following key:
7854 36A7 8258 6B71 829C 67A0  4169 AA27 ECB3 1663  in
https://dist.apache.org/repos/dist/dev/incubator/metron/0.4.0-RC2/KEYS

Please vote on releasing this package as Apache Metron 0.4.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open for at least 72 hours.
Please vote as follows:
+1 Release this package as Apache Metron 0.4.0
0 No opinion
  -1 Do not release this package because...

Thanks everybody for your on-going contributions to the Metron project!
--Matt
(Release Manager)