Resending the data as a text.
profiler content and the PUSH command output.
[root@krishna-4 zookeeper]# pwd
/usr/metron/0.3.1/config/zookeeper
[root@krishna-4 zookeeper]# ls -lrt
total 12
drwxr-xr-x. 2 root root 4096 Apr 28 10:16 parsers
drwxr-xr-x. 2 root root 89 Apr 28 10:16 enrichments
drwxr-xr-x. 2 root root 89 Apr 28 10:17 indexing
-rw-r--r--. 1 metron root 108 Jul 12 06:55 global.json
-rw-r--r--. 1 metron hadoop 196 Jul 13 03:30 profiler.json
[root@krishna-4 zookeeper]# cat profiler.json
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
[root@krishna-4 zookeeper]# cd ../..
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z
krishna-3:2181
log4j:WARN No appenders could be found for logger
(org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
[root@krishna-4 0.3.1]#
Regards,
Bala
From: Nick Allen [mailto:[email protected]]
Sent: Thursday, July 13, 2017 6:36 AM
To: [email protected]
Subject: RE: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
That's your problem then. You are not getting your profile definitions into
Zookeeper.
Send us a copy of your terminal session showing the location and content of
your profiler.json along with the command and ouput of you running the upload
script.
On Jul 12, 2017 8:41 PM, "Krishna Dhanekula"
<[email protected]<mailto:[email protected]>> wrote:
When I run DUMP I can see everything else under /config/zookeeper except the
profile (profiler.json) created by me.
________________________________
From: Nick Allen [[email protected]<mailto:[email protected]>]
Sent: Wednesday, July 12, 2017 2:44 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
If the Profiler definitions were uploaded to Zookeeper correctly, you should be
able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula
<[email protected]<mailto:[email protected]>> wrote:
When I run this command
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
(Krishna-3 is my node1 name)
[root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z
krishna-3:2181
log4j:WARN No appenders could be found for logger
(org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
I don’t see the configuration files getting uploaded to zookeeper. Should I
add out directory in this command.
Regards,
Balakrishna
From: Nick Allen [mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, July 12, 2017 8:15 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1
> it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is
wrong, the rest of the docs are correct.
Wrong:
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
expires<https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
Optional
Profile data is purged after this period of time, specified in
m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
Correct:
https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires
> Now I removed the entire field and from ‘profiler.json’ so that it profiled
> data never expires. I pushed and configuration and tested from kafka. I still
> don’t see the data storing to Hbase.
Did you follow the Getting Started guide? Can you follow that exactly and let
me know if that works for you.
On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula
<[email protected]<mailto:[email protected]>> wrote:
Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1
it’s mentioned as 30 days.
Now I removed the entire field and from ‘profiler.json’ so that it profiled
data never expires. I pushed and configuration and tested from kafka. I still
don’t see the data storing to Hbase.
Regards,
Balakrishna
From: Nick Allen [mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, July 12, 2017 6:55 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Quick correction for posterity... The 'expire' field is actually in days not
milliseconds. So that should not be a problem here.
On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball
<[email protected]<mailto:[email protected]>> wrote:
In your config for the individual profile though, you expire the content every
30ms (per the zip file)
Simon
On 12 Jul 2017, at 12:20, Krishna Dhanekula
<[email protected]<mailto:[email protected]>> wrote:
Thanks Simon for replying.
This is my ‘profiler.properties’ file content. TTL is 30 minutes and for every
30 seconds I am flushing to Hbase.
profiler.workers=1
profiler.executors=0
profiler.input.topic=indexing
profiler.period.duration=1
profiler.period.duration.units=MINUTES
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=1
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=10.10.110.184:2181<http://10.10.110.184:2181>
kafka.broker=10.10.110.184:6667<http://10.10.110.184:6667>
kafka.start=WHERE_I_LEFT_OFF
Regards,
Balakrishna
From: Simon Elliston Ball [mailto:[email protected]]
Sent: Wednesday, July 12, 2017 3:28 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance
Looks like you’ve set the profile to purge (expires) every 30 ms, and your
period is set to 30 minutes, so the data is being expired long before it has a
change to write.
Simon
On 12 Jul 2017, at 06:17, Krishna Dhanekula
<[email protected]<mailto:[email protected]>> wrote:
I have an problem where profiled data is not storing to Hbase instance:
Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
Attached my profiler.properties and profiler.json files, remote.yaml and
worker.log files.
Once I place the following message into kafka topic 'indexing'
[{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
It's picked by splitterBolt and passed till HBaseBolt. But data is not getting
stored to Hbase.
Attached metron-artifacts.zip
Attached storm-ui screenshot.storm-ui-screen.png
Please suggest what am I missing here.
Thanks in advance.
Attachments:
storm-ui-screen.png (17.4 kB)
metron-artifacts.zip (36.6 kB)
Regards,
Balakrishna
<metron_artifacts.zip><storm_ui_screen.png>
