If the Profiler definitions were uploaded to Zookeeper correctly, you
should be able to see them when using the DUMP option. See example below.
[root@node1 0.4.0]# bin/zk_load_configs.sh -m DUMP -z node1:2181
...
PROFILER Config: profiler
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
On Wed, Jul 12, 2017 at 2:26 PM, Krishna Dhanekula <
[email protected]> wrote:
> When I run this command
>
>
>
> bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z krishna-3:2181
>
> (Krishna-3 is my node1 name)
>
>
>
> [root@krishna-4 0.3.1]# bin/zk_load_configs.sh -m PUSH -i
> config/zookeeper/ -z krishna-3:2181
>
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
>
> I don’t see the configuration files getting uploaded to zookeeper. Should
> I add out directory in this command.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
> *From:* Nick Allen [mailto:[email protected]]
> *Sent:* Wednesday, July 12, 2017 8:15 PM
>
> *To:* [email protected]
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> > Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Yes, there is an inconsistency in the README. The sentence in the table
> is wrong, the rest of the docs are correct.
>
>
>
> Wrong: https://github.com/apache/metron/tree/master/
> metron-analytics/metron-profiler#creating-profiles
>
>
>
> expires
> <https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires>
>
> Optional
>
> Profile data is purged after this period of time, specified
> in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days]
>
> Correct: https://github.com/apache/metron/tree/master/
> metron-analytics/metron-profiler#expires
>
>
>
>
>
> > Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
> Did you follow the Getting Started guide? Can you follow that exactly and
> let me know if that works for you.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula <
> [email protected]> wrote:
>
> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In
> Example1 it’s mentioned as 30 days.
>
>
>
> Now I removed the entire field and from ‘profiler.json’ so that it
> profiled data never expires. I pushed and configuration and tested from
> kafka. I still don’t see the data storing to Hbase.
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna *
>
> *From:* Nick Allen [mailto:[email protected]]
> *Sent:* Wednesday, July 12, 2017 6:55 PM
>
>
> *To:* [email protected]
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Quick correction for posterity... The 'expire' field is actually in days
> not milliseconds. So that should not be a problem here.
>
>
>
> On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball <
> [email protected]> wrote:
>
> In your config for the individual profile though, you expire the content
> every 30ms (per the zip file)
>
>
>
> Simon
>
>
>
>
>
>
>
> On 12 Jul 2017, at 12:20, Krishna Dhanekula <[email protected]>
> wrote:
>
>
>
> Thanks Simon for replying.
>
>
>
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for
> every 30 seconds I am flushing to Hbase.
>
>
>
> profiler.workers=1
>
> profiler.executors=0
>
> profiler.input.topic=indexing
>
> profiler.period.duration=1
>
> profiler.period.duration.units=MINUTES
>
> profiler.ttl=30
>
> profiler.ttl.units=MINUTES
>
> profiler.hbase.salt.divisor=1000
>
> profiler.hbase.table=profiler
>
> profiler.hbase.column.family=P
>
> profiler.hbase.batch=1
>
> profiler.hbase.flush.interval.seconds=30
>
>
>
> ##### Kafka #####
>
>
>
> kafka.zk=10.10.110.184:2181
>
> kafka.broker=10.10.110.184:6667
>
> kafka.start=WHERE_I_LEFT_OFF
>
>
>
>
>
>
>
> *Regards,*
>
>
>
> *Balakrishna*
>
>
>
>
>
> *From:* Simon Elliston Ball [mailto:[email protected]
> <[email protected]>]
> *Sent:* Wednesday, July 12, 2017 3:28 PM
> *To:* [email protected]
> *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase
> Instance
>
>
>
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your
> period is set to 30 minutes, so the data is being expired long before it
> has a change to write.
>
>
>
> Simon
>
>
>
>
>
> On 12 Jul 2017, at 06:17, Krishna Dhanekula <[email protected]>
> wrote:
>
>
>
> I have an problem where profiled data is not storing to Hbase instance:
>
>
>
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>
>
>
> https://metron.apache.org/current-book/metron-analytics/
> metron-profiler/index.html
>
>
>
> Attached my profiler.properties and profiler.json files, remote.yaml and
> worker.log files.
>
>
>
> Once I place the following message into kafka topic 'indexing'
>
>
>
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"
> 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","
> protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_
> addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>
>
>
> It's picked by splitterBolt and passed till HBaseBolt. But data is not
> getting stored to Hbase.
>
>
>
> Attached metron-artifacts.zip
>
>
>
> Attached storm-ui screenshot.storm-ui-screen.png
>
>
>
> Please suggest what am I missing here.
>
>
>
> Thanks in advance.
>
>
>
> Attachments:
>
> storm-ui-screen.png (17.4 kB)
>
> metron-artifacts.zip (36.6 kB)
>
>
>
>
>
> Regards,
>
> Balakrishna
>
> <metron_artifacts.zip><storm_ui_screen.png>
>
>
>
>
>
>
>
