> Under section ‘Creating Profiles’ it’s mentioned as milli secs. In Example1 it’s mentioned as 30 days.
Yes, there is an inconsistency in the README. The sentence in the table is wrong, the rest of the docs are correct. Wrong: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles expires <https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires> Optional Profile data is purged after this period of time, specified in m̶i̶l̶l̶i̶s̶e̶c̶o̶n̶d̶s̶ [days] Correct: https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#expires > Now I removed the entire field and from ‘profiler.json’ so that it profiled data never expires. I pushed and configuration and tested from kafka. I still don’t see the data storing to Hbase. Did you follow the Getting Started guide? Can you follow that exactly and let me know if that works for you. On Wed, Jul 12, 2017 at 9:31 AM, Krishna Dhanekula < [email protected]> wrote: > Under section ‘Creating Profiles’ it’s mentioned as milli secs. In > Example1 it’s mentioned as 30 days. > > > > Now I removed the entire field and from ‘profiler.json’ so that it > profiled data never expires. I pushed and configuration and tested from > kafka. I still don’t see the data storing to Hbase. > > > > > > > > *Regards,* > > > > *Balakrishna * > > *From:* Nick Allen [mailto:[email protected]] > *Sent:* Wednesday, July 12, 2017 6:55 PM > > *To:* [email protected] > *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase > Instance > > > > Quick correction for posterity... The 'expire' field is actually in days > not milliseconds. So that should not be a problem here. > > > > On Wed, Jul 12, 2017 at 8:02 AM, Simon Elliston Ball < > [email protected]> wrote: > > In your config for the individual profile though, you expire the content > every 30ms (per the zip file) > > > > Simon > > > > > > > > On 12 Jul 2017, at 12:20, Krishna Dhanekula <[email protected]> > wrote: > > > > Thanks Simon for replying. > > > > This is my ‘profiler.properties’ file content. TTL is 30 minutes and for > every 30 seconds I am flushing to Hbase. > > > > profiler.workers=1 > > profiler.executors=0 > > profiler.input.topic=indexing > > profiler.period.duration=1 > > profiler.period.duration.units=MINUTES > > profiler.ttl=30 > > profiler.ttl.units=MINUTES > > profiler.hbase.salt.divisor=1000 > > profiler.hbase.table=profiler > > profiler.hbase.column.family=P > > profiler.hbase.batch=1 > > profiler.hbase.flush.interval.seconds=30 > > > > ##### Kafka ##### > > > > kafka.zk=10.10.110.184:2181 > > kafka.broker=10.10.110.184:6667 > > kafka.start=WHERE_I_LEFT_OFF > > > > > > > > *Regards,* > > > > *Balakrishna* > > > > > > *From:* Simon Elliston Ball [mailto:[email protected] > <[email protected]>] > *Sent:* Wednesday, July 12, 2017 3:28 PM > *To:* [email protected] > *Subject:* Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase > Instance > > > > Looks like you’ve set the profile to purge (expires) every 30 ms, and your > period is set to 30 minutes, so the data is being expired long before it > has a change to write. > > > > Simon > > > > > > On 12 Jul 2017, at 06:17, Krishna Dhanekula <[email protected]> > wrote: > > > > I have an problem where profiled data is not storing to Hbase instance: > > > > Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link: > > > > https://metron.apache.org/current-book/metron-analytics/ > metron-profiler/index.html > > > > Attached my profiler.properties and profiler.json files, remote.yaml and > worker.log files. > > > > Once I place the following message into kafka topic 'indexing' > > > > [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":" > 10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2"," > protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_ > addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}] > > > > It's picked by splitterBolt and passed till HBaseBolt. But data is not > getting stored to Hbase. > > > > Attached metron-artifacts.zip > > > > Attached storm-ui screenshot.storm-ui-screen.png > > > > Please suggest what am I missing here. > > > > Thanks in advance. > > > > Attachments: > > storm-ui-screen.png (17.4 kB) > > metron-artifacts.zip (36.6 kB) > > > > > > Regards, > > Balakrishna > > <metron_artifacts.zip><storm_ui_screen.png> > > > > >
