Hi Shoggi,
In addition to Nick's and Casey's comments, I noticed your global.json does
not specify a profiler period. Try adding the following:
"profiler.client.period.duration" : "15",
"profiler.client.period.duration.units" : "MINUTES"
This period duration should match the duration you've specified in the
profiler.properties file:
profiler.period.duration=15
profiler.period.duration.units=MINUTES
If you want to use a different period duration, you should change the value
to match in *both* locations.
Best,
Mike
On Sun, Mar 5, 2017 at 6:09 PM, Nick Allen <n...@nickallen.org> wrote:
> What version of Metron are you using? Based on what I am seeing in the
> stack trace it seems to be a few versions ago. Any chance you'd be willing
> to try something newer like 0.3.1 RC5? It would be easier to help
> troubleshoot that way.
>
> On Sun, Mar 5, 2017 at 5:48 PM, shoggi <sho...@gmail.com> wrote:
>
>> The quorum and kafka config was ok, the host is actually called node1
>> (same system). The variables were set like that because I wanted to see if
>> I can set it to another value. Anyway, changed everything back and did
>> another of this:
>> - killed the topology
>> - created an empty profiler config
>> - restarted system
>> - added profiler configuration again (started with just one profile)
>> - data gets added to hbase, I get the error as shown previously, every
>> couple of flush cycles
>> - still no luck querying hbase out from stellar or via the enrichment
>> parser. No errors anywhere but the profiler NPE's
>>
>> you mentioned tick time.. is that something I can tune?
>>
>>
>> 2017-03-05 23:25:06.583 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>> profile: profile=url-length, entity=google.ch
>> 2017-03-05 23:25:06.584 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
>> {}, [60]'
>> java.lang.NullPointerException
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>> ecute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.executeRe
>> sult(ProfileBuilderBolt.java:316) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> flush$4(ProfileBuilderBolt.java:245) ~[stormjar.jar:?]
>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>> ~[?:1.8.0_77]
>> at
>> org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>> [stormjar.jar:?]
>> at
>> org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> 2017-03-05 23:25:06.585 o.a.s.d.executor [ERROR]
>> java.lang.NullPointerException
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>> ecute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.executeRe
>> sult(ProfileBuilderBolt.java:316) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> flush$4(ProfileBuilderBolt.java:245) ~[stormjar.jar:?]
>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>> ~[?:1.8.0_77]
>> at
>> org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>> [stormjar.jar:?]
>> at
>> org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> 2017-03-05 23:25:08.628 o.a.s.k.ZkCoordinator [INFO] Task [1/1]
>> Refreshing partition manager connections
>>
>> On Sun, Mar 5, 2017 at 6:24 PM, Casey Stella <ceste...@gmail.com> wrote:
>>
>>> Ok, so a couple of things I see here that you might try:
>>>
>>>
>>> - You should set kafka.zk and kafka.broker in profiler.properties to
>>> your real zookeeper quorum and kafka broker respectively
>>>
>>> In your profiler.json, instead of:
>>> {
>>> "profile": "url-bytes",
>>> "foreach": "if exists(domain_without_subdomains) then
>>> domain_without_subdomains else 'n/a'",
>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>> 'squid'",
>>> "update": { "n": "STATS_ADD(m, bytes)" },
>>> "result": "n"
>>> },
>>> {
>>> "profile": "content-type",
>>> "foreach": "if exists(domain_content) then domain_content else
>>> 'n/a'",
>>> "onlyif": "exists(domain_content) && source.type == 'squid'",
>>> "update": { "o": "STATS_ADD(m, bytes)" },
>>> "result": "o"
>>> }
>>> You might want (note the change on the update statements)
>>> {
>>> "profile": "url-bytes",
>>> "foreach": "if exists(domain_without_subdomains) then
>>> domain_without_subdomains else 'n/a'",
>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>> 'squid'",
>>> "update": { "n": "STATS_ADD(n, bytes)" },
>>> "result": "n"
>>> },
>>> {
>>> "profile": "content-type",
>>> "foreach": "if exists(domain_content) then domain_content else
>>> 'n/a'",
>>> "onlyif": "exists(domain_content) && source.type == 'squid'",
>>> "update": { "o": "STATS_ADD(o, bytes)" },
>>> "result": "o"
>>> }
>>>
>>> Try restarting the profiler topology and if you could look at the storm
>>> logs and see if you see any issues show up in the logs for the profiler.
>>>
>>> On Sun, Mar 5, 2017 at 7:11 AM, shoggi <sho...@gmail.com> wrote:
>>>
>>>> Here is my config:
>>>>
>>>> # global config
>>>> {
>>>> "es.clustername": "metron",
>>>> "es.ip": "172.16.16.2",
>>>> "es.port": "9300",
>>>> "es.date.format": "yyyy.MM.dd.HH"
>>>> }
>>>>
>>>> # profiler config
>>>> {
>>>> "profiles": [
>>>> {
>>>> "profile": "url-length",
>>>> "foreach": "if exists(domain_without_subdomains) then
>>>> domain_without_subdomains else 'n/a'",
>>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>>> 'squid'",
>>>> "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>>>> "result": "m"
>>>> },
>>>> {
>>>> "profile": "url-bytes",
>>>> "foreach": "if exists(domain_without_subdomains) then
>>>> domain_without_subdomains else 'n/a'",
>>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>>> 'squid'",
>>>> "update": { "n": "STATS_ADD(m, bytes)" },
>>>> "result": "n"
>>>> },
>>>> {
>>>> "profile": "content-type",
>>>> "foreach": "if exists(domain_content) then domain_content else
>>>> 'n/a'",
>>>> "onlyif": "exists(domain_content) && source.type == 'squid'",
>>>> "update": { "o": "STATS_ADD(m, bytes)" },
>>>> "result": "o"
>>>> }
>>>> ]
>>>> }
>>>>
>>>> # profiler properties
>>>> ##### Storm #####
>>>>
>>>> profiler.workers=1
>>>> profiler.executors=0
>>>> profiler.input.topic=indexing
>>>> profiler.period.duration=15
>>>> profiler.period.duration.units=MINUTES
>>>> profiler.ttl=30
>>>> profiler.ttl.units=MINUTES
>>>> profiler.hbase.salt.divisor=1000
>>>> profiler.hbase.table=profiler
>>>> profiler.hbase.column.family=P
>>>> profiler.hbase.batch=10
>>>> profiler.hbase.flush.interval.seconds=30
>>>>
>>>> ##### Kafka #####
>>>>
>>>> kafka.zk=node1:2181
>>>> kafka.broker=node1:6667
>>>> kafka.start=WHERE_I_LEFT_OFF
>>>>
>>>> On Sun, Mar 5, 2017 at 2:37 AM, Casey Stella <ceste...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sorry you are having issues! :(. Sometimes this is due to a mismatch
>>>>> in the tick time in the profiler between write and read.
>>>>>
>>>>> What's in your global config (METRON_HOME/config/zookeeper/global.json),
>>>>> profiler config (METRON_HOME/config/zookeeper/profiler.json) and
>>>>> profiler topology properties (METRON_HOME/config/profiler.properties)?
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Mar 4, 2017 at 17:38 shoggi <sho...@gmail.com> wrote:
>>>>>
>>>>>> Hey all
>>>>>>
>>>>>> Very strange, I had a few profilers working and wanted to show
>>>>>> someone (left system alone for a few days) & now can't query data
>>>>>> anymore.
>>>>>> I went so far to reboot the system, deleted the profiler table in hbase
>>>>>> and
>>>>>> loaded new data.
>>>>>>
>>>>>> I see the data in base but stellar does not let me query it anymore.
>>>>>> The queries return empty as if data does not exist but it's definitely
>>>>>> there. The timeframe can not be an issue, tired to use a very wide
>>>>>> stellar
>>>>>> query and as mentioned, loaded fresh data.
>>>>>>
>>>>>> Any troubleshooting hints? This bugs me, as I have not touched the
>>>>>> system & even restarted it to get rid of any possible stale connections.
>>>>>>
>>>>>> [Stellar]>>> PROFILE_GET( "url-bytes","google.com",60,"MINUTES")
>>>>>> []
>>>>>>
>>>>>> [Stellar]>>> PROFILE_GET( "url-bytes","google.com",60,"HOURS")
>>>>>> []
>>>>>>
>>>>>> Base data is there:
>>>>>>
>>>>>> \xFF\xFF\xFFkurl-bytesgoogle.com\x00\x00\x00\x00\x0 column=P:value,
>>>>>> timestamp=1488664729500, value=\x01\x00org.apache.metro
>>>>>> n.statistics.OnlineStatisticsProvide\xF2\x01\x00\x00\x00\x1C
>>>>>> \x00\x00\x00\x01@b\x
>>>>>> 1z\x96F
>>>>>> C0\x00\x00\x00\x00\x00\x00\x00\x00\x01@\x82H\x00\x00\x00\x00
>>>>>> \x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01@\x82H\x
>>>>>> 00\x00\x00\x00\x00A\x14\xE3D\x0
>>>>>> 0\x00\x00\x00@
>>>>>> \x19|\x87\xD0\xEA\xAA\xFB@\x82H\x00\x00\x00\x00\x00@
>>>>>> \x82H\x00\x00\x00\x00\x00@\x82H\x00\x00\x00\x00\x00\x
>>>>>> 00\x00\x00\x00\x00\x00\x00\x00\x
>>>>>>
>>>>>> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
>>>>>>
>>>>>> Thanks
>>>>>> shoggi
>>>>>>
>>>>>
>>>>
>>>
>>
>
