If I select a durationAgo value which does not contain any profiler data I
get this, which is normal:
[Stellar]>>> PROFILE_GET('url-length', '20min.ch', 15, 'MINUTES')
[]
If I select a value which has profiler data available, then I always get an
exception. I can repeat that with any entity stored in the profiler.
[Stellar]>>> PROFILE_GET('url-length', '20min.ch', 120, 'MINUTES')
[!] Unable to execute:
com.tdunning.math.stats.TDigest.createAvlTreeDigest(D)Lcom/tdunning/math/stats/TDigest;
org.apache.metron.common.dsl.ParseException: Unable to execute:
com.tdunning.math.stats.TDigest.createAvlTreeDigest(D)Lcom/tdunning/math/stats/TDigest;
at
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
at
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:275)
at
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:373)
at
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:276)
at
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:412)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError:
com.tdunning.math.stats.TDigest.createAvlTreeDigest(D)Lcom/tdunning/math/stats/TDigest;
at
org.apache.metron.statistics.OnlineStatisticsProvider.<init>(OnlineStatisticsProvider.java:74)
at
org.apache.metron.statistics.OnlineStatisticsProviderConstructorAccess.newInstance(Unknown
Source)
at
org.apache.metron.common.utils.SerDeUtils$DefaultInstantiatorStrategy$1.newInstance(SerDeUtils.java:132)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1090)
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$KryoSerializableSerializer.read(DefaultSerializers.java:509)
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$KryoSerializableSerializer.read(DefaultSerializers.java:503)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at org.apache.metron.common.utils.SerDeUtils.fromBytes(SerDeUtils.java:246)
at
org.apache.metron.profiler.client.HBaseProfilerClient.lambda$get$3(HBaseProfilerClient.java:130)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
org.apache.metron.profiler.client.HBaseProfilerClient.get(HBaseProfilerClient.java:130)
at
org.apache.metron.profiler.client.HBaseProfilerClient.fetch(HBaseProfilerClient.java:109)
at
org.apache.metron.profiler.client.HBaseProfilerClient.fetch(HBaseProfilerClient.java:80)
at
org.apache.metron.profiler.client.stellar.GetProfile.apply(GetProfile.java:181)
at
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
at
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
at
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
at
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
... 8 more
On Sat, Feb 25, 2017 at 2:07 AM, Casey Stella <ceste...@gmail.com> wrote:
> What's the error exactly on 2?
>
> On Fri, Feb 24, 2017 at 19:36 shoggi <sho...@gmail.com> wrote:
>
>> Sorry for my late reply. From my previous post I still have two open
>> questions.
>>
>> 1) My domain_without_subdomains entries are not empty, why would I need
>> to add the 'if exists' statement? I'd like to understand the logic behind
>> that.
>>
>> 2) I get now data slowly into the profiler (even with 1 minute profiler
>> pushes). But I do get Java errors in Stellar on any queries based on that
>> profiler data. For example running "PROFILE_GET('url-length', '20min.ch',
>> 15, 'MINUTES')", throws an error. I can adjust the durationAgo were no data
>> is available and the command returns empty. I included a screenshot in my
>> previous mail.
>>
>> Do you have any insight into that?
>>
>> On Fri, Feb 24, 2017 at 4:02 PM, Casey Stella <ceste...@gmail.com> wrote:
>>
>> Yeah, we snapshot every 15 minutes. You can adjust that if you wnat to
>> snapshot the data in the profiler to hbase quicker. Glad your issue went
>> away :)
>>
>> On Fri, Feb 24, 2017 at 10:00 AM, shoggi <sho...@gmail.com> wrote:
>>
>> Hi
>>
>> The error has gone and I get now data into the profiler (base.png). The
>> time delay is large though. As a note, the domain_without_subdomains is in
>> my example never empty (es.png).
>>
>> When I try to query the data in stellar, then I get an error but only if
>> I the query matches data.
>>
>> Thank you
>> shoggi
>>
>>
>> On Fri, Feb 24, 2017 at 3:25 PM, Casey Stella <ceste...@gmail.com> wrote:
>>
>> I wonder if one of your records has a null domain_without_subdomains or
>> it doesn't exist for one of them. Try the following config:
>> {
>> "profiles": [
>> {
>> "profile": "url-length",
>> "foreach": "if exists(domain_without_subdomains) then
>> domain_without_subdomains else 'n/a'",
>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>> 'squid'",
>> "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>> "result": "m"
>> }
>> ]
>> }
>>
>>
>> On Fri, Feb 24, 2017 at 9:09 AM, shoggi <sho...@gmail.com> wrote:
>>
>> Sure
>>
>> Here is the output..
>>
>> # ./metron-deployment/scripts/platform-info.sh
>> Metron 0.3.0
>> --
>> * master
>> --
>> commit 59f8f03acbf66d5ac63f9156546833b0c98f57f2
>> Author: mmiklavc <michael.miklav...@gmail.com>
>> Date: Wed Nov 30 11:24:05 2016 -0700
>>
>> METRON-565: apps/metron/enrichment/indexed directory path does not
>> get created for metron cluster deployed via Ambari (mmiklavc) closes
>> apache/incubator-metron#365
>> --
>> .../roles/elasticsearch/tasks/elasticsearch.yml | 9 ++++-----
>> .../roles/pcap_replay/templates/pcap-replay | 9 ++++++---
>> 2 files changed, 10 insertions(+), 8 deletions(-)
>> --
>> ./metron-deployment/scripts/platform-info.sh: line 48: ansible: command
>> not found
>> --
>> ./metron-deployment/scripts/platform-info.sh: line 52: vagrant: command
>> not found
>> --
>> Python 2.6.6
>> --
>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>> 2015-11-10T17:41:47+01:00)
>> Maven home: /opt/apache-maven-3.3.9
>> Java version: 1.8.0_111, vendor: Oracle Corporation
>> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.
>> x86_64/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "linux", version: "2.6.32-642.11.1.el6.x86_64", arch: "amd64",
>> family: "unix"
>> --
>> Linux x 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>>
>> On Fri, Feb 24, 2017 at 2:55 PM, Nick Allen <n...@nickallen.org> wrote:
>>
>> Can you run the "./metron-deployment/scripts/platform-info.sh" script
>> and send the output to us?
>>
>> On Fri, Feb 24, 2017 at 8:38 AM, shoggi <sho...@gmail.com> wrote:
>>
>> Hi All
>>
>> I tried to set up following profiler config.
>>
>> > CONFIG_GET("PROFILER")
>> {
>> "profiles": [
>> {
>> "profile": "url-length",
>> "foreach": "domain_without_subdomains",
>> "onlyif": "source.type == 'squid'",
>> "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>> "result": "m"
>> }
>> ]
>> }
>>
>> Whenever I send squid data through, I get a null pointer exception & the
>> profiler dies. Other enrichments on the squid feed work fine and the
>> enriched logs reaches elasticsearch. I followed this guide:
>> https://cwiki.apache.org/confluence/display/METRON/Profiling+Your+Streams to
>> check my syntax and proceedings.
>>
>> Any help is highly appreciated.
>>
>> Cheers
>> Shoji
>>
>>
>>
>> 2017-02-24 14:20:35.832 o.a.s.util [ERROR] Async loop died!
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.
>> consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> Caused by: java.lang.NullPointerException
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.
>> execute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileSplitterBolt.
>> applyProfile(ProfileSplitterBolt.java:139) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileSplitterBolt.
>> doExecute(ProfileSplitterBolt.java:122) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileSplitterBolt.
>> execute(ProfileSplitterBolt.java:97) ~[stormjar.jar:?]
>> at org.apache.storm.daemon.executor$fn__6571$tuple_
>> action_fn__6573.invoke(executor.clj:734) ~[storm-core-1.0.1.2.5.0.0-
>> 1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> ... 6 more
>> 2017-02-24 14:20:35.834 o.a.s.d.executor [ERROR]
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.
>> consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> Caused by: java.lang.NullPointerException
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.
>> execute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileSplitterBolt.
>> applyProfile(ProfileSplitterBolt.java:139) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileSplitterBolt.
>> doExecute(ProfileSplitterBolt.java:122) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileSplitterBolt.
>> execute(ProfileSplitterBolt.java:97) ~[stormjar.jar:?]
>> at org.apache.storm.daemon.executor$fn__6571$tuple_
>> action_fn__6573.invoke(executor.clj:734) ~[storm-core-1.0.1.2.5.0.0-
>> 1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at
>> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> ... 6 more
>> 2017-02-24 14:20:35.908 o.a.s.util [ERROR] Halting process: ("Worker
>> died")
>> java.lang.RuntimeException: ("Worker died")
>> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.RestFn.invoke(RestFn.java:423)
>> [clojure-1.7.0.jar:?]
>> at
>> org.apache.storm.daemon.worker$fn__7176$fn__7177.invoke(worker.clj:765)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$mk_executor_data$fn__
>> 6388$fn__6389.invoke(executor.clj:275) [storm-core-1.0.1.2.5.0.0-
>> 1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:494)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
