Sorry we couldn't get you to a working state, but thanks so much for the
kind words!
On Mon, Mar 6, 2017 at 20:53 shoggi <sho...@gmail.com> wrote:
> Hi all
>
> I did add the 2 additional lines in the global.json file but also that did
> not help.
>
> I completely understand the issues running an out-versioned system and for
> that I am very thankful for all the help provided from all of you.
> Unfortunately I could not upgrade my 0.3.0 bare metal system as my time
> evaluating the system has run short. I wanted to use the time I had left to
> collect as much information as possible.
>
> I definitely like what I have seen and I absolutely believe this to be a
> great tool as it provides great flexibility in creating customized
> analytics & detection capabilities. Just thinking about the enrichment part
> and how I can tie in virtually anything I want, is exactly what I want from
> such a platform.
>
> You guys do an amazing job of helping the people wanting to know metron
> better. Undoubtedly it is very hard to work on the code and at the same
> time sharing your time to answer the many questions.
>
> If I may add two recommendations from my end.
>
> 1) A current and past threads point out to not loose touch with the
> 'window shoppers' (not sure if that term makes sense to you). Invest some
> time to get the installers right. People with various backgrounds look at
> your project right now. The simpler it is for them to get a system running,
> more likely it will be to get them excited about it. It took me more than a
> month to go through an installation, getting to know the various
> components, generating my own feeds prepared and ingested, included various
> data enrichments and creating relevant dashboards. You cannot expect this
> from someone who just wants to look at metron. True, it is still early days
> but the more you automate that 'acquisition' part, the more time you will
> have on your hands doing development work. It is easier said than done but
> you have here lots of people in the group who are willing to test &
> contribute. Use them..! Oh yeah.. I really like your youtube videos but you
> definitely need to promote those better. Add some proper titles so that
> they can be searched & found and add a short summary of what the video is
> about in the description field. It can be something very simple such as
> follows (Video from the 23rd of September):
> Topics covered:
> a) PCAP CLI
> b) Stellar introduction
> c) Ambari
> d) Profiler
>
> 2) Documentation is painful but grateful. Ask the community to help, you
> might even find someone with the passion to handle this for you. I did
> write my own installation manual for a bare metal rig, only to find out
> that others such as Dima did the same (and better). It might be helpful to
> have a dedicated person or a group of people to write metron documentation.
> Some things are for a developer just known facts. A newbie on the other
> hand can be easily deterred if not guided through properly. On that note..
> Apart of everything else I had to find out, one experience stuck with me.
> When I for example wanted to join a variable and a string with Stellar, I
> had to look at the source code to find the proper syntax. It never occurred
> to me to use square brackets and the short help only mentions to use a
> list. For the person who coded the function, it is crystal clear. Others
> might get to it eventually and then there are the rest who do not want to
> find out, it just needs to be clear. Also here, it is easier said than done
> but I strongly believe that you can gain lots of having someone oversee
> your docs and help getting more people excited about metron.
>
> Keep up with the great work !
>
> Regards
> Shoggi
>
> On Mon, Mar 6, 2017 at 8:12 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> Hi Shoggi,
>
> In addition to Nick's and Casey's comments, I noticed your global.json
> does not specify a profiler period. Try adding the following:
> "profiler.client.period.duration" : "15",
> "profiler.client.period.duration.units" : "MINUTES"
>
> This period duration should match the duration you've specified in the
> profiler.properties file:
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
>
> If you want to use a different period duration, you should change the
> value to match in *both* locations.
>
> Best,
> Mike
>
>
> On Sun, Mar 5, 2017 at 6:09 PM, Nick Allen <n...@nickallen.org> wrote:
>
> What version of Metron are you using? Based on what I am seeing in the
> stack trace it seems to be a few versions ago. Any chance you'd be willing
> to try something newer like 0.3.1 RC5? It would be easier to help
> troubleshoot that way.
>
> On Sun, Mar 5, 2017 at 5:48 PM, shoggi <sho...@gmail.com> wrote:
>
> The quorum and kafka config was ok, the host is actually called node1
> (same system). The variables were set like that because I wanted to see if
> I can set it to another value. Anyway, changed everything back and did
> another of this:
> - killed the topology
> - created an empty profiler config
> - restarted system
> - added profiler configuration again (started with just one profile)
> - data gets added to hbase, I get the error as shown previously, every
> couple of flush cycles
> - still no luck querying hbase out from stellar or via the enrichment
> parser. No errors anywhere but the profiler NPE's
>
> you mentioned tick time.. is that something I can tune?
>
>
> 2017-03-05 23:25:06.583 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
> profile: profile=url-length, entity=google.ch
> 2017-03-05 23:25:06.584 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
> {}, [60]'
> java.lang.NullPointerException
> at
> org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:117)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.executeResult(ProfileBuilderBolt.java:316)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:245)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2017-03-05 23:25:06.585 o.a.s.d.executor [ERROR]
> java.lang.NullPointerException
> at
> org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:117)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.executeResult(ProfileBuilderBolt.java:316)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:245)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
> ~[stormjar.jar:?]
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at
> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 2017-03-05 23:25:08.628 o.a.s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
> partition manager connections
>
> On Sun, Mar 5, 2017 at 6:24 PM, Casey Stella <ceste...@gmail.com> wrote:
>
> Ok, so a couple of things I see here that you might try:
>
>
> - You should set kafka.zk and kafka.broker in profiler.properties to
> your real zookeeper quorum and kafka broker respectively
>
> In your profiler.json, instead of:
> {
> "profile": "url-bytes",
> "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
> "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
> "update": { "n": "STATS_ADD(m, bytes)" },
> "result": "n"
> },
> {
> "profile": "content-type",
> "foreach": "if exists(domain_content) then domain_content else
> 'n/a'",
> "onlyif": "exists(domain_content) && source.type == 'squid'",
> "update": { "o": "STATS_ADD(m, bytes)" },
> "result": "o"
> }
> You might want (note the change on the update statements)
> {
> "profile": "url-bytes",
> "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
> "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
> "update": { "n": "STATS_ADD(n, bytes)" },
> "result": "n"
> },
> {
> "profile": "content-type",
> "foreach": "if exists(domain_content) then domain_content else
> 'n/a'",
> "onlyif": "exists(domain_content) && source.type == 'squid'",
> "update": { "o": "STATS_ADD(o, bytes)" },
> "result": "o"
> }
>
> Try restarting the profiler topology and if you could look at the storm
> logs and see if you see any issues show up in the logs for the profiler.
>
> On Sun, Mar 5, 2017 at 7:11 AM, shoggi <sho...@gmail.com> wrote:
>
> Here is my config:
>
> # global config
> {
> "es.clustername": "metron",
> "es.ip": "172.16.16.2",
> "es.port": "9300",
> "es.date.format": "yyyy.MM.dd.HH"
> }
>
> # profiler config
> {
> "profiles": [
> {
> "profile": "url-length",
> "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
> "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
> "update": { "m": "STATS_ADD(m, LENGTH(url))" },
> "result": "m"
> },
> {
> "profile": "url-bytes",
> "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
> "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
> "update": { "n": "STATS_ADD(m, bytes)" },
> "result": "n"
> },
> {
> "profile": "content-type",
> "foreach": "if exists(domain_content) then domain_content else
> 'n/a'",
> "onlyif": "exists(domain_content) && source.type == 'squid'",
> "update": { "o": "STATS_ADD(m, bytes)" },
> "result": "o"
> }
> ]
> }
>
> # profiler properties
> ##### Storm #####
>
> profiler.workers=1
> profiler.executors=0
> profiler.input.topic=indexing
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> profiler.ttl=30
> profiler.ttl.units=MINUTES
> profiler.hbase.salt.divisor=1000
> profiler.hbase.table=profiler
> profiler.hbase.column.family=P
> profiler.hbase.batch=10
> profiler.hbase.flush.interval.seconds=30
>
> ##### Kafka #####
>
> kafka.zk=node1:2181
> kafka.broker=node1:6667
> kafka.start=WHERE_I_LEFT_OFF
>
> On Sun, Mar 5, 2017 at 2:37 AM, Casey Stella <ceste...@gmail.com> wrote:
>
> Sorry you are having issues! :(. Sometimes this is due to a mismatch in
> the tick time in the profiler between write and read.
>
> What's in your global config (METRON_HOME/config/zookeeper/global.json),
> profiler config (METRON_HOME/config/zookeeper/profiler.json) and profiler
> topology properties (METRON_HOME/config/profiler.properties)?
>
>
>
> On Sat, Mar 4, 2017 at 17:38 shoggi <sho...@gmail.com> wrote:
>
> Hey all
>
> Very strange, I had a few profilers working and wanted to show someone
> (left system alone for a few days) & now can't query data anymore. I went
> so far to reboot the system, deleted the profiler table in hbase and loaded
> new data.
>
> I see the data in base but stellar does not let me query it anymore. The
> queries return empty as if data does not exist but it's definitely there.
> The timeframe can not be an issue, tired to use a very wide stellar query
> and as mentioned, loaded fresh data.
>
> Any troubleshooting hints? This bugs me, as I have not touched the system
> & even restarted it to get rid of any possible stale connections.
>
> [Stellar]>>> PROFILE_GET( "url-bytes","google.com",60,"MINUTES")
> []
>
> [Stellar]>>> PROFILE_GET( "url-bytes","google.com",60,"HOURS")
> []
>
> Base data is there:
>
> \xFF\xFF\xFFkurl-bytesgoogle.com\x00\x00\x00\x00\x0 column=P:value,
> timestamp=1488664729500,
> value=\x01\x00org.apache.metron.statistics.OnlineStatisticsProvide\xF2\x01\x00\x00\x00\x1C\x00\x00\x00\x01@b
> \x
> 1z\x96F
> C0\x00\x00\x00\x00\x00\x00\x00\x00\x01@
> \x82H\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01@
> \x82H\x00\x00\x00\x00\x00A\x14\xE3D\x0
> 0\x00\x00\x00@
> \x19|\x87\xD0\xEA\xAA\xFB@\x82H\x00\x00\x00\x00\x00@
> \x82H\x00\x00\x00\x00\x00@
> \x82H\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
>
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
>
> Thanks
> shoggi
>
>
>
>
>
>
>
>
