Re: [GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Thats great Nick, thank you! You were right, by default it is set to: "es.ip": "{{ es_url }}" which expands to http://:9300 I have changed it to: "es.ip": "", "es.port": "9300", I have tested end2end briefly and Metron is working with HDP 2.5 on bare-metal. However there are multiple issues I've

Re: [GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

I would guess that your global configuration value for 'es.ip' looks something like "http://...; which is incorrect. It should just be the hostname or IP address with no protocol specifier. For example, by default the global properties look like the following for the Quick Dev environment. {

Re: [GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Thank you Jon, I have resolved it by increasing "max user processes" for user storm using: # su - storm $ ulimit -u 257597 Topologies are working without crashes now, however in Indexing topology indexingBolt now gives me this error: [ERROR] Async loop died! java.lang.RuntimeException:

Re: [GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Hi Dima, You probably want to increase the -Xmx setting in "worker.childopts", which is available in ambari under $Server:8080/#/main/services/STORM/configs. Jon On Tue, Nov 8, 2016 at 2:47 PM DimaKovalyov wrote: > Github user DimaKovalyov commented on the issue: > >

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user DimaKovalyov commented on the issue: https://github.com/apache/incubator-metron/pull/318 Thank you James, > Once you have data in your kafka queue this should go away. That is true! Once I create a topic and stream data through it the error is gone.

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/318 Would it make sense to put an instantiation or genesis message on the topic to avoid this in the future? Are there other ways to suppress this message on initial startup?

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/318 Once you have data in your kafka queue this should go away. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user DimaKovalyov commented on the issue: https://github.com/apache/incubator-metron/pull/318 Hello, I am not sure if this is a good place for jumping it, but I have installed Metron with HDP 2.5 using this great article:

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/318 I checked the licensing changes @mmiklavc and they look sensible to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 @ottobackwards - concur. If it exceeds the start/stop timeout (defaults to 30 seconds), Monit will terminate the start/stop process and try again. So, 60 worked on my larger machines

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/318 I think this monit timeout issue is part of the problem on low resource machines, and with 'zombie' storm threads being left behind --- If your project is set up for it, you can

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 @mmiklavc PR sent. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 A note for the community - the /tmp file problem **did** reoccur for us. As it turns out, the timeout default for starting up topologies in Monit was set too low. Normally, Storm cleans

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 @dlyle65535 I don't have a strong opinion on it. I'm giving attribution to @justinleet on this PR, since he laid the foundation of pretty much everything here. If you file separately we

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 @mmiklavc - I had to make a small tweak to the Quick Dev Vagrantfile for the new image. It's backwardly compatible, fwiw. Just added ambari-slave to the default tags. Do you

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 Yes. The Packer stuff is part of Metron and those instructions are in a

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/318 Are there instructions for doing either of those things? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 This ran well on EC2- deployment was good, expected data flow was good, Kafka offset tracking worked as expected. I'm +1, but there's some things to do prior to pulling this in

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 Before we accept this, I want to point out that I've changed the dependencies_with_url.csv file and that it's probably worth a look. --- If your project is set up for it, you can reply

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 Modifying build versions for Storm removes the Storm Kafka lag error from the UI.

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 It looks like those are just testing defaults that we don't actually set. Do I have that right? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 I'm now seeing a unit test failure when swapping out Apache Storm 1.0.1 for the HDP repo version. Tests pass in IntelliJ, not on the CLI. Investigating. ``` Results :

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 I've added a profile and am currently testing this out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 @cestella - I do. It's something we'll need to do anyway, may as well. I can't think of anything easier that'd do it. --- If your project is set up for it, you can reply to this email

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/318 @dlyle65535 makes sense. It didn't bug me, but I get it. You thinking adding a HDP profile is a good fix? --- If your project is set up for it, you can reply to this email and have

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 Kicking Travis --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 The /tmp issue didn't re-occur. I'd like to see if we can get this out without the error display in the UI. I was confused by it and I suspect others would be as well. --- If

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 @cestella the /tmp issue did not reappear after a redeploy --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 There also appears to be a version issue with the Storm-kafka client and server versions due to a mismatch with HDP. HDP 2.5 pulls in some commits from a later version of Storm.

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/318 It appears that the first issue (snort) is being addressed as part of METRON-514. Any insight on what's going on with the `/tmp/` directory issue? --- If your project is set up for it,

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 What should the path be for EC2? In the Flume Ansible scripts I see the following - java_home: /usr/jdk64/jdk1.8.0_60 I've created a separate Jira to track Flume migration to Ambari

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/318 I ran this up on EC2 and found a couple of issues: 1) Flume has the wrong java_home so the flume-agent process will not start. This is an ongoing issue each time we change the

[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/318 The Dyn DNS attacks seem to be affecting the Travis website. Will check on this later. --- If your project is set up for it, you can reply to this email and have your reply appear on