Re: [DISCUSS] Storm topology.classpath setting

2017-04-25 Thread Michael Miklavcic
Wait, which restart isn't required? Justin and I just stepped through this and we were getting errors with the Maxmind DB without restarting the enrichment topology. The reason was that the topology had null for the topology.classpath until I restarted it. On Tue, Apr 25, 2017 at 3:21 PM, Michael

Re: [DISCUSS] Storm topology.classpath setting

2017-04-25 Thread Michael Miklavcic
Yeah, I had originally added that feature a while back in https://issues.apache.org/jira/browse/METRON-356. But strictly speaking I think it should be able to be specified at the topology level per the Storm docs, https://github.com/apache/storm/blob/master/conf/defaults.yaml#L224 On Tue, Apr 25,

Re: [DISCUSS] Storm topology.classpath setting

2017-04-25 Thread Justin Leet
topology.classpath is a topology level config, so Mike's potential alternative is to migrate that down from Ambari into the actual topologies we launch (assuming it works as expected). Basically we're trading ease of use in Ambari vs. additional setup for each topology that's run. I'm in favor of

Re: [DISCUSS] Storm topology.classpath setting

2017-04-25 Thread Casey Stella
I don't know that I'm offended by either approach, but the way we have it now isn't offending me overly, honestly. Casey On Tue, Apr 25, 2017 at 5:11 PM, David Lyle wrote: > I think you have one more restart than is strictly required but I get your > point. IIRC, when I did the original impleme

Re: [DISCUSS] Storm topology.classpath setting

2017-04-25 Thread David Lyle
I think you have one more restart than is strictly required but I get your point. IIRC, when I did the original implementation there was a bit of a wrinkle getting the right configs on the right classpath so I punted in favor of topology.classpath. If you've got a more efficient way, I'm all for i

[DISCUSS] Storm topology.classpath setting

2017-04-25 Thread Michael Miklavcic
We currently define topology.classpath at the global level for Storm. The reason for this feature was to enable setting the classpath to include things like hbase-site and core-site without modifying the topology jar files on the cluster. That made things simpler, but this also has the consequence