The Kafka Broker crashes are probably because you are running out of memory on the single virtual node that you are running.
On Oct 9, 2016 10:59 AM, "Dima Kovalyov" <[email protected]> wrote: Thank you Nick, I've re-build topologies. Another thing I've noticed after upgrade, my Kafka Broker crashes with started topologies after a while. The only relative to the crash message I was able to find is this: > [2016-10-09 11:30:35,013] ERROR Controller 1001 epoch 14 initiated > state change for partition [parser_invalid,0] from OfflinePartition to > OnlinePartition failed (state.change.logger) I have also attached kafka.tar.gz archive with my kafka logs from the moment of the Kafka Broker start till crash (15 minutes past). When I deploy topologies I receive following message, which I consider is just a warning, not error: > 20382 [main] WARN o.a.s.h.DefaultShader - Relocating > backtype/storm/security/auth/ICredentialsRenewer to > org/apache/storm/security/auth/ICredentialsRenewer in > org/apache/storm/hdfs/common/security/AutoHDFS.class. please modify > your code to use the new namespace Because I have following set in storm.yaml: client.jartransformer.class : 'org.apache.storm.hack.StormShadeTransformer' Which is exactly what you need to set for Storm 1.0+ to work with legacy jars according to: https://storm.apache.org/releases/1.0.0/ Last thing, when I start Kafka back again after crash and deploy topologies to Storm I have following message displaying in Storm UI: > Topology spouts lag error > kafkaSpout KAFKA Offset lags for kafka not supported for older > versions. Please update kafka spout to latest version. Does anyone know how to fix topology offset lag or how to update kafka spout to latest version? - Dima On 09/28/2016 03:08 PM, Nick Allen wrote: > You can re-build and deploy to your 'Quick Dev' environment by doing the > following from the root of Metron's source code. > > mvn clean package -DskipTests > cd metron-deployment/vagrant/quick-dev-platform > ./run_enrichment_role.sh > > > On Tue, Sep 27, 2016 at 11:57 PM, Dima Kovalyov <[email protected]> > wrote: > >> Thank you for reply James, >> >> I have performed Ambari 2.2 -> 2.4 and HDP 2.4 -> 2.5 upgrade on >> quick-dev-platform in Vagrant Metron setup. Most of it went just fine, >> except few intermittent errors. >> >> Can you please advise where do you want it to be documented and/or >> discussed? >> >> Right now I am battling issue with Storm. After the upgrade I need to >> re-build Storm Topology jars in order to propagate HDP 2.5 files in >> them. Can you please advise how I can do that on already running setup? >> (TLDR: how to build java jars manually?) >> >> Thank you! >> >> - Dima >> >> On 09/27/2016 06:59 PM, James Sirota wrote: >>> We didn't test it enough to know how well it works. We don't expect it >> to work at this time. If you can run it up for us and document all the >> issues that go wrong that would be greatly appreciated. >>> 26.09.2016, 16:54, "Satish Abburi" <[email protected]>: >>>> All, any one using/tested this combination. >>>> >>>> Thanks, >>>> Satish >>> ------------------- >>> Thank you, >>> >>> James Sirota >>> PPMC- Apache Metron (Incubating) >>> jsirota AT apache DOT org >>> >> >
