This should be handled fine by Storm. You can set topology.max.spout.pending as back pressure.
https://github.com/apache/metron/blob/master/metron-platform/Performance-tuning-guide.md#storm-tuning When there are bursts you may have some backup in the Kafka topics, but the topologies will catch up if tuned properly. Mike On Fri, Sep 14, 2018 at 12:10 PM Vets, Laurens <[email protected]> wrote: > For the record, here is how I 'fixed' this: > > 1. Stop Storm, it's crashing constantly anyway. Stop sending messages to > your Metron installation. > > 2. Export the messages from the Kafka topic that's crashing Storm so > that they're not lost. In my case that's the indexing topic. I have no > idea yet on how to re-ingest them. > > 3. Set the 'retention.ms' Kafka configuration setting to a small value, > then wait a minute. The command for this is > "/usr/hdp/current/kafka-broker/bin/kafka-configs.sh --zookeeper > localhost:2181 --entity-type topics --alter --add-config > retention.ms=1000 --entity-name indexing". > > 4. Make sure that the 'retention.ms' value is set: > "/usr/hdp/current/kafka-broker/bin/kafka-configs.sh --zookeeper > localhost:2181 --entity-type topics --describe --entity-name indexing" > > 5. Wait a couple of minutes, the Kafka log files should be empty. You > can check this with "ls -altr /tmp/kafka-logs/indexing/" or "du -h > /tmp/kafka-logs/indexing/". Replace "/tmp/kafka-logs/" with the correct > path to your Kafka logs directory. In my case, there was approx. 11GB of > data in the indexing topic. > > 6. Restore the default retention time: > "/usr/hdp/current/kafka-broker/bin/kafka-configs.sh --zookeeper > localhost:2181 --entity-type topics --alter --delete-config retention.ms > --entity-name indexing". > > (7. try to re-index the lost data. I have not found a way for this yet) > > At this point, start Storm again. It shouldn't crash anymore as there's > no data to index. > > Does this sound like a sound way to 'fix' these kinds of problems? I > suspect that I received a big burst of logs (Kibana seems to support > this) that Storm couldn't handle. Is there a way to better handle big > bursts? Or a rate control mechanism of some sort? > > On 13-Sep-18 11:39, Vets, Laurens wrote: > > 1. worker.childopts: -Xmx2048m > > > > 2. As in individual messages? Just small(-ish) JSON messages. A few > KBytes? > > > > On 13-Sep-18 11:21, Casey Stella wrote: > >> Two questions: > >> 1. How much memory are you giving the workers for the indexing topology? > >> 2. how large are the messages you're sending through? > >> > >> On Thu, Sep 13, 2018 at 2:00 PM Vets, Laurens <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> Hello list, > >> > >> I've installed OS updates on my Metron 0.4.2 yesterday, restarted > all > >> nodes and now my indexing topology keeps crashing. > >> > >> This is what I see in the Storm UI for the indexing topology: > >> > >> Topology stats: > >> 10m 0s 1304380 1953520 12499.833 1320 > >> 3h 0m 0s 1304380 1953520 12499.833 1320 > >> 1d 0h 0m 0s 1304380 1953520 12499.833 1320 > >> All time 1304380 1953520 12499.833 1320 > >> > >> Spouts: > >> kafkaSpout 1 1 1299940 1949080 12499.833 1320 > >> 0 > >> metron3 6702 java.lang.OutOfMemoryError: GC overhead limit > >> exceeded at java.lang.Long.valueOf(Long.java:840) at > >> > > org.apache.storm.kafka.spout.KafkaSpoutRetryExponentialBackoff$RetryEntryTimeStampComparator.compar > >> > >> Bolts: > >> hdfsIndexingBolt 1 1 1800 1800 0.278 7.022 > >> 1820 > >> 38.633 1800 0 metron3 6702 > >> java.lang.NullPointerException > >> at > >> > org.apache.metron.writer.hdfs.SourceHandler.handle(SourceHandler.java:80) > >> at > org.apache.metron.writer.hdfs.HdfsWriter.write(HdfsWriter.java:113) > >> at org.apache.metr Thur, 13 Sep 2018 07:35:02 > >> indexingBolt 1 1 1320 1320 0.217 7.662 1300 > >> 47.815 1300 0 metron3 6702 > >> java.lang.OutOfMemoryError: GC > >> overhead limit exceeded at > >> java.util.Arrays.copyOfRange(Arrays.java:3664) at > >> java.lang.String.<init>(String.java:207) at > >> org.json.simple.parser.Yylex.yytext(Yylex.jav Thur, 13 Sep 2018 > >> 07:37:33 > >> > >> When I check the Kafka topic, I can see that there's at least 3 > >> million > >> messages in the kafka indexing topic... I _suspect_ that the > indexing > >> topology tries to write those but fails, restarts, tries to write, > >> fails, etc... Metron is currently not ingesting any additional > >> messages, > >> but also can't seem to index the current ones... > >> > >> Any idea on how to proceed? > >> > >
