Hi Stig, I hope you are doing good. My topology has been working great until I restarted my Storm cluster. There were a few nimbus issues as it was refusing to restart. Currently I am getting Java Heap OutOfMemory issues with my Trident Topology. Can you please help me in increasing "Assigned Memory" to my Trident Topology ? I have posted my question here very long back https://stackoverflow.com/questions/50443737/increasing-assigned-memory-for-a-topology-in-storm . Please help me in resolving this issue.
Thank you, Shaik Asifullah On Tue, Jul 31, 2018 at 1:57 AM Stig Rohde Døssing <[email protected]> wrote: > Also forgot to mention, in Trident the offsetCommitPeriodMs and > maxUncommittedOffsets don't do anything. Trident spouts "commit" into > Storm's Zookeeper rather than Kafka, so the mentioned settings don't do > anything. > > 2018-07-30 19:58 GMT+02:00 Shaik Asifullah <[email protected]>: > >> Hi, >> I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea >> of my topology is to modify messages from Kafka a little and push it to >> Database. Each event of mine in Kafka is around 3 KB which is to be >> inserted in one of the tables in DB. I want 10K inserts per second in the >> database. Currently I am able to send around 1K messages per second with >> the current configuration. >> >> .setOffsetCommitPeriodMs(1_000) >> >> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST) >> >> .setMaxUncommittedOffsets(1000) >> >> .setPollTimeoutMs(2000) >> >> .setEmitNullTuples(false) >> >> .setProp(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") >> >> .setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE) >> >> .setProp("fetch.min.bytes",10*1024*1024) >> >> .setProp("max.partition.fetch.bytes",20*1024*1024) >> >> .setProp("fetch.max.wait.ms",5* 100) >> >> >> I am using 8 worker nodes spread over 2 machines of 4 cores each and I >> am using tumbling window of batch size 10K. I am using batch since writes >> to DB takes around 1-2 seconds. Can you please help me in increasing it >> performance ? >> >> >> Thanks, >> >> Shaik Asifullah >> >> >
