Also forgot to mention, in Trident the offsetCommitPeriodMs and maxUncommittedOffsets don't do anything. Trident spouts "commit" into Storm's Zookeeper rather than Kafka, so the mentioned settings don't do anything.
2018-07-30 19:58 GMT+02:00 Shaik Asifullah <[email protected]>: > Hi, > I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea > of my topology is to modify messages from Kafka a little and push it to > Database. Each event of mine in Kafka is around 3 KB which is to be > inserted in one of the tables in DB. I want 10K inserts per second in the > database. Currently I am able to send around 1K messages per second with > the current configuration. > > .setOffsetCommitPeriodMs(1_000) > > .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST) > > .setMaxUncommittedOffsets(1000) > > .setPollTimeoutMs(2000) > > .setEmitNullTuples(false) > > .setProp(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") > > .setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE) > > .setProp("fetch.min.bytes",10*1024*1024) > > .setProp("max.partition.fetch.bytes",20*1024*1024) > > .setProp("fetch.max.wait.ms",5* 100) > > > I am using 8 worker nodes spread over 2 machines of 4 cores each and I am > using tumbling window of batch size 10K. I am using batch since writes to > DB takes around 1-2 seconds. Can you please help me in increasing it > performance ? > > > Thanks, > > Shaik Asifullah > >
