Hi,
I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea
of my topology is to modify messages from Kafka a little and push it to
Database. Each event of mine in Kafka is around 3 KB which is to be
inserted in one of the tables in DB. I want 10K inserts per second in the
database. Currently I am able to send around 1K messages per second with
the current configuration.
.setOffsetCommitPeriodMs(1_000)
.setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST)
.setMaxUncommittedOffsets(1000)
.setPollTimeoutMs(2000)
.setEmitNullTuples(false)
.setProp(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
.setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
.setProp("fetch.min.bytes",10*1024*1024)
.setProp("max.partition.fetch.bytes",20*1024*1024)
.setProp("fetch.max.wait.ms",5* 100)
I am using 8 worker nodes spread over 2 machines of 4 cores each and I am
using tumbling window of batch size 10K. I am using batch since writes to
DB takes around 1-2 seconds. Can you please help me in increasing it
performance ?
Thanks,
Shaik Asifullah
