Hi kafka users,

I created a filtering stream with the Processor API;  input topic that have 
input rate at ~5 records per millisecond.  The filtering function on average 
takes 0.05milliseconds to complete which in ideal case would translate to 
(1/0.05)  20 records per millisecond.  However, when I benchmark the whole 
process, the streams is only processing 0.05 record per milliseconds.

Anyone have any idea on how to tune the steaming system to be faster as 0.05 
record is very far away from the theoretical max of 20?  The results above are 
per partition based where I have 16 partition for the input topic and all 
partitions have similar throughput.

I've only set the streams to have the following config:
        Properties config = new Properties();
        config.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap);
        config.put(StreamsConfig.STATE_DIR_CONFIG, stateDir);
        config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 50);

I'm not defining TimeExtractor so the default one is used.

Thanks for any help in advance.



This email and any attachment(s) are intended solely for the person(s) named 
above, and are or may contain information of a proprietary or confidential 
nature. If you are not the intended recipient(s), you should delete this 
message immediately. Any use, disclosure or distribution of this message 
without our prior consent is strictly prohibited.
This message may be subject to errors, incomplete or late delivery, 
interruption, interception, modification, or may contain viruses. Neither Daiwa 
Capital Markets Hong Kong Limited, its subsidiaries, affiliates nor their 
officers or employees represent or warrant the accuracy or completeness, nor 
accept any responsibility or liability whatsoever for any use of or reliance 
upon, this email or any of the contents hereof. The contents of this message 
are for information purposes only, and subject to change without notice.
This message is not and is not intended to be an offer or solicitation to buy 
or sell any securities or financial products, nor does any recommendation, 
opinion or advice necessarily reflect those of Daiwa Capital Markets Hong Kong 
Limited, its subsidiaries or affiliates.

Reply via email to