[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[
https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flower.min updated KAFKA-8106:
--
Description:
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
# Main config of Kafka
server:[num.network.threads=6;num.io.threads=128;queued.max.requests=500|http://num.network.threads%3D6%3Bnum.io.threads%3D128%3Bqueued.max.requests%3D500/]
# Number of TopicPartition : 50~2000
# Size of Single Message : 1024B
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It is negative for performance in common usage
scenarios .*_{color:#33}Therefore, we suggest that *_removing unnecessary
decompression operation_* when doing validation for compressed message when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.{color}
was:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
# Main config of Kafka
server:[num.network.threads=6;num.io.threads=128;queued.max.requests=500|http://num.network.threads%3D6%3Bnum.io.threads%3D128%3Bqueued.max.requests%3D500/]
# Number of TopicPartition : 50~2000
# Size of Single Message : 1024B
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[
https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flower.min updated KAFKA-8106:
--
Description:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
# Main config of Kafka
server:[num.network.threads=6;num.io.threads=128;queued.max.requests=500|http://num.network.threads%3D6%3Bnum.io.threads%3D128%3Bqueued.max.requests%3D500/]
# Number of TopicPartition : 50~2000
# Size of Single Message : 1024B
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It is negative for performance in common usage
scenarios .*_{color:#33}Therefore, we suggest that *_removing unnecessary
decompression operation_* when doing validation for compressed message when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.{color}
was:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
#
Main config of Kafka
server:[num.network.threads=6;num.io.threads=128;queued.max.requests=500|http://num.network.threads%3D6%3Bnum.io.threads%3D128%3Bqueued.max.requests%3D500/]
#
Number of TopicPartition : 50~2000
#
Size of Single Message : 1024B
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[
https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flower.min updated KAFKA-8106:
--
Description:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
#
Main config of Kafka
server:[num.network.threads=6;num.io.threads=128;queued.max.requests=500|http://num.network.threads%3D6%3Bnum.io.threads%3D128%3Bqueued.max.requests%3D500/]
#
Number of TopicPartition : 50~2000
#
Size of Single Message : 1024B
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It is negative for performance in common usage
scenarios .*_{color:#33}Therefore, we suggest that *_removing unnecessary
decompression operation_* when doing validation for compressed message when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.{color}
was:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
||server:num.network.threads||server:num.io.threads||server:queued.max.requests||Number
of TopicPartition||Size of Single Message||
|6|128|500|50~2000|1024B|
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[
https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flower.min updated KAFKA-8106:
--
Description:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
||server:num.network.threads||server:num.io.threads||server:queued.max.requests||Number
of TopicPartition||Size of Single Message||
|6|128|500|50~2000|1024B|
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_*
||Network inflow rate||CPU Used (%)||Disk write speed||Performance of
production||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |23,000,000 messages/s|
*_4.Phenomenon and my doubt_*
_The upper limit of CPU usage has been reached But it does not reach
the upper limit of the bandwidth of the server network. *We are doubtful about
which cost too much CPU time and we want to Improve performance and reduces
CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It is negative for performance in common usage
scenarios .*_{color:#33}Therefore, we suggest that *_removing unnecessary
decompression operation_* when doing validation for compressed message when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.{color}
was:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
||server:num.network.threads||server:num.io.threads||server:queued.max.requests||Number
of TopicPartition||Size of Single Message||
|6|128|500|50~2000|1024B|
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_* #
_*Performance*_:23,000,000 messages/s.
#
_*Resource usage*_:
||Network inflow rate||CPU(%)||Disk write speed||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |
*_4.Phenomenon and my doubt_*
__ The upper limit of CPU usage has been reached But it does not
reach the upper limit of the bandwidth of the server network. _*We are
doubtful about which cost too much CPU time and we want to Improve
performance and reduces CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It is negative for performance in common usage
scenarios .*_
h1. *Suggestion and Modifing
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[
https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flower.min updated KAFKA-8106:
--
Environment:
Server :
cpu:2*16 ;
MemTotal : 256G;
Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network
Connection ;
SSD.
Labels: performance (was: )
Description:
h1. *Introduction of Performance Testing*
We do performance testing about Kafka in specific scenarios as described
below .We build a kafka cluster with one broker,and create topics with
different number of partitions.Then we start lots of producer processes to send
large amounts of messages to one of the topics at one testing .
*_Specific Scenario_*
*_1.Main config of Kafka_*
||server:num.network.threads||server:num.io.threads||server:queued.max.requests||Number
of TopicPartition||Size of Single Message||
|6|128|500|50~2000|1024B|
*_2.Config of KafkaProducer_*
||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
|lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
*_3.The best result of performance testing_* #
_*Performance*_:23,000,000 messages/s.
#
_*Resource usage*_:
||Network inflow rate||CPU(%)||Disk write speed||
|550MB/s~610MB/s|97%～99%|:550MB/s~610MB/s |
*_4.Phenomenon and my doubt_*
__ The upper limit of CPU usage has been reached But it does not
reach the upper limit of the bandwidth of the server network. _*We are
doubtful about which cost too much CPU time and we want to Improve
performance and reduces CPU usage of Kafka server.*_
_*5.Analysis*_
We analysis the JFIR of Kafka server when doing performance testing .We
found the hot spot method is
*_"java.io.DataInputStream.readFully(byte[],int,int)"_* and
*_"org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(byte[],int,int)"_*.When
we checking thread stack information we also have found most CPU being
occupied by lots of thread which is busy decompressing messages.Then we read
source code of Kafka .
There is double-layer nested Iterator when LogValidator do validate
every record.And There is a decompression for each message when traversing
every RecordBatch iterator. It is consuming CPU and affect total performance
that decompress message._*The purpose of decompressing every messages just for
gain total size in bytes of one record and size in bytes of record body when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages.It is negative for performance in common usage
scenarios .*_
h1. *Suggestion and Modifing Code*
{color:#33} Therefore, we suggest that *_removing unnecessary
decompression operation_* when doing validation for compressed message when
magic value to use is above 1 and no format conversion or value overwriting is
required for compressed messages{color}
h1. *Performance Testing Again*
** We do performance testing again with package include of modified
code .the result of perfoemance testing is shown as below.*_The performance of
production improved by 40%~50%,and resource usage of CPU reduced by 30%~50%._*
#
*_Performance of production_* :32,000,000 messages/s.
# *_Resource usage_*:
||Network inflow rate||CPU(%)||Disk write speed||
|800M/s~1GB/s|10%～40%|800M/s~1GB/s |
Summary: Remove unnecessary decompression operation when logValidator
do validation. (was: Remove )
> Remove unnecessary decompression operation when logValidator do validation.
>
>
> Key: KAFKA-8106
> URL: https://issues.apache.org/jira/browse/KAFKA-8106
> Project: Kafka
> Issue Type: Bug
> Components: clients, core
>Affects Versions: 2.1.1
> Environment: Server :
> cpu:2*16 ;
> MemTotal : 256G;
> Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network
> Connection ;
> SSD.
>Reporter: Flower.min
>Priority: Major
> Labels: performance
>
> h1. *Introduction of Performance Testing*
> We do performance testing about Kafka in specific scenarios as
> described below .We build a kafka cluster with one broker,and create topics
> with different number of partitions.Then we start lots of producer processes
> to send large amounts of messages to one of the topics at one testing .
> *_Specific Scenario_*
>
> *_1.Main config of Kafka_*
> ||server:num.network.threads||server:num.io.threads||server:queued.max.requests||Number
> of TopicPartition||Size of Single Message||
> |6|128|500|50~2000|1024B|
>
> *_2.Config of KafkaProducer_*
> ||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
> |lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
>
> *_3.The best result of performance testing_* #
> _*Performance*_:23,000,000 messages/s.
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[ https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flower.min updated KAFKA-8106: -- Description: We do performance testing about kafka in specific scenarios as described below .We build a kafka cluster with one broker,and create topics with different number of partitions;then we start lots of producer processes to send large amounts of messages to one of the topics at one testing . *_specific scenario_* # *_server :_* cpu:2*16 ; MemTotal : 256G,Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection ; SSD. # _*Topics :*_ topic1:50 partitions,topic2:100partitions,topic3:200 partitions,...,2000 partiitons # _*size of Single Message* :_ 1024B *_Config of KafkaProducer :_ __* ** # _*compression.type*:_ lz4 # _*linger.ms*:_ 1000ms/2000ms/5000ms # *_batch.size:_* _1_6384B/10240B/102400B # _*buffer.memory:*_ 134217728B *_The best result of performance testing:_* # *_Pe_r*_*formance*:_2300 messages/s. # *_Resource usage:_* Network inflow rate : 550M/s~610MB/s,CPU(%) : 97%～99%,Disk write speed:550M/s~610MB/s . _*Phenomenon and my doubt：*_ _** The upper limit of CPU usage has been reached But it does not reach the upper limit of the bandwidth of the server network. We are doubtful about which cost too much CPU time and we want to Improve performance and reduces CPU usage of kafka server._ _**_ was: We do performance testing about kafka in specific scenarios as described below .We build a kafka cluster with one broker,and create topics with different number of partitions;then we start lots of producer processes to send large amounts of messages to one of the topics at one testing . *_specific scenario_* # *_server :_* cpu:2*16 ; MemTotal : 256G,Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection ; SSD. # _*Topics :* _ topic1:50 partitions,topic2:100partitions,topic3:200 partitions,...,2000 partiitons # _*size of Single Message* :_ 1024B *_Config of KafkaProducer :_ __* ** # _*compression.type*:_ lz4 # _*linger.ms*:_ 1000ms/2000ms/5000ms # *_batch.size:_* _1_6384B/10240B/102400B # _*buffer.memory:*_ 134217728B *_The best result of performance testing:_* # *_Performance:_*2300 messages/s. # *_Resource usage:_* Network inflow rate : 550M/s~610MB/s,CPU(%) : 97%～99%,Disk write speed:550M/s~610MB/s . _*Phenomenon and my doubt：*_ _** The upper limit of CPU usage has been reached But it does not reach the upper limit of the bandwidth of the server network. We are doubtful about which cost too much CPU time and we want to Improve performance and reduces CPU usage of kafka server._ _**_ > Remove unnecessary decompression operation when logValidator do validation. > > > Key: KAFKA-8106 > URL: https://issues.apache.org/jira/browse/KAFKA-8106 > Project: Kafka > Issue Type: Bug > Components: clients, core >Affects Versions: 2.1.1 >Reporter: Flower.min >Priority: Major > > We do performance testing about kafka in specific scenarios as > described below .We build a kafka cluster with one broker,and create topics > with different number of partitions;then we start lots of producer processes > to send large amounts of messages to one of the topics at one testing . > *_specific scenario_* > # *_server :_* cpu:2*16 ; MemTotal : 256G,Ethernet controller:Intel > Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection ; SSD. > # _*Topics :*_ topic1:50 partitions,topic2:100partitions,topic3:200 > partitions,...,2000 partiitons > # _*size of Single Message* :_ 1024B > *_Config of KafkaProducer :_ __* ** > # _*compression.type*:_ lz4 > # _*linger.ms*:_ 1000ms/2000ms/5000ms > # *_batch.size:_* _1_6384B/10240B/102400B > # _*buffer.memory:*_ 134217728B > *_The best result of performance testing:_* > # *_Pe_r*_*formance*:_2300 messages/s. > # *_Resource usage:_* Network inflow rate : 550M/s~610MB/s,CPU(%) : > 97%～99%,Disk write speed:550M/s~610MB/s . > _*Phenomenon and my doubt：*_ > _** The upper limit of CPU usage has been reached But it does > not reach the upper limit of the bandwidth of the server network. We are > doubtful about which cost too much CPU time and we want to Improve > performance and reduces CPU usage of kafka server._ > _**_ > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[ https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flower.min updated KAFKA-8106: -- Description: We do performance testing about kafka in specific scenarios as described below .We build a kafka cluster with one broker,and create topics with different number of partitions;then we start lots of producer processes to send large amounts of messages to one of the topics at one testing . *_specific scenario_* # *_server :_* cpu:2*16 ; MemTotal : 256G,Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection ; SSD. # _*Topics :* _ topic1:50 partitions,topic2:100partitions,topic3:200 partitions,...,2000 partiitons # _*size of Single Message* :_ 1024B *_Config of KafkaProducer :_ __* ** # _*compression.type*:_ lz4 # _*linger.ms*:_ 1000ms/2000ms/5000ms # *_batch.size:_* _1_6384B/10240B/102400B # _*buffer.memory:*_ 134217728B *_The best result of performance testing:_* # *_Performance:_*2300 messages/s. # *_Resource usage:_* Network inflow rate : 550M/s~610MB/s,CPU(%) : 97%～99%,Disk write speed:550M/s~610MB/s . _*Phenomenon and my doubt：*_ _** The upper limit of CPU usage has been reached But it does not reach the upper limit of the bandwidth of the server network. We are doubtful about which cost too much CPU time and we want to Improve performance and reduces CPU usage of kafka server._ _**_ > Remove unnecessary decompression operation when logValidator do validation. > > > Key: KAFKA-8106 > URL: https://issues.apache.org/jira/browse/KAFKA-8106 > Project: Kafka > Issue Type: Bug > Components: clients, core >Affects Versions: 2.1.1 >Reporter: Flower.min >Priority: Major > > We do performance testing about kafka in specific scenarios as > described below .We build a kafka cluster with one broker,and create topics > with different number of partitions;then we start lots of producer processes > to send large amounts of messages to one of the topics at one testing . > *_specific scenario_* > # *_server :_* cpu:2*16 ; MemTotal : 256G,Ethernet controller:Intel > Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection ; SSD. > # _*Topics :* _ topic1:50 partitions,topic2:100partitions,topic3:200 > partitions,...,2000 partiitons > # _*size of Single Message* :_ 1024B > *_Config of KafkaProducer :_ __* ** > # _*compression.type*:_ lz4 > # _*linger.ms*:_ 1000ms/2000ms/5000ms > # *_batch.size:_* _1_6384B/10240B/102400B > # _*buffer.memory:*_ 134217728B > *_The best result of performance testing:_* > # *_Performance:_*2300 messages/s. > # *_Resource usage:_* Network inflow rate : 550M/s~610MB/s,CPU(%) : > 97%～99%,Disk write speed:550M/s~610MB/s . > _*Phenomenon and my doubt：*_ > _** The upper limit of CPU usage has been reached But it does > not reach the upper limit of the bandwidth of the server network. We are > doubtful about which cost too much CPU time and we want to Improve > performance and reduces CPU usage of kafka server._ > _**_ > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[ https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flower.min updated KAFKA-8106: -- Description: (was: We do performance testing about kafka in specific scenarios as described below .We build a kafka cluster with one broker,and create topics with different number of partitions;then we start lots of producer processes to send large amounts of messages to one of the topics at one testing . *_specific scenario_* # *_server :_* cpu:2*16 ; MemTotal : 256G,Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection ; SSD. # _*Topics :*_ topic1:50 partitions,topic2:100partitions,topic3:200 partitions,...,2000 partiitons # _*size of Single Message* :_ 1024B *_Config of KafkaProducer :_ __* ** # _*compression.type*:_ lz4 # _*linger.ms*:_ 1000ms/2000ms/5000ms # *_batch.size:_* _1_6384B/10240B/102400B # _*buffer.memory:*_ 134217728B *_The best result of performance testing:_* # *_Pe_r*_*formance*:_2300 messages/s. # *_Resource usage:_* Network inflow rate : 550M/s~610MB/s,CPU(%) : 97%～99%,Disk write speed:550M/s~610MB/s . _*Phenomenon and my doubt：*_ _** The upper limit of CPU usage has been reached But it does not reach the upper limit of the bandwidth of the server network. We are doubtful about which cost too much CPU time and we want to Improve performance and reduces CPU usage of kafka server._ _**_ ) > Remove unnecessary decompression operation when logValidator do validation. > > > Key: KAFKA-8106 > URL: https://issues.apache.org/jira/browse/KAFKA-8106 > Project: Kafka > Issue Type: Bug > Components: clients, core >Affects Versions: 2.1.1 >Reporter: Flower.min >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8106) Remove unnecessary decompression operation when logValidator do validation.
[ https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flower.min updated KAFKA-8106: -- Summary: Remove unnecessary decompression operation when logValidator do validation. (was: Unnecessary ) > Remove unnecessary decompression operation when logValidator do validation. > > > Key: KAFKA-8106 > URL: https://issues.apache.org/jira/browse/KAFKA-8106 > Project: Kafka > Issue Type: Bug > Components: clients, core >Affects Versions: 2.1.1 >Reporter: Flower.min >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
