[
https://issues.apache.org/jira/browse/IGNITE-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Shishkov updated IGNITE-19910:
-----------------------------------
Description:
Currently, in CDC through Kafka applications, single timeout property
({{kafkaRequestTimeout)}} is used for all Kafka related operations instead of
built-in timeouts of Kafka clients (moreover, default value of 3 seconds does
not correspond to Kafka clients defaults):
||Client||Timeout||Default value, s||
|{{KafkaProducer}}|{{delivery.timeout.ms}}|120|
|{{KafkaProducer}}|{{request.timeout.ms}}|30|
|{{KafkaConsumer}}|{{default.api.timeout.ms}}|60|
|{{KafkaConsumer}}|{{request.timeout.ms}}|30|
----
Places where {{kafkaRequestTimeout}} is used:
*ignite-cdc.sh* (in {{IgniteToKafkaCdcStreamer}}):
# Waiting for {{KafkaProducer#send}} future to finish.
*kafka-to-ignite.sh:*
# In {{KafkaToIgniteCdcStreamerApplier}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#commitSync}}
# In {{KafkaToIgniteMetadataUpdater}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#endOffsets}}
## {{KafkaConsumer#partitionsFor}}
----
Also, there is difference in behavior of {{KafkaConsumer#poll}} and other used
methods of {{KafkaConsumer}}. {{KafkaConsumer#poll}} just waits for data until
specified timeout expires, but other methods will raise an exception, thus,
timeout should not be to low for them.
But, on the other hand, {{#poll}} should occur quite often and if we set up too
large timeout for it, then we can face delays of replication, because, when
there are no new data in topics, {{#poll}} will block until new data or until
timeout will expire. It is not desired behavior, because in this situation we
can have pending updates which will wait to be processed.
----
Links:
#
https://kafka.apache.org/27/documentation.html#producerconfigs_delivery.timeout.ms
#
https://kafka.apache.org/27/documentation.html#producerconfigs_request.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_default.api.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_request.timeout.ms
was:
Currently, in CDC through Kafka applications, single timeout property
({{kafkaRequestTimeout)}} is used for all Kafka related operations instead of
built-in timeouts of Kafka clients (moreover, default value of 3 seconds does
not correspond to Kafka clients defaults):
# {{KafkaProducer}}:
** {{delivery.timeout.ms}}, defaults to 120s [1]
** {{request.timeout.ms}}, defaults to 30s [2]
# {{KafkaKonsumer}}:
** {{default.api.timeout.ms}}, defaults to 60s [3]
** {{request.timeout.ms}}, defaults to 30s [4]
----
Places where {{kafkaRequestTimeout}} is used:
*ignite-cdc.sh* (in {{IgniteToKafkaCdcStreamer}}):
# Waiting for {{KafkaProducer#send}} future to finish.
*kafka-to-ignite.sh:*
# In {{KafkaToIgniteCdcStreamerApplier}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#commitSync}}
# In {{KafkaToIgniteMetadataUpdater}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#endOffsets}}
## {{KafkaConsumer#partitionsFor}}
----
Also, there is difference in behavior of {{KafkaConsumer#poll}} and other used
methods of {{KafkaConsumer}}. {{KafkaConsumer#poll}} just waits for data until
specified timeout expires, but other methods will raise an exception, thus,
timeout should not be to low for them.
But, on the other hand, {{#poll}} should occur quite often and if we set up too
large timeout for it, then we can face delays of replication, because, when
there are no new data in topics, {{#poll}} will block until new data or until
timeout will expire. It is not desired behavior, because in this situation we
can have pending updates which will wait to be processed.
----
Links:
#
https://kafka.apache.org/27/documentation.html#producerconfigs_delivery.timeout.ms
#
https://kafka.apache.org/27/documentation.html#producerconfigs_request.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_default.api.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_request.timeout.ms
> CDC through Kafka: refactor timeouts
> ------------------------------------
>
> Key: IGNITE-19910
> URL: https://issues.apache.org/jira/browse/IGNITE-19910
> Project: Ignite
> Issue Type: Task
> Components: extensions
> Reporter: Ilya Shishkov
> Priority: Minor
> Labels: IEP-59, ise
>
> Currently, in CDC through Kafka applications, single timeout property
> ({{kafkaRequestTimeout)}} is used for all Kafka related operations instead of
> built-in timeouts of Kafka clients (moreover, default value of 3 seconds does
> not correspond to Kafka clients defaults):
> ||Client||Timeout||Default value, s||
> |{{KafkaProducer}}|{{delivery.timeout.ms}}|120|
> |{{KafkaProducer}}|{{request.timeout.ms}}|30|
> |{{KafkaConsumer}}|{{default.api.timeout.ms}}|60|
> |{{KafkaConsumer}}|{{request.timeout.ms}}|30|
> ----
> Places where {{kafkaRequestTimeout}} is used:
> *ignite-cdc.sh* (in {{IgniteToKafkaCdcStreamer}}):
> # Waiting for {{KafkaProducer#send}} future to finish.
> *kafka-to-ignite.sh:*
> # In {{KafkaToIgniteCdcStreamerApplier}} as parameter of methods:
> ## {{KafkaConsumer#poll}}
> ## {{KafkaConsumer#commitSync}}
> # In {{KafkaToIgniteMetadataUpdater}} as parameter of methods:
> ## {{KafkaConsumer#poll}}
> ## {{KafkaConsumer#endOffsets}}
> ## {{KafkaConsumer#partitionsFor}}
> ----
> Also, there is difference in behavior of {{KafkaConsumer#poll}} and other
> used methods of {{KafkaConsumer}}. {{KafkaConsumer#poll}} just waits for data
> until specified timeout expires, but other methods will raise an exception,
> thus, timeout should not be to low for them.
> But, on the other hand, {{#poll}} should occur quite often and if we set up
> too large timeout for it, then we can face delays of replication, because,
> when there are no new data in topics, {{#poll}} will block until new data or
> until timeout will expire. It is not desired behavior, because in this
> situation we can have pending updates which will wait to be processed.
> ----
> Links:
> #
> https://kafka.apache.org/27/documentation.html#producerconfigs_delivery.timeout.ms
> #
> https://kafka.apache.org/27/documentation.html#producerconfigs_request.timeout.ms
> #
> https://kafka.apache.org/27/documentation.html#consumerconfigs_default.api.timeout.ms
> #
> https://kafka.apache.org/27/documentation.html#consumerconfigs_request.timeout.ms
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
