[
https://issues.apache.org/jira/browse/IGNITE-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Shishkov updated IGNITE-19910:
-----------------------------------
Description:
Currently, in CDC through Kafka applications, single timeout property
({{kafkaRequestTimeout)}} is used for all Kafka related operations instead of
built-in timeouts of Kafka clients (moreover, default value of 3 seconds does
not correspond to Kafka clients defaults):
# {{KafkaProducer}}:
** {{delivery.timeout.ms}}, defaults to 120s [1]
** {{request.timeout.ms}}, defaults to 30s [2]
# {{KafkaKonsumer}}:
** {{default.api.timeout.ms}}, defaults to 60s [3]
** {{request.timeout.ms}}, defaults to 30s [4]
----
Places where {{kafkaRequestTimeout}} is used:
*ignite-cdc.sh* (in {{IgniteToKafkaCdcStreamer}}):
# Waiting for {{KafkaProducer#send}} future to finish.
*kafka-to-ignite.sh:*
# In {{KafkaToIgniteCdcStreamerApplier}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#commitSync}}
# In {{KafkaToIgniteMetadataUpdater}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#endOffsets}}
## {{KafkaConsumer#partitionsFor}}
----
Also, there is difference of behavior of {{KafkaConsumer#poll}} and other used
methods of {{KafkaConsumer}}. {{KafkaConsumer#poll}} just waits for data until
specified timeout expires, but other methods will raise an exception, thus,
timeout should not be to low for them.
But, on the other hand, {{#poll}} should happen relatively frequent and if we
set up too large timeout for it, then we can face delays of replication,
because, when there are no new data in topics, {{#poll}} will block until new
data or until timeout will expire. It is not desired behavior, because in this
situation we can have pending updates which will wait to be processed.
----
Links:
#
https://kafka.apache.org/27/documentation.html#producerconfigs_delivery.timeout.ms
#
https://kafka.apache.org/27/documentation.html#producerconfigs_request.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_default.api.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_request.timeout.ms
was:
Currently, in CDC through Kafka applications, single timeout property
({{kafkaRequestTimeout)}} is used for all Kafka related operations instead of
built-in timeouts of Kafka clients (moreover, default value of 3 seconds does
not correspond to Kafka clients defaults):
# {{KafkaProducer}}:
** {{delivery.timeout.ms}}, defaults to 120s [1]
** {{request.timeout.ms}}, defaults to 30s [2]
# {{KafkaKonsumer}}:
** {{default.api.timeout.ms}}, defaults to 60s [3]
** {{request.timeout.ms}}, defaults to 30s [4]
----
Places where {{kafkaRequestTimeout}} is used:
*ignite-cdc.sh* (in {{IgniteToKafkaCdcStreamer}}):
# Waiting for {{KafkaProducer#send}} future to finish.
*kafka-to-ignite.sh:*
# In {{KafkaToIgniteCdcStreamerApplier}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#commitSync}}
# In {{KafkaToIgniteMetadataUpdater}} as parameter of methods:
## {{KafkaConsumer#poll}}
## {{KafkaConsumer#endOffsets}}
## {{KafkaConsumer#partitionsFor}}
----
Also, there is difference of behavior of {{KafkaConsumer#poll}} and other used
methods of {{KafkaConsumer}}. {{#poll}} just waits for data until specified
timeout expires, but other methods will raise an exception, thus, timeout
should not be to low for them.
But, on the other hand, {{#poll}} should happen relatively frequent and if we
set up too large timeout for it, when there are no new data in topics,
{{#poll}} poll will block until new data or until timeout will expire. It is
not desired behavior, because pending updates will wait to be processed.
----
Links:
#
https://kafka.apache.org/27/documentation.html#producerconfigs_delivery.timeout.ms
#
https://kafka.apache.org/27/documentation.html#producerconfigs_request.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_default.api.timeout.ms
#
https://kafka.apache.org/27/documentation.html#consumerconfigs_request.timeout.ms
> CDC through Kafka: refactor timeouts
> ------------------------------------
>
> Key: IGNITE-19910
> URL: https://issues.apache.org/jira/browse/IGNITE-19910
> Project: Ignite
> Issue Type: Task
> Components: extensions
> Reporter: Ilya Shishkov
> Priority: Minor
> Labels: IEP-59, ise
>
> Currently, in CDC through Kafka applications, single timeout property
> ({{kafkaRequestTimeout)}} is used for all Kafka related operations instead of
> built-in timeouts of Kafka clients (moreover, default value of 3 seconds does
> not correspond to Kafka clients defaults):
> # {{KafkaProducer}}:
> ** {{delivery.timeout.ms}}, defaults to 120s [1]
> ** {{request.timeout.ms}}, defaults to 30s [2]
> # {{KafkaKonsumer}}:
> ** {{default.api.timeout.ms}}, defaults to 60s [3]
> ** {{request.timeout.ms}}, defaults to 30s [4]
> ----
> Places where {{kafkaRequestTimeout}} is used:
> *ignite-cdc.sh* (in {{IgniteToKafkaCdcStreamer}}):
> # Waiting for {{KafkaProducer#send}} future to finish.
> *kafka-to-ignite.sh:*
> # In {{KafkaToIgniteCdcStreamerApplier}} as parameter of methods:
> ## {{KafkaConsumer#poll}}
> ## {{KafkaConsumer#commitSync}}
> # In {{KafkaToIgniteMetadataUpdater}} as parameter of methods:
> ## {{KafkaConsumer#poll}}
> ## {{KafkaConsumer#endOffsets}}
> ## {{KafkaConsumer#partitionsFor}}
> ----
> Also, there is difference of behavior of {{KafkaConsumer#poll}} and other
> used methods of {{KafkaConsumer}}. {{KafkaConsumer#poll}} just waits for data
> until specified timeout expires, but other methods will raise an exception,
> thus, timeout should not be to low for them.
> But, on the other hand, {{#poll}} should happen relatively frequent and if we
> set up too large timeout for it, then we can face delays of replication,
> because, when there are no new data in topics, {{#poll}} will block until new
> data or until timeout will expire. It is not desired behavior, because in
> this situation we can have pending updates which will wait to be processed.
> ----
> Links:
> #
> https://kafka.apache.org/27/documentation.html#producerconfigs_delivery.timeout.ms
> #
> https://kafka.apache.org/27/documentation.html#producerconfigs_request.timeout.ms
> #
> https://kafka.apache.org/27/documentation.html#consumerconfigs_default.api.timeout.ms
> #
> https://kafka.apache.org/27/documentation.html#consumerconfigs_request.timeout.ms
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
