[
https://issues.apache.org/jira/browse/KAFKA-12499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304537#comment-17304537
]
A. Sophie Blee-Goldman edited comment on KAFKA-12499 at 3/29/21, 7:39 PM:
--------------------------------------------------------------------------
Nice catch, but just wondering why choose 10x the commit interval? And should
we set a lower bound as well so we fall back on the default of 1 minute if 10 *
commit_interval is below that? For example with EOS that would mean a
transaction timeout of just one second -- that seems easy to miss, eg if there
are a large number of tasks or some heavy processing we'd just be constantly
timing out.
was (Author: ableegoldman):
Nice catch, but just wondering why choose 10x the commit interval? And should
we set a lower bound as well so we fall back on the default of 1 minute if 10 *
commit_interval is below that? For example with EOS that would mean a
transaction timeout of just one second -- we only check whether it's time to
commit after a loop of processing all tasks, if there are a large number of
tasks or some heavy processing we'd just be constantly timing out.
> Adjust transaction timeout according to commit interval on Streams EOS
> ----------------------------------------------------------------------
>
> Key: KAFKA-12499
> URL: https://issues.apache.org/jira/browse/KAFKA-12499
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Boyang Chen
> Assignee: Boyang Chen
> Priority: Blocker
> Fix For: 3.0.0
>
>
> The transaction timeout is set to 1 minute by default on Producer today,
> while the commit interval on the other hand could be set to a very large
> value, which makes the stream always hit transaction timeout and drop into
> rebalance. We should increase the transaction timeout correspondingly when
> commit interval is large.
> On the other hand, broker could have a limit on the max transaction timeout
> to be set. If we scale up client transaction timeout over the limit, stream
> will fail due to INVALID_TRANSACTION_TIMEOUT. To alleviate this problem,
> user could define their own customized transaction timeout to avoid hitting
> the limit, so we should still respect what user configures in the override.
> The new rule for configuring transaction timeout should look like:
> 1. If transaction timeout is set in streams config, use it
> 2. if not, transaction_timeout = max(default_transaction_timeout, 10 *
> commit_interval)
> Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when
> calling initTransaction(), we should wrap the exception to inform user that
> their setting for commit interval could potentially be too high and should
> adjust.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
