prateekm commented on a change in pull request #1489:
URL: https://github.com/apache/samza/pull/1489#discussion_r616970406
##########
File path: samza-core/src/main/java/org/apache/samza/config/TaskConfig.java
##########
@@ -53,6 +56,18 @@
// commit period in milliseconds
public static final String COMMIT_MS = "task.commit.ms";
static final long DEFAULT_COMMIT_MS = 60000L;
+ // upper bound for the task commit thread pool size in a container .
+ // num threads == min(num tasks in container, max thread pool size)
+ public static final String COMMIT_MAX_THREAD_POOL_SIZE =
"task.commit.max.thread.pool.size";
+ static final int DEFAULT_COMMIT_MAX_THREAD_POOL_SIZE = 64;
+ // maximum amount of time a task may continue processing while a previous
commit is pending
+ public static final String COMMIT_MAX_DELAY_MS = "task.commit.max.delay.ms";
+ static final long DEFAULT_COMMIT_MAX_DELAY_MS =
Duration.ofMinutes(10).toMillis();
Review comment:
For context, `task.commit.max.delay.ms` is the amount of time the task
may continue processing while the previous commit is still in progress. A
longer default value of this timeout is better if using blob store state
backend (since compactions may occasionally increase the state size to be
uploaded by a large amount), and will not hurt when using Kafka state backend
(it will typically finish much earlier).
`task.commit.timeout.ms` determines how long the task blocking-waits for the
commit to complete when the previous "grace period" is exceeded. If this value
is <= 0 it will throw an exception immediately, else it will wait for the
provided interval and then throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
