muralibasani commented on code in PR #22458:
URL: https://github.com/apache/kafka/pull/22458#discussion_r3625103280
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -1207,8 +1218,27 @@ private void subscribeConsumer() {
}
}
- public void resizeCache(final long size) {
- cacheResizeSize.set(size);
+ public void resizeCacheAndBufferMemory(final long cacheSize, final long
maxBufferSize) {
+ cacheResizeSize.set(cacheSize);
+ maxBufferSizeInBytes.set(maxBufferSize);
+ }
+
+ private void maybeResumePartitionsPausedForBufferOverflow() {
+ if (maxBufferSizeInBytes.get() == UNDEFINED_INPUT_BUFFER_MAX_BYTES) {
+ return;
+ }
+ if (partitionsPausedForBufferOverflow.isEmpty()) {
+ return;
+ }
+ if (taskManager.getInputBufferSizeInBytes() <=
maxBufferSizeInBytes.get()) {
+ // defensive copy — we clear the tracking set right after.
+ final Set<TopicPartition> toResume = new
HashSet<>(partitionsPausedForBufferOverflow);
+ toResume.retainAll(mainConsumer.assignment());
Review Comment:
Great find. Have taken that (b) approach.
Added StreamThread.removePartitionsFromBufferOverflowTracking(.....).
It now keeps the invariant at the lifecycle boundary
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -1207,8 +1218,27 @@ private void subscribeConsumer() {
}
}
- public void resizeCache(final long size) {
- cacheResizeSize.set(size);
+ public void resizeCacheAndBufferMemory(final long cacheSize, final long
maxBufferSize) {
+ cacheResizeSize.set(cacheSize);
+ maxBufferSizeInBytes.set(maxBufferSize);
+ }
+
+ private void maybeResumePartitionsPausedForBufferOverflow() {
+ if (maxBufferSizeInBytes.get() == UNDEFINED_INPUT_BUFFER_MAX_BYTES) {
+ return;
+ }
+ if (partitionsPausedForBufferOverflow.isEmpty()) {
+ return;
+ }
+ if (taskManager.getInputBufferSizeInBytes() <=
maxBufferSizeInBytes.get()) {
+ // defensive copy — we clear the tracking set right after.
+ final Set<TopicPartition> toResume = new
HashSet<>(partitionsPausedForBufferOverflow);
+ toResume.retainAll(mainConsumer.assignment());
Review Comment:
Great find. Have taken that (b) approach.
Added StreamThread.removePartitionsFromBufferOverflowTracking(.....).
It now keeps the invariant at the lifecycle boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
