Github user gliu6 commented on a diff in the pull request:
https://github.com/apache/flink/pull/6021#discussion_r196952063
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisProducer.java
---
@@ -326,6 +366,29 @@ private void checkAndPropagateAsyncError() throws
Exception {
}
}
+ /**
+ * If the internal queue of the {@link KinesisProducer} gets too long,
+ * flush some of the records until we are below the limit again.
+ * We don't want to flush _all_ records at this point since that would
+ * break record aggregation.
+ */
+ private void enforceQueueLimit() {
--- End diff --
Btw, I am not requesting any changes, pr looks good to me for it defined
purpose.
Just wonder how to config easily. Now I think about this a bit more. Will
it be better if we expose `queue size` to the user instead of `queue limit
(number)`, thus, Inside of the FKP class, define an integer recordSize, and
inside of the invoke function, do a moving average calculation of the
recordSize with `serialized.remaining()` dynamically.
---
