Github user siyuanh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/132#discussion_r48523654
--- Diff:
contrib/src/main/java/com/datatorrent/contrib/kafka/AbstractKafkaInputOperator.java
---
@@ -392,9 +412,20 @@ public void emitTuples()
if (maxTuplesPerWindow > 0) {
count = Math.min(count, maxTuplesPerWindow - emitCount);
}
- for (int i = 0; i < count; i++) {
+ if (count > 0) {
+ // If the total size transmitted in the window will be exceeded
don't transmit anymore messages in this window
+ // Make an exception for the case when no message has been
transmitted in the window and transmit at least one
+ // message even if the message size is greater than max message size
so that the processing doesn't get stuck
+ if ((emitCount > 0) && ((maxTotalMsgSizePerWindow -
emitTotalMsgSize) < consumer.peekMessage().msg.size())) {
--- End diff --
You can use pollMessage only to keep the message as a field. And if it
exceed the limit wait until the first emitTuples get called in the next window.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
