[
https://issues.apache.org/jira/browse/KAFKA-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897371#comment-15897371
]
Michal Borowiecki commented on KAFKA-3514:
------------------------------------------
Hi [~enothereska],
I have to disagree. It is perfectly clear to me (from the documentation) that
punctuate is based on event time, not system time. However, the problem is
event time is not advanced reliably, since a single input stream that doesn't
receive messages will cause the event time to not be advanced. In an extreme
case of a poorly partitioned topic, I can imagine some partition may never get
a message. That would cause a topology that has that partition as input to not
advance event time ever, hence not fire punctuate ever, regardless of the
presence of messages on its other input topics. In my opinion, if the purpose
of punctuate is to perform periodic operations, then this flaw makes it unfit
for that purpose.
> Stream timestamp computation needs some further thoughts
> --------------------------------------------------------
>
> Key: KAFKA-3514
> URL: https://issues.apache.org/jira/browse/KAFKA-3514
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Eno Thereska
> Labels: architecture
> Fix For: 0.11.0.0
>
>
> Our current stream task's timestamp is used for punctuate function as well as
> selecting which stream to process next (i.e. best effort stream
> synchronization). And it is defined as the smallest timestamp over all
> partitions in the task's partition group. This results in two unintuitive
> corner cases:
> 1) observing a late arrived record would keep that stream's timestamp low for
> a period of time, and hence keep being process until that late record. For
> example take two partitions within the same task annotated by their
> timestamps:
> {code}
> Stream A: 5, 6, 7, 8, 9, 1, 10
> {code}
> {code}
> Stream B: 2, 3, 4, 5
> {code}
> The late arrived record with timestamp "1" will cause stream A to be selected
> continuously in the thread loop, i.e. messages with timestamp 5, 6, 7, 8, 9
> until the record itself is dequeued and processed, then stream B will be
> selected starting with timestamp 2.
> 2) an empty buffered partition will cause its timestamp to be not advanced,
> and hence the task timestamp as well since it is the smallest among all
> partitions. This may not be a severe problem compared with 1) above though.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
