[
https://issues.apache.org/jira/browse/BEAM-10691?focusedWorklogId=470696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-470696
]
ASF GitHub Bot logged work on BEAM-10691:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Aug/20 13:32
Start Date: 14/Aug/20 13:32
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #12551:
URL: https://github.com/apache/beam/pull/12551#discussion_r470623744
##########
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -1226,7 +1227,7 @@ public TimerInternals timerInternals() {
* fire time of the timer. Used for calculating the output watermark hold.
This avoids fetching
* timer data from the state backend which is expensive if done for each
timer.
*/
- private final PriorityQueue<Long> outputTimestampQueue;
+ private final TreeMultiset<Long> outputTimestamps = TreeMultiset.create();
Review comment:
We could also consider making this even more efficient by using a
`TreeMap<Long, Integer>` where the key is the output timestamp and the value
the number of timers which have set it, similar to how it's done in
`FlinkStateInternals` for the watermark holds.
Further, we could remove `outputTimestamps` entirely and simply use
`stateInternals.addWatermarkHoldUsage(output_timestamp)` and
`stateInternals.removeWatermarkHoldUsage(output_timestamp)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 470696)
Time Spent: 5h 40m (was: 5.5h)
> FlinkRunner: pipeline might get stuck due to timer watermark hold not being
> released
> ------------------------------------------------------------------------------------
>
> Key: BEAM-10691
> URL: https://issues.apache.org/jira/browse/BEAM-10691
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.23.0, 2.24.0
> Reporter: Jan Lukavský
> Assignee: Jan Lukavský
> Priority: P1
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Pipeline might stop progressing watermark in certain cases due to timer
> output timestamp not being released from
> FlinkTimerInternals#outputTimestampQueue. The pipeline has to be restarted
> from checkpoint to reload the cache and free watermark hold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
