[
https://issues.apache.org/jira/browse/FLINK-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895878#comment-16895878
]
Piotr Nowojski commented on FLINK-13384:
----------------------------------------
>From your two proposals, as a hotfix I think the first one (exposing
>{{LegacySourceFunctionThread}} via some {{Task.getMainExecutionThread()}} or
>sth like that) sounds better to me.
However this is not the first time that backpressure monitor has been broken.
Also I think it's inherently broken with the current threading model, since it
only detects if the task is backpressured when a record was produced from the
main thread. That's not always true: processing time timers,
{{AsyncWaitOperator}}, {{ContinousFileReaderOperator}}, custom sources that
emit records from independent threads are all braking this assumption. Sometime
ago I was proposing a more general fix for that: use output buffer usage
metrics. If at least one output buffer is full, we assume there is a
backpressure. However that would have been probably a bigger change.
> The back pressure monitoring does not work for StreamSources
> ------------------------------------------------------------
>
> Key: FLINK-13384
> URL: https://issues.apache.org/jira/browse/FLINK-13384
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics, Runtime / Task
> Affects Versions: 1.9.0
> Reporter: Dawid Wysakowicz
> Assignee: Gary Yao
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think it is caused by: [FLINK-12483]. The reason is that the
> {{BackPressureStatsTrackerImpl}} samples only the main thread. FLINK-12483
> introduced a separate thread for executing the sources in the mailbox model.
> It is similar to other old bug that concerned only Kafka source: [FLINK-3456]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
