[
https://issues.apache.org/jira/browse/FLINK-26548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504182#comment-17504182
]
zl edited comment on FLINK-26548 at 3/10/22, 10:57 AM:
-------------------------------------------------------
Hi [~wanglijie95], thanks for replying.
-> this problem only occurs when using legacy file sources, other sources and
the new source(FLIP-27) will not have two operators.
yes, this problem only occurs when using legacy file sources
We can solve this problem in this way: before create
*_{{ContinuousFileMonitoringFunction}}_ ,* we get the value of
*_default.parallelism_* , if the value is -1 and AdaptiveBatchScheduler is
enabled, we pass the value of
*_jobmanager.adaptive-batch-scheduler.default-source-parallelism_* {_}to
*{{ContinuousFileMonitoringFunction.}}*{_}{{{}Then we need to set the
parallelism of real source reader to
*_jobmanager.adaptive-batch-scheduler.default-source-parallelism._*{}}}
btw, can you assign this issue to me?
was (Author: leo zhou):
Hi [~wanglijie95], thanks for replying.
-> this problem only occurs when using legacy file sources, other sources and
the new source(FLIP-27) will not have two operators.
yes, this problem only occurs when using legacy file sources
We can solve this problem in this way: before create
*_{{ContinuousFileMonitoringFunction}}_ ,* we get the value of
*_default.parallelism_* , if the value is -1 and AdaptiveBatchScheduler is
enabled, we pass the value of
*_jobmanager.adaptive-batch-scheduler.default-source-parallelism_* {_}to
*{{ContinuousFileMonitoringFunction.}}*{_}{{{}Then we need to set the
parallelism of real source reader to
*_jobmanager.adaptive-batch-scheduler.default-source-parallelism._*{}}}
> the source parallelism is not set correctly with AdaptiveBatchScheduler
> -----------------------------------------------------------------------
>
> Key: FLINK-26548
> URL: https://issues.apache.org/jira/browse/FLINK-26548
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Affects Versions: 1.15.0
> Reporter: zl
> Priority: Blocker
> Fix For: 1.15.0
>
> Attachments: image-2022-03-09-19-00-18-396.png
>
>
> When running *_org.apache.flink.table.tpcds.TpcdsTestProgram_* with
> {_}*AdaptiveBatchScheduler*{_}, I ran into a problem：the num of records sent
> by the source operator is always 1, and the parallelism of source operator is
> also 1 even I set
> *_jobmanager.adaptive-batch-scheduler.default-source-parallelism_* to 8.
> !image-2022-03-09-19-00-18-396.png!
> After some research, I found that the operator A is not the actual file
> reader, it just splits files and assigns splits to downstream tasks for
> further processing, and the operator B is the actual file reader task. Here,
> the parallelism of operator B is 64, and the records sent by operator A is 1,
> this means, operator A assigned all splits to a task of operator B, {*}_the
> other 63 tasks of operator B is idle_{*}, it is unreasonable.
> In this case, the parallelism of operator B should be
> *_jobmanager.adaptive-batch-scheduler.default-source-parallelism_* and the
> num of records sent by operator A also should be
> {*}_jobmanager.adaptive-batch-scheduler.default-source-parallelism_{*}.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
