[
https://issues.apache.org/jira/browse/FLINK-28274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561038#comment-17561038
]
Robert Metzger commented on FLINK-28274:
----------------------------------------
Thanks for your research. I agree that we shouldn't include source-specific
logic into the adaptive scheduler for this.
I like the idea of setting maxParallelism in the SEE (I currently can not
imagine any issues with that, but we'll see if all the tests are passing
properly once you've done the change ;) )
> ContinuousFileMonitoringFunction doesn't work with reactive mode
> ----------------------------------------------------------------
>
> Key: FLINK-28274
> URL: https://issues.apache.org/jira/browse/FLINK-28274
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Coordination
> Affects Versions: 1.16.0
> Reporter: Robert Metzger
> Priority: Major
>
> This issue was first reported in the Flink Slack:
> https://apache-flink.slack.com/archives/C03G7LJTS2G/p1656257678477659
> It seems that reactive mode is changing the parallelism of the
> `ContinuousFileMonitoringFunction`, which is supposed to always run with a
> parallelism of 1.
> This is the error
> {code}
> INITIALIZING to FAILED with failure cause:
> java.lang.IllegalArgumentException: ContinuousFileMonitoringFunction
> retrieved invalid state.
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> {code}
> You can see from the logs that the parallelism is changing on a rescale event:
> {code}
> 2022-06-27 13:38:54,979 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/1) (cbaad20beee908b95c9fe5c34ba76bfa) switched from
> RUNNING to CANCELING.
> 2022-06-27 13:38:55,254 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/1) (cbaad20beee908b95c9fe5c34ba76bfa) switched from
> CANCELING to CANCELED.
> 2022-06-27 13:38:55,657 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (6ceaacbe8d9aa507b0a56c850082da8c) switched from
> DEPLOYING to INITIALIZING.
> 2022-06-27 13:38:55,722 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (6ceaacbe8d9aa507b0a56c850082da8c) switched from
> INITIALIZING to RUNNING.
> 2022-06-27 13:44:54,058 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/2) (665b12194741744d6bba4408a252fa45) switched from
> RUNNING to CANCELING.
> 2022-06-27 13:45:00,825 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/2) (3cc408fd0eb9ddfa97b22f4dfc09d8dc) switched from
> DEPLOYING to INITIALIZING.
> 2022-06-27 13:45:00,826 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/2) (3cc408fd0eb9ddfa97b22f4dfc09d8dc) switched from
> INITIALIZING to RUNNING.
> 2022-06-27 13:45:01,434 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (79338042d84b6458c34760bc85145512) switched from
> DEPLOYING to INITIALIZING.
> 2022-06-27 13:45:02,427 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (79338042d84b6458c34760bc85145512) switched from
> INITIALIZING to RUNNING.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
