[
https://issues.apache.org/jira/browse/FLINK-28274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17562423#comment-17562423
]
Martijn Visser commented on FLINK-28274:
----------------------------------------
-- The legacy source are still widely used, for example, When reading files
with datastream api, the first thought is to use readFile/readTextFile()
methods, especially for starters. Since the fix work won't take much effort,
may be it's worthy.
I don't think that the usage is a good reason, because that's always the case
when something new is introduced. Without an incentive for users to migrate,
they will never do so. It probably makes sense to fix it in this case, but I
still think that we are doing something wrong as a Flink community since people
shouldn't use {Internal} interfaces at all.
> ContinuousFileMonitoringFunction doesn't work with reactive mode
> ----------------------------------------------------------------
>
> Key: FLINK-28274
> URL: https://issues.apache.org/jira/browse/FLINK-28274
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / Coordination
> Affects Versions: 1.16.0
> Reporter: Robert Metzger
> Assignee: zl
> Priority: Major
> Labels: pull-request-available
>
> This issue was first reported in the Flink Slack:
> https://apache-flink.slack.com/archives/C03G7LJTS2G/p1656257678477659
> It seems that reactive mode is changing the parallelism of the
> `ContinuousFileMonitoringFunction`, which is supposed to always run with a
> parallelism of 1.
> This is the error
> {code}
> INITIALIZING to FAILED with failure cause:
> java.lang.IllegalArgumentException: ContinuousFileMonitoringFunction
> retrieved invalid state.
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> {code}
> You can see from the logs that the parallelism is changing on a rescale event:
> {code}
> 2022-06-27 13:38:54,979 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/1) (cbaad20beee908b95c9fe5c34ba76bfa) switched from
> RUNNING to CANCELING.
> 2022-06-27 13:38:55,254 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/1) (cbaad20beee908b95c9fe5c34ba76bfa) switched from
> CANCELING to CANCELED.
> 2022-06-27 13:38:55,657 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (6ceaacbe8d9aa507b0a56c850082da8c) switched from
> DEPLOYING to INITIALIZING.
> 2022-06-27 13:38:55,722 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (6ceaacbe8d9aa507b0a56c850082da8c) switched from
> INITIALIZING to RUNNING.
> 2022-06-27 13:44:54,058 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/2) (665b12194741744d6bba4408a252fa45) switched from
> RUNNING to CANCELING.
> 2022-06-27 13:45:00,825 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/2) (3cc408fd0eb9ddfa97b22f4dfc09d8dc) switched from
> DEPLOYING to INITIALIZING.
> 2022-06-27 13:45:00,826 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (1/2) (3cc408fd0eb9ddfa97b22f4dfc09d8dc) switched from
> INITIALIZING to RUNNING.
> 2022-06-27 13:45:01,434 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (79338042d84b6458c34760bc85145512) switched from
> DEPLOYING to INITIALIZING.
> 2022-06-27 13:45:02,427 | INFO | .executiongraph.ExecutionGraph | Source:
> Custom File Source (2/2) (79338042d84b6458c34760bc85145512) switched from
> INITIALIZING to RUNNING.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
