snuyanzin opened a new pull request, #29274: URL: https://github.com/apache/flink/pull/29274
## What is the purpose of the change There is a problem with tests that FileConnector might mark partially written file as already processed, as a result it will not see further update and tests will fail with timeout example https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=79340&view=logs&j=a9db68b9-a7e0-54b6-0f98-010e0aff39e2&t=feeea8c8-fb6c-541e-ab85-af75c9efb8e4 also a way to repro locally ```java List<Integer> actual = new ArrayList<>(); try (CloseableIterator<Row> it = tEnv().sqlQuery("SELECT * FROM t").execute().collect()) { for (int i = 0; i < 3; i++) { actual.add(it.next().getFieldAs(0)); // 1, 2, 3 } Path late = Paths.get(dir0.getPath(), "input_1.csv"); Files.createFile(late); // discovered empty -> path marked processed Thread.sleep(3_000); // let a monitor scan record the empty file Files.write(late, Arrays.asList("4", "5", "6"), StandardOpenOption.WRITE); for (int i = 0; i < 3; i++) { actual.add(it.next().getFieldAs(0)); // hangs: 4, 5, 6 never arrive } } assertThat(actual).containsExactlyInAnyOrder(1, 2, 3, 4, 5, 6); ``` ## Brief change log Make writing file atomically just by renaming when writing is finished ## Verifying this change The change is test itself ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable ) --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change the checkbox below to `[X]` and replace the placeholder in the "Generated-by" line with the tool name and version. Otherwise remove the "Generated-by" line. See the ASF Generative Tooling Guidance for details: https://www.apache.org/legal/generative-tooling.html You are responsible for the quality and correctness of every change in this PR regardless of the tooling used. Low-effort AI-generated PRs will be closed. See AGENTS.md for the full guidance. --> - [ ] Yes (please specify the tool below) Generated-by: [Tool Name and Version] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
