DanielLeens commented on issue #10523: URL: https://github.com/apache/seatunnel/issues/10523#issuecomment-4713921093
Thanks for the extra details. After checking your follow-up together with the current file sink write path, this still looks like a real issue in the file connector checkpoint/close coordination path rather than a Hive-only or JuiceFS-only configuration problem. The key signal is that you can reproduce it in a single-table pipeline and across different file formats, while the current file sink implementation still closes active output streams during `prepareCommit()` and keeps mutable output-stream state around the write path. Under the normal flow, a checkpoint should not race ongoing writes into a closed stream, but the current implementation can still end up there. As a temporary workaround, increasing the checkpoint interval or reducing very frequent file commit/rotation pressure may reduce the failure rate, but this issue is worth keeping open for a code-side fix. We've labeled this as `help wanted` as well, so contributions are very welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
