DanielLeens commented on issue #11306: URL: https://github.com/apache/seatunnel/issues/11306#issuecomment-4880124706
Thanks for the detailed write-up and the code-path check. I traced the current file-source initialization path, and this does look like a real issue rather than expected REST behavior. `BaseFileSourceConfig` resolves file paths during source config creation, and that immediately calls `readStrategy.getFileNamesByPath(...)`. In the default batch path, `AbstractReadStrategy.collectFileInfoByPath(...)` lists the directory first and only then applies modification-time filtering locally. So even when only a small time window matches, the expensive SFTP enumeration is still happening synchronously before job creation. That is also why the symptom shows up the way you described: - `submit-job` stays blocked long before `jobId` is returned - `parallelism` does not help this stage - `full` and `update` both remain slow because the dominant cost is still source-side enumeration in the submit path - ingress / proxy timeout can fire even though the backend job later continues So this should be treated as a real bug / performance issue in the current submit-time source preparation path, not just as a large-directory expected delay. A reasonable first fix direction would be: 1. avoid full file enumeration in the REST submit thread before `jobId` is returned 2. add finer-grained timing logs around remote listing, local filtering, and update-mode target comparison 3. keep the optimization especially focused on SFTP / large flat-directory cases, since that is where the submit path becomes operationally painful We will keep this open for follow-up. We also labeled it as `help wanted`, since the reporter did not opt into a PR and this looks like a useful contribution target for the community. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
