goutamadwant opened a new pull request, #11377:
URL: https://github.com/apache/seatunnel/pull/11377
### Purpose of this pull request
Fixes #11306.
For SftpFile binary sources in once discovery mode, file discovery was done
while the source config was being created. In the REST submit path that happens
before `jobId` is returned, so a large SFTP directory can keep `submit-job`
waiting while the connector lists and filters remote files.
This change keeps schema/config construction lightweight for SftpFile binary
sources and defers full file discovery until the once-mode split enumerator
opens. Other file formats keep the existing eager behavior because some formats
still infer schema from discovered files.
This also adds a timing log for file discovery with plugin name, path,
discovered file count, and elapsed time.
### Does this PR introduce _any_ user-facing change?
Yes. For SftpFile binary sources, REST submit can return the `jobId` before
full source file discovery runs. If deferred discovery fails, the job fails
during split enumeration instead of failing during submit.
Non-binary file formats keep the existing behavior.
### How was this patch tested?
Added unit coverage for the deferred discovery path:
- deferred file discovery does not call `listStatus` during source config
creation
- eager file discovery keeps the existing config-time discovery behavior
- the split enumerator discovers deferred file paths when `open()` runs
Verified with:
```shell
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-base
-Dtest=MultipleTableFileSourceSplitEnumeratorTest test
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-base
-DfailIfNoTests=false package
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-sftp -am
-DfailIfNoTests=false package
```
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
