goutamadwant opened a new pull request, #11789: URL: https://github.com/apache/seatunnel/pull/11789
### Purpose of this pull request Part of #11758. The shared file source already supports polling-based continuous discovery, but the S3File and OssFile factories did not expose the required configuration options. This PR enables the existing continuous discovery and update-mode options for both connectors: - expose polling, update comparison, and post-sync options through the connector factories - require `target_path` when `sync_mode=update` - preserve the existing `discovery_mode=once` and `sync_mode=full` defaults - add factory validation coverage for S3File and OssFile - add a MinIO E2E test that verifies an object added after job startup is discovered and copied - document the options and example configurations in English and Chinese This is the polling-based first slice. It does not add object-storage event notifications, delete event propagation, changelog records, ETag-based deduplication, or lazy object-content fetching. ### Does this PR introduce _any_ user-facing change? Yes. S3File and OssFile sources can now be configured with `discovery_mode=continuous` to periodically discover new or changed objects in streaming jobs. The existing bounded behavior remains unchanged because `discovery_mode` still defaults to `once`. ### How was this patch tested? Added factory tests for S3File and OssFile covering: - exposure of the continuous discovery and update-mode options - validation requiring `target_path` when `sync_mode=update` Verified the connector tests with: ```shell JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" \ ./mvnw -B -pl :connector-file-s3,:connector-file-oss -DskipITs test ``` Results: - S3File: 19 tests passed - OssFile: 3 tests passed Added a MinIO E2E test that: - starts with an existing S3 object - runs S3File in polling continuous mode - verifies the initial object is copied - uploads another object after the job has started - verifies the later object is discovered and copied - cancels the streaming job cleanly Verified with: ```shell JAVA_HOME=$(/usr/libexec/java_home -v 11) PATH="$JAVA_HOME/bin:$PATH" \ ./mvnw -B -T 1 verify \ -DskipUT=true \ -DskipIT=false \ -DfailIfNoTests=false \ -Dlicense.skipAddThirdParty=true \ -Dskip.ui=true \ -Dtestcontainer.version=1.21.4 \ -Dit.test=S3FileWithFilterIT#testS3BinaryUpdateModeContinuousDiscovery \ -pl :connector-file-s3-e2e \ -Pci ``` The Testcontainers override was needed locally because Docker 29 rejects the API version used by the repository default. The focused E2E passed with 1 test and no failures. OSS uses the same shared continuous enumerator path. Its factory and validation behavior are covered locally, but an OSS integration test was not run because it requires external OSS credentials. ### Check list * [x] No new Jar binary package is added. * [x] English and Chinese connector documentation is updated. * [x] This change preserves the existing default behavior and does not require an incompatible change entry. * [x] No new connector package is introduced, so plugin mapping, distribution POM, label configuration, and plugin configuration updates are not required. * [x] A focused S3 connector E2E test is included. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
