goutamadwant opened a new pull request, #12242:
URL: https://github.com/apache/seatunnel/pull/12242
<!-- PR title: [Feature][Connector-V2] Support S3 source connectivity
dry-run -->
### Purpose of this pull request
Closes #12239.
Add S3 source support for the existing `--dry-run connect` validation SPI.
Previously,
the S3 source factory did not implement the SPI, so connectivity validation
was skipped.
The check uses Hadoop S3A client configuration to request object metadata or
one
prefix-listing entry. It does not read file contents or modify objects.
This uses the current Hadoop/AWS APIs without a dependency upgrade. If the
migration
in #11648 lands first, this change will need rebasing and revalidation.
### Does this PR introduce _any_ user-facing change?
Yes. Connect dry-run supports single-table S3A sources with an inline schema
for
text, CSV, JSON or XML, `parse_partition_from_path=false`, and no
`read_columns`.
Missing batch paths and denied requests fail early. A successful empty prefix
listing is valid for continuous discovery. Unsupported configurations fail
with
an explanation rather than reporting successful validation.
Ordinary jobs and their defaults are unchanged. Validation uses capped
connection
and socket timeouts with no request retries; these are not a total deadline
for
DNS or credential initialization. English and Chinese documentation describe
the
supported scope and limitations.
### How was this patch tested?
- Reproduced the missing SPI on unchanged production code on Java 8 and Java
11.
- All 68 connector unit tests and 13 factory integration tests passed on
each of
Java 8 and Java 11, with no failures or skipped tests.
- Connector unit tests cover schema eligibility, bucket/path handling,
configuration
reuse, timeout overrides, bounded listing, permission failures and empty
prefixes.
- Factory integration tests use a pinned MinIO service for existing objects,
prefixes, empty roots, missing buckets and incorrect credentials.
- HTTP-wire tests verify metadata-only requests, no pagination, denial and
timeout
handling, and resource cleanup after success, request failure and endpoint
setup failure.
From the repository root, repeat the following on Java 8 and Java 11 after
building
the required reactor dependencies:
```bash
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-s3 install
./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-s3-e2e \
-DskipIT=false -Dit.test=S3FileConnectDryRunIT,S3FileConnectDryRunWireIT
verify
```
These are factory-level integration tests, not submitted engine jobs.
Root formatting and the complete no-tests build passed on Java 11:
```bash
./mvnw spotless:apply
./mvnw -q -DskipTests verify
```
The unchanged upstream baseline also passed the same no-tests build. This
checks
compilation and packaging, not execution of the full Java test suite.
The current [fork
Build](https://github.com/goutamadwant/seatunnel/actions/runs/34320638878) is
not fully green: the Java 8 SFTP job failed before tests because downloading
the Maven wrapper JAR from Maven Central returned HTTP 403. This is separate
from the passing local S3 tests above.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md).
No new dependencies or JARs.
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs (EN/ZH connector
documentation updated.)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. Ordinary runtime behavior is unchanged.
* [x] If you are contributing the connector code, please check that the
following files are updated: Existing registration, distribution and labels
remain valid; factory integration tests added in the existing S3 E2E module.
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
