goutamadwant opened a new pull request, #12146:
URL: https://github.com/apache/seatunnel/pull/12146
### Purpose of this pull request
Adds the GCS sink slice of #10394 and #10753, building on the GCS source
merged in #11967. This moves the sink work from goutamadwant/seatunnel#1 into
upstream review.
- Register `GcsFile` as a sink in the existing `connector-file-gcs` module.
- Reuse the shared file sink writers, partitioning, multiple-table handling
and commit lifecycle.
- Reuse the GCS source's Hadoop configuration and authentication. No new
dependency or connector module is added.
- Add save-mode catalog support, including recursive existing-data detection
without collecting the directory listing in memory.
- Reject bucket-root destinations for destructive save modes, including
normalized paths and resolved table placeholders.
- Add English and Chinese documentation, portable unit tests and emulator
E2E coverage.
### Does this PR introduce _any_ user-facing change?
Yes. `GcsFile` currently supports reading from GCS on `dev`; this PR adds
file output with the shared file sink formats, partitioning, multiple-table
configuration and save modes.
For example:
```hocon
sink {
GcsFile {
bucket = "gs://my-bucket"
path = "/warehouse/orders"
tmp_path = "/staging/orders"
file_format_type = "parquet"
data_save_mode = "APPEND_DATA"
}
}
```
Existing GCS source behavior and other file connectors' defaults are
unchanged. Spark and Flink deployments still require a compatible Hadoop 3
runtime, as documented for the source.
Transactional commits do not make an entire directory atomically visible to
external readers. The documentation explains staging, rename permissions and
the copy/delete publication boundary.
### How was this patch tested?
- Java 8: 301 unit tests passed across `connector-file-base` and
`connector-file-gcs`, with no failures, errors or skips.
- Java 11: all 28 GCS tests and 6 shared filesystem proxy tests passed.
- Java 11 E2E reactor: all 66 modules passed verification. Four E2E cases
passed with no failures, errors or skips: the source smoke test and the sink
write/replace/readback scenario on both Zeta and Spark 3.3.
- Tests cover factory discovery, required options, bucket validation,
destructive-path guards, recursive lazy listing, missing paths, save-mode
behavior and prefix-scoped cleanup.
- Added a GCS E2E scenario that writes partitioned JSON, reads it back,
repeats the job with `DROP_DATA`, and verifies the output still contains
exactly two rows. The existing source-only smoke test is retained.
Unit test command:
```bash
./mvnw -B -pl :connector-file-base,:connector-file-gcs
-Dlicense.skipAddThirdParty=true test
```
Focused Java 11 command:
```bash
./mvnw -B -pl :connector-file-base,:connector-file-gcs
'-Dtest=Gcs*Test,HadoopFileSystemProxyTest'
-Dsurefire.failIfNoSpecifiedTests=false -Dlicense.skipAddThirdParty=true test
```
E2E command:
```bash
./mvnw -B -pl :connector-file-gcs-e2e -am -Pci -DskipUT=true -DskipIT=false
-Dit.test=GcsFileIT -Dfailsafe.failIfNoSpecifiedTests=false
-DfailIfNoTests=false -Dlicense.skipAddThirdParty=true -Dskip.ui=true verify
```
The emulator uses its memory backend to preserve Hadoop directory markers.
The sink test sets `fs.gs.operation.move.enable=false` because fake-gcs-server
1.56.1 does not implement the native move endpoint; it exercises the driver's
copy/delete commit path. Production defaults are unchanged. This is emulator
coverage, not a live-GCS credential or native-move validation.
On Docker 29, the local run used the process-local setting
`JAVA_TOOL_OPTIONS=-Dapi.version=1.44` for compatibility with the existing
Testcontainers client. No Docker-wide or production configuration was changed.
The broader Java 11 file suite on Apple Silicon hit the existing Snappy
dependency's missing native library. The same full module suites passed under
Java 8.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
No new third-party JAR or incompatible change is introduced. Distribution
packaging, the file connector label scope and `plugin_config` already include
this module from #11967; this PR adds the sink mapping and extends its E2E test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
