goutamadwant opened a new pull request, #11877:
URL: https://github.com/apache/seatunnel/pull/11877
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
Contributes to #10753.
This PR adds the Google Pub/Sub sink side as a focused first connector slice.
- Publishes one Pub/Sub message for each SeaTunnel row
- Supports JSON and delimited text payloads
- Supports Application Default Credentials, a service account key file, and
the Pub/Sub emulator
- Waits for outstanding asynchronous publishes during checkpoints and
shutdown
- Reports synchronous and asynchronous publish failures to the SeaTunnel task
- Isolates the Google client dependency stack through connector-local shading
- Registers the connector in the distribution, plugin mapping, plugin
config, CI labels, and E2E workflow
- Adds English and Chinese documentation
Source support is intentionally not part of this PR because subscriber
acknowledgement and checkpoint recovery need a separate source-focused design.
### Does this PR introduce _any_ user-facing change?
Yes.
Users can configure a new `GooglePubSub` sink for batch or streaming jobs.
The connector publishes serialized row payloads with at-least-once delivery
semantics.
Message attributes, ordering keys, and per-row topic routing are not
included in this first slice.
### How was this patch tested?
Added unit coverage for:
- Required and mutually exclusive options
- JSON and delimited text serialization
- Checkpoint flush behavior
- Synchronous and asynchronous publish failures
- Publisher cleanup after a failed publish
- Factory identifier and option registration
Added an emulator E2E test that creates a topic and subscription, runs
`FakeSource` to `GooglePubSub`, pulls the published message, verifies its
payload, and acknowledges it.
Verified with:
```shell
JAVA_HOME=<jdk8-home> ./mvnw -B -T 1 -pl :connector-google-pubsub test
-Dskip.ui=true --no-snapshot-updates
JAVA_HOME=<jdk11-home> ./mvnw -B -T 1 -pl :connector-google-pubsub verify
-Dskip.ui=true --no-snapshot-updates
JAVA_HOME=<jdk11-home> ./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false
-Dlicense.skipAddThirdParty=true -Dskip.ui=true --no-snapshot-updates -pl
:connector-google-pubsub-e2e -am -Pci
```
The emulator E2E test passed on Flink 1.13, Flink 1.15, Flink 1.18, Flink
1.20, SeaTunnel Zeta, Spark 2, and Spark 3.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
No Jar binary package is added to the repository.
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
No incompatible change is introduced.
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
