SEPURI-SAI-KRISHNA opened a new pull request, #13166:
URL: https://github.com/apache/seatunnel/pull/13166
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Part of #13078 (slice 1, the queue source) for the "JMS / AMQP (generic)"
row of #10753. The sink follows in a separate PR, as agreed there.
This adds the `connector-jms` module with the `Jms` source. It reads one
queue of any broker that has a JMS 1.1 (`javax.jms`) client, through that
broker's own client library. `connector-activemq` and `connector-rabbitmq` are
not changed.
The design is the one reviewed in #13078:
- **One transacted session per checkpoint epoch** on one connection per
reader. `snapshotState(N)` seals the open session with no JMS call, the next
`pollNext` closes the sealed consumer and opens the next session, and
`notifyCheckpointComplete(N)` commits every sealed session up to N. An aborted
checkpoint commits nothing; the next completed one commits it too. Uncommitted
sessions are rolled back on close, so delivery is at least once.
- **Receive and emit happen under the checkpoint lock**, one message at a
time with `receiveNoWait()`, so no message is received before a barrier and
emitted after it. An empty queue never holds the lock.
- **A failed commit fails the reader**, and a connection loss reported to
the `ExceptionListener` fails it before any commit is attempted. Received but
uncommitted messages are bounded by `max_in_flight_messages` (default 1000).
- **Connection factory**: `connection_factory_class` with
`connection_factory_properties` applied through setters (dotted keys such as
`prefetchPolicy.queuePrefetch` walk getters), or a JNDI lookup. Errors name the
key, never the value.
- **`cancel()`** closes the connection without taking any lock, which JMS
allows from another thread.
- `TextMessage` and `BytesMessage` bodies go through the existing `json` and
`text` formats. Other types and empty bodies fail the reader without a commit;
`ObjectMessage` is never deserialized; errors carry the message id, not the
body.
As confirmed by @SEZ9 on #13078: plugin `Jms`, module `connector-jms`, the
only new dependency is `geronimo-jms_1.1_spec` 1.1.1 (Apache License 2.0), and
vendor clients go in `${SEATUNNEL_HOME}/plugins/connector-jms/`. The docs list
the jars for each tested client.
Differences from the issue text:
- The source's queue option is `destination`, the key the issue gives the
sink.
- **Correction to item 9.** The spike saw ActiveMQ Classic return `null`
from `receiveNoWait()` right after a broker side drop. In the connector tests
every provider, Classic included, threw on the first poll after the drop (2 to
26 ms later), so a reader without the listener would not "idle forever" on
these clients. The listener stays: it fails the reader before a commit is
attempted, and
`JmsSourceReaderTest#connectionLossReportedOnlyToTheExceptionListenerFailsTheReader`
covers a provider that reports the loss only to the listener.
- The Java baseline is now 11 (#11545). The test clients are still the
versions from the issue: ActiveMQ Classic 5.15.16, Artemis 2.19.1, Qpid JMS
0.61.0.
### Does this PR introduce _any_ user-facing change?
Yes, a new source connector `Jms`, documented in EN and ZH
(`docs/*/connectors/source/Jms.md`). No existing option or behavior changes.
### How was this patch tested?
**Unit tests** (84) in `connector-jms`, against brokers embedded in the test
JVM and reached over TCP: ActiveMQ Classic (default prefetch, and
`queuePrefetch` 0 set through a dotted property), Artemis core, and Artemis
over AMQP 1.0 through Qpid JMS. The readers are configured the way users
configure them, through `connection_factory_class` and its properties.
- `JmsSourceReaderTest` (55): nothing committed before a completed
checkpoint; messages received after a snapshot are not committed by it; an
aborted checkpoint is committed by the next one; the final checkpoint commits
from another thread after polling stopped; repeated notifications are harmless;
a failed commit and a broker side drop fail the reader and lose nothing; 15
rotations over a 30 message backlog read each message once; the in-flight
bound; parallel readers; selector and `BytesMessage`; unsupported and empty
messages; conversion errors do not expose the body; JNDI; cancel while
connecting and before the first poll.
- `JmsConnectionFactoriesTest` (16), `JmsSourceFactoryTest` (9),
`JmsSourceSplitEnumeratorTest` (4, with a `DefaultSerializer` round trip of
split and state).
- Passing on JDK 17 and with the test JVM on JDK 11.
Each design rule was also broken on purpose and the tests failed: committing
the open epoch on completion (the naive single session), no exception listener,
no in-flight bound, accepting empty bodies, ignoring a cancel during connect,
committing only the completed checkpoint's own epoch, and not closing sealed
consumers (the last one stalled Classic with prefetch 0, Artemis and Qpid).
**E2E** `JmsIT` against `apache/activemq-artemis:2.43.0`, with the 27 jars
of `artemis-jms-client` 2.19.1 copied into `plugins/connector-jms/` as the docs
describe:
- Zeta: a message is committed once a checkpoint completes (checked with the
broker's `messageCount`), and with no completed checkpoint it stays on the
queue and comes back after cancel.
- Flink 1.13.6, 1.15.3, 1.18.0 and 1.20.1: a message is committed after a
Flink checkpoint.
- Spark is not supported and is disabled in the test.
The test turns off the broker's disk usage check, because Artemis blocks
producers once its disk is 90% full.
Also run: `spotless:check`, `seatunnel-ci-tools` tests, `verify` for both
new modules, and `ruff check` and `pytest` in `seatunnel-cli` for the alias
change (335 passed).
The branch is based on 26ad719a48. Merged locally with `dev` at e25c90a016,
there are no conflicts, the 84 unit tests, `ConnectorOptionCheckTest` and the
`seatunnel-cli` tests pass, and `JmsIT` compiles. I did not run `JmsIT` on that
merge; its runs above are on the branch.
### Check list
* [X] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md):
`geronimo-jms_1.1_spec` 1.1.1 is Apache License 2.0 and is shaded into the
connector jar, as `connector-activemq` already does through `activemq-client`;
connector dependencies are outside `tools/dependencies/checkLicense.sh`. No
vendor client is bundled.
* [X] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs: `Jms` source and
changelog, EN and ZH.
* [X] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR: not needed, new connector.
* [X] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it: `seatunnel.source.Jms`
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml):
done
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml):
`jms`
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/):
`connector-jms-e2e`
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config):
done
Also added the `jms` keyword to the AI CLI's connector aliases in
`seatunnel-cli/seatunnel_cli/connectors.py`, next to `activemq`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]