tkaymak opened a new pull request, #38966:
URL: https://github.com/apache/beam/pull/38966
## What
Follow-up to #38493 (now merged). Sets up a PostCommit suite for the
cross-language MQTT transforms (`ReadFromMqtt` / `WriteToMqtt`) served by the
messaging expansion service, and re-adds the CHANGES.md announcement that was
deferred from #38493.
## How
Clones the existing `beam_PostCommit_Python_Xlang_IO_Direct` pattern for the
messaging expansion service:
- New workflow `beam_PostCommit_Python_Xlang_Messaging_Direct.yml` + trigger
file + `.github/workflows/README.md` row.
- New `messagingCrossLanguage` `CrossLanguageTask` (collect marker
`uses_messaging_java_expansion_service`) in
`sdks/python/test-suites/xlang/build.gradle`, plus a
`messagingCrossLanguagePostCommit` aggregator task and the pytest marker
registration.
- New integration test `apache_beam/io/external/xlang_mqttio_it_test.py`
that starts an Eclipse Mosquitto broker via testcontainers and covers:
- bounded `ReadFromMqtt` on the DirectRunner (`max_num_records`, with a
continuous `mosquitto_pub` publisher inside the container since MQTT has no
message retention),
- `WriteToMqtt` on the DirectRunner (verified with a `mosquitto_sub`
subscriber),
- the unbounded (streaming) path on the **Prism** runner: a streaming
`ReadFromMqtt` feeding `WriteToMqtt` on a second topic, observed via
`mosquitto_sub`, then cancelled.
- The tests skip on Dataflow, where a testcontainers broker is not reachable
from the workers. A Dataflow variant would need a remotely-hosted broker (like
the hosted Kafka cluster used by the Xlang IO Dataflow suite) and is out of
scope here.
## Testing
All three integration tests pass locally against the messaging
expansion-service shadowJar (~2 min).
If Prism binary provisioning proves flaky on CI workers, the streaming test
can be sickbayed.
R: @Abacn
This addresses your post-approval comment on #38493 about setting up the
Xlang Messaging PostCommit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
