Abacn opened a new pull request, #36141:
URL: https://github.com/apache/beam/pull/36141
#31078. Fix #30688, fix #30152; fix #21714, fix #21560
The main issue for current PulsarIO.read is
* It is based on Pulsar Reader instead of PulsarConsumer, which then do not
support acknowledgement
* The while() block in reader DoFn would never return until topic
termination, this basically means pipeline stuck
* The restriction is on publishTime, and tryClaim assumes its ordering. This
is not true. reader returning message is ordered on messageId. This is a wrong
choice. Currently unresolved
* PulsarMessage's coder implementation dropped message. This causes Data
loss if the PulsarIO.read do not follow an immediate mapping
* Tests are defunct and errors are suppressed, making them succeed spuriously
Current PulsarIO.write is even more primitive. Pipeline expansion actually
fails. It is not idempotent.
Major fixes include
* Allow Pulsar reader to have a timeout
* Fix PulsarMessage and coder to include serializable fields from message
* Fix mock client/reader and add a full read pipeline in test
* Fix issues prevent PulsarIO.write from expanding. now it works minimally,
that is publish every message received (at least once).
* Working integration tests for read and write
This has made PulsarIO.read minimally functionable. Although it won't split
and can only run single thread.
Going forward, we should re-implement reader DoFn based on Pulsar consumer.
Thoughs rename the current DoFn to "NaiveReadFromPulsarDoFn"
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
