lostluck opened a new pull request, #25982:
URL: https://github.com/apache/beam/pull/25982
Support receiving and directing timers to the *exec.DataSource from the
DataManager.
This doesn't fully implement timers, and only comprises the required data
plane changes. This unblocks their development however, though it includes an
untested TimerWriter path, and unused callback invocations in the DataSource to
support this later.
The FnAPI Data Stream multiplexes all data for all instructions onto a
single channel from the Runner harness. Each message may contain Data or Timers
bytes for one or more instructions executing on the SDK harness, to be decoded
by the execution plane's DataSource. The Data or Timers are directed to a
specfic PTransform.
The data and timers may arrive before or after the associated instruction
begins execution. This means that the Data and Timers must be buffered in the
SDK, until they are consumed, or their associated instruction is marked as
ended. Since a single instruction may have more than one consuming PTransform
it's important to track the wanted number of "islast" signals, to compare
against the number received, and only signal that all byte streams for this
instruction are complete when they match.
Specific things to note:
* In elementChan we use a mutex as well as the channel to avoid concurrent
changes to the "want" and "got", but use an atomic to check if elementChan is
closed. This is lighterweight than using the mutex to check the condition all
the time.
* The channel index in DataSource now defaults to 0 instead of -1, and we
now do a "post processing check" to validate if a split has occurred. This
ensures that the expected primary and residuals are respected in the same way
they were before.
Unit tests were updated to match the current possible cases, and validate
the above mentioned semantics. A "read" benchmark was added as well. Additional
larger scale testing was also done to ensure continued correctness of the data
plane's behavior.
Also Closes #21164, ensuring that the last element is always consumed.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
