Mola-maker opened a new pull request, #40520:
URL: https://github.com/apache/beam/pull/40520
DaskRunner snapshots receiver outputs before calling `DoFn.finish_bundle()`,
so a buffering DoFn can complete successfully while emitting an empty
collection. Move output collection after `invoke_finish_bundle()` and before
teardown, add regression tests for finish-only and mixed process/finish output,
and record the fix in `CHANGES.md`.
Fixes #40519.
### Validation
- Original code: existing DaskRunner suite `27 passed, 1 xfailed`; both new
regression tests fail with missing records.
- Fixed code on fresh workers: full `apache_beam/runners/dask/` suite `29
passed, 1 xfailed, 5 subtests passed`. The existing expected failure is
`test_pardo_windowed_side_inputs`.
- Real local execution on Ubuntu x86_64, Python 3.12.14, Beam 2.78.0.dev,
Dask/distributed 2026.8.0: two worker processes, two threads per worker, one
input partition. Before the fix, the pipeline returned three process records
and lost `('finish', 6)`; afterward all four records match the expected output.
- Ruff, Yapf formatting check, and `git diff --check` pass. `pip check`: `No
broken requirements found.`
The SDK was installed from the editable source checkout with
`[dask,test,dataframe]`. [Dependency installation steps and verification
commands](https://github.com/Mola-maker/beam/blob/2437bf6c395b7470a579e51102f0b5827bc96f77/README.md),
[installed dependency
versions](https://github.com/Mola-maker/beam/blob/2437bf6c395b7470a579e51102f0b5827bc96f77/dependencies.txt),
[regression
log](https://github.com/Mola-maker/beam/blob/2437bf6c395b7470a579e51102f0b5827bc96f77/dask-red.log),
and [full DaskRunner test
log](https://github.com/Mola-maker/beam/blob/2437bf6c395b7470a579e51102f0b5827bc96f77/dask-green.log)
are available at a fixed evidence commit.
Validation used the pure-Python SDK without compiled Cython extensions.
Java, cloud runners, and the full multi-language repository suite were not
tested. No new project runtime dependency is introduced.
### Captured local verification
These screenshots show a read-only local evidence viewer displaying the
actual gathered worker outputs and recorded test results. This is a backend
runner, so they document execution results rather than an application UI. The
evidence files are kept in a separate branch of the fork.
Before fix:
After fix:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
