shunping opened a new pull request, #40467:
URL: https://github.com/apache/beam/pull/40467
## Summary
Fixes flaky `ErrorHandlingTest.test_error_handling_outputs` and
`ErrorHandlingTest.test_strip_error_metadata` in `yaml_transform_test.py`,
which intermittently fail with:
```
RuntimeError: Unable to pickle fn CallableWrapperDoFn(...):
cannot pickle 'google._upb._message.Descriptor' object
```
## Root cause
- `Scope.create_ptransform` monkey-patches `annotations` and `expand` onto
the user's transform instance. The patched `expand` (`recording_expand`) closes
over the YAML `Scope`, which references every provider, including the
process-global `standard_providers()` (cached via `functools.cache`).
- Any DoFn that captures its enclosing transform (e.g. `SizeLimiter`'s
`raise_on_big` uses `self._limit`) therefore pickles the whole Scope and all
providers. Providers hold lazily-populated, process-wide state (e.g.
`ExternalProvider._service` / `_schema_transforms`), so whether the pickle
succeeds depends on which tests ran earlier on the same xdist worker. Hence the
flakiness.
- Locally, the pickled `SizeLimiter` closure is ~90 KB and contains the
Scope, 28 `ExternalJavaProvider`s, 20 `RenamingProvider`s, etc., all to carry a
single int.
## Fix
Add `Scope.__reduce__` returning a string stub, mirroring the existing
`Pipeline.__reduce__`.
The Scope is only used during pipeline construction:
- `recording_expand` only runs during `runner.apply_PTransform`; workers
never call a composite's `expand`.
- Annotations are read once when the transform is applied and encoded into
the pipeline proto.
- Transforms restored via `PICKLED_TRANSFORM` are never re-expanded.
This keeps the existing monkey-patching unchanged, so there's no change to
transform labels (update compatibility) or to type-hint behavior. After the
fix, the pickled closure drops from ~90 KB to ~3.9 KB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
