Mazen050 opened a new pull request, #40501: URL: https://github.com/apache/beam/pull/40501
## Summary Fixes #20226 This PR reduces runtime type-checking overhead in the Python SDK by consolidating two nested `DoFn` wrappers into a single `RuntimeTypeCheckWrapperDoFn`. Previously, input and output checks were handled by separate wrappers. The merged wrapper aims to preserve existing type-checking behavior while eliminating one level of Python call indirection per processed element. ## Changes - Consolidate `OutputCheckWrapperDoFn` and `TypeCheckWrapperDoFn` into `RuntimeTypeCheckWrapperDoFn`. - Cache the unbound `process` function while avoiding storing a bound method on the wrapper, preserving stateful `DoFn` validation. - Preserve input and output type checking, output-shape validation, and transform error labeling. - Add regression tests for wrapper application, error reporting, result preservation, and stateful `DoFn` compatibility. ## Tests - `apache_beam/typehints/typecheck_test.py`: 18 passed. - `apache_beam/transforms/ptransform_test.py`: 202 passed, 4 skipped. ## Performance Compared the original and modified implementations using Beam's existing `runtime_type_check_microbenchmark`. Configuration: 20 `DoFn`s and three timing repetitions per configuration. | Elements | Simple types (before → after) | Nested types (before → after) | |---:|---:|---:| | 501 | 3.22s → 3.06s | 3.90s → 3.41s | | 1,001 | 5.44s → 5.39s | 6.44s → 6.07s | | 1,501 | 7.57s → 7.39s | 8.93s → 8.83s | | 2,001 | 9.91s → 9.46s | 11.84s → 11.22s | At 2,001 elements, the measured runtime was approximately 4.5% lower for simple type hints and 5.2% lower for nested type hints in this local run. These measurements are preliminary and may vary by environment. They demonstrate the results of this benchmark configuration, not a guaranteed improvement for all pipelines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
