jonathansudhakar1 commented on issue #25374: URL: https://github.com/apache/beam/issues/25374#issuecomment-6881133387
@robertwb, a polite follow-up, since you added the `use_subprocess` and `timeout` modes of `with_exception_handling` (#26526). The fix touches `transforms/core.py`, so I'd like a maintainer's view on the approach in [my comment above](https://github.com/apache/beam/issues/25374#issuecomment-6710411217) before I send a PR. It still reproduces on master (de0e417a696, Python 3.13). On the DirectRunner, an `AsIter` side input sends every element to the dead-letter output with `'_thread._local' object has no attribute 'side_input_cache_tokens'`. That happens with `use_subprocess=True` and with `timeout=30`. The BundleBasedDirectRunner passes, because its side inputs are already lists. In short: 1. **Subprocess mode:** pickle the call's arguments in the dispatching thread, which has the state context, instead of leaving it to the pool's queue feeder thread. 2. **Timeout mode:** turn lazy `AsIter` side inputs into lists before handing the call to the worker thread. Is materializing the side input in the timeout path OK with you? Or would you rather carry the state context into the worker thread, or fix only the subprocess path? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
