praneetnadella opened a new issue, #41002: URL: https://github.com/apache/beam/issues/41002
### What happened? With stable code identifier pickling enabled, `cloudpickle_pickler.dumps` looks like it can crash on a module-level or class-level lambda that normal pickling handles fine. To find a lambda's identifier, `_search_module_or_class` walks every name in `dir(node)` and reads each one with `getattr(node, name)` until it finds the lambda ([code_object_pickler.py#L177-L196](https://github.com/apache/beam/blob/71c11d44598ec69886b399a8c21505d646b2e7a9/sdks/python/apache_beam/internal/code_object_pickler.py#L177-L196)). If a neighboring attribute raises when read, the error escapes `get_code_object_identifier`, and cloudpickle doesn't catch it ([cloudpickle.py#L1368-L1374](https://github.com/apache/beam/blob/71c11d44598ec69886b399a8c21505d646b2e7a9/sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py#L1368-L1374)). I'd expect an unrelated attribute that fails to read not to break pickling: the lambda should still be found, or, per the `GetCodeObjectParams` docstring ("If no stable identifier can be generated, it should return None"), pickling should fall back to by-value. **Repro** (master @ 71c11d4, Python 3.13): ```python from apache_beam.internal import cloudpickle_pickler class InstanceOnly: def __get__(self, instance, owner=None): if instance is None: raise TypeError("only valid on instances") return 1 class Helpers: a_setting = InstanceOnly() fn = lambda self, x: x + 1 cloudpickle_pickler.dumps(Helpers.fn) # works cloudpickle_pickler.dumps(Helpers.fn, enable_stable_code_identifier_pickling=True) # TypeError: only valid on instances ``` The same happens for a module-level lambda in a module that uses PEP 562 `__getattr__`/`__dir__` for a lazy attribute whose optional dependency isn't installed (raises `ModuleNotFoundError`). Whether it triggers depends on name order: `dir()` is alphabetical and the scan stops once it finds the lambda, so a named lambda that sorts before the failing attribute pickles fine. A lambda that isn't bound to any name (e.g. one passed straight to `beam.Map` at module scope) is never matched, so the scan always reads every name. Found while reviewing #40711 (not introduced by it). ### Issue Priority Priority: 2 (default / most bugs should be filed as P2) ### Issue Components - [x] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
