Holden Karau created SPARK-59029:
------------------------------------
Summary: Transpilation: resolve `self.<attr>` reads on a callable
instance
Key: SPARK-59029
URL: https://issues.apache.org/jira/browse/SPARK-59029
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.3.0
Reporter: Holden Karau
SPARK-55207 adds free-variable capture for closure cells and module globals. A
callable instance whose `__call__` reads instance state is not covered: a
reference to the receiver is refused outright in
`_LiteralNormalizer.visit_Name`, so
```python
class AddK:
def __init__(self, k):
self.k = k
def __call__(self, x):
return x + self.k
```
falls back to interpreted Python even though `self.k` is a plain bakeable int.
This is a deliberate omission, not an oversight. Resolving it faithfully means
reproducing:
- Python's attribute lookup — the descriptor protocol and the MRO, including
`__slots__`, properties, and `__getattr__` / `__getattribute__` overrides,
none
of which we want to be *running* while deciding what to transpile
(`_call_dunder` already uses `getattr_static` specifically to avoid firing
user
code); and
- cloudpickle's instance-state rules, since baking is only sound when the
interpreted path would have shipped the same value. The existing gate for a
callable instance already has to check that both the class owning `__call__`
and the receiver's own class are pickled by value
(`test_udf_transpile_callable_instance_gated_on_call_owner`); instance
attributes add the object's own `__reduce__` / `__getstate__` to that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
