[ 
https://issues.apache.org/jira/browse/SPARK-62009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62009:
------------------------------------

    Assignee: Hyukjin Kwon

> Make the Observations of the Python client PySpark's objects: get as the live 
> _result dict, the plan holding the Observation itself, and the AssertionError 
> of an unused Observation given to SparkConnectClient.to_table
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62009
>                 URL: https://issues.apache.org/jira/browse/SPARK-62009
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> pyspark.sql.connect.observation.Observation of pyspark-client 4.3.0 is a 
> plain object of _name, _result and _error (connect/observation.py):
> - `_on` sets `_result = {}`, and `get` returns that dict. The client updates 
> it in place, `observations[name]._result.update(...)` (client/core.py, 
> `_execute_and_fetch_as_iterator`), so `get` is one dict, whose changes stay 
> and which a shallow copy shares.
> - Its CollectMetrics plan holds the Observation itself 
> (connect/plan.py:1803-1837), so `df._plan.observations` gives the object that 
> observe was given, of its subclass and attributes, while the DataFrame lives.
> - The client asserts that the Observation of the name of observed metrics is 
> used (`assert observation_result is not None`), so `client.to_table(plan, 
> {"x": Observation("x")})` with an Observation that is not used raises 
> AssertionError when the metrics arrive.
> The drop-in differs in each (probes p_obs_get.py and kd-peer N7):
> | | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `o.get is o.get` | True | False |
> | a dict read before the action, after `collect()` | `{'n': 3}` | `{}` |
> | `o.get["extra"] = 1`, then `o.get` | `{'n': 3, 'extra': 1}` | `{'n': 3}` |
> | `o._result`, `o._error` | None, None; then the dict | AttributeError |
> | `df._plan.observations` once a subclass Observation with an attribute is 
> dropped | the object, of its subclass and attribute | a new `Observation`, 
> without the attribute |
> | `client.to_table(plan, {"x": Observation("x")})` | `AssertionError` | sets 
> nothing |
> Expected: the drop-in's Observation as PySpark's in each row.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to