[ 
https://issues.apache.org/jira/browse/SPARK-62072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62072:
------------------------------------

    Assignee: Hyukjin Kwon

> An Observation's _name, _result and _error are its attributes, its plan is 
> built of its _name each time, its metrics are set at the action, and a signal 
> handler's Exception is caught by the plan compression lookup, as PySpark 
> Connect
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62072
>                 URL: https://issues.apache.org/jira/browse/SPARK-62072
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Follow-ups of SPARK-62009 (#853) and SPARK-62005 (#850). Differences of the 
> drop-in from pyspark-client 4.3.0:
> 1. The attributes of an Observation. PySpark Connect's Observation 
> (python/pyspark/sql/connect/observation.py:34-86) is a plain class whose 
> `__init__` sets `self._name`, `self._result` and `self._error`, which `get` 
> and `_on` read, and which its client sets at each action 
> (connect/client/core.py:1766-1799): the client reads 
> `observations[name]._result`, asserts it is not None, converts the metrics 
> with LiteralExpression._to_value and updates it in place, or gives the error 
> of collecting them to `_set_error`. The drop-in's Observation had them as 
> read-only getters of its native state:
>    - vars(Observation("o")) was {} where PySpark's is {'_name': 'o', 
> '_result': None, '_error': None};
>    - `o._name = ...`, `o._result = None` (which PySpark's `_on` then uses 
> again) and `o._error = exc` (which `get` then raises) raised AttributeError: 
> attribute '_name' of 'Observation' objects is not writable;
>    - each read of `_error` was a new exception, where PySpark's is the one 
> object that its client set;
>    - a reference cycle through the result dict (`o.get["self"] = o`) was 
> never freed, as the native state held the dict;
>    - the metrics were converted when `get` was read: a metric of a timestamp 
> of the year 1, whose conversion to a local datetime raises ValueError: year 0 
> is out of range, made `get` raise where PySpark's action raises it and leaves 
> `get` empty ({}).
>    Once `_name` is an attribute that a program sets, the plan must read it as 
> PySpark's does (3).
> 2. A SIGINT handler that raises an Exception subclass while a call waits. 
> PySpark's lookup of the plan compression settings of the first plan 
> (_get_plan_compression_threshold_and_algorithm, core.py:2714-2740) catches 
> Exception, so the exception that a handler raises while the lookup waits is 
> caught, the compression is disabled, and the call goes on; its execution 
> interrupts the operation of a KeyboardInterrupt only (core.py:1943-1950), and 
> raises the exception of another handler at once. So the clean_cache of the 
> exit hook of a session that ran no plan, which looks the settings up first, 
> needs two signals of such a handler to end against a server that does not 
> answer: in a probe, PySpark's exit hook ended after 2 signals, the drop-in's 
> after 1; and the first action of such a session raised the second signal's 
> exception in PySpark, the first in the drop-in. The drop-in also interrupted 
> the operation of any handler's exception, so the next signal only ended the 
> wait of the interrupt: an action against a server that does not answer raised 
> the second signal's exception after 3 signals.
> 3. The name of an Observation at each build of its plan. PySpark's 
> CollectMetrics (python/pyspark/sql/connect/plan.py:1817-1837) sets 
> `str(self._observation._name)` in the name of its message each time it builds 
> the plan, after its input, and its `observations` name the Observation by 
> that str, by which the client sets the metrics; Observation._on does not read 
> the name. The drop-in's plan, which the Rust core builds, is of the name that 
> the Observation had at `observe`, so once a program can set `_name` (1):
>    - after `o._name = "other"`, the requests of its DataFrames would name the 
> first name, where PySpark's name "other", by which its metrics are set;
>    - a rename to another object, such as 5, None or b"n", which PySpark sends 
> as its str ("5", "None", "b'n'"), or to a name whose `__str__` raises or a 
> str that UTF-8 cannot encode, which fail each of PySpark's requests with that 
> error (the exception of `__str__`, or UnicodeEncodeError) until it is renamed 
> again, would change nothing;
>    - a name of another object set before `observe` would be taken as protobuf 
> takes a string field (bytes as their text, 5 as protobuf's TypeError of each 
> request), not as its str.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to