Hyukjin Kwon created SPARK-62072:
------------------------------------

             Summary: An Observation's _name, _result and _error are its 
attributes, its plan is built of its _name each time, its metrics are set at 
the action, and a signal handler's Exception is caught by the plan compression 
lookup, as PySpark Connect
                 Key: SPARK-62072
                 URL: https://issues.apache.org/jira/browse/SPARK-62072
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Follow-ups of SPARK-62009 (#853) and SPARK-62005 (#850). Differences of the 
drop-in from pyspark-client 4.3.0:

1. The attributes of an Observation. PySpark Connect's Observation 
(python/pyspark/sql/connect/observation.py:34-86) is a plain class whose 
`__init__` sets `self._name`, `self._result` and `self._error`, which `get` and 
`_on` read, and which its client sets at each action 
(connect/client/core.py:1766-1799): the client reads 
`observations[name]._result`, asserts it is not None, converts the metrics with 
LiteralExpression._to_value and updates it in place, or gives the error of 
collecting them to `_set_error`. The drop-in's Observation had them as 
read-only getters of its native state:
   - vars(Observation("o")) was {} where PySpark's is {'_name': 'o', '_result': 
None, '_error': None};
   - `o._name = ...`, `o._result = None` (which PySpark's `_on` then uses 
again) and `o._error = exc` (which `get` then raises) raised AttributeError: 
attribute '_name' of 'Observation' objects is not writable;
   - each read of `_error` was a new exception, where PySpark's is the one 
object that its client set;
   - a reference cycle through the result dict (`o.get["self"] = o`) was never 
freed, as the native state held the dict;
   - the metrics were converted when `get` was read: a metric of a timestamp of 
the year 1, whose conversion to a local datetime raises ValueError: year 0 is 
out of range, made `get` raise where PySpark's action raises it and leaves 
`get` empty ({}).

   Once `_name` is an attribute that a program sets, the plan must read it as 
PySpark's does (3).

2. A SIGINT handler that raises an Exception subclass while a call waits. 
PySpark's lookup of the plan compression settings of the first plan 
(_get_plan_compression_threshold_and_algorithm, core.py:2714-2740) catches 
Exception, so the exception that a handler raises while the lookup waits is 
caught, the compression is disabled, and the call goes on; its execution 
interrupts the operation of a KeyboardInterrupt only (core.py:1943-1950), and 
raises the exception of another handler at once. So the clean_cache of the exit 
hook of a session that ran no plan, which looks the settings up first, needs 
two signals of such a handler to end against a server that does not answer: in 
a probe, PySpark's exit hook ended after 2 signals, the drop-in's after 1; and 
the first action of such a session raised the second signal's exception in 
PySpark, the first in the drop-in. The drop-in also interrupted the operation 
of any handler's exception, so the next signal only ended the wait of the 
interrupt: an action against a server that does not answer raised the second 
signal's exception after 3 signals.

3. The name of an Observation at each build of its plan. PySpark's 
CollectMetrics (python/pyspark/sql/connect/plan.py:1817-1837) sets 
`str(self._observation._name)` in the name of its message each time it builds 
the plan, after its input, and its `observations` name the Observation by that 
str, by which the client sets the metrics; Observation._on does not read the 
name. The drop-in's plan, which the Rust core builds, is of the name that the 
Observation had at `observe`, so once a program can set `_name` (1):
   - after `o._name = "other"`, the requests of its DataFrames would name the 
first name, where PySpark's name "other", by which its metrics are set;
   - a rename to another object, such as 5, None or b"n", which PySpark sends 
as its str ("5", "None", "b'n'"), or to a name whose `__str__` raises or a str 
that UTF-8 cannot encode, which fail each of PySpark's requests with that error 
(the exception of `__str__`, or UnicodeEncodeError) until it is renamed again, 
would change nothing;
   - a name of another object set before `observe` would be taken as protobuf 
takes a string field (bytes as their text, 5 as protobuf's TypeError of each 
request), not as its str.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to