[
https://issues.apache.org/jira/browse/SPARK-62072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62072:
-----------------------------------
Labels: pull-request-available (was: )
> An Observation's _name, _result and _error are its attributes, its plan is
> built of its _name each time, its metrics are set at the action, and a signal
> handler's Exception is caught by the plan compression lookup, as PySpark
> Connect
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62072
> URL: https://issues.apache.org/jira/browse/SPARK-62072
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Follow-ups of SPARK-62009 (#853) and SPARK-62005 (#850). Differences of the
> drop-in from pyspark-client 4.3.0:
> 1. The attributes of an Observation. PySpark Connect's Observation
> (python/pyspark/sql/connect/observation.py:34-86) is a plain class whose
> `__init__` sets `self._name`, `self._result` and `self._error`, which `get`
> and `_on` read, and which its client sets at each action
> (connect/client/core.py:1766-1799): the client reads
> `observations[name]._result`, asserts it is not None, converts the metrics
> with LiteralExpression._to_value and updates it in place, or gives the error
> of collecting them to `_set_error`. The drop-in's Observation had them as
> read-only getters of its native state:
> - vars(Observation("o")) was {} where PySpark's is {'_name': 'o',
> '_result': None, '_error': None};
> - `o._name = ...`, `o._result = None` (which PySpark's `_on` then uses
> again) and `o._error = exc` (which `get` then raises) raised AttributeError:
> attribute '_name' of 'Observation' objects is not writable;
> - each read of `_error` was a new exception, where PySpark's is the one
> object that its client set;
> - a reference cycle through the result dict (`o.get["self"] = o`) was
> never freed, as the native state held the dict;
> - the metrics were converted when `get` was read: a metric of a timestamp
> of the year 1, whose conversion to a local datetime raises ValueError: year 0
> is out of range, made `get` raise where PySpark's action raises it and leaves
> `get` empty ({}).
> Once `_name` is an attribute that a program sets, the plan must read it as
> PySpark's does (3).
> 2. A SIGINT handler that raises an Exception subclass while a call waits.
> PySpark's lookup of the plan compression settings of the first plan
> (_get_plan_compression_threshold_and_algorithm, core.py:2714-2740) catches
> Exception, so the exception that a handler raises while the lookup waits is
> caught, the compression is disabled, and the call goes on; its execution
> interrupts the operation of a KeyboardInterrupt only (core.py:1943-1950), and
> raises the exception of another handler at once. So the clean_cache of the
> exit hook of a session that ran no plan, which looks the settings up first,
> needs two signals of such a handler to end against a server that does not
> answer: in a probe, PySpark's exit hook ended after 2 signals, the drop-in's
> after 1; and the first action of such a session raised the second signal's
> exception in PySpark, the first in the drop-in. The drop-in also interrupted
> the operation of any handler's exception, so the next signal only ended the
> wait of the interrupt: an action against a server that does not answer raised
> the second signal's exception after 3 signals.
> 3. The name of an Observation at each build of its plan. PySpark's
> CollectMetrics (python/pyspark/sql/connect/plan.py:1817-1837) sets
> `str(self._observation._name)` in the name of its message each time it builds
> the plan, after its input, and its `observations` name the Observation by
> that str, by which the client sets the metrics; Observation._on does not read
> the name. The drop-in's plan, which the Rust core builds, is of the name that
> the Observation had at `observe`, so once a program can set `_name` (1):
> - after `o._name = "other"`, the requests of its DataFrames would name the
> first name, where PySpark's name "other", by which its metrics are set;
> - a rename to another object, such as 5, None or b"n", which PySpark sends
> as its str ("5", "None", "b'n'"), or to a name whose `__str__` raises or a
> str that UTF-8 cannot encode, which fail each of PySpark's requests with that
> error (the exception of `__str__`, or UnicodeEncodeError) until it is renamed
> again, would change nothing;
> - a name of another object set before `observe` would be taken as protobuf
> takes a string field (bytes as their text, 5 as protobuf's TypeError of each
> request), not as its str.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]