Hyukjin Kwon created SPARK-62023:
------------------------------------
Summary: Make SparkSession.client PySpark's SparkConnectClient:
its state per session, a threading.local of live tags and extensions, PySpark's
argument conversions and error order in its registrations and executions, and
the members PySpark's tests read
Key: SPARK-62023
URL: https://issues.apache.org/jira/browse/SPARK-62023
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Three parts, one commit each.
1. The client's Python state, one per session.
PySpark 4.3.0's session has one SparkConnectClient (connect/session.py
`self._client`), a plain object whose attributes a caller may set, of a
`thread_local` (a `threading.local`, core.py:885) whose `tags` and
`user_context_extensions` exist once a thread uses them (core.py:2239-2315), of
a `global_user_context_extensions` list and its lock (core.py:961-962), and
whose requests serialize the extensions when they are built
(core.py:1500-1516). SparkSession.addTag/getTags delegate to it, and getTags
returns the live set.
The drop-in's `spark.client` was a new object whenever the last reference was
gone, each with a list and a lock of its own, so:
| (probes p_client_identity.py, p_global_stale_min.py, p_threadlocal.py,
p_globalext.py) | pyspark-client 4.3.0 | drop-in |
|---|---|---|
| `spark.client.x = 1; spark.client.x` | 1 | AttributeError |
| the lock across statements | the same object | a new one |
| a list held before `add_global_user_context_extension(e1)`, appended e2 |
['e1', 'e2'] | ['e2'] (the held list overwrote the session's) |
| `thread_local` | a `threading.local` | a plain object of properties |
| `thread_local.tags` / `user_context_extensions` before use | AttributeError |
`set()` / `[]` |
| identity of the list and of its `Any` | the same objects | new ones at each
read |
| `thread_local.user_context_extensions.append(...)`,
`thread_local.tags.add(...)` | sent | not sent |
| `thread_local.tags = {...}` and `thread_local.user_context_extensions =
[...]` | sent | AttributeError (no setter) |
| another thread's attributes | AttributeError, its own | the drop-in's empty
values; attributes shared |
| `get_tags()` / `spark.getTags()` | the live set | a copy |
| an `Any` changed in place after it was added | sent as changed | sent as it
was |
Expected: as pyspark-client 4.3.0 in each row.
2. The arguments of the registrations and the executions.
PySpark 4.3.0's register_udf (client/core.py:1078-1115,
CommonInlineUserDefinedFunction.to_plan_udf and PythonUDF.to_plan in
connect/expressions.py:735-850) sets the name and `deterministic` in the fields
of its command, then parses a DDL return type (an RPC), converts it, sets the
eval type and pickles the function; register_java (core.py:1168-1188,
JavaUDF.to_plan) sets the name, the class name, the return type and then
`aggregate` (of no return type only); register_udtf reads the handler's
`__name__`, sets the name and `deterministic`, converts the return type and
sets the eval type before it pickles. Each raises protobuf's error of its field
in that order (an int is a bool; a bool is not an int32). PySpark's
spark.udf.register, registerJavaFunction and registerJavaUDAF register through
the client (connect/udf.py). Its executions read `observations` only when
observed metrics arrive (`x.name in observations`). Its copyFromLocalToFs runs
`urlparse(dest_path)` first (connect/session.py:1205), and its ArtifactManager
names the artifact `FORWARD_TO_FS_PREFIX + dest_path` (artifact.py:266).
Probes p_reg_order.py, p_reg_order2.py, p_udtf_vals.py, p_reg_cases.py,
p_udfreg_deleg.py, p_client_args.py and p_paths_more.py, against a Spark 4.3.0
server:
| | pyspark-client 4.3.0 | drop-in |
|---|---|---|
| `register_udf(unpicklable, "int", name=5)` | TypeError: bad argument type for
built-in operation | PicklingError, after a ddl_parse RPC |
| `register_udf(f, "not a type ###", name=5)` / `deterministic="x"` | the
TypeError of the name / of `deterministic`, before the parse | ParseException |
| `register_udf(unpicklable, IntegerType(), eval_type="x")` | TypeError: 'str'
object cannot be interpreted as an integer | PicklingError |
| `register_udf(..., deterministic=1)` | accepted | TypeError: 'int' object is
not an instance of 'bool' |
| `register_udf(..., eval_type=True)` | TypeError: Field
spark.connect.PythonUDF.eval_type: Expected an int, got a boolean. | sent; the
server answers UnknownException (scala.MatchError) 1 |
| `register_udf(..., eval_type=2**40)` | ValueError: Value out of range:
1099511627776 | OverflowError: number too large to fit in target type |
| `register_java(5, "x.Y", aggregate="x")` / `aggregate=1` | the name's
TypeError / accepted | aggregate's TypeError / TypeError |
| `register_udtf(..., deterministic=1)` / `eval_type=True` / `2**40` | accepted
/ TypeError / ValueError | TypeError / registered as 1 / OverflowError |
| `spark.udf.register(5, udf(unpicklable))`, `("\udc80", ...)` | TypeError,
UnicodeEncodeError | PicklingError |
| `client.to_table(plan, None)` / `[]` | works; TypeError of `in` when observed
metrics arrive | AttributeError: no attribute 'items' |
| `spark.copyFromLocalToFs(local, Path(...))` | AttributeError: 'PosixPath'
object has no attribute 'decode' | TypeError of PyO3 |
| `client.copy_from_local_to_fs(local, Path(...))` | TypeError: can only
concatenate str (not "PosixPath") to str | TypeError of PyO3 |
Expected: as pyspark-client 4.3.0 in each row.
3. The members that PySpark's tests and libraries read.
PySpark 4.3.0's SparkConnectClient keeps attributes and members that its tests
and libraries read (client/core.py): `_session_id`, which its requests read, so
that a set makes the next response of another server-side session close the
client (test_connect_session.py:280 relies on it); `_closed`, which reopens the
client when it is set to False; `_user_id`; `_use_reattachable_execute`;
`_builder`, a DefaultChannelBuilder of the connection; and
`_get_operation_statuses`, a GetStatus RPC (core.py:2183-2237).
Probe p_c6_c7_n5.py, against a Spark 4.3.0 server:
| | pyspark-client 4.3.0 | drop-in |
|---|---|---|
| `client._session_id`, `_user_id`, `_use_reattachable_execute` | the values |
AttributeError |
| `spark.client._builder.endpoint`, `.userAgent` | the endpoint of the
connection, `_SPARK_CONNECT_PYTHON spark/...` | AttributeError of None |
| `client._get_operation_statuses()`, `([op_id])` | a GetStatusResponse of the
session, the statuses | AttributeError |
| `client._session_id = uuid4()`, then an execution | NO_ACTIVE_SESSION, and
the session is stopped | a plain attribute; the execution works |
| `client._closed = True`, then `= False`, then an execution |
NO_ACTIVE_SESSION, then the execution works | NO_ACTIVE_SESSION, then still
NO_ACTIVE_SESSION |
| `client._use_reattachable_execute = False`, then
`enable_reattachable_execute()` | True | a plain attribute; False |
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]