Hyukjin Kwon created SPARK-62023:
------------------------------------

             Summary: Make SparkSession.client PySpark's SparkConnectClient: 
its state per session, a threading.local of live tags and extensions, PySpark's 
argument conversions and error order in its registrations and executions, and 
the members PySpark's tests read
                 Key: SPARK-62023
                 URL: https://issues.apache.org/jira/browse/SPARK-62023
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Three parts, one commit each.

1. The client's Python state, one per session.

PySpark 4.3.0's session has one SparkConnectClient (connect/session.py 
`self._client`), a plain object whose attributes a caller may set, of a 
`thread_local` (a `threading.local`, core.py:885) whose `tags` and 
`user_context_extensions` exist once a thread uses them (core.py:2239-2315), of 
a `global_user_context_extensions` list and its lock (core.py:961-962), and 
whose requests serialize the extensions when they are built 
(core.py:1500-1516). SparkSession.addTag/getTags delegate to it, and getTags 
returns the live set.

The drop-in's `spark.client` was a new object whenever the last reference was 
gone, each with a list and a lock of its own, so:

| (probes p_client_identity.py, p_global_stale_min.py, p_threadlocal.py, 
p_globalext.py) | pyspark-client 4.3.0 | drop-in |
|---|---|---|
| `spark.client.x = 1; spark.client.x` | 1 | AttributeError |
| the lock across statements | the same object | a new one |
| a list held before `add_global_user_context_extension(e1)`, appended e2 | 
['e1', 'e2'] | ['e2'] (the held list overwrote the session's) |
| `thread_local` | a `threading.local` | a plain object of properties |
| `thread_local.tags` / `user_context_extensions` before use | AttributeError | 
`set()` / `[]` |
| identity of the list and of its `Any` | the same objects | new ones at each 
read |
| `thread_local.user_context_extensions.append(...)`, 
`thread_local.tags.add(...)` | sent | not sent |
| `thread_local.tags = {...}` and `thread_local.user_context_extensions = 
[...]` | sent | AttributeError (no setter) |
| another thread's attributes | AttributeError, its own | the drop-in's empty 
values; attributes shared |
| `get_tags()` / `spark.getTags()` | the live set | a copy |
| an `Any` changed in place after it was added | sent as changed | sent as it 
was |

Expected: as pyspark-client 4.3.0 in each row.

2. The arguments of the registrations and the executions.

PySpark 4.3.0's register_udf (client/core.py:1078-1115, 
CommonInlineUserDefinedFunction.to_plan_udf and PythonUDF.to_plan in 
connect/expressions.py:735-850) sets the name and `deterministic` in the fields 
of its command, then parses a DDL return type (an RPC), converts it, sets the 
eval type and pickles the function; register_java (core.py:1168-1188, 
JavaUDF.to_plan) sets the name, the class name, the return type and then 
`aggregate` (of no return type only); register_udtf reads the handler's 
`__name__`, sets the name and `deterministic`, converts the return type and 
sets the eval type before it pickles. Each raises protobuf's error of its field 
in that order (an int is a bool; a bool is not an int32). PySpark's 
spark.udf.register, registerJavaFunction and registerJavaUDAF register through 
the client (connect/udf.py). Its executions read `observations` only when 
observed metrics arrive (`x.name in observations`). Its copyFromLocalToFs runs 
`urlparse(dest_path)` first (connect/session.py:1205), and its ArtifactManager 
names the artifact `FORWARD_TO_FS_PREFIX + dest_path` (artifact.py:266).

Probes p_reg_order.py, p_reg_order2.py, p_udtf_vals.py, p_reg_cases.py, 
p_udfreg_deleg.py, p_client_args.py and p_paths_more.py, against a Spark 4.3.0 
server:

| | pyspark-client 4.3.0 | drop-in |
|---|---|---|
| `register_udf(unpicklable, "int", name=5)` | TypeError: bad argument type for 
built-in operation | PicklingError, after a ddl_parse RPC |
| `register_udf(f, "not a type ###", name=5)` / `deterministic="x"` | the 
TypeError of the name / of `deterministic`, before the parse | ParseException |
| `register_udf(unpicklable, IntegerType(), eval_type="x")` | TypeError: 'str' 
object cannot be interpreted as an integer | PicklingError |
| `register_udf(..., deterministic=1)` | accepted | TypeError: 'int' object is 
not an instance of 'bool' |
| `register_udf(..., eval_type=True)` | TypeError: Field 
spark.connect.PythonUDF.eval_type: Expected an int, got a boolean. | sent; the 
server answers UnknownException (scala.MatchError) 1 |
| `register_udf(..., eval_type=2**40)` | ValueError: Value out of range: 
1099511627776 | OverflowError: number too large to fit in target type |
| `register_java(5, "x.Y", aggregate="x")` / `aggregate=1` | the name's 
TypeError / accepted | aggregate's TypeError / TypeError |
| `register_udtf(..., deterministic=1)` / `eval_type=True` / `2**40` | accepted 
/ TypeError / ValueError | TypeError / registered as 1 / OverflowError |
| `spark.udf.register(5, udf(unpicklable))`, `("\udc80", ...)` | TypeError, 
UnicodeEncodeError | PicklingError |
| `client.to_table(plan, None)` / `[]` | works; TypeError of `in` when observed 
metrics arrive | AttributeError: no attribute 'items' |
| `spark.copyFromLocalToFs(local, Path(...))` | AttributeError: 'PosixPath' 
object has no attribute 'decode' | TypeError of PyO3 |
| `client.copy_from_local_to_fs(local, Path(...))` | TypeError: can only 
concatenate str (not "PosixPath") to str | TypeError of PyO3 |

Expected: as pyspark-client 4.3.0 in each row.

3. The members that PySpark's tests and libraries read.

PySpark 4.3.0's SparkConnectClient keeps attributes and members that its tests 
and libraries read (client/core.py): `_session_id`, which its requests read, so 
that a set makes the next response of another server-side session close the 
client (test_connect_session.py:280 relies on it); `_closed`, which reopens the 
client when it is set to False; `_user_id`; `_use_reattachable_execute`; 
`_builder`, a DefaultChannelBuilder of the connection; and 
`_get_operation_statuses`, a GetStatus RPC (core.py:2183-2237).

Probe p_c6_c7_n5.py, against a Spark 4.3.0 server:

| | pyspark-client 4.3.0 | drop-in |
|---|---|---|
| `client._session_id`, `_user_id`, `_use_reattachable_execute` | the values | 
AttributeError |
| `spark.client._builder.endpoint`, `.userAgent` | the endpoint of the 
connection, `_SPARK_CONNECT_PYTHON spark/...` | AttributeError of None |
| `client._get_operation_statuses()`, `([op_id])` | a GetStatusResponse of the 
session, the statuses | AttributeError |
| `client._session_id = uuid4()`, then an execution | NO_ACTIVE_SESSION, and 
the session is stopped | a plain attribute; the execution works |
| `client._closed = True`, then `= False`, then an execution | 
NO_ACTIVE_SESSION, then the execution works | NO_ACTIVE_SESSION, then still 
NO_ACTIVE_SESSION |
| `client._use_reattachable_execute = False`, then 
`enable_reattachable_execute()` | True | a plain attribute; False |




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to