[ 
https://issues.apache.org/jira/browse/SPARK-62023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62023:
------------------------------------

    Assignee: Hyukjin Kwon

> Make SparkSession.client PySpark's SparkConnectClient: its state per session, 
> a threading.local of live tags and extensions, PySpark's argument conversions 
> and error order in its registrations and executions, and the members 
> PySpark's tests read
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62023
>                 URL: https://issues.apache.org/jira/browse/SPARK-62023
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Three parts, one commit each.
> 1. The client's Python state, one per session.
> PySpark 4.3.0's session has one SparkConnectClient (connect/session.py 
> `self._client`), a plain object whose attributes a caller may set, of a 
> `thread_local` (a `threading.local`, core.py:885) whose `tags` and 
> `user_context_extensions` exist once a thread uses them (core.py:2239-2315), 
> of a `global_user_context_extensions` list and its lock (core.py:961-962), 
> and whose requests serialize the extensions when they are built 
> (core.py:1500-1516). SparkSession.addTag/getTags delegate to it, and getTags 
> returns the live set.
> The drop-in's `spark.client` was a new object whenever the last reference was 
> gone, each with a list and a lock of its own, so:
> | (probes p_client_identity.py, p_global_stale_min.py, p_threadlocal.py, 
> p_globalext.py) | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `spark.client.x = 1; spark.client.x` | 1 | AttributeError |
> | the lock across statements | the same object | a new one |
> | a list held before `add_global_user_context_extension(e1)`, appended e2 | 
> ['e1', 'e2'] | ['e2'] (the held list overwrote the session's) |
> | `thread_local` | a `threading.local` | a plain object of properties |
> | `thread_local.tags` / `user_context_extensions` before use | AttributeError 
> | `set()` / `[]` |
> | identity of the list and of its `Any` | the same objects | new ones at each 
> read |
> | `thread_local.user_context_extensions.append(...)`, 
> `thread_local.tags.add(...)` | sent | not sent |
> | `thread_local.tags = {...}` and `thread_local.user_context_extensions = 
> [...]` | sent | AttributeError (no setter) |
> | another thread's attributes | AttributeError, its own | the drop-in's empty 
> values; attributes shared |
> | `get_tags()` / `spark.getTags()` | the live set | a copy |
> | an `Any` changed in place after it was added | sent as changed | sent as it 
> was |
> Expected: as pyspark-client 4.3.0 in each row.
> 2. The arguments of the registrations and the executions.
> PySpark 4.3.0's register_udf (client/core.py:1078-1115, 
> CommonInlineUserDefinedFunction.to_plan_udf and PythonUDF.to_plan in 
> connect/expressions.py:735-850) sets the name and `deterministic` in the 
> fields of its command, then parses a DDL return type (an RPC), converts it, 
> sets the eval type and pickles the function; register_java 
> (core.py:1168-1188, JavaUDF.to_plan) sets the name, the class name, the 
> return type and then `aggregate` (of no return type only); register_udtf 
> reads the handler's `__name__`, sets the name and `deterministic`, converts 
> the return type and sets the eval type before it pickles. Each raises 
> protobuf's error of its field in that order (an int is a bool; a bool is not 
> an int32). PySpark's spark.udf.register, registerJavaFunction and 
> registerJavaUDAF register through the client (connect/udf.py). Its executions 
> read `observations` only when observed metrics arrive (`x.name in 
> observations`). Its copyFromLocalToFs runs `urlparse(dest_path)` first 
> (connect/session.py:1205), and its ArtifactManager names the artifact 
> `FORWARD_TO_FS_PREFIX + dest_path` (artifact.py:266).
> Probes p_reg_order.py, p_reg_order2.py, p_udtf_vals.py, p_reg_cases.py, 
> p_udfreg_deleg.py, p_client_args.py and p_paths_more.py, against a Spark 
> 4.3.0 server:
> | | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `register_udf(unpicklable, "int", name=5)` | TypeError: bad argument type 
> for built-in operation | PicklingError, after a ddl_parse RPC |
> | `register_udf(f, "not a type ###", name=5)` / `deterministic="x"` | the 
> TypeError of the name / of `deterministic`, before the parse | ParseException 
> |
> | `register_udf(unpicklable, IntegerType(), eval_type="x")` | TypeError: 
> 'str' object cannot be interpreted as an integer | PicklingError |
> | `register_udf(..., deterministic=1)` | accepted | TypeError: 'int' object 
> is not an instance of 'bool' |
> | `register_udf(..., eval_type=True)` | TypeError: Field 
> spark.connect.PythonUDF.eval_type: Expected an int, got a boolean. | sent; 
> the server answers UnknownException (scala.MatchError) 1 |
> | `register_udf(..., eval_type=2**40)` | ValueError: Value out of range: 
> 1099511627776 | OverflowError: number too large to fit in target type |
> | `register_java(5, "x.Y", aggregate="x")` / `aggregate=1` | the name's 
> TypeError / accepted | aggregate's TypeError / TypeError |
> | `register_udtf(..., deterministic=1)` / `eval_type=True` / `2**40` | 
> accepted / TypeError / ValueError | TypeError / registered as 1 / 
> OverflowError |
> | `spark.udf.register(5, udf(unpicklable))`, `("\udc80", ...)` | TypeError, 
> UnicodeEncodeError | PicklingError |
> | `client.to_table(plan, None)` / `[]` | works; TypeError of `in` when 
> observed metrics arrive | AttributeError: no attribute 'items' |
> | `spark.copyFromLocalToFs(local, Path(...))` | AttributeError: 'PosixPath' 
> object has no attribute 'decode' | TypeError of PyO3 |
> | `client.copy_from_local_to_fs(local, Path(...))` | TypeError: can only 
> concatenate str (not "PosixPath") to str | TypeError of PyO3 |
> Expected: as pyspark-client 4.3.0 in each row.
> 3. The members that PySpark's tests and libraries read.
> PySpark 4.3.0's SparkConnectClient keeps attributes and members that its 
> tests and libraries read (client/core.py): `_session_id`, which its requests 
> read, so that a set makes the next response of another server-side session 
> close the client (test_connect_session.py:280 relies on it); `_closed`, which 
> reopens the client when it is set to False; `_user_id`; 
> `_use_reattachable_execute`; `_builder`, a DefaultChannelBuilder of the 
> connection; and `_get_operation_statuses`, a GetStatus RPC 
> (core.py:2183-2237).
> Probe p_c6_c7_n5.py, against a Spark 4.3.0 server:
> | | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `client._session_id`, `_user_id`, `_use_reattachable_execute` | the values 
> | AttributeError |
> | `spark.client._builder.endpoint`, `.userAgent` | the endpoint of the 
> connection, `_SPARK_CONNECT_PYTHON spark/...` | AttributeError of None |
> | `client._get_operation_statuses()`, `([op_id])` | a GetStatusResponse of 
> the session, the statuses | AttributeError |
> | `client._session_id = uuid4()`, then an execution | NO_ACTIVE_SESSION, and 
> the session is stopped | a plain attribute; the execution works |
> | `client._closed = True`, then `= False`, then an execution | 
> NO_ACTIVE_SESSION, then the execution works | NO_ACTIVE_SESSION, then still 
> NO_ACTIVE_SESSION |
> | `client._use_reattachable_execute = False`, then 
> `enable_reattachable_execute()` | True | a plain attribute; False |



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to