[ 
https://issues.apache.org/jira/browse/SPARK-62008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62008.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 852
[https://github.com/apache/spark-connect-rust/pull/852]

> Make pyspark.sql.types.__all__ and the package pyspark.sql.streaming export 
> PySpark's names only, as pyspark-client 4.3.0
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62008
>                 URL: https://issues.apache.org/jira/browse/SPARK-62008
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The drop-in for pyspark-client 4.3.0 exports names from pyspark.sql.types and 
> pyspark.sql.streaming that PySpark does not, so code that star-imports them, 
> or imports one of these names from the package, works on the drop-in and 
> fails or differs on pyspark-client.
> 1. pyspark.sql.types.__all__. PySpark 4.3.0's lists 32 names 
> (python/pyspark/sql/types.py:79-112). The drop-in's listed 36: also 
> UserDefinedType, _drop_metadata, _parse_datatype_json_string and 
> _parse_datatype_json_value.
>      import pyspark.sql.types as t
>      ns = {}
>      exec("from pyspark.sql.types import *", ns)
>      print(len(t.__all__), "UserDefinedType" in ns)
>    pyspark-client 4.3.0: 32 False
>    drop-in: 36 True
>    The order of the list was the drop-in's own too.
> 2. pyspark.sql.streaming. PySpark 4.3.0's 
> python/pyspark/sql/streaming/__init__.py imports StreamingQuery, 
> StreamingQueryManager, DataStreamReader, DataStreamWriter, 
> StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle, TwsTester 
> and StreamingQueryException, and has no __all__. The drop-in's file had an 
> __all__ of 17 names, among them Trigger (a native class of the drop-in's 
> pyspark.sql.streaming.readwriter only) and the read limits of 
> pyspark.sql.streaming.datasource (ReadLimit, ReadAllAvailable, ReadMinRows, 
> ReadMaxRows, ReadMaxFiles, ReadMaxBytes, SupportsTriggerAvailableNow).
>      from pyspark.sql.streaming import ReadLimit, Trigger
>    pyspark-client 4.3.0: ImportError: cannot import name 'ReadLimit' from 
> 'pyspark.sql.streaming'
>    drop-in: imports both.
>      ns = {}
>      exec("from pyspark.sql.streaming import *", ns)
>      print(sorted(k for k in ns if not k.startswith("__")))
>    pyspark-client 4.3.0: the 9 classes and the loaded submodules 
> list_state_client, listener, map_state_client, query, readwriter, state, 
> stateful_processor, stateful_processor_api_client, tws_tester and 
> value_state_client.
>    drop-in: the 17 names of its __all__.
>    PySpark's `import pyspark.sql` loads pyspark.sql.streaming.state through 
> pyspark.sql.pandas.group_ops, which imports GroupStateTimeout and 
> StatefulProcessor (python/pyspark/sql/pandas/group_ops.py:25-26). The 
> drop-in's group_ops shim did not, so with the package vendored the star 
> import would still lack `state`.
> Expected: pyspark.sql.types.__all__ is PySpark's list, and 
> pyspark/sql/streaming/__init__.py is PySpark's file (kept byte-identical to 
> the tag by scripts/check_vendored_upstream.sh), with the namespace and the 
> star import of pyspark-client 4.3.0.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to