[
https://issues.apache.org/jira/browse/SPARK-62008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62008.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 852
[https://github.com/apache/spark-connect-rust/pull/852]
> Make pyspark.sql.types.__all__ and the package pyspark.sql.streaming export
> PySpark's names only, as pyspark-client 4.3.0
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62008
> URL: https://issues.apache.org/jira/browse/SPARK-62008
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The drop-in for pyspark-client 4.3.0 exports names from pyspark.sql.types and
> pyspark.sql.streaming that PySpark does not, so code that star-imports them,
> or imports one of these names from the package, works on the drop-in and
> fails or differs on pyspark-client.
> 1. pyspark.sql.types.__all__. PySpark 4.3.0's lists 32 names
> (python/pyspark/sql/types.py:79-112). The drop-in's listed 36: also
> UserDefinedType, _drop_metadata, _parse_datatype_json_string and
> _parse_datatype_json_value.
> import pyspark.sql.types as t
> ns = {}
> exec("from pyspark.sql.types import *", ns)
> print(len(t.__all__), "UserDefinedType" in ns)
> pyspark-client 4.3.0: 32 False
> drop-in: 36 True
> The order of the list was the drop-in's own too.
> 2. pyspark.sql.streaming. PySpark 4.3.0's
> python/pyspark/sql/streaming/__init__.py imports StreamingQuery,
> StreamingQueryManager, DataStreamReader, DataStreamWriter,
> StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle, TwsTester
> and StreamingQueryException, and has no __all__. The drop-in's file had an
> __all__ of 17 names, among them Trigger (a native class of the drop-in's
> pyspark.sql.streaming.readwriter only) and the read limits of
> pyspark.sql.streaming.datasource (ReadLimit, ReadAllAvailable, ReadMinRows,
> ReadMaxRows, ReadMaxFiles, ReadMaxBytes, SupportsTriggerAvailableNow).
> from pyspark.sql.streaming import ReadLimit, Trigger
> pyspark-client 4.3.0: ImportError: cannot import name 'ReadLimit' from
> 'pyspark.sql.streaming'
> drop-in: imports both.
> ns = {}
> exec("from pyspark.sql.streaming import *", ns)
> print(sorted(k for k in ns if not k.startswith("__")))
> pyspark-client 4.3.0: the 9 classes and the loaded submodules
> list_state_client, listener, map_state_client, query, readwriter, state,
> stateful_processor, stateful_processor_api_client, tws_tester and
> value_state_client.
> drop-in: the 17 names of its __all__.
> PySpark's `import pyspark.sql` loads pyspark.sql.streaming.state through
> pyspark.sql.pandas.group_ops, which imports GroupStateTimeout and
> StatefulProcessor (python/pyspark/sql/pandas/group_ops.py:25-26). The
> drop-in's group_ops shim did not, so with the package vendored the star
> import would still lack `state`.
> Expected: pyspark.sql.types.__all__ is PySpark's list, and
> pyspark/sql/streaming/__init__.py is PySpark's file (kept byte-identical to
> the tag by scripts/check_vendored_upstream.sh), with the namespace and the
> star import of pyspark-client 4.3.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]