Hyukjin Kwon created SPARK-62008:
------------------------------------
Summary: Make pyspark.sql.types.__all__ and the package
pyspark.sql.streaming export PySpark's names only, as pyspark-client 4.3.0
Key: SPARK-62008
URL: https://issues.apache.org/jira/browse/SPARK-62008
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in for pyspark-client 4.3.0 exports names from pyspark.sql.types and
pyspark.sql.streaming that PySpark does not, so code that star-imports them, or
imports one of these names from the package, works on the drop-in and fails or
differs on pyspark-client.
1. pyspark.sql.types.__all__. PySpark 4.3.0's lists 32 names
(python/pyspark/sql/types.py:79-112). The drop-in's listed 36: also
UserDefinedType, _drop_metadata, _parse_datatype_json_string and
_parse_datatype_json_value.
import pyspark.sql.types as t
ns = {}
exec("from pyspark.sql.types import *", ns)
print(len(t.__all__), "UserDefinedType" in ns)
pyspark-client 4.3.0: 32 False
drop-in: 36 True
The order of the list was the drop-in's own too.
2. pyspark.sql.streaming. PySpark 4.3.0's
python/pyspark/sql/streaming/__init__.py imports StreamingQuery,
StreamingQueryManager, DataStreamReader, DataStreamWriter,
StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle, TwsTester
and StreamingQueryException, and has no __all__. The drop-in's file had an
__all__ of 17 names, among them Trigger (a native class of the drop-in's
pyspark.sql.streaming.readwriter only) and the read limits of
pyspark.sql.streaming.datasource (ReadLimit, ReadAllAvailable, ReadMinRows,
ReadMaxRows, ReadMaxFiles, ReadMaxBytes, SupportsTriggerAvailableNow).
from pyspark.sql.streaming import ReadLimit, Trigger
pyspark-client 4.3.0: ImportError: cannot import name 'ReadLimit' from
'pyspark.sql.streaming'
drop-in: imports both.
ns = {}
exec("from pyspark.sql.streaming import *", ns)
print(sorted(k for k in ns if not k.startswith("__")))
pyspark-client 4.3.0: the 9 classes and the loaded submodules
list_state_client, listener, map_state_client, query, readwriter, state,
stateful_processor, stateful_processor_api_client, tws_tester and
value_state_client.
drop-in: the 17 names of its __all__.
PySpark's `import pyspark.sql` loads pyspark.sql.streaming.state through
pyspark.sql.pandas.group_ops, which imports GroupStateTimeout and
StatefulProcessor (python/pyspark/sql/pandas/group_ops.py:25-26). The drop-in's
group_ops shim did not, so with the package vendored the star import would
still lack `state`.
Expected: pyspark.sql.types.__all__ is PySpark's list, and
pyspark/sql/streaming/__init__.py is PySpark's file (kept byte-identical to the
tag by scripts/check_vendored_upstream.sh), with the namespace and the star
import of pyspark-client 4.3.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]