Hyukjin Kwon created SPARK-62008:
------------------------------------

             Summary: Make pyspark.sql.types.__all__ and the package 
pyspark.sql.streaming export PySpark's names only, as pyspark-client 4.3.0
                 Key: SPARK-62008
                 URL: https://issues.apache.org/jira/browse/SPARK-62008
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 exports names from pyspark.sql.types and 
pyspark.sql.streaming that PySpark does not, so code that star-imports them, or 
imports one of these names from the package, works on the drop-in and fails or 
differs on pyspark-client.

1. pyspark.sql.types.__all__. PySpark 4.3.0's lists 32 names 
(python/pyspark/sql/types.py:79-112). The drop-in's listed 36: also 
UserDefinedType, _drop_metadata, _parse_datatype_json_string and 
_parse_datatype_json_value.

     import pyspark.sql.types as t
     ns = {}
     exec("from pyspark.sql.types import *", ns)
     print(len(t.__all__), "UserDefinedType" in ns)

   pyspark-client 4.3.0: 32 False
   drop-in: 36 True

   The order of the list was the drop-in's own too.

2. pyspark.sql.streaming. PySpark 4.3.0's 
python/pyspark/sql/streaming/__init__.py imports StreamingQuery, 
StreamingQueryManager, DataStreamReader, DataStreamWriter, 
StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle, TwsTester 
and StreamingQueryException, and has no __all__. The drop-in's file had an 
__all__ of 17 names, among them Trigger (a native class of the drop-in's 
pyspark.sql.streaming.readwriter only) and the read limits of 
pyspark.sql.streaming.datasource (ReadLimit, ReadAllAvailable, ReadMinRows, 
ReadMaxRows, ReadMaxFiles, ReadMaxBytes, SupportsTriggerAvailableNow).

     from pyspark.sql.streaming import ReadLimit, Trigger

   pyspark-client 4.3.0: ImportError: cannot import name 'ReadLimit' from 
'pyspark.sql.streaming'
   drop-in: imports both.

     ns = {}
     exec("from pyspark.sql.streaming import *", ns)
     print(sorted(k for k in ns if not k.startswith("__")))

   pyspark-client 4.3.0: the 9 classes and the loaded submodules 
list_state_client, listener, map_state_client, query, readwriter, state, 
stateful_processor, stateful_processor_api_client, tws_tester and 
value_state_client.
   drop-in: the 17 names of its __all__.

   PySpark's `import pyspark.sql` loads pyspark.sql.streaming.state through 
pyspark.sql.pandas.group_ops, which imports GroupStateTimeout and 
StatefulProcessor (python/pyspark/sql/pandas/group_ops.py:25-26). The drop-in's 
group_ops shim did not, so with the package vendored the star import would 
still lack `state`.

Expected: pyspark.sql.types.__all__ is PySpark's list, and 
pyspark/sql/streaming/__init__.py is PySpark's file (kept byte-identical to the 
tag by scripts/check_vendored_upstream.sh), with the namespace and the star 
import of pyspark-client 4.3.0.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to