Hyukjin Kwon created SPARK-62074:
------------------------------------

             Summary: pyspark.sql.utils lacks 17 of PySpark's names, and the 
package pyspark.sql.connect lacks check_dependencies and utils and has udf and 
udtf, in the Python client
                 Key: SPARK-62074
                 URL: https://issues.apache.org/jira/browse/SPARK-62074
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Differences of the drop-in from pyspark-client 4.3.0:

1. pyspark.sql.utils. PySpark's module (python/pyspark/sql/utils.py, 513 lines) 
has 56 names, of the helpers and the decorators that its modules import: the 
decorators that run the functions of PySpark Connect's modules 
(try_remote_functions, try_partitioning_remote_functions, 
try_remote_avro_functions, try_remote_protobuf_functions, utils.py:273-331) and 
the methods of the classes of Spark Connect (dispatch_df_method, 
dispatch_col_method, dispatch_window_method, dispatch_table_arg_method, 
utils.py:363-455), StringConcat, escape_meta_characters, require_test_compiled, 
to_java_array, and the names that it imports (is_remote_only, JVM_INT_MAX, 
_find_spark_home, Dict, List). The drop-in's module was a subset of its own, 
which lacked 17 of them:
   Dict JVM_INT_MAX List StringConcat _find_spark_home dispatch_col_method 
dispatch_df_method dispatch_table_arg_method dispatch_window_method 
escape_meta_characters is_remote_only require_test_compiled to_java_array 
try_partitioning_remote_functions try_remote_avro_functions 
try_remote_functions try_remote_protobuf_functions
   Repro: `from pyspark.sql.utils import try_remote_functions` raises 
ImportError with the drop-in and gives the decorator with pyspark-client; so 
does the `from pyspark.sql.utils import require_test_compiled` of 
pyspark.testing.sqlutils (sqlutils.py:165-167), which takes the ImportError for 
test classes that are not compiled.

2. The package pyspark.sql.connect. PySpark's __init__ 
(python/pyspark/sql/connect/__init__.py:20-22) runs `from 
pyspark.sql.connect.utils import check_dependencies; check_dependencies()`, and 
importing pyspark imports no module of the package. Repro, of a script file: 
`import pyspark.sql.connect as c; print(sorted(n for n in vars(c) if not 
n.startswith("__")))`.
   - pyspark-client: ['check_dependencies', 'utils'], and only 
pyspark.sql.connect.utils is loaded.
   - The drop-in: ['udf', 'udtf'], as pyspark.sql.functions imported udf, udtf 
and arrow_udtf of pyspark.sql.connect.udf and pyspark.sql.connect.udtf when 
pyspark was imported, so those modules were loaded by `import pyspark`.
   The check that PySpark's import runs (utils.py:24-46) raises 
PACKAGE_NOT_INSTALLED without pandas 2.2.0, PyArrow 18.0.0, grpcio 1.48.1, 
grpcio-status 1.48.1, googleapis-common-protos 1.56.4 or zstandard 0.25.0. Of a 
main module without a file, as of `python -c "import pyspark.sql.connect"`, it 
takes the import for a run of the doctests and imports 
pyspark.testing.connectutils, whose import of pyspark.testing.sqlutils runs 
require_test_compiled, whose _find_spark_home exits the process with 255 when 
no SPARK_HOME is found; without the dependencies of the Connect tests, the 
check exits with 0 ("Skipping doctests"). The drop-in needs none of these 
packages (its transport is native, and it runs without PyArrow and pandas), so 
its import should not run the check, and its names should be PySpark's.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to