[ 
https://issues.apache.org/jira/browse/SPARK-62074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62074:
-----------------------------------
    Labels: pull-request-available  (was: )

> pyspark.sql.utils lacks 17 of PySpark's names, and the package 
> pyspark.sql.connect lacks check_dependencies and utils and has udf and udtf, 
> in the Python client
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62074
>                 URL: https://issues.apache.org/jira/browse/SPARK-62074
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Differences of the drop-in from pyspark-client 4.3.0:
> 1. pyspark.sql.utils. PySpark's module (python/pyspark/sql/utils.py, 513 
> lines) has 56 names, of the helpers and the decorators that its modules 
> import: the decorators that run the functions of PySpark Connect's modules 
> (try_remote_functions, try_partitioning_remote_functions, 
> try_remote_avro_functions, try_remote_protobuf_functions, utils.py:273-331) 
> and the methods of the classes of Spark Connect (dispatch_df_method, 
> dispatch_col_method, dispatch_window_method, dispatch_table_arg_method, 
> utils.py:363-455), StringConcat, escape_meta_characters, 
> require_test_compiled, to_java_array, and the names that it imports 
> (is_remote_only, JVM_INT_MAX, _find_spark_home, Dict, List). The drop-in's 
> module was a subset of its own, which lacked 17 of them:
>    Dict JVM_INT_MAX List StringConcat _find_spark_home dispatch_col_method 
> dispatch_df_method dispatch_table_arg_method dispatch_window_method 
> escape_meta_characters is_remote_only require_test_compiled to_java_array 
> try_partitioning_remote_functions try_remote_avro_functions 
> try_remote_functions try_remote_protobuf_functions
>    Repro: `from pyspark.sql.utils import try_remote_functions` raises 
> ImportError with the drop-in and gives the decorator with pyspark-client; so 
> does the `from pyspark.sql.utils import require_test_compiled` of 
> pyspark.testing.sqlutils (sqlutils.py:165-167), which takes the ImportError 
> for test classes that are not compiled.
> 2. The package pyspark.sql.connect. PySpark's __init__ 
> (python/pyspark/sql/connect/__init__.py:20-22) runs `from 
> pyspark.sql.connect.utils import check_dependencies; check_dependencies()`, 
> and importing pyspark imports no module of the package. Repro, of a script 
> file: `import pyspark.sql.connect as c; print(sorted(n for n in vars(c) if 
> not n.startswith("__")))`.
>    - pyspark-client: ['check_dependencies', 'utils'], and only 
> pyspark.sql.connect.utils is loaded.
>    - The drop-in: ['udf', 'udtf'], as pyspark.sql.functions imported udf, 
> udtf and arrow_udtf of pyspark.sql.connect.udf and pyspark.sql.connect.udtf 
> when pyspark was imported, so those modules were loaded by `import pyspark`.
>    The check that PySpark's import runs (utils.py:24-46) raises 
> PACKAGE_NOT_INSTALLED without pandas 2.2.0, PyArrow 18.0.0, grpcio 1.48.1, 
> grpcio-status 1.48.1, googleapis-common-protos 1.56.4 or zstandard 0.25.0. Of 
> a main module without a file, as of `python -c "import pyspark.sql.connect"`, 
> it takes the import for a run of the doctests and imports 
> pyspark.testing.connectutils, whose import of pyspark.testing.sqlutils runs 
> require_test_compiled, whose _find_spark_home exits the process with 255 when 
> no SPARK_HOME is found; without the dependencies of the Connect tests, the 
> check exits with 0 ("Skipping doctests"). The drop-in needs none of these 
> packages (its transport is native, and it runs without PyArrow and pandas), 
> so its import should not run the check, and its names should be PySpark's.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to