[
https://issues.apache.org/jira/browse/SPARK-62074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62074:
------------------------------------
Assignee: Hyukjin Kwon
> pyspark.sql.utils lacks 17 of PySpark's names, and the package
> pyspark.sql.connect lacks check_dependencies and utils and has udf and udtf,
> in the Python client
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62074
> URL: https://issues.apache.org/jira/browse/SPARK-62074
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Differences of the drop-in from pyspark-client 4.3.0:
> 1. pyspark.sql.utils. PySpark's module (python/pyspark/sql/utils.py, 513
> lines) has 56 names, of the helpers and the decorators that its modules
> import: the decorators that run the functions of PySpark Connect's modules
> (try_remote_functions, try_partitioning_remote_functions,
> try_remote_avro_functions, try_remote_protobuf_functions, utils.py:273-331)
> and the methods of the classes of Spark Connect (dispatch_df_method,
> dispatch_col_method, dispatch_window_method, dispatch_table_arg_method,
> utils.py:363-455), StringConcat, escape_meta_characters,
> require_test_compiled, to_java_array, and the names that it imports
> (is_remote_only, JVM_INT_MAX, _find_spark_home, Dict, List). The drop-in's
> module was a subset of its own, which lacked 17 of them:
> Dict JVM_INT_MAX List StringConcat _find_spark_home dispatch_col_method
> dispatch_df_method dispatch_table_arg_method dispatch_window_method
> escape_meta_characters is_remote_only require_test_compiled to_java_array
> try_partitioning_remote_functions try_remote_avro_functions
> try_remote_functions try_remote_protobuf_functions
> Repro: `from pyspark.sql.utils import try_remote_functions` raises
> ImportError with the drop-in and gives the decorator with pyspark-client; so
> does the `from pyspark.sql.utils import require_test_compiled` of
> pyspark.testing.sqlutils (sqlutils.py:165-167), which takes the ImportError
> for test classes that are not compiled.
> 2. The package pyspark.sql.connect. PySpark's __init__
> (python/pyspark/sql/connect/__init__.py:20-22) runs `from
> pyspark.sql.connect.utils import check_dependencies; check_dependencies()`,
> and importing pyspark imports no module of the package. Repro, of a script
> file: `import pyspark.sql.connect as c; print(sorted(n for n in vars(c) if
> not n.startswith("__")))`.
> - pyspark-client: ['check_dependencies', 'utils'], and only
> pyspark.sql.connect.utils is loaded.
> - The drop-in: ['udf', 'udtf'], as pyspark.sql.functions imported udf,
> udtf and arrow_udtf of pyspark.sql.connect.udf and pyspark.sql.connect.udtf
> when pyspark was imported, so those modules were loaded by `import pyspark`.
> The check that PySpark's import runs (utils.py:24-46) raises
> PACKAGE_NOT_INSTALLED without pandas 2.2.0, PyArrow 18.0.0, grpcio 1.48.1,
> grpcio-status 1.48.1, googleapis-common-protos 1.56.4 or zstandard 0.25.0. Of
> a main module without a file, as of `python -c "import pyspark.sql.connect"`,
> it takes the import for a run of the doctests and imports
> pyspark.testing.connectutils, whose import of pyspark.testing.sqlutils runs
> require_test_compiled, whose _find_spark_home exits the process with 255 when
> no SPARK_HOME is found; without the dependencies of the Connect tests, the
> check exits with 0 ("Skipping doctests"). The drop-in needs none of these
> packages (its transport is native, and it runs without PyArrow and pandas),
> so its import should not run the check, and its names should be PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]