Hyukjin Kwon created SPARK-62074:
------------------------------------
Summary: pyspark.sql.utils lacks 17 of PySpark's names, and the
package pyspark.sql.connect lacks check_dependencies and utils and has udf and
udtf, in the Python client
Key: SPARK-62074
URL: https://issues.apache.org/jira/browse/SPARK-62074
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Differences of the drop-in from pyspark-client 4.3.0:
1. pyspark.sql.utils. PySpark's module (python/pyspark/sql/utils.py, 513 lines)
has 56 names, of the helpers and the decorators that its modules import: the
decorators that run the functions of PySpark Connect's modules
(try_remote_functions, try_partitioning_remote_functions,
try_remote_avro_functions, try_remote_protobuf_functions, utils.py:273-331) and
the methods of the classes of Spark Connect (dispatch_df_method,
dispatch_col_method, dispatch_window_method, dispatch_table_arg_method,
utils.py:363-455), StringConcat, escape_meta_characters, require_test_compiled,
to_java_array, and the names that it imports (is_remote_only, JVM_INT_MAX,
_find_spark_home, Dict, List). The drop-in's module was a subset of its own,
which lacked 17 of them:
Dict JVM_INT_MAX List StringConcat _find_spark_home dispatch_col_method
dispatch_df_method dispatch_table_arg_method dispatch_window_method
escape_meta_characters is_remote_only require_test_compiled to_java_array
try_partitioning_remote_functions try_remote_avro_functions
try_remote_functions try_remote_protobuf_functions
Repro: `from pyspark.sql.utils import try_remote_functions` raises
ImportError with the drop-in and gives the decorator with pyspark-client; so
does the `from pyspark.sql.utils import require_test_compiled` of
pyspark.testing.sqlutils (sqlutils.py:165-167), which takes the ImportError for
test classes that are not compiled.
2. The package pyspark.sql.connect. PySpark's __init__
(python/pyspark/sql/connect/__init__.py:20-22) runs `from
pyspark.sql.connect.utils import check_dependencies; check_dependencies()`, and
importing pyspark imports no module of the package. Repro, of a script file:
`import pyspark.sql.connect as c; print(sorted(n for n in vars(c) if not
n.startswith("__")))`.
- pyspark-client: ['check_dependencies', 'utils'], and only
pyspark.sql.connect.utils is loaded.
- The drop-in: ['udf', 'udtf'], as pyspark.sql.functions imported udf, udtf
and arrow_udtf of pyspark.sql.connect.udf and pyspark.sql.connect.udtf when
pyspark was imported, so those modules were loaded by `import pyspark`.
The check that PySpark's import runs (utils.py:24-46) raises
PACKAGE_NOT_INSTALLED without pandas 2.2.0, PyArrow 18.0.0, grpcio 1.48.1,
grpcio-status 1.48.1, googleapis-common-protos 1.56.4 or zstandard 0.25.0. Of a
main module without a file, as of `python -c "import pyspark.sql.connect"`, it
takes the import for a run of the doctests and imports
pyspark.testing.connectutils, whose import of pyspark.testing.sqlutils runs
require_test_compiled, whose _find_spark_home exits the process with 255 when
no SPARK_HOME is found; without the dependencies of the Connect tests, the
check exits with 0 ("Skipping doctests"). The drop-in needs none of these
packages (its transport is native, and it runs without PyArrow and pandas), so
its import should not run the check, and its names should be PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]