Hyukjin Kwon created SPARK-62082:
------------------------------------
Summary: Make pyspark.sql.functions PySpark's package of the
modules builtin and partitioning, whose functions run those of
pyspark.sql.connect.functions, as in pyspark-client
Key: SPARK-62082
URL: https://issues.apache.org/jira/browse/SPARK-62082
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in for pyspark-client 4.3.0 has pyspark.sql.functions as a module of
its own (pyspark/sql/functions.py), whose functions are its Python wrappers of
the Rust core. SPARK-62071 and SPARK-62074 made the classic modules of the
objects of a session and pyspark.sql.utils PySpark's, of which
try_remote_functions is. PySpark's pyspark.sql.functions is a package
(python/pyspark/sql/functions/__init__.py, builtin.py and partitioning.py of
v4.3.0), whose functions are decorated with try_remote_functions or
try_partitioning_remote_functions of pyspark.sql.utils: on Spark Connect, a
call runs the function of pyspark.sql.connect.functions (or of its
partitioning) of the same name, which it looks up at each call. The drop-in's
module makes the modules builtin and partitioning as ModuleType objects that it
puts in sys.modules, and next to it the directory pyspark/sql/functions/ holds
the type stubs of PySpark's package, which Python does not import.
Repro, without a server:
import importlib, inspect, os, pkgutil
import pyspark.sql
import pyspark.sql.connect.functions as CF
import pyspark.sql.functions as F
F.__file__ # .../pyspark/sql/functions.py;
pyspark-client: .../pyspark/sql/functions/__init__.py
F.__path__ # AttributeError; pyspark-client:
[.../pyspark/sql/functions]
[m for m in pkgutil.iter_modules(pyspark.sql.__path__) if
m.name.startswith("functions")]
# functions (not a package) and
functions_generated; pyspark-client: functions, a package
importlib.util.find_spec("pyspark.sql.functions.builtin") # ValueError:
__spec__ is None; pyspark-client: the spec of builtin.py
hasattr(F.col, "__wrapped__") # False; pyspark-client: True
(try_remote_functions)
inspect.signature(F.col) # (col); pyspark-client: (col: str) ->
pyspark.sql.column.Column
F.col.__doc__ # "Returns a Column based on the given
column name."; pyspark-client: PySpark's docstring
CF.upper = CF.lower
F.upper("a") # Column<'upper(a)'>; pyspark-client:
Column<'lower(a)'>, the function of pyspark.sql.connect.functions
os.environ["PYSPARK_NO_NAMESPACE_SHARE"] = "1"
F.col("a") # Column<'a'>; pyspark-client: ImportError:
cannot import name 'SparkContext' from 'pyspark'
F.sum("a") # Column<'sum(a)'>; pyspark-client:
ModuleNotFoundError: No module named 'pyspark.sql.classic'
So also: dir(F) has 654 names, where pyspark-client's has 614 (41 private names
of the drop-in, and no __path__); pyspark.sql.functions.partitioning has an
__all__ and 11 names, PySpark's has no __all__ and 23; F.partitioning.bucket
has no docstring; udf, udtf and arrow_udtf have docstrings of their own
("Create a user-defined function (UDF).", 793 characters, where PySpark's is
6127); with PYSPARK_NO_NAMESPACE_SHARE, F.udf(f) raises the ImportError of
SparkContext, where pyspark-client's gives PySpark's classic UDF; the functions
of pyspark.sql.connect.functions have the drop-in's docstrings and no
annotations, where PySpark Connect's copy PySpark's docstrings (col.__doc__ =
pysparkfuncs.col.__doc__) and are annotated, and the functions of
pyspark.sql.connect.functions that the drop-in generates call a lit of a module
of their own, so a patch of pyspark.sql.connect.functions.builtin.lit does not
reach them, where PySpark Connect's call that of their module.
Type checkers: mypy run on the source tree by path (mypy
pyspark/sql/functions.py pyspark/sql/functions) reports `Duplicate module named
"pyspark.sql.functions"` and checks nothing, and pyright warns that `from
pyspark.sql.functions import partitioning` and an import of
pyspark.sql.functions.builtin "could not be resolved from source"
(reportMissingModuleSource), where pyspark-client's are PySpark's source.
docs/compatibility.md documents the module as kept on purpose ("PySpark's
`pyspark.sql.functions` is a package ..."), as a package would change its
run-time facts.
Expected, as in pyspark-client: pyspark.sql.functions is PySpark's package, its
__init__.py, builtin.py and partitioning.py of Apache Spark 4.3.0 as they are
(python/pyspark/sql/functions/ of the v4.3.0 tag), whose functions run those of
pyspark.sql.connect.functions (python/pyspark/sql/utils.py,
try_remote_functions and try_partitioning_remote_functions), which are the
drop-in's, of the Rust core, of the module
pyspark.sql.connect.functions.builtin, with the docstrings and the annotations
of PySpark's functions, as PySpark Connect's are
(python/pyspark/sql/connect/functions/builtin.py of v4.3.0), and
pyspark.sql.connect.functions.partitioning is PySpark Connect's
(python/pyspark/sql/connect/functions/partitioning.py of v4.3.0).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]