Hyukjin Kwon created SPARK-62082:
------------------------------------

             Summary: Make pyspark.sql.functions PySpark's package of the 
modules builtin and partitioning, whose functions run those of 
pyspark.sql.connect.functions, as in pyspark-client
                 Key: SPARK-62082
                 URL: https://issues.apache.org/jira/browse/SPARK-62082
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 has pyspark.sql.functions as a module of 
its own (pyspark/sql/functions.py), whose functions are its Python wrappers of 
the Rust core. SPARK-62071 and SPARK-62074 made the classic modules of the 
objects of a session and pyspark.sql.utils PySpark's, of which 
try_remote_functions is. PySpark's pyspark.sql.functions is a package 
(python/pyspark/sql/functions/__init__.py, builtin.py and partitioning.py of 
v4.3.0), whose functions are decorated with try_remote_functions or 
try_partitioning_remote_functions of pyspark.sql.utils: on Spark Connect, a 
call runs the function of pyspark.sql.connect.functions (or of its 
partitioning) of the same name, which it looks up at each call. The drop-in's 
module makes the modules builtin and partitioning as ModuleType objects that it 
puts in sys.modules, and next to it the directory pyspark/sql/functions/ holds 
the type stubs of PySpark's package, which Python does not import.

Repro, without a server:

    import importlib, inspect, os, pkgutil
    import pyspark.sql
    import pyspark.sql.connect.functions as CF
    import pyspark.sql.functions as F

    F.__file__                      # .../pyspark/sql/functions.py; 
pyspark-client: .../pyspark/sql/functions/__init__.py
    F.__path__                      # AttributeError; pyspark-client: 
[.../pyspark/sql/functions]
    [m for m in pkgutil.iter_modules(pyspark.sql.__path__) if 
m.name.startswith("functions")]
                                    # functions (not a package) and 
functions_generated; pyspark-client: functions, a package
    importlib.util.find_spec("pyspark.sql.functions.builtin")   # ValueError: 
__spec__ is None; pyspark-client: the spec of builtin.py
    hasattr(F.col, "__wrapped__")   # False; pyspark-client: True 
(try_remote_functions)
    inspect.signature(F.col)        # (col); pyspark-client: (col: str) -> 
pyspark.sql.column.Column
    F.col.__doc__                   # "Returns a Column based on the given 
column name."; pyspark-client: PySpark's docstring

    CF.upper = CF.lower
    F.upper("a")                    # Column<'upper(a)'>; pyspark-client: 
Column<'lower(a)'>, the function of pyspark.sql.connect.functions

    os.environ["PYSPARK_NO_NAMESPACE_SHARE"] = "1"
    F.col("a")                      # Column<'a'>; pyspark-client: ImportError: 
cannot import name 'SparkContext' from 'pyspark'
    F.sum("a")                      # Column<'sum(a)'>; pyspark-client: 
ModuleNotFoundError: No module named 'pyspark.sql.classic'

So also: dir(F) has 654 names, where pyspark-client's has 614 (41 private names 
of the drop-in, and no __path__); pyspark.sql.functions.partitioning has an 
__all__ and 11 names, PySpark's has no __all__ and 23; F.partitioning.bucket 
has no docstring; udf, udtf and arrow_udtf have docstrings of their own 
("Create a user-defined function (UDF).", 793 characters, where PySpark's is 
6127); with PYSPARK_NO_NAMESPACE_SHARE, F.udf(f) raises the ImportError of 
SparkContext, where pyspark-client's gives PySpark's classic UDF; the functions 
of pyspark.sql.connect.functions have the drop-in's docstrings and no 
annotations, where PySpark Connect's copy PySpark's docstrings (col.__doc__ = 
pysparkfuncs.col.__doc__) and are annotated, and the functions of 
pyspark.sql.connect.functions that the drop-in generates call a lit of a module 
of their own, so a patch of pyspark.sql.connect.functions.builtin.lit does not 
reach them, where PySpark Connect's call that of their module.

Type checkers: mypy run on the source tree by path (mypy 
pyspark/sql/functions.py pyspark/sql/functions) reports `Duplicate module named 
"pyspark.sql.functions"` and checks nothing, and pyright warns that `from 
pyspark.sql.functions import partitioning` and an import of 
pyspark.sql.functions.builtin "could not be resolved from source" 
(reportMissingModuleSource), where pyspark-client's are PySpark's source.

docs/compatibility.md documents the module as kept on purpose ("PySpark's 
`pyspark.sql.functions` is a package ..."), as a package would change its 
run-time facts.

Expected, as in pyspark-client: pyspark.sql.functions is PySpark's package, its 
__init__.py, builtin.py and partitioning.py of Apache Spark 4.3.0 as they are 
(python/pyspark/sql/functions/ of the v4.3.0 tag), whose functions run those of 
pyspark.sql.connect.functions (python/pyspark/sql/utils.py, 
try_remote_functions and try_partitioning_remote_functions), which are the 
drop-in's, of the Rust core, of the module 
pyspark.sql.connect.functions.builtin, with the docstrings and the annotations 
of PySpark's functions, as PySpark Connect's are 
(python/pyspark/sql/connect/functions/builtin.py of v4.3.0), and 
pyspark.sql.connect.functions.partitioning is PySpark Connect's 
(python/pyspark/sql/connect/functions/partitioning.py of v4.3.0).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to