Hyukjin Kwon created SPARK-61949:
------------------------------------
Summary: Make the type stubs declare what the drop-in's modules
have at run time, check that their imports and names resolve, and add the stubs
of pyspark.sql.connect.datasource and pyspark.ml.stat
Key: SPARK-61949
URL: https://issues.apache.org/jira/browse/SPARK-61949
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
1. Declare the private functions and constants of PySpark's modules in the type
stubs when the drop-in's modules have them
Problem
The type stubs of scripts/gen_type_stubs.py keep a private module-level
function, class or constant
of PySpark's module only when the stub refers to it. 54 private names that both
pyspark-client 4.3.0
and the drop-in have at run time are missing for type checkers, such as
pyspark.sql.types._parse_datatype_string, _parse_datatype_json_string,
_infer_schema and _merge_type,
which applications and libraries use. Some stubs import them too (udf.pyi
imports
_parse_datatype_string from pyspark.sql.types), which resolves to nothing.
Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
$ cat app.py
from pyspark.sql.types import _parse_datatype_string
reveal_type(_parse_datatype_string("a int"))
$ mypy app.py
app.py:1: error: Module "pyspark.sql.types" has no attribute
"_parse_datatype_string" [attr-defined]
app.py:2: note: Revealed type is "Any"
pyright 1.1.414: error: "_parse_datatype_string" is unknown import symbol
Expected (as with pyspark-client 4.3.0)
app.py:2: note: Revealed type is "pyspark.sql.types.DataType"
with no error, under mypy and pyright.
2. Keep PySpark's type ignore comments and the import of the annotation of
__all__ in the type stubs, and check that their imports resolve in the drop-in
Problem
Three robustness gaps of scripts/gen_type_stubs.py:
1. The reference pass returns on __all__ before it reads its annotation, so
pyspark/util.pyi writes
`__all__: List[str] = []` without importing List.
2. ast.unparse drops comments, so the stubs lose PySpark's `# type:
ignore[...]`, such as [override]
on Column.__eq__ and [assignment] on DataFrame.groupby.
3. A stub can import a name that the drop-in does not have (pyspark.sql.types
imports StringConcat
from pyspark.sql.utils), which makes PySpark's annotations of it Any without
a word.
Repro (master 0dbe860b, the drop-in on the path, mypy 2.4.1)
$ echo 'import pyspark.util, pyspark.sql.column, pyspark.sql.types' >
imports.py
$ mypy --no-silence-site-packages imports.py
pyspark/util.pyi:30: error: Name "List" is not defined [name-defined]
pyspark/sql/column.pyi:46: error: Return type "Column" of "__eq__"
incompatible with return type "bool" in supertype "builtins.object" [override]
pyspark/sql/types.pyi:24: error: Module "pyspark.sql.utils" has no attribute
"StringConcat" [attr-defined]
PySpark's column.py ignores the second with `# type: ignore[override]`.
Expected
The stubs import what their annotations read and keep PySpark's type ignore
comments, so that mypy
reports in them only what it reports of PySpark's typing; and the generator
fails on an import of a
name that the drop-in does not have, unless a table lists it with the reason.
3. Declare in the type stubs only the names that the drop-in has at run time,
and check it
Problem
A stub replaces its module for a type checker, so a name that only the stub
declares type-checks
and then fails at run time. The stubs of master (0dbe860b) declare 244 public
names that the drop-in
does not have: names that PySpark's modules import, which a stub re-exports
(DataFrame of pyspark.sql.catalog,
DataFrameWriter of pyspark.sql.dataframe, ...), PySpark's plumbing (the py4j
converters of pyspark.sql.types,
StreamingCheckpointManager), and PySpark's aliases PathOrPaths and
TupleOrListOfString of
pyspark.sql.readwriter. Nothing checks that what a stub declares exists.
Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
$ cat app.py
from pyspark.sql.types import DateConverter
from pyspark.sql.catalog import DataFrame
from pyspark.sql.readwriter import PathOrPaths
$ mypy app.py # no error
$ python app.py
ImportError: cannot import name 'DateConverter' from 'pyspark.sql.types'
With pyspark-client 4.3.0 all three import, and PathOrPaths is PySpark's alias
that applications
annotate their own readers with.
Expected
Every name that a stub declares exists at the drop-in's run time, but a short,
reasoned list of
names that PySpark's annotations need, and a test checks it. PySpark's API that
the drop-in lacks
(PathOrPaths, TupleOrListOfString) is given to the drop-in; PySpark's plumbing
and incidental imports
are left out of the stubs.
4. Add the type stubs of pyspark.sql.connect.datasource and pyspark.ml.stat,
whose classes are native
Problem
pyspark.sql.connect.datasource (DataSourceRegistration, spark.dataSource of a
Connect session) and
pyspark.ml.stat (ChiSquareTest, Correlation, KolmogorovSmirnovTest, Summarizer,
SummaryBuilder)
re-export native classes and have no stub, so type checkers see Any for them.
Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
$ cat app.py
from pyspark.ml.stat import Correlation
from pyspark.sql.connect.session import SparkSession
spark = SparkSession.builder.getOrCreate()
reveal_type(spark.dataSource)
Correlation.corr(spark.range(1), 1)
spark.dataSource.register(1)
$ mypy app.py
app.py:4: note: Revealed type is "Any"
(no error)
Expected (as with pyspark-client 4.3.0)
app.py:4: note: Revealed type is
"pyspark.sql.connect.datasource.DataSourceRegistration"
app.py:5: error: Argument 2 to "corr" of "Correlation" has incompatible type
"int"; expected "str" [arg-type]
app.py:6: error: Argument 1 to "register" of "DataSourceRegistration" has
incompatible type "int"; expected "type[DataSource]" [arg-type]
and the same under pyright 1.1.414.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]