[
https://issues.apache.org/jira/browse/SPARK-61949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61949:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Make the type stubs declare what the drop-in's modules have at run time,
> check that their imports and names resolve, and add the stubs of
> pyspark.sql.connect.datasource and pyspark.ml.stat
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61949
> URL: https://issues.apache.org/jira/browse/SPARK-61949
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> 1. Declare the private functions and constants of PySpark's modules in the
> type stubs when the drop-in's modules have them
> Problem
> The type stubs of scripts/gen_type_stubs.py keep a private module-level
> function, class or constant
> of PySpark's module only when the stub refers to it. 54 private names that
> both pyspark-client 4.3.0
> and the drop-in have at run time are missing for type checkers, such as
> pyspark.sql.types._parse_datatype_string, _parse_datatype_json_string,
> _infer_schema and _merge_type,
> which applications and libraries use. Some stubs import them too (udf.pyi
> imports
> _parse_datatype_string from pyspark.sql.types), which resolves to nothing.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
> $ cat app.py
> from pyspark.sql.types import _parse_datatype_string
> reveal_type(_parse_datatype_string("a int"))
> $ mypy app.py
> app.py:1: error: Module "pyspark.sql.types" has no attribute
> "_parse_datatype_string" [attr-defined]
> app.py:2: note: Revealed type is "Any"
> pyright 1.1.414: error: "_parse_datatype_string" is unknown import symbol
> Expected (as with pyspark-client 4.3.0)
> app.py:2: note: Revealed type is "pyspark.sql.types.DataType"
> with no error, under mypy and pyright.
> 2. Keep PySpark's type ignore comments and the import of the annotation of
> __all__ in the type stubs, and check that their imports resolve in the drop-in
> Problem
> Three robustness gaps of scripts/gen_type_stubs.py:
> 1. The reference pass returns on __all__ before it reads its annotation, so
> pyspark/util.pyi writes
> `__all__: List[str] = []` without importing List.
> 2. ast.unparse drops comments, so the stubs lose PySpark's `# type:
> ignore[...]`, such as [override]
> on Column.__eq__ and [assignment] on DataFrame.groupby.
> 3. A stub can import a name that the drop-in does not have (pyspark.sql.types
> imports StringConcat
> from pyspark.sql.utils), which makes PySpark's annotations of it Any
> without a word.
> Repro (master 0dbe860b, the drop-in on the path, mypy 2.4.1)
> $ echo 'import pyspark.util, pyspark.sql.column, pyspark.sql.types' >
> imports.py
> $ mypy --no-silence-site-packages imports.py
> pyspark/util.pyi:30: error: Name "List" is not defined [name-defined]
> pyspark/sql/column.pyi:46: error: Return type "Column" of "__eq__"
> incompatible with return type "bool" in supertype "builtins.object"
> [override]
> pyspark/sql/types.pyi:24: error: Module "pyspark.sql.utils" has no
> attribute "StringConcat" [attr-defined]
> PySpark's column.py ignores the second with `# type: ignore[override]`.
> Expected
> The stubs import what their annotations read and keep PySpark's type ignore
> comments, so that mypy
> reports in them only what it reports of PySpark's typing; and the generator
> fails on an import of a
> name that the drop-in does not have, unless a table lists it with the reason.
> 3. Declare in the type stubs only the names that the drop-in has at run time,
> and check it
> Problem
> A stub replaces its module for a type checker, so a name that only the stub
> declares type-checks
> and then fails at run time. The stubs of master (0dbe860b) declare 244 public
> names that the drop-in
> does not have: names that PySpark's modules import, which a stub re-exports
> (DataFrame of pyspark.sql.catalog,
> DataFrameWriter of pyspark.sql.dataframe, ...), PySpark's plumbing (the py4j
> converters of pyspark.sql.types,
> StreamingCheckpointManager), and PySpark's aliases PathOrPaths and
> TupleOrListOfString of
> pyspark.sql.readwriter. Nothing checks that what a stub declares exists.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
> $ cat app.py
> from pyspark.sql.types import DateConverter
> from pyspark.sql.catalog import DataFrame
> from pyspark.sql.readwriter import PathOrPaths
> $ mypy app.py # no error
> $ python app.py
> ImportError: cannot import name 'DateConverter' from 'pyspark.sql.types'
> With pyspark-client 4.3.0 all three import, and PathOrPaths is PySpark's
> alias that applications
> annotate their own readers with.
> Expected
> Every name that a stub declares exists at the drop-in's run time, but a
> short, reasoned list of
> names that PySpark's annotations need, and a test checks it. PySpark's API
> that the drop-in lacks
> (PathOrPaths, TupleOrListOfString) is given to the drop-in; PySpark's
> plumbing and incidental imports
> are left out of the stubs.
> 4. Add the type stubs of pyspark.sql.connect.datasource and pyspark.ml.stat,
> whose classes are native
> Problem
> pyspark.sql.connect.datasource (DataSourceRegistration, spark.dataSource of a
> Connect session) and
> pyspark.ml.stat (ChiSquareTest, Correlation, KolmogorovSmirnovTest,
> Summarizer, SummaryBuilder)
> re-export native classes and have no stub, so type checkers see Any for them.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
> $ cat app.py
> from pyspark.ml.stat import Correlation
> from pyspark.sql.connect.session import SparkSession
> spark = SparkSession.builder.getOrCreate()
> reveal_type(spark.dataSource)
> Correlation.corr(spark.range(1), 1)
> spark.dataSource.register(1)
> $ mypy app.py
> app.py:4: note: Revealed type is "Any"
> (no error)
> Expected (as with pyspark-client 4.3.0)
> app.py:4: note: Revealed type is
> "pyspark.sql.connect.datasource.DataSourceRegistration"
> app.py:5: error: Argument 2 to "corr" of "Correlation" has incompatible
> type "int"; expected "str" [arg-type]
> app.py:6: error: Argument 1 to "register" of "DataSourceRegistration" has
> incompatible type "int"; expected "type[DataSource]" [arg-type]
> and the same under pyright 1.1.414.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]