[ 
https://issues.apache.org/jira/browse/SPARK-61949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61949:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the type stubs declare what the drop-in's modules have at run time, 
> check that their imports and names resolve, and add the stubs of 
> pyspark.sql.connect.datasource and pyspark.ml.stat
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61949
>                 URL: https://issues.apache.org/jira/browse/SPARK-61949
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> 1. Declare the private functions and constants of PySpark's modules in the 
> type stubs when the drop-in's modules have them
> Problem
> The type stubs of scripts/gen_type_stubs.py keep a private module-level 
> function, class or constant
> of PySpark's module only when the stub refers to it. 54 private names that 
> both pyspark-client 4.3.0
> and the drop-in have at run time are missing for type checkers, such as
> pyspark.sql.types._parse_datatype_string, _parse_datatype_json_string, 
> _infer_schema and _merge_type,
> which applications and libraries use. Some stubs import them too (udf.pyi 
> imports
> _parse_datatype_string from pyspark.sql.types), which resolves to nothing.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
>   $ cat app.py
>   from pyspark.sql.types import _parse_datatype_string
>   reveal_type(_parse_datatype_string("a int"))
>   $ mypy app.py
>   app.py:1: error: Module "pyspark.sql.types" has no attribute 
> "_parse_datatype_string"  [attr-defined]
>   app.py:2: note: Revealed type is "Any"
> pyright 1.1.414: error: "_parse_datatype_string" is unknown import symbol
> Expected (as with pyspark-client 4.3.0)
>   app.py:2: note: Revealed type is "pyspark.sql.types.DataType"
> with no error, under mypy and pyright.
> 2. Keep PySpark's type ignore comments and the import of the annotation of 
> __all__ in the type stubs, and check that their imports resolve in the drop-in
> Problem
> Three robustness gaps of scripts/gen_type_stubs.py:
> 1. The reference pass returns on __all__ before it reads its annotation, so 
> pyspark/util.pyi writes
>    `__all__: List[str] = []` without importing List.
> 2. ast.unparse drops comments, so the stubs lose PySpark's `# type: 
> ignore[...]`, such as [override]
>    on Column.__eq__ and [assignment] on DataFrame.groupby.
> 3. A stub can import a name that the drop-in does not have (pyspark.sql.types 
> imports StringConcat
>    from pyspark.sql.utils), which makes PySpark's annotations of it Any 
> without a word.
> Repro (master 0dbe860b, the drop-in on the path, mypy 2.4.1)
>   $ echo 'import pyspark.util, pyspark.sql.column, pyspark.sql.types' > 
> imports.py
>   $ mypy --no-silence-site-packages imports.py
>   pyspark/util.pyi:30: error: Name "List" is not defined  [name-defined]
>   pyspark/sql/column.pyi:46: error: Return type "Column" of "__eq__" 
> incompatible with return type "bool" in supertype "builtins.object"  
> [override]
>   pyspark/sql/types.pyi:24: error: Module "pyspark.sql.utils" has no 
> attribute "StringConcat"  [attr-defined]
> PySpark's column.py ignores the second with `# type: ignore[override]`.
> Expected
> The stubs import what their annotations read and keep PySpark's type ignore 
> comments, so that mypy
> reports in them only what it reports of PySpark's typing; and the generator 
> fails on an import of a
> name that the drop-in does not have, unless a table lists it with the reason.
> 3. Declare in the type stubs only the names that the drop-in has at run time, 
> and check it
> Problem
> A stub replaces its module for a type checker, so a name that only the stub 
> declares type-checks
> and then fails at run time. The stubs of master (0dbe860b) declare 244 public 
> names that the drop-in
> does not have: names that PySpark's modules import, which a stub re-exports 
> (DataFrame of pyspark.sql.catalog,
> DataFrameWriter of pyspark.sql.dataframe, ...), PySpark's plumbing (the py4j 
> converters of pyspark.sql.types,
> StreamingCheckpointManager), and PySpark's aliases PathOrPaths and 
> TupleOrListOfString of
> pyspark.sql.readwriter. Nothing checks that what a stub declares exists.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
>   $ cat app.py
>   from pyspark.sql.types import DateConverter
>   from pyspark.sql.catalog import DataFrame
>   from pyspark.sql.readwriter import PathOrPaths
>   $ mypy app.py          # no error
>   $ python app.py
>   ImportError: cannot import name 'DateConverter' from 'pyspark.sql.types'
> With pyspark-client 4.3.0 all three import, and PathOrPaths is PySpark's 
> alias that applications
> annotate their own readers with.
> Expected
> Every name that a stub declares exists at the drop-in's run time, but a 
> short, reasoned list of
> names that PySpark's annotations need, and a test checks it. PySpark's API 
> that the drop-in lacks
> (PathOrPaths, TupleOrListOfString) is given to the drop-in; PySpark's 
> plumbing and incidental imports
> are left out of the stubs.
> 4. Add the type stubs of pyspark.sql.connect.datasource and pyspark.ml.stat, 
> whose classes are native
> Problem
> pyspark.sql.connect.datasource (DataSourceRegistration, spark.dataSource of a 
> Connect session) and
> pyspark.ml.stat (ChiSquareTest, Correlation, KolmogorovSmirnovTest, 
> Summarizer, SummaryBuilder)
> re-export native classes and have no stub, so type checkers see Any for them.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
>   $ cat app.py
>   from pyspark.ml.stat import Correlation
>   from pyspark.sql.connect.session import SparkSession
>   spark = SparkSession.builder.getOrCreate()
>   reveal_type(spark.dataSource)
>   Correlation.corr(spark.range(1), 1)
>   spark.dataSource.register(1)
>   $ mypy app.py
>   app.py:4: note: Revealed type is "Any"
>   (no error)
> Expected (as with pyspark-client 4.3.0)
>   app.py:4: note: Revealed type is 
> "pyspark.sql.connect.datasource.DataSourceRegistration"
>   app.py:5: error: Argument 2 to "corr" of "Correlation" has incompatible 
> type "int"; expected "str"  [arg-type]
>   app.py:6: error: Argument 1 to "register" of "DataSourceRegistration" has 
> incompatible type "int"; expected "type[DataSource]"  [arg-type]
> and the same under pyright 1.1.414.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to