Hyukjin Kwon created SPARK-61949:
------------------------------------

             Summary: Make the type stubs declare what the drop-in's modules 
have at run time, check that their imports and names resolve, and add the stubs 
of pyspark.sql.connect.datasource and pyspark.ml.stat
                 Key: SPARK-61949
                 URL: https://issues.apache.org/jira/browse/SPARK-61949
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


1. Declare the private functions and constants of PySpark's modules in the type 
stubs when the drop-in's modules have them

Problem
The type stubs of scripts/gen_type_stubs.py keep a private module-level 
function, class or constant
of PySpark's module only when the stub refers to it. 54 private names that both 
pyspark-client 4.3.0
and the drop-in have at run time are missing for type checkers, such as
pyspark.sql.types._parse_datatype_string, _parse_datatype_json_string, 
_infer_schema and _merge_type,
which applications and libraries use. Some stubs import them too (udf.pyi 
imports
_parse_datatype_string from pyspark.sql.types), which resolves to nothing.

Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
  $ cat app.py
  from pyspark.sql.types import _parse_datatype_string
  reveal_type(_parse_datatype_string("a int"))
  $ mypy app.py
  app.py:1: error: Module "pyspark.sql.types" has no attribute 
"_parse_datatype_string"  [attr-defined]
  app.py:2: note: Revealed type is "Any"
pyright 1.1.414: error: "_parse_datatype_string" is unknown import symbol

Expected (as with pyspark-client 4.3.0)
  app.py:2: note: Revealed type is "pyspark.sql.types.DataType"
with no error, under mypy and pyright.

2. Keep PySpark's type ignore comments and the import of the annotation of 
__all__ in the type stubs, and check that their imports resolve in the drop-in

Problem
Three robustness gaps of scripts/gen_type_stubs.py:
1. The reference pass returns on __all__ before it reads its annotation, so 
pyspark/util.pyi writes
   `__all__: List[str] = []` without importing List.
2. ast.unparse drops comments, so the stubs lose PySpark's `# type: 
ignore[...]`, such as [override]
   on Column.__eq__ and [assignment] on DataFrame.groupby.
3. A stub can import a name that the drop-in does not have (pyspark.sql.types 
imports StringConcat
   from pyspark.sql.utils), which makes PySpark's annotations of it Any without 
a word.

Repro (master 0dbe860b, the drop-in on the path, mypy 2.4.1)
  $ echo 'import pyspark.util, pyspark.sql.column, pyspark.sql.types' > 
imports.py
  $ mypy --no-silence-site-packages imports.py
  pyspark/util.pyi:30: error: Name "List" is not defined  [name-defined]
  pyspark/sql/column.pyi:46: error: Return type "Column" of "__eq__" 
incompatible with return type "bool" in supertype "builtins.object"  [override]
  pyspark/sql/types.pyi:24: error: Module "pyspark.sql.utils" has no attribute 
"StringConcat"  [attr-defined]
PySpark's column.py ignores the second with `# type: ignore[override]`.

Expected
The stubs import what their annotations read and keep PySpark's type ignore 
comments, so that mypy
reports in them only what it reports of PySpark's typing; and the generator 
fails on an import of a
name that the drop-in does not have, unless a table lists it with the reason.

3. Declare in the type stubs only the names that the drop-in has at run time, 
and check it

Problem
A stub replaces its module for a type checker, so a name that only the stub 
declares type-checks
and then fails at run time. The stubs of master (0dbe860b) declare 244 public 
names that the drop-in
does not have: names that PySpark's modules import, which a stub re-exports 
(DataFrame of pyspark.sql.catalog,
DataFrameWriter of pyspark.sql.dataframe, ...), PySpark's plumbing (the py4j 
converters of pyspark.sql.types,
StreamingCheckpointManager), and PySpark's aliases PathOrPaths and 
TupleOrListOfString of
pyspark.sql.readwriter. Nothing checks that what a stub declares exists.

Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
  $ cat app.py
  from pyspark.sql.types import DateConverter
  from pyspark.sql.catalog import DataFrame
  from pyspark.sql.readwriter import PathOrPaths
  $ mypy app.py          # no error
  $ python app.py
  ImportError: cannot import name 'DateConverter' from 'pyspark.sql.types'
With pyspark-client 4.3.0 all three import, and PathOrPaths is PySpark's alias 
that applications
annotate their own readers with.

Expected
Every name that a stub declares exists at the drop-in's run time, but a short, 
reasoned list of
names that PySpark's annotations need, and a test checks it. PySpark's API that 
the drop-in lacks
(PathOrPaths, TupleOrListOfString) is given to the drop-in; PySpark's plumbing 
and incidental imports
are left out of the stubs.

4. Add the type stubs of pyspark.sql.connect.datasource and pyspark.ml.stat, 
whose classes are native

Problem
pyspark.sql.connect.datasource (DataSourceRegistration, spark.dataSource of a 
Connect session) and
pyspark.ml.stat (ChiSquareTest, Correlation, KolmogorovSmirnovTest, Summarizer, 
SummaryBuilder)
re-export native classes and have no stub, so type checkers see Any for them.

Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
  $ cat app.py
  from pyspark.ml.stat import Correlation
  from pyspark.sql.connect.session import SparkSession
  spark = SparkSession.builder.getOrCreate()
  reveal_type(spark.dataSource)
  Correlation.corr(spark.range(1), 1)
  spark.dataSource.register(1)
  $ mypy app.py
  app.py:4: note: Revealed type is "Any"
  (no error)

Expected (as with pyspark-client 4.3.0)
  app.py:4: note: Revealed type is 
"pyspark.sql.connect.datasource.DataSourceRegistration"
  app.py:5: error: Argument 2 to "corr" of "Correlation" has incompatible type 
"int"; expected "str"  [arg-type]
  app.py:6: error: Argument 1 to "register" of "DataSourceRegistration" has 
incompatible type "int"; expected "type[DataSource]"  [arg-type]
and the same under pyright 1.1.414.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to