[ 
https://issues.apache.org/jira/browse/SPARK-61950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61950:
-----------------------------------
    Labels: pull-request-available  (was: )

> Run pyright in the offline test of the type stubs, and describe what type 
> checkers see of the drop-in
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61950
>                 URL: https://issues.apache.org/jira/browse/SPARK-61950
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> 1. Run pyright in the offline test of the type stubs, as it runs mypy, and 
> install it in the coverage job
> Problem
> The type stubs are for mypy and pyright, and docs/compatibility.md says both 
> see PySpark's API as
> with pyspark-client, but CI runs only mypy 
> (python/tests/test_type_stubs_offline.py, mypy 2.4.1 in
> coverage.yml). pyright treats stubs differently: it does not narrow in a 
> stub, and it reports private
> imports. A change that breaks the stubs for pyright alone passes CI.
> Repro
> Drop the StructType.fields entry of ATTRIBUTE_TYPES in 
> scripts/gen_type_stubs.py, regenerate, and
> type-check `for f in df.schema.fields: ...` with pyright 1.1.414: pyright 
> reports
> `Object of type "None" cannot be used as iterable value 
> (reportOptionalIterable)`, as it gives the
> attribute the Optional of the parameter of the kept __init__; mypy, and so 
> CI, report nothing.
> Expected
> The offline test runs pyright, pinned in coverage.yml as mypy is, on the user 
> application and on the
> probes of the functions, the names and the instance attributes of the stubs, 
> and fails on such a
> change.
> 2. Describe what type checkers see of the drop-in, with the differences from 
> pyspark-client that remain
> Problem
> docs/compatibility.md says that mypy and pyright type an application as with 
> pyspark-client but for
> the client of SparkSession.client, SQLContext and the internals of PySpark 
> Connect. On master
> 0dbe860b that is not the whole story: spark.dataSource of a Connect session 
> and pyspark.ml.stat are
> Any, and the stubs declare names that the drop-in does not have; and 
> SPARK-61856 gave
> SparkSession.client the public API of PySpark Connect's SparkConnectClient, 
> so all 46 of its public
> members are typed now, 30 of them exactly as with pyspark-client and 16 apart 
> only in the module of
> the class (pyspark.sql.connect.client, not .client.core) and in Any for 
> protobuf messages.
> docs/contributing.md does not say that a module that becomes vendored leaves 
> STUB_MODULES, nor that
> a name of EXTRAS that the drop-in's module loses must go, as pyspark.sql.conf 
> and its RuntimeConf had
> to in SPARK-61854.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
>   from pyspark.sql.connect.session import SparkSession
>   spark = SparkSession.builder.getOrCreate()
>   reveal_type(spark.dataSource)              # Any; the docs do not mention it
>   reveal_type(spark.client.execute_command)  # typed; the docs call the 
> client another class
> Expected
> The docs list the modules with stubs and each remaining difference from 
> pyspark-client with an
> example, as the follow-ups leave them, and dev/parity/known-differences.md 
> says whether each is
> intentional.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to