[
https://issues.apache.org/jira/browse/SPARK-61950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61950:
-----------------------------------
Labels: pull-request-available (was: )
> Run pyright in the offline test of the type stubs, and describe what type
> checkers see of the drop-in
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-61950
> URL: https://issues.apache.org/jira/browse/SPARK-61950
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> 1. Run pyright in the offline test of the type stubs, as it runs mypy, and
> install it in the coverage job
> Problem
> The type stubs are for mypy and pyright, and docs/compatibility.md says both
> see PySpark's API as
> with pyspark-client, but CI runs only mypy
> (python/tests/test_type_stubs_offline.py, mypy 2.4.1 in
> coverage.yml). pyright treats stubs differently: it does not narrow in a
> stub, and it reports private
> imports. A change that breaks the stubs for pyright alone passes CI.
> Repro
> Drop the StructType.fields entry of ATTRIBUTE_TYPES in
> scripts/gen_type_stubs.py, regenerate, and
> type-check `for f in df.schema.fields: ...` with pyright 1.1.414: pyright
> reports
> `Object of type "None" cannot be used as iterable value
> (reportOptionalIterable)`, as it gives the
> attribute the Optional of the parameter of the kept __init__; mypy, and so
> CI, report nothing.
> Expected
> The offline test runs pyright, pinned in coverage.yml as mypy is, on the user
> application and on the
> probes of the functions, the names and the instance attributes of the stubs,
> and fails on such a
> change.
> 2. Describe what type checkers see of the drop-in, with the differences from
> pyspark-client that remain
> Problem
> docs/compatibility.md says that mypy and pyright type an application as with
> pyspark-client but for
> the client of SparkSession.client, SQLContext and the internals of PySpark
> Connect. On master
> 0dbe860b that is not the whole story: spark.dataSource of a Connect session
> and pyspark.ml.stat are
> Any, and the stubs declare names that the drop-in does not have; and
> SPARK-61856 gave
> SparkSession.client the public API of PySpark Connect's SparkConnectClient,
> so all 46 of its public
> members are typed now, 30 of them exactly as with pyspark-client and 16 apart
> only in the module of
> the class (pyspark.sql.connect.client, not .client.core) and in Any for
> protobuf messages.
> docs/contributing.md does not say that a module that becomes vendored leaves
> STUB_MODULES, nor that
> a name of EXTRAS that the drop-in's module loses must go, as pyspark.sql.conf
> and its RuntimeConf had
> to in SPARK-61854.
> Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
> from pyspark.sql.connect.session import SparkSession
> spark = SparkSession.builder.getOrCreate()
> reveal_type(spark.dataSource) # Any; the docs do not mention it
> reveal_type(spark.client.execute_command) # typed; the docs call the
> client another class
> Expected
> The docs list the modules with stubs and each remaining difference from
> pyspark-client with an
> example, as the follow-ups leave them, and dev/parity/known-differences.md
> says whether each is
> intentional.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]