Hyukjin Kwon created SPARK-61950:
------------------------------------
Summary: Run pyright in the offline test of the type stubs, and
describe what type checkers see of the drop-in
Key: SPARK-61950
URL: https://issues.apache.org/jira/browse/SPARK-61950
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
1. Run pyright in the offline test of the type stubs, as it runs mypy, and
install it in the coverage job
Problem
The type stubs are for mypy and pyright, and docs/compatibility.md says both
see PySpark's API as
with pyspark-client, but CI runs only mypy
(python/tests/test_type_stubs_offline.py, mypy 2.4.1 in
coverage.yml). pyright treats stubs differently: it does not narrow in a stub,
and it reports private
imports. A change that breaks the stubs for pyright alone passes CI.
Repro
Drop the StructType.fields entry of ATTRIBUTE_TYPES in
scripts/gen_type_stubs.py, regenerate, and
type-check `for f in df.schema.fields: ...` with pyright 1.1.414: pyright
reports
`Object of type "None" cannot be used as iterable value
(reportOptionalIterable)`, as it gives the
attribute the Optional of the parameter of the kept __init__; mypy, and so CI,
report nothing.
Expected
The offline test runs pyright, pinned in coverage.yml as mypy is, on the user
application and on the
probes of the functions, the names and the instance attributes of the stubs,
and fails on such a
change.
2. Describe what type checkers see of the drop-in, with the differences from
pyspark-client that remain
Problem
docs/compatibility.md says that mypy and pyright type an application as with
pyspark-client but for
the client of SparkSession.client, SQLContext and the internals of PySpark
Connect. On master
0dbe860b that is not the whole story: spark.dataSource of a Connect session and
pyspark.ml.stat are
Any, and the stubs declare names that the drop-in does not have; and
SPARK-61856 gave
SparkSession.client the public API of PySpark Connect's SparkConnectClient, so
all 46 of its public
members are typed now, 30 of them exactly as with pyspark-client and 16 apart
only in the module of
the class (pyspark.sql.connect.client, not .client.core) and in Any for
protobuf messages.
docs/contributing.md does not say that a module that becomes vendored leaves
STUB_MODULES, nor that
a name of EXTRAS that the drop-in's module loses must go, as pyspark.sql.conf
and its RuntimeConf had
to in SPARK-61854.
Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
from pyspark.sql.connect.session import SparkSession
spark = SparkSession.builder.getOrCreate()
reveal_type(spark.dataSource) # Any; the docs do not mention it
reveal_type(spark.client.execute_command) # typed; the docs call the client
another class
Expected
The docs list the modules with stubs and each remaining difference from
pyspark-client with an
example, as the follow-ups leave them, and dev/parity/known-differences.md says
whether each is
intentional.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]