Hyukjin Kwon created SPARK-61950:
------------------------------------

             Summary: Run pyright in the offline test of the type stubs, and 
describe what type checkers see of the drop-in
                 Key: SPARK-61950
                 URL: https://issues.apache.org/jira/browse/SPARK-61950
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


1. Run pyright in the offline test of the type stubs, as it runs mypy, and 
install it in the coverage job

Problem
The type stubs are for mypy and pyright, and docs/compatibility.md says both 
see PySpark's API as
with pyspark-client, but CI runs only mypy 
(python/tests/test_type_stubs_offline.py, mypy 2.4.1 in
coverage.yml). pyright treats stubs differently: it does not narrow in a stub, 
and it reports private
imports. A change that breaks the stubs for pyright alone passes CI.

Repro
Drop the StructType.fields entry of ATTRIBUTE_TYPES in 
scripts/gen_type_stubs.py, regenerate, and
type-check `for f in df.schema.fields: ...` with pyright 1.1.414: pyright 
reports
`Object of type "None" cannot be used as iterable value 
(reportOptionalIterable)`, as it gives the
attribute the Optional of the parameter of the kept __init__; mypy, and so CI, 
report nothing.

Expected
The offline test runs pyright, pinned in coverage.yml as mypy is, on the user 
application and on the
probes of the functions, the names and the instance attributes of the stubs, 
and fails on such a
change.

2. Describe what type checkers see of the drop-in, with the differences from 
pyspark-client that remain

Problem
docs/compatibility.md says that mypy and pyright type an application as with 
pyspark-client but for
the client of SparkSession.client, SQLContext and the internals of PySpark 
Connect. On master
0dbe860b that is not the whole story: spark.dataSource of a Connect session and 
pyspark.ml.stat are
Any, and the stubs declare names that the drop-in does not have; and 
SPARK-61856 gave
SparkSession.client the public API of PySpark Connect's SparkConnectClient, so 
all 46 of its public
members are typed now, 30 of them exactly as with pyspark-client and 16 apart 
only in the module of
the class (pyspark.sql.connect.client, not .client.core) and in Any for 
protobuf messages.
docs/contributing.md does not say that a module that becomes vendored leaves 
STUB_MODULES, nor that
a name of EXTRAS that the drop-in's module loses must go, as pyspark.sql.conf 
and its RuntimeConf had
to in SPARK-61854.

Repro (master 0dbe860b, a venv with the drop-in's wheel and mypy 2.4.1)
  from pyspark.sql.connect.session import SparkSession
  spark = SparkSession.builder.getOrCreate()
  reveal_type(spark.dataSource)              # Any; the docs do not mention it
  reveal_type(spark.client.execute_command)  # typed; the docs call the client 
another class

Expected
The docs list the modules with stubs and each remaining difference from 
pyspark-client with an
example, as the follow-ups leave them, and dev/parity/known-differences.md says 
whether each is
intentional.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to