Hyukjin Kwon created SPARK-62011:
------------------------------------

             Summary: Make type checkers see the messages of 
pyspark.sql.connect.proto, the Avro and Protobuf functions of Spark Connect, 
the classes of pyspark.ml.connect and PySpark's classic SQLContext, as with 
pyspark-client
                 Key: SPARK-62011
                 URL: https://issues.apache.org/jira/browse/SPARK-62011
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 ships stubs (.pyi) of the modules whose 
classes are native or whose names are made at import time 
(scripts/gen_type_stubs.py), so that mypy and pyright type an application as 
with pyspark-client. Several modules had no stub, or a stub that missed a name, 
and SQLContext was the drop-in's own class of fewer methods.

Problems, each with mypy 2.4.1 and pyright 1.1.414 against pyspark-client 4.3.0 
and the drop-in:

1. pyspark.sql.connect.proto. PySpark's package star-imports the messages of 
its generated modules (python/pyspark/sql/connect/proto/__init__.py). The 
drop-in's installs its native messages in globals(), so type checkers saw none:
     import pyspark.sql.connect.proto as pb2
     reveal_type(pb2.Plan(root=pb2.Relation()))
   pyspark-client: mypy `Revealed type is 
"pyspark.sql.connect.proto.base_pb2.Plan"`, pyright `Plan`.
   Drop-in: mypy `Module has no attribute "Plan"` and `Any`; pyright `"Plan" is 
not a known attribute of module "pyspark.sql.connect.proto"`. Every annotation 
of a message in PySpark's modules (SparkConnectClient.to_table's plan, 
MergeIntoWriter._matched_actions, ...) was Any.

2. pyspark.sql.connect.avro.functions and 
pyspark.sql.connect.protobuf.functions copy their functions into globals(), 
with no stub: `from pyspark.sql.connect.avro.functions import from_avro` is an 
error (`Module has no attribute "from_avro"`, pyright `unknown import symbol`), 
where pyspark-client types `from_avro(data: ColumnOrName, jsonFormatSchema: 
str, options: Optional[Dict[str, str]] = None) -> Column`.

3. pyspark.ml.connect. The drop-in's modules of its native ML classes 
(classification, clustering, evaluation, fpm, pipeline, recommendation, 
regression, tuning, the package and the native stages of the feature module) 
had no stub, so their classes were Any for mypy, pyright reported `"Pipeline" 
is not exported from module "pyspark.ml.connect.pipeline"`, and `from 
pyspark.ml.connect import VectorAssembler` was an error. At run time each is a 
subclass of PySpark's class of its name of pyspark.ml 
(pyspark.ml._native_classes.publish).

4. SQLContext. pyspark-client ships PySpark's classic 
python/pyspark/sql/context.py, whose SQLContext has all its methods (sql, 
table, read, ...) and whose getOrCreate() returns the SQLContext of 
pyspark.sql.connect.context (context.py:146-200). The drop-in's had 
getOrCreate() only: `hasattr(SQLContext, "sql")` was False (True with 
pyspark-client), and for `ctx = SQLContext.getOrCreate()` mypy reported 
`"SQLContext" has no attribute "sql"` of ctx.sql(...), where pyspark-client 
types `SQLContext` and `DataFrame`.

5. The classic names of pyspark. PySpark's pyspark/__init__.py imports 
SparkContext, RDD, RDDBarrier, SparkFiles, Broadcast, StatusTracker, 
SparkJobInfo, SparkStageInfo and SparkExecutorInfo of pyspark.core when not 
is_remote_only() (python/pyspark/__init__.py:52-63, 118-127), which type 
checkers do not evaluate: with pyspark-client mypy takes `from pyspark import 
SparkContext` as Any. The drop-in's module had none: mypy reported `Module 
"pyspark" has no attribute "SparkContext"`.

6. The stub of pyspark.sql.connect.table_arg left out TableArg.to_plan, as 
PySpark Connect's plumbing, after the drop-in had the method: 
`ta.to_plan(client)` was an error (`"TableArg" has no attribute "to_plan"`), 
typed `(session: SparkConnectClient) -> Expression` with pyspark-client.

7. The stub test checked only the module-level names of stubbed modules: a 
class member that a stub missed (6) and a name of DROPPED that the drop-in had 
passed.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to