Hyukjin Kwon created SPARK-62011:
------------------------------------
Summary: Make type checkers see the messages of
pyspark.sql.connect.proto, the Avro and Protobuf functions of Spark Connect,
the classes of pyspark.ml.connect and PySpark's classic SQLContext, as with
pyspark-client
Key: SPARK-62011
URL: https://issues.apache.org/jira/browse/SPARK-62011
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in for pyspark-client 4.3.0 ships stubs (.pyi) of the modules whose
classes are native or whose names are made at import time
(scripts/gen_type_stubs.py), so that mypy and pyright type an application as
with pyspark-client. Several modules had no stub, or a stub that missed a name,
and SQLContext was the drop-in's own class of fewer methods.
Problems, each with mypy 2.4.1 and pyright 1.1.414 against pyspark-client 4.3.0
and the drop-in:
1. pyspark.sql.connect.proto. PySpark's package star-imports the messages of
its generated modules (python/pyspark/sql/connect/proto/__init__.py). The
drop-in's installs its native messages in globals(), so type checkers saw none:
import pyspark.sql.connect.proto as pb2
reveal_type(pb2.Plan(root=pb2.Relation()))
pyspark-client: mypy `Revealed type is
"pyspark.sql.connect.proto.base_pb2.Plan"`, pyright `Plan`.
Drop-in: mypy `Module has no attribute "Plan"` and `Any`; pyright `"Plan" is
not a known attribute of module "pyspark.sql.connect.proto"`. Every annotation
of a message in PySpark's modules (SparkConnectClient.to_table's plan,
MergeIntoWriter._matched_actions, ...) was Any.
2. pyspark.sql.connect.avro.functions and
pyspark.sql.connect.protobuf.functions copy their functions into globals(),
with no stub: `from pyspark.sql.connect.avro.functions import from_avro` is an
error (`Module has no attribute "from_avro"`, pyright `unknown import symbol`),
where pyspark-client types `from_avro(data: ColumnOrName, jsonFormatSchema:
str, options: Optional[Dict[str, str]] = None) -> Column`.
3. pyspark.ml.connect. The drop-in's modules of its native ML classes
(classification, clustering, evaluation, fpm, pipeline, recommendation,
regression, tuning, the package and the native stages of the feature module)
had no stub, so their classes were Any for mypy, pyright reported `"Pipeline"
is not exported from module "pyspark.ml.connect.pipeline"`, and `from
pyspark.ml.connect import VectorAssembler` was an error. At run time each is a
subclass of PySpark's class of its name of pyspark.ml
(pyspark.ml._native_classes.publish).
4. SQLContext. pyspark-client ships PySpark's classic
python/pyspark/sql/context.py, whose SQLContext has all its methods (sql,
table, read, ...) and whose getOrCreate() returns the SQLContext of
pyspark.sql.connect.context (context.py:146-200). The drop-in's had
getOrCreate() only: `hasattr(SQLContext, "sql")` was False (True with
pyspark-client), and for `ctx = SQLContext.getOrCreate()` mypy reported
`"SQLContext" has no attribute "sql"` of ctx.sql(...), where pyspark-client
types `SQLContext` and `DataFrame`.
5. The classic names of pyspark. PySpark's pyspark/__init__.py imports
SparkContext, RDD, RDDBarrier, SparkFiles, Broadcast, StatusTracker,
SparkJobInfo, SparkStageInfo and SparkExecutorInfo of pyspark.core when not
is_remote_only() (python/pyspark/__init__.py:52-63, 118-127), which type
checkers do not evaluate: with pyspark-client mypy takes `from pyspark import
SparkContext` as Any. The drop-in's module had none: mypy reported `Module
"pyspark" has no attribute "SparkContext"`.
6. The stub of pyspark.sql.connect.table_arg left out TableArg.to_plan, as
PySpark Connect's plumbing, after the drop-in had the method:
`ta.to_plan(client)` was an error (`"TableArg" has no attribute "to_plan"`),
typed `(session: SparkConnectClient) -> Expression` with pyspark-client.
7. The stub test checked only the module-level names of stubbed modules: a
class member that a stub missed (6) and a name of DROPPED that the drop-in had
passed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]