[
https://issues.apache.org/jira/browse/SPARK-62011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62011:
-----------------------------------
Labels: pull-request-available (was: )
> Make type checkers see the messages of pyspark.sql.connect.proto, the Avro
> and Protobuf functions of Spark Connect, the classes of pyspark.ml.connect
> and PySpark's classic SQLContext, as with pyspark-client
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62011
> URL: https://issues.apache.org/jira/browse/SPARK-62011
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> The drop-in for pyspark-client 4.3.0 ships stubs (.pyi) of the modules whose
> classes are native or whose names are made at import time
> (scripts/gen_type_stubs.py), so that mypy and pyright type an application as
> with pyspark-client. Several modules had no stub, or a stub that missed a
> name, and SQLContext was the drop-in's own class of fewer methods.
> Problems, each with mypy 2.4.1 and pyright 1.1.414 against pyspark-client
> 4.3.0 and the drop-in:
> 1. pyspark.sql.connect.proto. PySpark's package star-imports the messages of
> its generated modules (python/pyspark/sql/connect/proto/__init__.py). The
> drop-in's installs its native messages in globals(), so type checkers saw
> none:
> import pyspark.sql.connect.proto as pb2
> reveal_type(pb2.Plan(root=pb2.Relation()))
> pyspark-client: mypy `Revealed type is
> "pyspark.sql.connect.proto.base_pb2.Plan"`, pyright `Plan`.
> Drop-in: mypy `Module has no attribute "Plan"` and `Any`; pyright `"Plan"
> is not a known attribute of module "pyspark.sql.connect.proto"`. Every
> annotation of a message in PySpark's modules (SparkConnectClient.to_table's
> plan, MergeIntoWriter._matched_actions, ...) was Any.
> 2. pyspark.sql.connect.avro.functions and
> pyspark.sql.connect.protobuf.functions copy their functions into globals(),
> with no stub: `from pyspark.sql.connect.avro.functions import from_avro` is
> an error (`Module has no attribute "from_avro"`, pyright `unknown import
> symbol`), where pyspark-client types `from_avro(data: ColumnOrName,
> jsonFormatSchema: str, options: Optional[Dict[str, str]] = None) -> Column`.
> 3. pyspark.ml.connect. The drop-in's modules of its native ML classes
> (classification, clustering, evaluation, fpm, pipeline, recommendation,
> regression, tuning, the package and the native stages of the feature module)
> had no stub, so their classes were Any for mypy, pyright reported `"Pipeline"
> is not exported from module "pyspark.ml.connect.pipeline"`, and `from
> pyspark.ml.connect import VectorAssembler` was an error. At run time each is
> a subclass of PySpark's class of its name of pyspark.ml
> (pyspark.ml._native_classes.publish).
> 4. SQLContext. pyspark-client ships PySpark's classic
> python/pyspark/sql/context.py, whose SQLContext has all its methods (sql,
> table, read, ...) and whose getOrCreate() returns the SQLContext of
> pyspark.sql.connect.context (context.py:146-200). The drop-in's had
> getOrCreate() only: `hasattr(SQLContext, "sql")` was False (True with
> pyspark-client), and for `ctx = SQLContext.getOrCreate()` mypy reported
> `"SQLContext" has no attribute "sql"` of ctx.sql(...), where pyspark-client
> types `SQLContext` and `DataFrame`.
> 5. The classic names of pyspark. PySpark's pyspark/__init__.py imports
> SparkContext, RDD, RDDBarrier, SparkFiles, Broadcast, StatusTracker,
> SparkJobInfo, SparkStageInfo and SparkExecutorInfo of pyspark.core when not
> is_remote_only() (python/pyspark/__init__.py:52-63, 118-127), which type
> checkers do not evaluate: with pyspark-client mypy takes `from pyspark import
> SparkContext` as Any. The drop-in's module had none: mypy reported `Module
> "pyspark" has no attribute "SparkContext"`.
> 6. The stub of pyspark.sql.connect.table_arg left out TableArg.to_plan, as
> PySpark Connect's plumbing, after the drop-in had the method:
> `ta.to_plan(client)` was an error (`"TableArg" has no attribute "to_plan"`),
> typed `(session: SparkConnectClient) -> Expression` with pyspark-client.
> 7. The stub test checked only the module-level names of stubbed modules: a
> class member that a stub missed (6) and a name of DROPPED that the drop-in
> had passed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]