Haotian Sun created SPARK-58244:
-----------------------------------
Summary: Remove type: ignore[name-defined] in pyspark.sql
Key: SPARK-58244
URL: https://issues.apache.org/jira/browse/SPARK-58244
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0
Reporter: Haotian Sun
1. Some forward-reference annotations in pyspark.sql reference a type only
imported inside a function body to avoid a circular import, so it is
unresolvable to the type checker. Adding it to the module's if TYPE_CHECKING:
block resolves the reference.
2. In the _test() doctest runners of streaming/query.py and
streaming/listener.py, the except Py4JError: fallback calls SparkSession(sc)
but never defines sc. The _test() helpers in readwriter.py define sc =
SparkContext("local[4]", "PythonTest") before the try block. Applying the same
fixes it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
