This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 12c176214c60 [SPARK-58410][PYTHON] Remove unnecessary type: ignore
comments in the DataFrame read/write API
12c176214c60 is described below
commit 12c176214c60d2e773e1f34a3f7ad825ca7adc35
Author: Spenser Sun <[email protected]>
AuthorDate: Wed Aug 12 00:24:07 2026 -0700
[SPARK-58410][PYTHON] Remove unnecessary type: ignore comments in the
DataFrame read/write API
### What changes were proposed in this pull request?
This PR removes 3 unnecessary `# type: ignore` comments in the PySpark
DataFrame read/write API (`DataFrameReader` / `DataFrameWriter` and their Spark
Connect variants) by improving the underlying annotations:
1. **`OptionUtils._set_opts` (classic + Connect)** — the `OptionUtils`
mixin calls `self.option(...)` but does not declare that its subclasses provide
it, so the call needed a `# type: ignore[attr-defined]`. `OptionUtils` is now
an `ABC` with `option` declared as an `abstractmethod`, which states the
contract via inheritance and lets the call type-check. (The sibling
`self.schema(...)` call keeps its ignore on purpose: `schema` is a reader-only
method, but `_set_opts` is shared with w [...]
2. **`DataFrameReader.load` (Connect)** — a `paths` local was assigned from
the wide `Optional[Union[str, List[str]]]` parameter and then reused across an
`isinstance` narrowing, so mypy kept the wide first-assignment type and needed
a `# type: ignore[arg-type]`. Declaring `paths: Optional[List[str]]` up front
and normalizing the input removes the ignore.
### Why are the changes needed?
These ignores sit on fixable annotation gaps rather than genuine type
deviations. Removing them documents the intended contracts and lets mypy
type-check the affected code paths.
Note: an earlier revision of this PR also retyped
`DataStreamWriter.partitionBy` / `clusterBy`. Per review, that
vararg-that-also-accepts-a-sequence typing pattern is a repo-wide issue (it
appears in many methods across many files) and should be handled uniformly
following the approach in SPARK-55967 (#54764), so it has been reverted here
and will be addressed in a dedicated follow-up.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests. Verified with mypy over the full `python/pyspark` scope.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57611 from Spenserrrr/typeignore3.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 9a007724f35ed883933efcf791aaa5d3170be85e)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/sql/connect/readwriter.py | 16 +++++++++++-----
python/pyspark/sql/readwriter.py | 10 +++++++---
2 files changed, 18 insertions(+), 8 deletions(-)
diff --git a/python/pyspark/sql/connect/readwriter.py
b/python/pyspark/sql/connect/readwriter.py
index 5c2c0c80ccdf..3c2c9e93dc81 100644
--- a/python/pyspark/sql/connect/readwriter.py
+++ b/python/pyspark/sql/connect/readwriter.py
@@ -14,8 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from abc import ABC, abstractmethod
from typing import Dict
-from typing import Optional, Union, List, overload, Tuple, cast, Callable
+from typing import Any, Optional, Union, List, overload, Tuple, cast, Callable
from typing import TYPE_CHECKING
from pyspark.sql.connect.plan import (
@@ -55,7 +56,10 @@ PathOrPaths = Union[str, List[str]]
TupleOrListOfString = Union[List[str], Tuple[str, ...]]
-class OptionUtils:
+class OptionUtils(ABC):
+ @abstractmethod
+ def option(self, key: str, value: "OptionalPrimitiveType") -> Any: ...
+
def _set_opts(
self,
schema: Optional[Union[StructType, str]] = None,
@@ -68,7 +72,7 @@ class OptionUtils:
self.schema(schema) # type: ignore[attr-defined]
for k, v in options.items():
if v is not None:
- self.option(k, v) # type: ignore[attr-defined]
+ self.option(k, v)
class DataFrameReader(OptionUtils):
@@ -130,15 +134,17 @@ class DataFrameReader(OptionUtils):
self.schema(schema)
self.options(**options)
- paths = path
+ paths: Optional[List[str]]
if isinstance(path, str):
paths = [path]
+ else:
+ paths = path
plan = DataSource(
format=self._format,
schema=self._schema,
options=self._options,
- paths=paths, # type: ignore[arg-type]
+ paths=paths,
)
return self._df(plan)
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index afe8000b5c45..eeb31654fbe7 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -15,7 +15,8 @@
# limitations under the License.
#
import sys
-from typing import cast, overload, Dict, Iterable, List, Optional, Tuple,
TYPE_CHECKING, Union
+from abc import ABC, abstractmethod
+from typing import Any, cast, overload, Dict, Iterable, List, Optional, Tuple,
TYPE_CHECKING, Union
from pyspark.util import is_remote_only
from pyspark.sql.types import StructType
@@ -37,7 +38,10 @@ PathOrPaths = Union[str, List[str]]
TupleOrListOfString = Union[List[str], Tuple[str, ...]]
-class OptionUtils:
+class OptionUtils(ABC):
+ @abstractmethod
+ def option(self, key: str, value: "OptionalPrimitiveType") -> Any: ...
+
def _set_opts(
self,
schema: Optional[Union[StructType, str]] = None,
@@ -50,7 +54,7 @@ class OptionUtils:
self.schema(schema) # type: ignore[attr-defined]
for k, v in options.items():
if v is not None:
- self.option(k, v) # type: ignore[attr-defined]
+ self.option(k, v)
class DataFrameReader(OptionUtils):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
