Spenserrrr opened a new pull request, #57611:
URL: https://github.com/apache/spark/pull/57611
### What changes were proposed in this pull request?
This PR removes 7 unnecessary `# type: ignore` comments in the PySpark
DataFrame
read/write API (`DataFrameReader` / `DataFrameWriter` and their streaming and
Spark Connect variants) by improving the underlying annotations:
1. **`OptionUtils._set_opts` (classic + Connect)** — the mixin calls
`self.option(...)` without declaring that its host provides it, requiring
a
`# type: ignore[attr-defined]`. A small `SupportsOption` `Protocol` is
added and
`self` is annotated with it, so the call type-checks and the contract is
explicit. (The sibling `self.schema(...)` call keeps its ignore on
purpose:
`schema` is a reader-only method, but `_set_opts` is shared with writers,
so
requiring `schema` on `self` would break the writer call sites.)
2. **`DataFrameReader.load` (Connect)** — a `paths` local was assigned from
the
wide `Optional[Union[str, List[str]]]` parameter and then reused across an
`isinstance` narrowing, so mypy kept the wide first-assignment type and
needed a
`# type: ignore[arg-type]`. Declaring `paths: Optional[List[str]]` up
front and
normalizing the input removes the ignore and makes the None/str/list cases
explicit.
3. **`DataStreamWriter.partitionBy` / `clusterBy` (classic + Connect)** — the
overload implementation signatures were `*cols: str`, which contradicts
the
`List[str]` overload and required `# type: ignore[misc]`; it also left the
list-handling branch as dead code to the type checker. The implementation
signatures are widened to `*cols: Union[str, List[str]]` (matching the
non-streaming `DataFrameWriter`), and the columns are normalized into a
correctly typed `Sequence[str]` local, so no new ignore is introduced.
Some `# type: ignore` comments in these files are intentionally left in
place, as
removing them would only trade one escape hatch for another rather than
improve
readability:
- the reader-only `self.schema(...)` in `_set_opts` (see above);
- the `x = x[0]` varargs-unwrap pattern (`[assignment]`), which is a widely
used
idiom across the codebase;
- `keyed._bypass_serializer = True` (`[attr-defined]`), a real private `RDD`
attribute whose proper typing belongs in `pyspark.core.rdd`;
- the optional `f.open` / `f.close` calls in `foreach` (`[attr-defined]`),
which
are genuine duck-typing guarded at runtime by existence checks.
### Why are the changes needed?
These ignores sit on fixable annotation gaps rather than genuine type
deviations.
Removing them documents the intended contracts and lets mypy type-check the
affected code paths (for example, the streaming `partitionBy` list branch was
previously unreachable to the type checker).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests. Verified with mypy over the full `python/pyspark` scope.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
