This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e0c6db19f1d4 [SPARK-60195][PYTHON] Improve type annotation for
StatCounter callables
e0c6db19f1d4 is described below
commit e0c6db19f1d4583c25fb252efd586e02d920c623
Author: Tim Hsiung <[email protected]>
AuthorDate: Wed Nov 4 19:55:58 2026 +0000
[SPARK-60195][PYTHON] Improve type annotation for StatCounter callables
### What changes were proposed in this pull request?
Declare callable types for `maximum`, `minimum`, and `sqrt` in
`StatCounter.__init__`, then remove the three assignment type ignores on the
standard-library fallbacks.
### Why are the changes needed?
The initial NumPy imports infer NumPy-specific callable types. Explicit
scalar callable signatures describe both the NumPy implementations and their
`max`, `min`, and `math.sqrt` fallbacks without suppressions.
### Does this PR introduce _any_ user-facing change?
No. Function selection and statistical calculations are unchanged.
### How was this patch tested?
- Existing `pyspark.tests.test_statcounter` suite passed.
- Local probes produced identical baseline/modified results with NumPy and
with its import forced to fail, including empty input, merging, self-merging,
copying, and statistics output.
- Mypy on `python/pyspark/statcounter.py`, Ruff lint/format checks, and
`git diff --check` passed.
No new tests were added; validation used the existing suite and local
equivalence probes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 1.1.0
Closes #59396 from bearomorphism/typing-statcounter-callables.
Authored-by: Tim Hsiung <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/statcounter.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/statcounter.py b/python/pyspark/statcounter.py
index 92bf0b11a3e1..05c93dfc8d5a 100644
--- a/python/pyspark/statcounter.py
+++ b/python/pyspark/statcounter.py
@@ -19,17 +19,20 @@
import copy
import math
-from typing import Dict, Iterable, Optional
+from typing import Callable, Dict, Iterable, Optional
class StatCounter:
def __init__(self, values: Optional[Iterable[float]] = None):
+ maximum: Callable[[float, float], float]
+ minimum: Callable[[float, float], float]
+ sqrt: Callable[[float], float]
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max # type: ignore[assignment]
- minimum = min # type: ignore[assignment]
- sqrt = math.sqrt # type: ignore[assignment]
+ maximum = max
+ minimum = min
+ sqrt = math.sqrt
self.maximum = maximum
self.minimum = minimum
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]