bearomorphism opened a new pull request, #59586:
URL: https://github.com/apache/spark/pull/59586
### What changes were proposed in this pull request?
Improve typing of ML and legacy MLlib vector operations independently of the
Vector ABC proposal in #59585:
- Annotate vector arrays, array conversions, dense values, and dense slices
as `NDArray[np.float64]` instead of losing their dtype through bare
`np.ndarray` annotations.
- Use `object` for equality/inequality operands and explicit dense/sparse
vector, `ArrayLike`, or `SupportsFloat` arithmetic operands instead of `Any`.
Preserve working sparse-vector and `Fraction` arithmetic.
- Make the arithmetic-operation factory a typed `staticmethod`, removing its
invalid-instance-method suppression while preserving the delegated NumPy
operations.
- Overload concrete `dot()` methods: ordinary vector/sequence inputs retain
`np.float64` results, while NumPy inputs of unknown shape/dtype allow array and
numeric scalar results. ML dot annotations also accept the nominal vector
operands already supported at runtime.
- Preserve concrete dense/sparse types through overloads of legacy
`Vectors.fromML()`.
- Remove avoidable assignment suppressions in dense factories and
sparse-vector parsing without changing their inputs, numerical implementations,
or errors.
- Add typing regression coverage for dtype preservation, arithmetic
operands, dot results, factories, and ML-to-MLlib conversion.
This PR is based directly on `master`, not on #59585. It does not make
Vector abstract, add abstract methods, change norm annotations, or modify
either `NormType` or `VectorLike`. Dynamic NumPy attribute forwarding retains
`Any`; unrelated matrix and overloaded sparse-factory cleanup is not included.
### Why are the changes needed?
Current annotations lose known float64 array dtypes, allow unsupported
arithmetic operands without a diagnostic, always describe dot products as
float64 scalars even when NumPy returns arrays or other numeric scalars, and
reject valid sparse/base-vector inputs to `Vectors.fromML()`.
These issues exist independently of whether Vector is an abstract class. The
new typing fixtures reproduce the incorrect behavior before the implementation
changes.
### Does this PR introduce _any_ user-facing change?
Yes, for static typing. Callers receive more precise array and conversion
types; unsupported arithmetic operands such as `dense + object()` are rejected;
and callers of `dot()` with an array of unknown shape/dtype must account for
its possible array or scalar result.
The numerical algorithms and serialization formats are unchanged.
DenseVector/SparseVector construction, supported mathematical operations,
parsing behavior, and ML/MLlib conversion behavior remain the same. The Vector
bases remain instantiable, and the existing aliases are unchanged.
### How was this patch tested?
- Added 11 typing fixture cases. Against the unchanged implementation, 7
failed and 4 passed, reproducing lost array dtypes, unchecked object operands,
incorrect dot-result types, and incorrect ML-to-MLlib conversion types.
- **All 22 ML typing fixtures passed**, including the 11 new cases.
```bash
source .venv/bin/activate
MYPYPATH=python pytest -q -o 'python_files=test_*.yml test_*.yaml' \
--rootdir python --mypy-only-local-stub --mypy-ini-file python/mypy.ini \
python/pyspark/ml/tests/typing
```
- **Repository-wide mypy, example mypy, Ruff lint, and formatting checks
passed**:
```bash
dev/lint-python --mypy --mypy-examples --ruff
```
- `git diff --check` passed.
- **37 existing vector/UDT tests passed**: both complete `VectorTests`
classes plus both UDT serialization/schema-inference selections.
```bash
source .venv/bin/activate
SPARK_LOCAL_IP=localhost python/run-tests --keep-test-output --parallelism
2 \
--testnames 'pyspark.ml.tests.test_linalg VectorTests
VectorUDTTests.test_serialization
VectorUDTTests.test_infer_schema,pyspark.mllib.tests.test_linalg VectorTests
VectorUDTTests.test_serialization VectorUDTTests.test_infer_schema'
```
- **174 module doctests passed**: 85 ML and 89 MLlib examples, using the
modules' existing NumPy legacy print setting.
- Additional runtime smoke coverage passed for operators, factory/parsing
behavior, dense/sparse conversion, and scalar/matrix dot products with float64,
complex128, and object arrays. Dense/sparse and Fraction forward/reflected
arithmetic was also checked explicitly.
- Confirmed that both Vector bases remain instantiable and both existing
aliases are unchanged.
Validation used Python 3.13.3 with the repository's development
dependencies. The local pandas stubs were aligned to Spark's existing CI pin
(`pandas-stubs==2.3.3.260113`); no dependency files were changed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 1.3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
