bearomorphism opened a new pull request, #59345:
URL: https://github.com/apache/spark/pull/59345
### What changes were proposed in this pull request?
Add type annotations to the vendored `LooseVersion` comparison methods and
remove their six `no-untyped-def` suppressions:
- Accept `object` for equality and `LooseVersion | str` for ordering, with
`bool` comparison results.
- Annotate `_cmp` as returning `int | NotImplementedType`.
- Return `NotImplemented` explicitly in the sentinel branches so mypy does
not treat the returned variable as the full comparison-result union.
- Make the final greater-than result unconditional after the equality and
less-than checks, avoiding an implicit `None` return for the annotated helper.
- Use pipe syntax and postponed annotation evaluation for the class
self-reference; replace `Optional[str]` with `str | None` and remove the
`typing` import.
- Move the existing runtime coverage into a plain `unittest.TestCase`,
expand public comparison coverage, and add caller-facing typing regression
tests.
### Why are the changes needed?
The comparison methods are currently untyped, so expressions such as the
following infer as `Any` rather than `bool`:
```python
from typing_extensions import assert_type
from pyspark.loose_version import LooseVersion
version = LooseVersion("1.2.3")
assert_type(version < "1.2.4", bool)
```
This assertion fails against the original implementation and passes with
this change. The annotations also allow mypy to reject unsupported ordering
operands instead of accepting code that fails at runtime.
#### Upstream investigation
An upstream refresh was considered before making this local change:
- [CPython removed `distutils` in Python
3.12](https://docs.python.org/3/library/distutils.html). Python 3.11 is the
last release containing it; there is no replacement `Lib/distutils/version.py`
in current CPython.
- The current [CPython 3.11
source](https://github.com/python/cpython/blob/3.11/Lib/distutils/version.py)
still leaves the comparison methods and `LooseVersion._cmp` untyped. Copying
that source would not resolve the typing issue.
- The maintained [setuptools
copy](https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/version.py)
also leaves these methods untyped and introduces deprecation-warning behavior
through its base class. It does not provide a drop-in typing fix.
- Spark intentionally embedded this implementation in [SPARK-45390 /
#43192](https://github.com/apache/spark/pull/43192) to remove its `distutils`
dependency without adding a `packaging` dependency. Spark's file is already an
adaptation that combines upstream base/subclass methods and adds partial
annotations.
Replacing it with `packaging.version.Version` would be a separate
compatibility change: its accepted version strings and ordering semantics
differ. This PR therefore keeps the existing implementation and completes its
comparison annotations locally.
### Does this PR introduce _any_ user-facing change?
Static typing improves: comparison expressions infer as `bool`, and
unsupported ordering operands such as `LooseVersion("1.2.3") < 1.1` are
rejected by mypy. Equality continues to accept arbitrary objects.
No runtime behavior change is intended for parsed version strings. String
and `LooseVersion` operands remain supported; unsupported equality resolves to
`False`, and unsupported ordering continues to raise `TypeError`.
### How was this patch tested?
Verified locally with Python 3.13.5, mypy 1.19.1, pytest-mypy-plugins 3.2.0,
and Ruff 0.14.8:
```bash
# All 8 core typing cases passed, including the 2 new cases.
MYPYPATH=python python -m pytest -q \
-o 'python_files=test_*.yml' \
--rootdir python \
--mypy-only-local-stub \
--mypy-ini-file python/mypy.ini \
python/pyspark/tests/typing
# All 4 runtime tests passed; no Spark session or JVM is needed.
PYTHONPATH=python python -m unittest
pyspark.tests.test_util.LooseVersionTests -v
python -m mypy --config-file python/mypy.ini --follow-imports=silent \
python/pyspark/loose_version.py
python -m ruff check python/pyspark/loose_version.py
python/pyspark/tests/test_util.py
python -m ruff format --check python/pyspark/loose_version.py
python/pyspark/tests/test_util.py
git diff --check
```
The runtime tests cover string representations, all five comparison
operators with strings and version objects in both operand orders, unsupported
operands, and incomparable numeric/alphabetic components. The typing tests
verify boolean results and rejection of unsupported ordering operands. The
original implementation was also checked with a caller-side `assert_type`
reproducer, which failed before the fix.
A full Spark build or JVM-backed test suite was not run for this Python-only
typing change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
