This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 82177686fa4e [SPARK-60139][PYTHON] Remove dead code
UNSUPPORTED_COMBINATIONS
82177686fa4e is described below
commit 82177686fa4e0e3c3fbf3ddfe98a7aa0c1863141
Author: Tim Hsiung <[email protected]>
AuthorDate: Thu Oct 29 15:14:58 2026 -0700
[SPARK-60139][PYTHON] Remove dead code UNSUPPORTED_COMBINATIONS
### What changes were proposed in this pull request?
Remove the unused `UNSUPPORTED_COMBINATIONS` constant from
`python/pyspark/install.py`, together with its import and dead loop in
`test_checked_versions`.
This supersedes the original annotation-only approach. The existing
supported-version checks and explicit test rejecting Hive 1.2 remain unchanged.
### Why are the changes needed?
History investigation shows that this constant has been unused by
production code since September 2020:
- **September 23, 2020:** [SPARK-32017, commit
942f577b6e3](https://github.com/apache/spark/commit/942f577b6e34a4f24203dfcad53f5fb98232a6e4)
introduced the constant to reject Hive 1.2 paired with either `without-hadoop`
or Hadoop 3.2.
- **September 24, 2020:** [SPARK-32982, commit
688d016c7ac](https://github.com/apache/spark/commit/688d016c7acc4b9d96d75b40123be9f40b7b2693)
removed Hive 1.2 support, emptied the list, and deleted the production
membership check in `checked_versions`. The empty declaration and test loop
were left behind.
- **September 27, 2020:** [SPARK-32714, commit
c65b64552f9](https://github.com/apache/spark/commit/c65b64552f947a7eaf4f379edbdce05daa923363)
added `# type: ignore` to silence mypy's missing-annotation error for the
empty list.
Before this cleanup, a repository-wide search found only the declaration,
the test import, and the test loop. The loop executes zero iterations because
the list is empty. Removing these leftovers is preferable to adding a type
annotation to dead code.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The following focused tests passed both before and after removal (2 tests):
```bash
source .venv/bin/activate
PYTHONPATH="$PWD/python:$PWD/python/lib/py4j-0.10.9.9-src.zip" python -m
unittest \
pyspark.tests.test_install_spark.SparkInstallationTestCase.test_checked_versions
\
pyspark.tests.test_install_spark.SparkInstallationTestCase.test_package_name
```
Additional checks passed:
- Ruff lint and format checks on both changed files.
- Standalone mypy checking of the full `install.py` source using
`python/mypy.ini`, without importing the parent PySpark package.
- `git diff --check`.
- Search confirmed no remaining `UNSUPPORTED_COMBINATIONS` references under
`python`.
The network-dependent installation/download test was not run. An earlier
package-aware mypy invocation encountered 23 errors elsewhere in PySpark; this
is not a claim that the full package type check passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 1.1.0
Closes #59343 from bearomorphism/fix-unsupported-combinations-typing.
Authored-by: Tim Hsiung <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 3877d03b697335319bd99e8c47ce9e6c8a02daad)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/install.py | 1 -
python/pyspark/tests/test_install_spark.py | 9 ---------
2 files changed, 10 deletions(-)
diff --git a/python/pyspark/install.py b/python/pyspark/install.py
index c2ae38e05c14..4ed4d578eeb0 100644
--- a/python/pyspark/install.py
+++ b/python/pyspark/install.py
@@ -35,7 +35,6 @@ DEFAULT_HADOOP = "hadoop3"
DEFAULT_HIVE = "hive2.3"
SUPPORTED_HADOOP_VERSIONS = ["hadoop3", "without-hadoop"]
SUPPORTED_HIVE_VERSIONS = ["hive2.3"]
-UNSUPPORTED_COMBINATIONS = [] # type: ignore
# Official ASF-controlled hosts serving release checksums. Integrity metadata
# must come from these origins, never from a (community) mirror, so that a
diff --git a/python/pyspark/tests/test_install_spark.py
b/python/pyspark/tests/test_install_spark.py
index 19332897d2c3..12c4d9c4e01b 100644
--- a/python/pyspark/tests/test_install_spark.py
+++ b/python/pyspark/tests/test_install_spark.py
@@ -26,7 +26,6 @@ import urllib.request
from pyspark.install import (
DEFAULT_HADOOP,
DEFAULT_HIVE,
- UNSUPPORTED_COMBINATIONS,
_extract_tar,
_parse_sha512_checksum,
_verify_checksum,
@@ -190,14 +189,6 @@ class SparkInstallationTestCase(unittest.TestCase):
)
# Negative test cases
- for hadoop_version, hive_version in UNSUPPORTED_COMBINATIONS:
- with self.assertRaisesRegex(RuntimeError, "Hive.*should.*Hadoop"):
- checked_versions(
- spark_version=test_version,
- hadoop_version=hadoop_version,
- hive_version=hive_version,
- )
-
with self.assertRaisesRegex(RuntimeError, "Spark version should start
with 'spark-'"):
checked_versions(
spark_version="malformed", hadoop_version=DEFAULT_HADOOP,
hive_version=DEFAULT_HIVE
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]