This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9e776eeb059a [SPARK-58127][INFRA] Remove flake8 from the Python lint
tooling
9e776eeb059a is described below
commit 9e776eeb059af320257d1867e54b4a15717caef4
Author: Tian Gao <[email protected]>
AuthorDate: Mon Jul 20 15:54:05 2026 -0700
[SPARK-58127][INFRA] Remove flake8 from the Python lint tooling
### What changes were proposed in this pull request?
This PR removes `flake8` from the PySpark lint tooling, since it has been
fully superseded by `ruff`.
- `dev/lint-python`: drop the flake8 binary/version variables, the
`--flake8` option, and the `flake8_test` function and its invocation.
- `dev/tox.ini`: deleted. It contained only a `[flake8]` section, and its
rules (ignores, per-file-ignores, excludes, `max-line-length`) are already
covered by the `[tool.ruff]` config in `pyproject.toml`.
- `dev/run-tests.py`: the Python style-check change trigger was keyed on
`tox.ini`; repoint it to `pyproject.toml`, where the ruff config now lives.
- Drop the `flake8==3.9.0` install from the lint / docs / release Docker
images (`dev/spark-test-image/lint/Dockerfile`,
`dev/spark-test-image/docs/Dockerfile`,
`dev/create-release/spark-rm/Dockerfile`).
### Why are the changes needed?
`flake8` was already dead in the lint pipeline:
- `dev/lint-python` only ran flake8 behind an explicit `--flake8` flag; the
default run enables compile, custom-errors, `ruff`, and mypy. Nothing (CI,
`.pre-commit-config.yaml`, `dev/run-tests.py`) passes `--flake8`.
- The `_lint` dependency group in `pyproject.toml` no longer installs
flake8 - only `ruff` and `mypy`.
- `dev/tox.ini`'s flake8 configuration is duplicated by `[tool.ruff]` in
`pyproject.toml` (same `line-length = 100`, `E402` ignore, and
per-file-ignores).
Removing the dead code and config reduces confusion and drops an unused
dependency from the Docker images and docs workflow.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `bash -n dev/lint-python` to verify the script still parses.
- Confirmed `git grep -niI flake8` returns no matches after the change.
- The lint pipeline (`ruff`) continues to run via `dev/lint-python` and
pre-commit; the changed Docker images and workflow still install `ruff`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57258 from gaogaotiantian/remove-flake8.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
dev/create-release/spark-rm/Dockerfile | 2 +-
dev/lint-python | 48 +-------------------------
dev/run-tests.py | 2 +-
dev/spark-test-image/docs/Dockerfile | 2 +-
dev/spark-test-image/lint/Dockerfile | 1 -
dev/tox.ini | 62 ----------------------------------
6 files changed, 4 insertions(+), 113 deletions(-)
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index e3b1504fba1d..f34f008aa9a6 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -59,7 +59,7 @@ RUN python3.11 -m pip install --ignore-installed
'blinker>=1.6.2' && \
RUN python3.11 -m pip install 'sphinx==8.2.3' mkdocs
'pydata_sphinx_theme>=0.13' \
sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2' pyarrow
pandas \
- 'plotly>=4.8' 'docutils' 'flake8==3.9.0' 'mypy==1.19.1' 'pytest==7.1.3' \
+ 'plotly>=4.8' 'docutils' 'mypy==1.19.1' 'pytest==7.1.3' \
'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' 'pandas-stubs==1.2.0.53' \
'grpcio==1.76.0' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0'
diff --git a/dev/lint-python b/dev/lint-python
index efea64fe101a..fb785c13301d 100755
--- a/dev/lint-python
+++ b/dev/lint-python
@@ -16,8 +16,6 @@
# limitations under the License.
#
# define test binaries + versions
-FLAKE8_BUILD="flake8"
-MINIMUM_FLAKE8="3.9.0"
RUFF_BUILD="ruff"
MINIMUM_RUFF="0.14.0"
MINIMUM_MYPY="1.19.1"
@@ -46,9 +44,6 @@ while (( "$#" )); do
--custom-pyspark-error)
PYSPARK_CUSTOM_ERRORS_CHECK_TEST=true
;;
- --flake8)
- FLAKE8_TEST=true
- ;;
--ruff)
RUFF_TEST=true
;;
@@ -69,7 +64,7 @@ while (( "$#" )); do
shift
done
-if [[ -z
"$COMPILE_TEST$PYSPARK_CUSTOM_ERRORS_CHECK_TEST$FLAKE8_TEST$RUFF_TEST$MYPY_TEST$MYPY_EXAMPLES_TEST$MYPY_DATA_TEST"
]]; then
+if [[ -z
"$COMPILE_TEST$PYSPARK_CUSTOM_ERRORS_CHECK_TEST$RUFF_TEST$MYPY_TEST$MYPY_EXAMPLES_TEST$MYPY_DATA_TEST"
]]; then
COMPILE_TEST=true
PYSPARK_CUSTOM_ERRORS_CHECK_TEST=true
RUFF_TEST=true
@@ -234,44 +229,6 @@ function mypy_test {
}
-function flake8_test {
- local FLAKE8_VERSION=
- local EXPECTED_FLAKE8=
- local FLAKE8_REPORT=
- local FLAKE8_STATUS=
-
- if ! hash "$FLAKE8_BUILD" 2> /dev/null; then
- echo "The flake8 command was not found. Skipping for now."
- return
- fi
-
- _FLAKE8_VERSION=($($FLAKE8_BUILD --version))
- FLAKE8_VERSION="${_FLAKE8_VERSION[0]}"
- EXPECTED_FLAKE8="$(satisfies_min_version $FLAKE8_VERSION $MINIMUM_FLAKE8)"
-
- if [[ "$EXPECTED_FLAKE8" == "False" ]]; then
- echo "\
-The minimum flake8 version needs to be $MINIMUM_FLAKE8. Your current version
is $FLAKE8_VERSION
-
-flake8 checks failed."
- exit 1
- fi
-
- echo "starting $FLAKE8_BUILD test..."
- FLAKE8_REPORT=$( ($FLAKE8_BUILD --append-config dev/tox.ini --count
--show-source --statistics .) 2>&1)
- FLAKE8_STATUS=$?
-
- if [ "$FLAKE8_STATUS" -ne 0 ]; then
- echo "flake8 checks failed:"
- echo "$FLAKE8_REPORT"
- echo "$FLAKE8_STATUS"
- exit "$FLAKE8_STATUS"
- else
- echo "flake8 checks passed."
- echo
- fi
-}
-
function ruff_test {
local RUFF_VERSION=
local EXPECTED_RUFF=
@@ -355,9 +312,6 @@ fi
if [[ "$PYSPARK_CUSTOM_ERRORS_CHECK_TEST" == "true" ]]; then
pyspark_custom_errors_check_test
fi
-if [[ "$FLAKE8_TEST" == "true" ]]; then
- flake8_test
-fi
if [[ "$RUFF_TEST" == "true" ]]; then
ruff_test
fi
diff --git a/dev/run-tests.py b/dev/run-tests.py
index 685621193dd6..d50307513bb0 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -631,7 +631,7 @@ def main():
# Run SBT Checkstyle after the build to prevent a side-effect to
the build.
should_run_java_style_checks = True
if not changed_files or any(
- f.endswith("lint-python") or f.endswith("tox.ini") or
f.endswith(".py")
+ f.endswith("lint-python") or f.endswith("pyproject.toml") or
f.endswith(".py")
for f in changed_files
):
run_python_style_checks()
diff --git a/dev/spark-test-image/docs/Dockerfile
b/dev/spark-test-image/docs/Dockerfile
index fd4bc7f1e6ec..fdc364dbc690 100644
--- a/dev/spark-test-image/docs/Dockerfile
+++ b/dev/spark-test-image/docs/Dockerfile
@@ -93,6 +93,6 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# See 'docutils<0.18.0' in SPARK-39421
RUN python3.12 -m pip install 'sphinx==8.2.3' mkdocs
'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2
markupsafe \
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2'
'pyarrow>=23.0.0' 'pandas==2.3.3' 'plotly>=4.8' 'docutils' \
- 'flake8==3.9.0' 'mypy==1.19.1' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3'
'ruff==0.14.8' \
+ 'mypy==1.19.1' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' \
'pandas-stubs==1.2.0.53' 'grpcio==1.76.0' 'grpcio-status==1.76.0'
'protobuf==6.33.5' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0' \
&& python3.12 -m pip cache purge
diff --git a/dev/spark-test-image/lint/Dockerfile
b/dev/spark-test-image/lint/Dockerfile
index b7cecae654cb..f3968b0498d3 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -82,7 +82,6 @@ RUN python3.12 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN python3.12 -m pip install \
- 'flake8==3.9.0' \
'ruff==0.14.8' \
'googleapis-common-protos-stubs==2.2.0' \
'grpc-stubs==1.24.11' \
diff --git a/dev/tox.ini b/dev/tox.ini
deleted file mode 100644
index 721266ec2bfa..000000000000
--- a/dev/tox.ini
+++ /dev/null
@@ -1,62 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-[flake8]
-ignore =
- E203, # Skip as black formatter adds a whitespace around ':'.
- E402, # Module top level import is disabled for optional import check, etc.
- # There are too many instances to fix. Ignored for now.
- W503,
- W504,
-per-file-ignores =
- # E501 is ignored as shared.py is auto-generated.
- python/pyspark/ml/param/shared.py: E501,
- # E501 is ignored as we should keep the json string format in
error_classes.py.
- python/pyspark/errors/error_classes.py: E501,
- # Examples contain some unused variables.
- examples/src/main/python/sql/datasource.py: F841,
- # Exclude * imports in test files
- python/pyspark/errors/tests/*.py: F403,
- python/pyspark/logger/tests/*.py: F403,
- python/pyspark/logger/tests/connect/*.py: F403,
- python/pyspark/ml/tests/*.py: F403,
- python/pyspark/mllib/tests/*.py: F403,
- python/pyspark/pandas/tests/*.py: F401 F403,
- python/pyspark/pandas/tests/connect/*.py: F401 F403,
- python/pyspark/resource/tests/*.py: F403,
- python/pyspark/sql/tests/*.py: F403,
- python/pyspark/streaming/tests/*.py: F403,
- python/pyspark/tests/*.py: F403,
- python/pyspark/testing/*.py: F401,
- python/pyspark/testing/tests/*.py: F403
-exclude =
- */target/*,
- docs/.local_ruby_bundle/,
- *python/pyspark/cloudpickle/*.py,
- *python/pyspark/ml/deepspeed/tests/*.py
- *python/docs/build/*,
- *python/docs/source/conf.py,
- *python/.eggs/*,
- dist/*,
- .git/*,
- *python/pyspark/sql/pandas/functions.pyi,
- *python/pyspark/sql/column.pyi,
- *python/pyspark/worker.pyi,
- *python/pyspark/java_gateway.pyi,
- *python/pyspark/sql/connect/proto/*,
- *python/pyspark/sql/streaming/proto/*,
- */venv/*
-max-line-length = 100
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
