nchammas commented on a change in pull request #34655:
URL: https://github.com/apache/spark/pull/34655#discussion_r752775008
##########
File path: dev/lint-python
##########
@@ -275,8 +275,7 @@ SPARK_ROOT_DIR="$(dirname "${SCRIPT_DIR}")"
pushd "$SPARK_ROOT_DIR" &> /dev/null
-# skipping local ruby bundle directory from the search
-PYTHON_SOURCE="$(find . -path ./docs/.local_ruby_bundle -prune -false -o -name
"*.py")"
+PYTHON_SOURCE="$(git ls-files '*.py')"
Review comment:
This is an improvement, but I don't think this is that great, either. In
general, I don't like that we are building a huge string of every Python file
and passing it as an argument.
Instead, we should using the appropriate include and exclude filters (e.g.
via tox.ini) to capture everything to be tested. But I don't want to get into
that here.
##########
File path: dev/requirements.txt
##########
@@ -19,6 +19,7 @@ coverage
# Linter
mypy
+git+https://github.com/typeddjango/pytest-mypy-plugins.git@b0020061f48e85743ee3335bd62a3a608d17c6bd
Review comment:
This was being used [in
CI](https://github.com/apache/spark/blob/4bba110d9b66af8c0bafd26283ea92c733d76b53/.github/workflows/build_and_test.yml#L468)
but it was missing from here, so I added it in.
In a future PR, I want to reorganize things so that CI references the
requirements file. I don't think it's good that we have dependencies specified
here and then specified again inside our CI script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
