HyukjinKwon opened a new pull request, #38991:
URL: https://github.com/apache/spark/pull/38991
### What changes were proposed in this pull request?
This PR proposes to:
- Print out the correct error message when dependencies are not installed
for `pyspark.sql.connect`
- Ignore all errors from tests:
- Initial intention was to annotate types for public APIs only.
- Also removed unused ignores in other modules
- Print out the correct message if 1. dependencies are missing or 2. build
is not there.
- Move `python/mypy.ini` to `dev/mypy.ini` to be consistentl with
`dev/tox.ini`
Note that, for Spark Connect, we should follow both below to make sure they
are properly skipped when dependences are not there, or jars do not exist.
- Always use `if should_test_connect` in `pyspark.sql.connect` imports in
tests.
- Use `@unittest.skipIf(not should_test_connect,
connect_requirement_message)`
### Why are the changes needed?
- To make the development easier and faster.
- To make sure our Jenkins build pass without Python dependencies installed
- See [Scaleaway](https://apache-spark.s3.fr-par.scw.cloud/index.html (),
see also
https://github.com/apache/spark-website/blob/asf-site/developer-tools.md#test-coverage)
### Does this PR introduce _any_ user-facing change?
No, dev and test-only.
### How was this patch tested?
Manually tested without pandas, pyarrow and grpc, and verified the error
messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
