kevinjqliu opened a new issue, #2492: URL: https://github.com/apache/iceberg-python/issues/2492
### Feature Request / Improvement A common feedback ive heard about our testing infra is that the integration tests error when running on different environments. I think a big reason for that is how we run tests using spark. There are 2 places where we run spark queries: - [dev/provision.py](https://github.com/apache/iceberg-python/blob/main/dev/provision.py) - pytest fixture, [tests/conftest.py](https://github.com/apache/iceberg-python/blob/6935b4115557a0bf225de6d86dbfbbdb39a33618/tests/conftest.py#L2528-L2529) provision.py is ran inside a docker container so the env is consistent. pytests are ran locally and will install spark (via pyspark) and its packages on the machine running these tests. This step is brittle. #2491 changes pytest to use spark connect which will run all spark queries based on the spark server env that is inside the docker container. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
