kevinjqliu opened a new pull request, #4155:
URL: https://github.com/apache/iceberg-python/pull/4155
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Bump the integration test infra to the latest Iceberg Java release (1.12.0),
along with the related dependencies. Follows #3389.
| Component | File | Before | After |
|---|---|---|---|
| Iceberg (Spark runtime + AWS bundle) | `dev/spark/Dockerfile` | 1.11.0 |
1.12.0 |
| Spark base image | `dev/spark/Dockerfile` | 4.0.1 | 4.1.3 |
| Iceberg Spark runtime artifact | `dev/spark/Dockerfile` | `4.0_2.13` |
`4.1_2.13` |
| `hadoop-aws` | `dev/spark/Dockerfile` | 3.4.1 | 3.4.2 |
| AWS SDK bundle | `dev/spark/Dockerfile` | 2.24.6 | 2.57.3 |
| Hive metastore client | `dev/spark/Dockerfile` | 4.0.1 | 4.2.1 |
| `hive-standalone-metastore-client` | `dev/spark/Dockerfile` | — | 4.2.1
(new) |
| `amazon/aws-cli` | `dev/docker-compose*.yml` | 2.37.2 | 2.38.5 |
| `pyspark[connect]` | `pyproject.toml`, `uv.lock` | 4.0.1 | 4.1.3 |
- **Iceberg 1.12.0**: latest Java release; keeps integration tests on
current Java behavior.
- **Spark 4.1.3**: latest 4.1 patch; Iceberg 1.12.0 ships a `4.1_2.13`
runtime. Spark 4.2 has no Iceberg runtime yet.
- **`pyspark` 4.1.3**: the Spark Connect client must match the server
version.
- **`hadoop-aws` 3.4.2**: must match the Hadoop version bundled with Spark
4.1.3.
- **AWS SDK 2.57.3**: 2.24.6 was over two years old.
- **Hive metastore client 4.2.1**: matches the Hive 4.2.1 server image
(#3924).
- **`hive-standalone-metastore-client`**: Hive 4.2 moved `IMetaStoreClient`
out of `hive-standalone-metastore-common` into this new artifact; without it,
provisioning fails with `NoClassDefFoundError`.
- **aws-cli 2.38.5**: routine image refresh.
## Are these changes tested?
Yes, locally:
- `docker compose -f dev/docker-compose-integration.yml up -d --build
--wait` + `python dev/provision.py`: passed
- `pytest tests/ -m integration`: 1362 passed, 71 skipped
- `uv lock --check`: passed
## Are there any user-facing changes?
No, dev/test infra only.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
