This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new c3c6c55574 [#11293] feat(iceberg): Upgrade server-side Iceberg to
1.11.0 for REST service and lakehouse catalog (#11219)
c3c6c55574 is described below
commit c3c6c55574a844c421023ee61582e5fdb590a17e
Author: MaSai <[email protected]>
AuthorDate: Sat May 30 00:05:32 2026 +0800
[#11293] feat(iceberg): Upgrade server-side Iceberg to 1.11.0 for REST
service and lakehouse catalog (#11219)
### What changes were proposed in this pull request?
Upgrade Gravitino **server-side** Apache Iceberg from **1.10.1** to
**1.11.0** for the Iceberg REST Catalog (IRC) service and the lakehouse
Iceberg catalog. Spark/Flink connectors keep **per-engine** Iceberg
runtime versions; this PR aligns their IT classpath and test
expectations with the server upgrade.
**Iceberg REST and lakehouse**
- Bump `iceberg` in `gradle/libs.versions.toml` to `1.11.0`.
- Adapt IRC handlers to Iceberg 1.11 APIs (`CatalogHandlers.loadTable`
with `SnapshotMode.ALL`, namespace encode/decode via `RESTUtil` in
`IcebergRESTUtils`).
- Build `PlanTableScanResponse` with
`PlanTableScanResponse.builder().withFileScanTasks(...).withSpecsById(table.specs())`
(Iceberg 1.11 `file-scan-tasks` JSON).
- Remove legacy scan-plan endpoint workarounds; align
namespace/table/view operations and auth filters.
- Update lakehouse Iceberg unit/IT tests and docs
(`iceberg-rest-service.md`, `lakehouse-iceberg-catalog.md`).
**Spark and Flink connectors**
- Pin a single `icebergVersion` per engine module so compile/test
Iceberg jars match each engine's `iceberg-*-runtime`.
- Use Gravitino `iceberg-core` **1.11** on IT classpath where needed;
skip low-version Iceberg connector ITs in **embedded** mode (run in
**deploy** only) to avoid mixed Iceberg versions in one JVM.
- Adjust Spark Iceberg IT expectations (7 metadata columns on 1.11;
Spark 3.3 + Iceberg 1.8.x overrides for 5 columns).
**Other**
- Stabilize `web-v2` `MetalakePageTest` auxiliary window cleanup
(unrelated UI flake fix).
### Why are the changes needed?
Iceberg 1.11 updates REST catalog APIs and behavior. Staying on 1.10.1
blocks IRC/lakehouse alignment with 1.11 clients and the current REST
spec. Connector engines cannot all move to 1.11 in one step; IT and
classpath rules are updated so CI stays green while the server upgrades.
Fix: #11293
### Does this PR introduce _any_ user-facing change?
- **Yes (docs / server):** IRC and lakehouse Iceberg catalog
documentation reflect **Iceberg 1.11.0**. Plan-scan responses use
Iceberg 1.11 `file-scan-tasks` wire format via the standard builder
path.
- **Connectors:** User-facing connector Iceberg runtime versions are
unchanged; only test/CI behavior and embedded-vs-deploy IT coverage
differ.
- **No API break** intended for existing IRC clients beyond Iceberg 1.11
spec alignment.
### How was this patch tested?
- `./gradlew :iceberg:iceberg-rest-server:compileJava
:iceberg:iceberg-rest-server:compileTestJava -PskipITs`
- `./gradlew :iceberg:iceberg-rest-server:test -PskipITs --tests
org.apache.gravitino.iceberg.service.TestCatalogWrapperForREST --tests
org.apache.gravitino.iceberg.service.rest.TestScanPlanCache`
- Lakehouse Iceberg catalog tests; Spark/Flink Iceberg connector ITs
(deploy mode for version-sensitive cases)
- CI: BackendIT, SparkIT, FlinkIT, build, and related workflows
---
.../lakehouse/iceberg/TestIcebergTable.java | 15 ++--
.../iceberg/converter/TestConvertUtil.java | 16 ++--
.../integration/test/CatalogIcebergBaseIT.java | 13 ++--
docs/iceberg-rest-service.md | 8 +-
docs/lakehouse-iceberg-catalog.md | 2 +-
.../test/iceberg/FlinkIcebergHiveCatalogIT118.java | 5 ++
.../test/iceberg/FlinkIcebergRestCatalogIT118.java | 6 ++
.../test/iceberg/FlinkIcebergHiveCatalogIT119.java | 5 ++
.../test/iceberg/FlinkIcebergRestCatalogIT119.java | 6 ++
gradle/libs.versions.toml | 11 ++-
.../iceberg/common/ops/IcebergCatalogWrapper.java | 4 +-
.../iceberg/service/CatalogWrapperForREST.java | 91 ++++++++++------------
.../iceberg/service/IcebergRESTUtils.java | 9 +++
.../service/rest/IcebergConfigOperations.java | 8 +-
.../service/rest/IcebergNamespaceOperations.java | 20 +++--
.../service/rest/IcebergTableOperations.java | 27 ++++---
.../service/rest/IcebergViewOperations.java | 18 +++--
.../web/filter/IcebergLoadAuthzHandlerHelper.java | 5 +-
...bergMetadataAuthorizationMethodInterceptor.java | 4 +-
.../service/rest/IcebergNamespaceTestBase.java | 7 +-
.../iceberg/service/rest/IcebergRestTestUtil.java | 7 +-
.../iceberg/service/rest/IcebergTestBase.java | 29 +++++--
.../iceberg/service/rest/TestIcebergConfig.java | 6 +-
.../rest/TestIcebergNamespaceOperations.java | 7 +-
.../service/rest/TestIcebergTableOperations.java | 23 ++++--
.../iceberg/service/rest/TestScanPlanCache.java | 18 ++---
...bergMetadataAuthorizationMethodInterceptor.java | 9 ++-
maintenance/jobs/build.gradle.kts | 2 +-
maintenance/optimizer/build.gradle.kts | 2 +-
spark-connector/spark-common/build.gradle.kts | 2 +-
.../connector/integration/test/SparkCommonIT.java | 2 +-
.../test/iceberg/SparkIcebergCatalogIT.java | 6 +-
spark-connector/v3.3/spark/build.gradle.kts | 2 +-
.../SparkIcebergCatalogHiveBackendIT33.java | 33 ++++++++
.../SparkIcebergCatalogRestBackendIT33.java | 31 ++++++++
spark-connector/v3.4/spark/build.gradle.kts | 2 +-
spark-connector/v3.5/spark/build.gradle.kts | 2 +-
37 files changed, 316 insertions(+), 147 deletions(-)
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergTable.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergTable.java
index 3cff762850..5b1b0a2874 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergTable.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergTable.java
@@ -32,7 +32,6 @@ import java.util.List;
import java.util.Map;
import java.util.UUID;
import org.apache.commons.lang3.ArrayUtils;
-import org.apache.commons.lang3.StringUtils;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.catalog.PropertiesMetadataHelpers;
@@ -663,18 +662,20 @@ public class TestIcebergTable {
IllegalArgumentException.class,
() -> icebergTable.transformDistribution(Distributions.HASH));
Assertions.assertTrue(
- StringUtils.contains(
- illegalArgumentException.getMessage(),
- "Iceberg's Distribution Mode.HASH is distributed based on
partition, but the partition is empty"));
+ illegalArgumentException
+ .getMessage()
+ .contains(
+ "Iceberg's Distribution Mode.HASH is distributed based on
partition, but the partition is empty"));
illegalArgumentException =
Assertions.assertThrows(
IllegalArgumentException.class,
() -> icebergTable.transformDistribution(Distributions.RANGE));
Assertions.assertTrue(
- StringUtils.contains(
- illegalArgumentException.getMessage(),
- "Iceberg's Distribution Mode.RANGE is distributed based on
sortOrder or partition, but both are empty"));
+ illegalArgumentException
+ .getMessage()
+ .contains(
+ "Iceberg's Distribution Mode.RANGE is distributed based on
sortOrder or partition, but both are empty"));
IcebergTable newTable =
IcebergTable.builder()
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
index 28952ae218..f9099666d4 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
@@ -460,8 +460,8 @@ public class TestConvertUtil extends TestBaseConvert {
@Test
public void testFromNestedField() {
- String colName = RandomStringUtils.randomAlphabetic(10);
- String doc = RandomStringUtils.randomAlphabetic(20);
+ String colName = RandomStringUtils.secure().nextAlphabetic(10);
+ String doc = RandomStringUtils.secure().nextAlphabetic(20);
Types.NestedField colField =
Types.NestedField.optional(1, colName, Types.IntegerType.get(), doc);
IcebergColumn icebergColumn = ConvertUtil.fromNestedField(colField);
@@ -471,8 +471,8 @@ public class TestConvertUtil extends TestBaseConvert {
Assertions.assertTrue(
icebergColumn.dataType() instanceof
org.apache.gravitino.rel.types.Types.IntegerType);
- colName = RandomStringUtils.randomAlphabetic(10);
- doc = RandomStringUtils.randomAlphabetic(20);
+ colName = RandomStringUtils.secure().nextAlphabetic(10);
+ doc = RandomStringUtils.secure().nextAlphabetic(20);
colField = Types.NestedField.required(1, colName, Types.StringType.get(),
doc);
icebergColumn = ConvertUtil.fromNestedField(colField);
Assertions.assertEquals(icebergColumn.name(), colName);
@@ -481,8 +481,8 @@ public class TestConvertUtil extends TestBaseConvert {
Assertions.assertTrue(
icebergColumn.dataType() instanceof
org.apache.gravitino.rel.types.Types.StringType);
- colName = RandomStringUtils.randomAlphabetic(10);
- doc = RandomStringUtils.randomAlphabetic(20);
+ colName = RandomStringUtils.secure().nextAlphabetic(10);
+ doc = RandomStringUtils.secure().nextAlphabetic(20);
colField = Types.NestedField.required(1, colName, Types.DateType.get(),
doc);
icebergColumn = ConvertUtil.fromNestedField(colField);
Assertions.assertEquals(icebergColumn.name(), colName);
@@ -491,8 +491,8 @@ public class TestConvertUtil extends TestBaseConvert {
Assertions.assertTrue(
icebergColumn.dataType() instanceof
org.apache.gravitino.rel.types.Types.DateType);
- colName = RandomStringUtils.randomAlphabetic(10);
- doc = RandomStringUtils.randomAlphabetic(20);
+ colName = RandomStringUtils.secure().nextAlphabetic(10);
+ doc = RandomStringUtils.secure().nextAlphabetic(20);
colField =
Types.NestedField.required(
1,
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
index a91a6aa0bd..f4748d82ad 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
@@ -40,7 +40,6 @@ import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
import org.apache.commons.lang3.ArrayUtils;
-import org.apache.commons.lang3.StringUtils;
import org.apache.gravitino.Catalog;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
@@ -1128,9 +1127,9 @@ public abstract class CatalogIcebergBaseIT extends BaseIT
{
sortOrders);
});
Assertions.assertTrue(
- StringUtils.contains(
- illegalArgumentException.getMessage(),
- "Iceberg's Distribution Mode.HASH does not support set
expressions."));
+ illegalArgumentException
+ .getMessage()
+ .contains("Iceberg's Distribution Mode.HASH does not support set
expressions."));
distribution = Distributions.RANGE;
// Create a data table for Distributions.hash
@@ -1173,9 +1172,9 @@ public abstract class CatalogIcebergBaseIT extends BaseIT
{
sortOrders);
});
Assertions.assertTrue(
- StringUtils.contains(
- illegalArgumentException.getMessage(),
- "Iceberg's Distribution Mode.RANGE not support set expressions."));
+ illegalArgumentException
+ .getMessage()
+ .contains("Iceberg's Distribution Mode.RANGE not support set
expressions."));
}
@Test
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index d623fba839..d050e6d21e 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -19,7 +19,7 @@ There are some key difference between Gravitino Iceberg REST
server and Gravitin
### Capabilities
-- Supports the Apache Iceberg REST API defined in Iceberg 1.10, and supports
most namespace, table and view interfaces. The following interfaces are not
implemented yet:
+- Supports the Apache Iceberg REST API defined in Iceberg 1.11, and supports
most namespace, table and view interfaces. The following interfaces are not
implemented yet:
- multi table transaction
- pagination
- register view
@@ -310,7 +310,7 @@ Please refer the following configuration If you are using
Spark to access Iceber
```shell
./bin/spark-sql -v \
---conf
spark.jars=/Users/fanng/deploy/demo/jars/iceberg-spark-runtime-3.5_2.12-1.10.0.jar
\
+--conf
spark.jars=/Users/fanng/deploy/demo/jars/iceberg-spark-runtime-3.5_2.12-1.11.0.jar
\
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.rest.rest.auth.type=oauth2 \
@@ -520,7 +520,7 @@ View operations are supported when using the JDBC catalog
backend with schema ve
### Other Apache Iceberg catalog properties
-You can add other properties defined in [Iceberg catalog
properties](https://iceberg.apache.org/docs/1.10.0/configuration/#catalog-properties).
+You can add other properties defined in [Iceberg catalog
properties](https://iceberg.apache.org/docs/1.11.0/configuration/#catalog-properties).
The `clients` property for example:
| Configuration item | Description |
Default value | Required |
@@ -576,6 +576,8 @@ Gravitino provides the build-in
`org.apache.gravitino.iceberg.common.cache.Local
Gravitino caches scan plan results to speed up repeated queries with identical
parameters. The cache uses snapshot ID as part of the cache key, so queries
against different snapshots will not use stale cached data.
+Plan scan responses follow the Iceberg 1.11 REST API: completed plans return
structured `file-scan-tasks` only. Legacy `plan-tasks` JSON strings (used by
some Iceberg 1.9.x–1.10.x clients) are not emitted.
+
| Configuration item | Description
| Default value | Required | Since
Version |
|------------------------------------------------------------|----------------------------------------------------------|---------------|----------|---------------|
| `gravitino.iceberg-rest.scan-plan-cache-impl` | The
implementation of the scan plan cache. | (none) | No
| 1.2.0 |
diff --git a/docs/lakehouse-iceberg-catalog.md
b/docs/lakehouse-iceberg-catalog.md
index adf142e74a..203b055048 100644
--- a/docs/lakehouse-iceberg-catalog.md
+++ b/docs/lakehouse-iceberg-catalog.md
@@ -18,7 +18,7 @@ Apache Gravitino provides the ability to manage Apache
Iceberg metadata.
### Requirements and limitations
:::info
-Builds with Apache Iceberg `1.10.0`. The Apache Iceberg table format version
is `2` by default.
+Builds with Apache Iceberg `1.11.0`. The Apache Iceberg table format version
is `2` by default.
:::
## Catalog
diff --git
a/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT118.java
b/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT118.java
index cf4caaa306..7db8604055 100644
---
a/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT118.java
+++
b/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT118.java
@@ -19,4 +19,9 @@
package org.apache.gravitino.flink.connector.integration.test.iceberg;
+import org.junit.jupiter.api.condition.DisabledIf;
+
+// Flink 1.18 uses Iceberg 1.9.x; lakehouse-iceberg (1.11) runs in embedded
MiniGravitino in the
+// same JVM. Run Hive-backend Iceberg IT in deploy mode only.
+@DisabledIf("org.apache.gravitino.integration.test.util.ITUtils#isEmbedded")
public class FlinkIcebergHiveCatalogIT118 extends FlinkIcebergHiveCatalogIT {}
diff --git
a/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT118.java
b/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT118.java
index 94c0803035..24a0802edf 100644
---
a/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT118.java
+++
b/flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT118.java
@@ -19,4 +19,10 @@
package org.apache.gravitino.flink.connector.integration.test.iceberg;
+import org.junit.jupiter.api.condition.DisabledIf;
+
+// Flink 1.18 uses Iceberg 1.9.x; lakehouse-iceberg (1.11) runs in embedded
MiniGravitino in the
+// same JVM. Run REST-catalog Iceberg IT in deploy mode only. @DisabledIf is
not @Inherited, so
+// each concrete subclass must declare it explicitly.
+@DisabledIf("org.apache.gravitino.integration.test.util.ITUtils#isEmbedded")
public class FlinkIcebergRestCatalogIT118 extends FlinkIcebergRestCatalogIT {}
diff --git
a/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT119.java
b/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT119.java
index 9a744e02e8..7193642787 100644
---
a/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT119.java
+++
b/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT119.java
@@ -19,4 +19,9 @@
package org.apache.gravitino.flink.connector.integration.test.iceberg;
+import org.junit.jupiter.api.condition.DisabledIf;
+
+// Flink 1.19 uses Iceberg 1.10.x; lakehouse-iceberg (1.11) runs in embedded
MiniGravitino in the
+// same JVM. Run Hive-backend Iceberg IT in deploy mode only.
+@DisabledIf("org.apache.gravitino.integration.test.util.ITUtils#isEmbedded")
public class FlinkIcebergHiveCatalogIT119 extends FlinkIcebergHiveCatalogIT {}
diff --git
a/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT119.java
b/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT119.java
index b84263ea0f..948c74eda5 100644
---
a/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT119.java
+++
b/flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT119.java
@@ -19,4 +19,10 @@
package org.apache.gravitino.flink.connector.integration.test.iceberg;
+import org.junit.jupiter.api.condition.DisabledIf;
+
+// Flink 1.19 uses Iceberg 1.10.x; lakehouse-iceberg (1.11) runs in embedded
MiniGravitino in the
+// same JVM. Run REST-catalog Iceberg IT in deploy mode only. @DisabledIf is
not @Inherited, so
+// each concrete subclass must declare it explicitly.
+@DisabledIf("org.apache.gravitino.integration.test.util.ITUtils#isEmbedded")
public class FlinkIcebergRestCatalogIT119 extends FlinkIcebergRestCatalogIT {}
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
index 39c7a730e8..df44277b4a 100644
--- a/gradle/libs.versions.toml
+++ b/gradle/libs.versions.toml
@@ -65,8 +65,11 @@ commons-beanutils = "1.11.0"
commons-configuration1 = "1.6"
commons-dbcp2 = "2.11.0"
caffeine = "2.9.3"
-iceberg = '1.10.1' # used for Gravitino Iceberg catalog and Iceberg REST
service
-iceberg4connector = "1.6.1" # used for compile connectors like Spark, Flink,
etc
+iceberg = '1.11.0'
+# Last Iceberg release compatible with Spark 3.3; Spark 3.3 deploy ITs cover
basic CRUD only.
+iceberg4spark33 = "1.8.1"
+iceberg4spark34 = "1.11.0"
+iceberg4spark35 = "1.11.0"
paimon = '1.2.0'
spark33 = "3.3.4"
spark34 = "3.4.3"
@@ -110,8 +113,8 @@ flinkjdbc18 = "3.2.0-1.18"
flinkjdbc119 = "3.3.0-1.19"
flinkjdbc120 = "3.3.0-1.20"
iceberg4flink18 = "1.9.2"
-iceberg4flink119 = "1.9.2"
-iceberg4flink120 = "1.9.2"
+iceberg4flink119 = "1.10.2"
+iceberg4flink120 = "1.11.0"
paimon4flink18 = "1.2.0"
paimon4flink119 = "1.2.0"
paimon4flink120 = "1.2.0"
diff --git
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergCatalogWrapper.java
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergCatalogWrapper.java
index 0ab9f215f4..987fd329f6 100644
---
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergCatalogWrapper.java
+++
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergCatalogWrapper.java
@@ -47,6 +47,7 @@ import org.apache.iceberg.catalog.ViewCatalog;
import org.apache.iceberg.jdbc.JdbcCatalogWithMetadataLocationSupport;
import org.apache.iceberg.rest.CatalogHandlers;
import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.rest.RESTCatalogProperties.SnapshotMode;
import org.apache.iceberg.rest.requests.CreateNamespaceRequest;
import org.apache.iceberg.rest.requests.CreateTableRequest;
import org.apache.iceberg.rest.requests.CreateViewRequest;
@@ -246,7 +247,8 @@ public class IcebergCatalogWrapper implements AutoCloseable
{
return
LoadTableResponse.builder().withTableMetadata(tableMetadataOptional.get()).build();
}
- LoadTableResponse loadTableResponse =
CatalogHandlers.loadTable(getCatalog(), tableIdentifier);
+ LoadTableResponse loadTableResponse =
+ CatalogHandlers.loadTable(getCatalog(), tableIdentifier,
SnapshotMode.ALL);
if (loadTableResponse != null) {
getMetadataCache().updateTableMetadata(tableIdentifier,
loadTableResponse.tableMetadata());
}
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/CatalogWrapperForREST.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/CatalogWrapperForREST.java
index 264ec3c994..39fc375d7b 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/CatalogWrapperForREST.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/CatalogWrapperForREST.java
@@ -57,13 +57,11 @@ import org.apache.gravitino.utils.PrincipalUtils;
import org.apache.iceberg.BaseMetadataTable;
import org.apache.iceberg.BaseTable;
import org.apache.iceberg.BaseTransaction;
-import org.apache.iceberg.DeleteFile;
import org.apache.iceberg.FileScanTask;
import org.apache.iceberg.IncrementalAppendScan;
import org.apache.iceberg.MetadataUpdate;
import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Scan;
-import org.apache.iceberg.ScanTaskParser;
import org.apache.iceberg.Schema;
import org.apache.iceberg.SortOrder;
import org.apache.iceberg.Table;
@@ -419,16 +417,18 @@ public class CatalogWrapperForREST extends
IcebergCatalogWrapper {
* <p>This method performs server-side scan planning to optimize query
performance by reducing
* client-side metadata loading and enabling parallel task execution.
*
- * <p>Implementation uses synchronous scan planning (COMPLETED status) where
tasks are returned
- * immediately as serialized JSON strings. This is different from
asynchronous mode (SUBMITTED
- * status) where a plan ID is returned for later retrieval.
+ * <p>Implementation uses synchronous scan planning (COMPLETED status) and
returns structured
+ * {@code file-scan-tasks} per the Iceberg 1.11 REST spec. It does not emit
legacy {@code
+ * plan-tasks} JSON strings, so clients built for Iceberg < 1.11 are not
supported. This is
+ * different from asynchronous mode (SUBMITTED status) where a plan ID is
returned for later
+ * retrieval.
*
* <p>Referenced from Iceberg PR #13400 for scan planning implementation.
*
* @param tableIdentifier The table identifier.
* @param scanRequest The scan request parameters including filters,
projections, snapshot-id,
* etc.
- * @return PlanTableScanResponse with status=COMPLETED and serialized
planTasks.
+ * @return PlanTableScanResponse with status=COMPLETED and file scan tasks.
* @throws IllegalArgumentException if scan request validation fails
* @throws org.apache.gravitino.exceptions.NoSuchTableException if table
doesn't exist
* @throws RuntimeException for other scan planning failures
@@ -454,63 +454,36 @@ public class CatalogWrapperForREST extends
IcebergCatalogWrapper {
return cachedResponse.get();
}
- List<String> planTasks = new ArrayList<>();
- Map<Integer, PartitionSpec> specsById = new HashMap<>();
- List<DeleteFile> deleteFiles = new ArrayList<>();
+ List<FileScanTask> fileScanTasks = new ArrayList<>();
- try (CloseableIterable<FileScanTask> fileScanTasks =
+ try (CloseableIterable<FileScanTask> scanTasks =
createFilePlanScanTasks(table, tableIdentifier, scanRequest)) {
- for (FileScanTask fileScanTask : fileScanTasks) {
- try {
- String taskString = ScanTaskParser.toJson(fileScanTask);
- planTasks.add(taskString);
-
- int specId = fileScanTask.spec().specId();
- if (!specsById.containsKey(specId)) {
- specsById.put(specId, fileScanTask.spec());
- }
-
- if (!fileScanTask.deletes().isEmpty()) {
- deleteFiles.addAll(fileScanTask.deletes());
- }
- } catch (Exception e) {
- throw new RuntimeException(
- String.format(
- "Failed to serialize scan task for table: %s. Error: %s",
- tableIdentifier, e.getMessage()),
- e);
- }
+ for (FileScanTask fileScanTask : scanTasks) {
+ fileScanTasks.add(fileScanTask);
}
} catch (IOException e) {
LOG.error("Failed to close scan task iterator for table: {}",
tableIdentifier, e);
throw new RuntimeException("Failed to plan scan tasks: " +
e.getMessage(), e);
}
- List<DeleteFile> uniqueDeleteFiles =
- deleteFiles.stream().distinct().collect(Collectors.toList());
-
- if (planTasks.isEmpty()) {
+ if (fileScanTasks.isEmpty()) {
LOG.info(
"Scan planning returned no tasks for table: {}. Table may be empty
or fully filtered.",
tableIdentifier);
}
- PlanTableScanResponse.Builder responseBuilder =
- PlanTableScanResponse.builder()
- .withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(planTasks)
- .withSpecsById(specsById);
-
- if (!uniqueDeleteFiles.isEmpty()) {
- responseBuilder.withDeleteFiles(uniqueDeleteFiles);
- LOG.debug(
- "Included {} delete files in scan plan for table: {}",
- uniqueDeleteFiles.size(),
- tableIdentifier);
+ PlanTableScanResponse response;
+ try {
+ response = buildCompletedPlanTableScanResponse(table, fileScanTasks);
+ } catch (Exception e) {
+ LOG.error("Failed to build scan plan response for table: {}",
tableIdentifier, e);
+ throw new RuntimeException(
+ String.format(
+ "Failed to build scan plan response for table: %s. Error: %s",
+ tableIdentifier, e.getMessage()),
+ e);
}
- PlanTableScanResponse response = responseBuilder.build();
-
// Cache the scan plan response
scanPlanCache.put(ScanPlanCacheKey.create(tableIdentifier, table,
scanRequest), response);
return response;
@@ -528,6 +501,28 @@ public class CatalogWrapperForREST extends
IcebergCatalogWrapper {
}
}
+ /**
+ * Builds a synchronous COMPLETED scan plan response for Iceberg 1.11+ REST
clients only.
+ *
+ * <p>Matches {@link CatalogHandlers#planTableScan}: {@code file-scan-tasks}
plus {@code
+ * specs-by-id} from {@link Table#specs()}. Does not populate legacy {@code
plan-tasks} JSON
+ * strings.
+ *
+ * <p>{@code specs-by-id} uses the table's full spec map ({@link
Table#specs()}), not only
+ * partition specs referenced by the returned {@code fileScanTasks}. That
matches Iceberg 1.11
+ * REST behavior and may include historical specs from prior partition
evolution, including when a
+ * filtered scan returns zero tasks.
+ */
+ @SuppressWarnings("deprecation")
+ private static PlanTableScanResponse buildCompletedPlanTableScanResponse(
+ Table table, List<FileScanTask> fileScanTasks) {
+ return PlanTableScanResponse.builder()
+ .withPlanStatus(PlanStatus.COMPLETED)
+ .withFileScanTasks(fileScanTasks)
+ .withSpecsById(table.specs())
+ .build();
+ }
+
/**
* Create and plan a scan based on the scan request.
*
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergRESTUtils.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergRESTUtils.java
index 42ccd94ee4..c095ac231b 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergRESTUtils.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergRESTUtils.java
@@ -43,6 +43,7 @@ import org.apache.gravitino.NameIdentifier;
import
org.apache.gravitino.iceberg.service.authorization.IcebergRESTServerContext;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.rest.RESTUtil;
import org.apache.iceberg.rest.responses.ErrorResponse;
import org.apache.iceberg.rest.responses.LoadTableResponse;
import org.slf4j.Logger;
@@ -52,6 +53,14 @@ public class IcebergRESTUtils {
private static final Logger LOG =
LoggerFactory.getLogger(IcebergRESTUtils.class);
+ /**
+ * URL-encoded UTF-8 namespace separator ({@code 0x1F}) required by the
Iceberg REST catalog spec.
+ *
+ * <p>Iceberg 1.11 allows a configurable separator in {@link
RESTUtil#decodeNamespace}; use this
+ * value so decoding stays aligned with the wire format used in existing
deployments and tests.
+ */
+ public static final String NAMESPACE_SEPARATOR_URLENCODED_UTF_8 = "%1F";
+
public static final String SNAPSHOT_ALL = "all";
public static final String SNAPSHOT_REFS = "refs";
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergConfigOperations.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergConfigOperations.java
index f8c0c28036..719d75ce93 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergConfigOperations.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergConfigOperations.java
@@ -57,12 +57,6 @@ public class IcebergConfigOperations {
private final IcebergCatalogWrapperManager catalogWrapperManager;
- // TODO: Iceberg 1.10.1's Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN uses a broken
path that is missing
- // the namespaces/{namespace} segment (fixed in apache/iceberg#14120,
targeting 1.11.x).
- // We override it here with the correct namespace-scoped path until we
upgrade.
- private static final Endpoint V1_SUBMIT_TABLE_SCAN_PLAN =
- Endpoint.create("POST",
"/v1/{prefix}/namespaces/{namespace}/tables/{table}/plan");
-
private static final List<Endpoint> DEFAULT_ENDPOINTS =
ImmutableList.<Endpoint>builder()
.add(Endpoint.V1_LIST_NAMESPACES)
@@ -81,7 +75,7 @@ public class IcebergConfigOperations {
.add(Endpoint.V1_REGISTER_TABLE)
.add(Endpoint.V1_REPORT_METRICS)
.add(Endpoint.V1_TABLE_CREDENTIALS)
- .add(V1_SUBMIT_TABLE_SCAN_PLAN)
+ .add(Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN)
.build();
private static final List<Endpoint> DEFAULT_VIEW_ENDPOINTS =
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceOperations.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceOperations.java
index 3f486859dc..ca5807e2f4 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceOperations.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceOperations.java
@@ -104,7 +104,10 @@ public class IcebergNamespaceOperations {
@AuthorizationMetadata(type = Entity.EntityType.CATALOG)
@PathParam("prefix") String prefix) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
Namespace parentNamespace =
- parent.isEmpty() ? Namespace.empty() :
RESTUtil.decodeNamespace(parent);
+ parent.isEmpty()
+ ? Namespace.empty()
+ : RESTUtil.decodeNamespace(
+ parent, IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info(
"List Iceberg namespaces, catalog: {}, parentNamespace: {}",
catalogName, parentNamespace);
try {
@@ -142,7 +145,8 @@ public class IcebergNamespaceOperations {
@AuthorizationMetadata(type = Entity.EntityType.SCHEMA) @Encoded()
@PathParam("namespace")
String namespace) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info("Load Iceberg namespace, catalog: {}, namespace: {}",
catalogName, icebergNS);
try {
return Utils.doAs(
@@ -173,7 +177,8 @@ public class IcebergNamespaceOperations {
@AuthorizationMetadata(type = Entity.EntityType.SCHEMA) @Encoded()
@PathParam("namespace")
String namespace) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info("Check Iceberg namespace exists, catalog: {}, namespace: {}",
catalogName, icebergNS);
try {
return Utils.doAs(
@@ -207,7 +212,8 @@ public class IcebergNamespaceOperations {
String namespace) {
// todo check if table exists in namespace after table ops is added
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info("Drop Iceberg namespace, catalog: {}, namespace: {}",
catalogName, icebergNS);
try {
return Utils.doAs(
@@ -269,7 +275,8 @@ public class IcebergNamespaceOperations {
String namespace,
UpdateNamespacePropertiesRequest updateNamespacePropertiesRequest) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info(
"Update Iceberg namespace, catalog: {}, namespace: {},
updateNamespacePropertiesRequest: {}",
catalogName,
@@ -310,7 +317,8 @@ public class IcebergNamespaceOperations {
@HeaderParam(IcebergTableOperations.X_ICEBERG_ACCESS_DELEGATION) String
accessDelegation) {
boolean isCredentialVending =
IcebergTableOperations.isCredentialVending(accessDelegation);
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info(
"Register Iceberg table, catalog: {}, namespace: {},
registerTableRequest: {}, "
+ "accessDelegation: {}, isCredentialVending: {}",
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergTableOperations.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergTableOperations.java
index 96c8a861ec..a3eec15dd0 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergTableOperations.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergTableOperations.java
@@ -125,7 +125,8 @@ public class IcebergTableOperations {
@AuthorizationMetadata(type = EntityType.SCHEMA) @Encoded()
@PathParam("namespace")
String namespace) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info("List Iceberg tables, catalog: {}, namespace: {}", catalogName,
icebergNS);
try {
return Utils.doAs(
@@ -167,7 +168,8 @@ public class IcebergTableOperations {
@HeaderParam(X_ICEBERG_ACCESS_DELEGATION) String accessDelegation) {
boolean isCredentialVending = isCredentialVending(accessDelegation);
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info(
"Create Iceberg table, catalog: {}, namespace: {}, create table
request: {}, "
+ "accessDelegation: {}, isCredentialVending: {}",
@@ -210,7 +212,8 @@ public class IcebergTableOperations {
String table,
UpdateTableRequest updateTableRequest) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
if (LOG.isInfoEnabled()) {
LOG.info(
@@ -255,7 +258,8 @@ public class IcebergTableOperations {
String table,
@DefaultValue("false") @QueryParam("purgeRequested") boolean
purgeRequested) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
LOG.info(
"Drop Iceberg table, catalog: {}, namespace: {}, table: {},
purgeRequested: {}",
@@ -298,7 +302,8 @@ public class IcebergTableOperations {
@HeaderParam(X_ICEBERG_ACCESS_DELEGATION) String accessDelegation,
@HeaderParam(IF_NONE_MATCH) String ifNoneMatch) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
boolean isCredentialVending = isCredentialVending(accessDelegation);
LOG.info(
@@ -361,7 +366,8 @@ public class IcebergTableOperations {
@AuthorizationMetadata(type = Entity.EntityType.TABLE) @Encoded()
@PathParam("table")
String table) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
LOG.info(
"Check Iceberg table exists, catalog: {}, namespace: {}, table: {}",
@@ -398,7 +404,8 @@ public class IcebergTableOperations {
@Encoded() @PathParam("table") String table,
ReportMetricsRequest request) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
LOG.info(
"Report Iceberg table metrics, catalog: {}, namespace: {}, table: {}",
@@ -456,7 +463,8 @@ public class IcebergTableOperations {
String namespace,
@AuthorizationMetadata(type = EntityType.TABLE) @Encoded()
@PathParam("table") String table) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
LOG.info(
"Get Iceberg table credentials, catalog: {}, namespace: {}, table: {}",
@@ -511,7 +519,8 @@ public class IcebergTableOperations {
@Encoded() @PathParam("table") @AuthorizationMetadata(type =
EntityType.TABLE) String table,
PlanTableScanRequest scanRequest) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String tableName = RESTUtil.decodeString(table);
LOG.info(
"Plan table scan, catalog: {}, namespace: {}, table: {}",
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergViewOperations.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergViewOperations.java
index 3685333162..ff01399e0a 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergViewOperations.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergViewOperations.java
@@ -99,7 +99,8 @@ public class IcebergViewOperations {
@AuthorizationMetadata(type = EntityType.SCHEMA) @Encoded()
@PathParam("namespace")
String namespace) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info("List Iceberg views, catalog: {}, namespace: {}", catalogName,
icebergNS);
try {
return Utils.doAs(
@@ -136,7 +137,8 @@ public class IcebergViewOperations {
String namespace,
CreateViewRequest createViewRequest) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
LOG.info(
"Create Iceberg view, catalog: {}, namespace: {}, createViewRequest:
{}",
catalogName,
@@ -178,7 +180,8 @@ public class IcebergViewOperations {
@PathParam("view")
String view) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String viewName = RESTUtil.decodeString(view);
LOG.info(
"Load Iceberg view, catalog: {}, namespace: {}, view: {}",
@@ -216,7 +219,8 @@ public class IcebergViewOperations {
@AuthorizationMetadata(type = EntityType.VIEW) @Encoded()
@PathParam("view") String view,
UpdateTableRequest replaceViewRequest) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String viewName = RESTUtil.decodeString(view);
LOG.info(
"Replace Iceberg view, catalog: {}, namespace: {}, view: {},
replaceViewRequest: {}",
@@ -254,7 +258,8 @@ public class IcebergViewOperations {
String namespace,
@AuthorizationMetadata(type = EntityType.VIEW) @Encoded()
@PathParam("view") String view) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String viewName = RESTUtil.decodeString(view);
LOG.info(
"Drop Iceberg view, catalog: {}, namespace: {}, view: {}",
@@ -290,7 +295,8 @@ public class IcebergViewOperations {
String namespace,
@AuthorizationMetadata(type = EntityType.VIEW) @Encoded()
@PathParam("view") String view) {
String catalogName = IcebergRESTUtils.getCatalogName(prefix);
- Namespace icebergNS = RESTUtil.decodeNamespace(namespace);
+ Namespace icebergNS =
+ RESTUtil.decodeNamespace(namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
String viewName = RESTUtil.decodeString(view);
LOG.info(
"Check Iceberg view exists, catalog: {}, namespace: {}, view: {}",
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergLoadAuthzHandlerHelper.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergLoadAuthzHandlerHelper.java
index 3f18f82d6c..fc03d8aa65 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergLoadAuthzHandlerHelper.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergLoadAuthzHandlerHelper.java
@@ -29,6 +29,7 @@ import org.apache.gravitino.Entity.EntityType;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.authorization.AuthorizationRequestContext;
import org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper;
+import org.apache.gravitino.iceberg.service.IcebergRESTUtils;
import
org.apache.gravitino.iceberg.service.authorization.IcebergRESTServerContext;
import
org.apache.gravitino.server.authorization.annotations.AuthorizationExpression;
import
org.apache.gravitino.server.authorization.annotations.AuthorizationMetadata;
@@ -61,7 +62,9 @@ final class IcebergLoadAuthzHandlerHelper {
AuthorizationMetadata authMetadata =
parameter.getAnnotation(AuthorizationMetadata.class);
if (authMetadata != null && authMetadata.type() == EntityType.SCHEMA) {
- namespace = RESTUtil.decodeNamespace(String.valueOf(args[i]));
+ namespace =
+ RESTUtil.decodeNamespace(
+ String.valueOf(args[i]),
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
}
}
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergMetadataAuthorizationMethodInterceptor.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergMetadataAuthorizationMethodInterceptor.java
index 43ab6d6c03..17f5076eb6 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergMetadataAuthorizationMethodInterceptor.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergMetadataAuthorizationMethodInterceptor.java
@@ -72,7 +72,9 @@ public class IcebergMetadataAuthorizationMethodInterceptor
Entity.EntityType.CATALOG,
NameIdentifierUtil.ofCatalog(metalakeName, catalog));
break;
case SCHEMA:
- rawNamespace = RESTUtil.decodeNamespace(value);
+ rawNamespace =
+ RESTUtil.decodeNamespace(
+ value,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
schema = String.join(separator, rawNamespace.levels());
nameIdentifierMap.put(
Entity.EntityType.SCHEMA,
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceTestBase.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceTestBase.java
index 4cd8ac46bd..2aa584ad98 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceTestBase.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergNamespaceTestBase.java
@@ -28,6 +28,7 @@ import javax.ws.rs.client.Entity;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
+import org.apache.gravitino.iceberg.service.IcebergRESTUtils;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.rest.RESTUtil;
import org.apache.iceberg.rest.requests.CreateNamespaceRequest;
@@ -74,7 +75,11 @@ public class IcebergNamespaceTestBase extends
IcebergTestBase {
private Response doListNamespace(Optional<Namespace> parent) {
Optional<Map<String, String>> queryParam =
parent.isPresent()
- ? Optional.of(ImmutableMap.of("parent",
RESTUtil.encodeNamespace(parent.get())))
+ ? Optional.of(
+ ImmutableMap.of(
+ "parent",
+ RESTUtil.encodeNamespace(
+ parent.get(),
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8)))
: Optional.empty();
return getNamespaceClientBuilder(Optional.empty(), Optional.empty(),
queryParam).get();
}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergRestTestUtil.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergRestTestUtil.java
index a75c9c76ac..0f2c141800 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergRestTestUtil.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergRestTestUtil.java
@@ -36,6 +36,7 @@ import org.apache.gravitino.iceberg.common.IcebergConfig;
import org.apache.gravitino.iceberg.service.IcebergCatalogWrapperManager;
import org.apache.gravitino.iceberg.service.IcebergExceptionMapper;
import org.apache.gravitino.iceberg.service.IcebergObjectMapperProvider;
+import org.apache.gravitino.iceberg.service.IcebergRESTUtils;
import
org.apache.gravitino.iceberg.service.authorization.IcebergRESTServerContext;
import
org.apache.gravitino.iceberg.service.dispatcher.IcebergNamespaceEventDispatcher;
import
org.apache.gravitino.iceberg.service.dispatcher.IcebergNamespaceOperationDispatcher;
@@ -74,7 +75,11 @@ public class IcebergRestTestUtil {
Namespace.of("gravitino-test-2", "nested");
public static final String VIEW_PATH =
- NAMESPACE_PATH + "/" + RESTUtil.encodeNamespace(TEST_NAMESPACE_NAME) +
"/views";
+ NAMESPACE_PATH
+ + "/"
+ + RESTUtil.encodeNamespace(
+ TEST_NAMESPACE_NAME,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8)
+ + "/views";
public static final String RENAME_TABLE_PATH = V_1 + "/tables/rename";
public static final String RENAME_VIEW_PATH = V_1 + "/views/rename";
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergTestBase.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergTestBase.java
index d9e51cac8f..403bb39b08 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergTestBase.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/IcebergTestBase.java
@@ -29,6 +29,7 @@ import javax.ws.rs.core.MediaType;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.gravitino.iceberg.service.IcebergObjectMapperProvider;
+import org.apache.gravitino.iceberg.service.IcebergRESTUtils;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.rest.RESTUtil;
import org.glassfish.jersey.server.ResourceConfig;
@@ -56,7 +57,11 @@ public class IcebergTestBase extends JerseyTest {
Joiner.on("/")
.skipNulls()
.join(
- IcebergRestTestUtil.NAMESPACE_PATH + "/" +
RESTUtil.encodeNamespace(ns) + "/tables",
+ IcebergRestTestUtil.NAMESPACE_PATH
+ + "/"
+ + RESTUtil.encodeNamespace(
+ ns,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8)
+ + "/tables",
name.orElseGet(() -> null));
return getIcebergClientBuilder(path, Optional.empty());
}
@@ -70,7 +75,11 @@ public class IcebergTestBase extends JerseyTest {
Joiner.on("/")
.skipNulls()
.join(
- IcebergRestTestUtil.NAMESPACE_PATH + "/" +
RESTUtil.encodeNamespace(ns) + "/views",
+ IcebergRestTestUtil.NAMESPACE_PATH
+ + "/"
+ + RESTUtil.encodeNamespace(
+ ns,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8)
+ + "/views",
name.orElseGet(() -> null));
return getIcebergClientBuilder(path, Optional.empty());
}
@@ -80,7 +89,11 @@ public class IcebergTestBase extends JerseyTest {
Joiner.on("/")
.skipNulls()
.join(
- IcebergRestTestUtil.NAMESPACE_PATH + "/" +
RESTUtil.encodeNamespace(ns) + "/tables",
+ IcebergRestTestUtil.NAMESPACE_PATH
+ + "/"
+ + RESTUtil.encodeNamespace(
+ ns,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8)
+ + "/tables",
name,
IcebergRestTestUtil.REPORT_METRICS_POSTFIX);
return getIcebergClientBuilder(path, Optional.empty());
@@ -103,7 +116,12 @@ public class IcebergTestBase extends JerseyTest {
.skipNulls()
.join(
IcebergRestTestUtil.NAMESPACE_PATH,
- namespace.map(RESTUtil::encodeNamespace).orElseGet(() -> null),
+ namespace
+ .map(
+ ns ->
+ RESTUtil.encodeNamespace(
+ ns,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8))
+ .orElseGet(() -> null),
extraPath.orElseGet(() -> null));
return getIcebergClientBuilder(path, queryParams);
}
@@ -114,7 +132,8 @@ public class IcebergTestBase extends JerseyTest {
.skipNulls()
.join(
IcebergRestTestUtil.NAMESPACE_PATH,
- RESTUtil.encodeNamespace(namespace),
+ RESTUtil.encodeNamespace(
+ namespace,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8),
IcebergRestTestUtil.UPDATE_NAMESPACE_POSTFIX);
return getIcebergClientBuilder(path, Optional.empty());
}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergConfig.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergConfig.java
index 3a33ae5d9e..f60eebf36f 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergConfig.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergConfig.java
@@ -128,9 +128,9 @@ public class TestIcebergConfig extends IcebergTestBase {
@Test
public void testConfigScanPlanEndpointPathIsNamespaceScoped() {
- // Iceberg 1.10.1's Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN advertises the
wrong path
- // (missing namespaces/{namespace}) — fixed in apache/iceberg#14120
(targeting 1.11.x).
- // This test guards against regressing to the broken path after an Iceberg
upgrade.
+ // Iceberg 1.11+ advertises the namespace-scoped scan plan path via
+ // Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN. This test guards against regressing
to the broken
+ // path that existed in Iceberg 1.10.1 (apache/iceberg#14120).
Response resp = getConfigClientBuilder().get();
Assertions.assertEquals(Response.Status.OK.getStatusCode(),
resp.getStatus());
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergNamespaceOperations.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergNamespaceOperations.java
index fcda9dd3f9..c7c8f95bfb 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergNamespaceOperations.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergNamespaceOperations.java
@@ -164,11 +164,14 @@ public class TestIcebergNamespaceOperations extends
IcebergNamespaceTestBase {
verifyNamespaceExistsStatusCode(204, Namespace.of("exists_foo2", "a"));
verifyNamespaceExistsStatusCode(404, Namespace.of("exists_foo2", "b"));
- verifyDropNamespaceSucc(Namespace.of("exists_foo2"));
- verifyNamespaceExistsStatusCode(404, Namespace.of("exists_foo2"));
+ // Iceberg 1.11: dropping a namespace that still has child namespaces
returns 409 Conflict.
+ verifyDropNamespaceFail(409, Namespace.of("exists_foo2"));
+ verifyNamespaceExistsStatusCode(204, Namespace.of("exists_foo2"));
verifyNamespaceExistsStatusCode(204, Namespace.of("exists_foo2", "a"));
verifyDropNamespaceSucc(Namespace.of("exists_foo2", "a"));
+ verifyNamespaceExistsStatusCode(404, Namespace.of("exists_foo2", "a"));
+ verifyDropNamespaceSucc(Namespace.of("exists_foo2"));
verifyNamespaceExistsStatusCode(404, Namespace.of("exists_foo2"));
verifyNamespaceExistsStatusCode(404, Namespace.of("exists_foo2", "a"));
}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergTableOperations.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergTableOperations.java
index 4025fe26e2..13accb3c73 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergTableOperations.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergTableOperations.java
@@ -37,6 +37,7 @@ import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.credential.Credential;
+import org.apache.gravitino.iceberg.service.IcebergRESTUtils;
import org.apache.gravitino.iceberg.service.extension.DummyCredentialProvider;
import org.apache.gravitino.listener.api.event.Event;
import org.apache.gravitino.listener.api.event.IcebergCreateTableEvent;
@@ -183,9 +184,12 @@ public class TestIcebergTableOperations extends
IcebergNamespaceTestBase {
JsonNode planResponse = verifyPlanTableScanSucc(namespace,
"plan_scan_table", snapshotId);
Assertions.assertEquals(PlanStatus.COMPLETED.status(),
planResponse.get("status").asText());
- Assertions.assertTrue(planResponse.has("plan-tasks"));
- Assertions.assertTrue(planResponse.get("plan-tasks").isArray());
- Assertions.assertTrue(planResponse.get("plan-tasks").size() > 0);
+ Assertions.assertTrue(planResponse.has("file-scan-tasks"));
+ Assertions.assertTrue(planResponse.get("file-scan-tasks").isArray());
+ Assertions.assertTrue(planResponse.get("file-scan-tasks").size() > 0);
+ Assertions.assertFalse(
+ planResponse.has("plan-tasks"),
+ "Iceberg 1.11+ plan scan must not emit legacy plan-tasks JSON");
Assertions.assertTrue(dummyEventListener.popPreEvent() instanceof
IcebergPlanTableScanPreEvent);
Assertions.assertTrue(dummyEventListener.popPostEvent() instanceof
IcebergPlanTableScanEvent);
@@ -229,8 +233,11 @@ public class TestIcebergTableOperations extends
IcebergNamespaceTestBase {
verifyPlanTableScanSuccWithRange(
namespace, "incremental_scan_valid_table", startSnapshotId,
endSnapshotId);
Assertions.assertEquals(PlanStatus.COMPLETED.status(),
planResponse.get("status").asText());
- Assertions.assertTrue(planResponse.has("plan-tasks"));
- Assertions.assertTrue(planResponse.get("plan-tasks").isArray());
+ Assertions.assertTrue(planResponse.has("file-scan-tasks"));
+ Assertions.assertTrue(planResponse.get("file-scan-tasks").isArray());
+ Assertions.assertFalse(
+ planResponse.has("plan-tasks"),
+ "Iceberg 1.11+ plan scan must not emit legacy plan-tasks JSON");
Assertions.assertTrue(dummyEventListener.popPreEvent() instanceof
IcebergPlanTableScanPreEvent);
Assertions.assertTrue(dummyEventListener.popPostEvent() instanceof
IcebergPlanTableScanEvent);
@@ -493,7 +500,11 @@ public class TestIcebergTableOperations extends
IcebergNamespaceTestBase {
private Response doLoadTableWithSnapshots(Namespace ns, String name, String
snapshots) {
String path =
- IcebergRestTestUtil.NAMESPACE_PATH + "/" +
RESTUtil.encodeNamespace(ns) + "/tables/" + name;
+ IcebergRestTestUtil.NAMESPACE_PATH
+ + "/"
+ + RESTUtil.encodeNamespace(ns,
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8)
+ + "/tables/"
+ + name;
Map<String, String> queryParams = ImmutableMap.of("snapshots", snapshots);
return getIcebergClientBuilder(path, Optional.of(queryParams)).get();
}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestScanPlanCache.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestScanPlanCache.java
index 015f1ad9f1..8fb5afec21 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestScanPlanCache.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestScanPlanCache.java
@@ -76,7 +76,7 @@ public class TestScanPlanCache {
PlanTableScanResponse response =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task1"))
+ .withFileScanTasks(Collections.emptyList())
.build();
Optional<PlanTableScanResponse> cached1 =
@@ -104,13 +104,13 @@ public class TestScanPlanCache {
PlanTableScanResponse response1 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task1"))
+ .withFileScanTasks(Collections.emptyList())
.build();
PlanTableScanResponse response2 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task2"))
+ .withFileScanTasks(Collections.emptyList())
.build();
scanPlanCache.put(ScanPlanCacheKey.create(tableIdentifier, mockTable,
request1), response1);
@@ -145,7 +145,7 @@ public class TestScanPlanCache {
PlanTableScanResponse response =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task1"))
+ .withFileScanTasks(Collections.emptyList())
.build();
scanPlanCache.put(ScanPlanCacheKey.create(tableIdentifier, mockTable,
request1), response);
@@ -167,13 +167,13 @@ public class TestScanPlanCache {
PlanTableScanResponse response1 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task1"))
+ .withFileScanTasks(Collections.emptyList())
.build();
PlanTableScanResponse response2 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task2"))
+ .withFileScanTasks(Collections.emptyList())
.build();
scanPlanCache.put(ScanPlanCacheKey.create(table1, mockTable, scanRequest),
response1);
@@ -199,7 +199,7 @@ public class TestScanPlanCache {
PlanTableScanResponse response1 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task1"))
+ .withFileScanTasks(Collections.emptyList())
.build();
// Cache with snapshot ID 1
@@ -237,13 +237,13 @@ public class TestScanPlanCache {
PlanTableScanResponse response1 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task1"))
+ .withFileScanTasks(Collections.emptyList())
.build();
PlanTableScanResponse response2 =
PlanTableScanResponse.builder()
.withPlanStatus(PlanStatus.COMPLETED)
- .withPlanTasks(Collections.singletonList("task2"))
+ .withFileScanTasks(Collections.emptyList())
.build();
scanPlanCache.put(ScanPlanCacheKey.create(tableIdentifier, mockTable,
request1), response1);
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/server/web/filter/TestIcebergMetadataAuthorizationMethodInterceptor.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/server/web/filter/TestIcebergMetadataAuthorizationMethodInterceptor.java
index b33b85bb8e..cd0be14d34 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/server/web/filter/TestIcebergMetadataAuthorizationMethodInterceptor.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/server/web/filter/TestIcebergMetadataAuthorizationMethodInterceptor.java
@@ -40,6 +40,7 @@ import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
import org.apache.gravitino.iceberg.service.CatalogWrapperForREST;
import org.apache.gravitino.iceberg.service.IcebergCatalogWrapperManager;
+import org.apache.gravitino.iceberg.service.IcebergRESTUtils;
import
org.apache.gravitino.iceberg.service.authorization.IcebergRESTServerContext;
import org.apache.gravitino.iceberg.service.provider.IcebergConfigProvider;
import
org.apache.gravitino.server.authorization.annotations.AuthorizationExpression;
@@ -192,7 +193,9 @@ public class
TestIcebergMetadataAuthorizationMethodInterceptor {
"testTableOperation", String.class, String.class, String.class);
Parameter[] parameters = testMethod.getParameters();
- String encodedNamespace = RESTUtil.encodeNamespace(Namespace.of("A", "B",
"C"));
+ String encodedNamespace =
+ RESTUtil.encodeNamespace(
+ Namespace.of("A", "B", "C"),
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
Object[] args = new Object[] {TEST_CATALOG + "/", encodedNamespace,
"my_table"};
Config mockConfig = Mockito.mock(Config.class);
@@ -222,7 +225,9 @@ public class
TestIcebergMetadataAuthorizationMethodInterceptor {
"testTableOperation", String.class, String.class, String.class);
Parameter[] parameters = testMethod.getParameters();
- String encodedNamespace =
RESTUtil.encodeNamespace(Namespace.of("my_schema"));
+ String encodedNamespace =
+ RESTUtil.encodeNamespace(
+ Namespace.of("my_schema"),
IcebergRESTUtils.NAMESPACE_SEPARATOR_URLENCODED_UTF_8);
Object[] args = new Object[] {TEST_CATALOG + "/", encodedNamespace,
"my_table"};
Config mockConfig = Mockito.mock(Config.class);
diff --git a/maintenance/jobs/build.gradle.kts
b/maintenance/jobs/build.gradle.kts
index b0a60e39b7..203494f128 100644
--- a/maintenance/jobs/build.gradle.kts
+++ b/maintenance/jobs/build.gradle.kts
@@ -31,7 +31,7 @@ repositories {
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark35.get()
-val icebergVersion: String = libs.versions.iceberg4connector.get()
+val icebergVersion: String = libs.versions.iceberg4spark35.get()
val sparkMajorVersion = "3.5"
dependencies {
diff --git a/maintenance/optimizer/build.gradle.kts
b/maintenance/optimizer/build.gradle.kts
index 8d6f618050..abb18c589b 100644
--- a/maintenance/optimizer/build.gradle.kts
+++ b/maintenance/optimizer/build.gradle.kts
@@ -28,7 +28,7 @@ val scalaVersion: String =
project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark33.get()
val sparkMajorVersion: String = sparkVersion.substringBeforeLast(".")
-val icebergVersion: String = libs.versions.iceberg4connector.get()
+val icebergVersion: String = libs.versions.iceberg4spark33.get()
dependencies {
implementation(project(":api"))
diff --git a/spark-connector/spark-common/build.gradle.kts
b/spark-connector/spark-common/build.gradle.kts
index 80c6ea8faa..afd0324df8 100644
--- a/spark-connector/spark-common/build.gradle.kts
+++ b/spark-connector/spark-common/build.gradle.kts
@@ -30,7 +30,7 @@ repositories {
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark33.get()
val sparkMajorVersion: String = sparkVersion.substringBeforeLast(".")
-val icebergVersion: String = libs.versions.iceberg4connector.get()
+val icebergVersion: String = libs.versions.iceberg4spark33.get()
val paimonVersion: String = libs.versions.paimon.get()
val kyuubiVersion: String = libs.versions.kyuubi4spark.get()
val scalaJava8CompatVersion: String = libs.versions.scala.java.compat.get()
diff --git
a/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/SparkCommonIT.java
b/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/SparkCommonIT.java
index ebc3f07d83..8a96f3ab44 100644
---
a/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/SparkCommonIT.java
+++
b/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/SparkCommonIT.java
@@ -929,7 +929,7 @@ public abstract class SparkCommonIT extends SparkEnvIT {
.map(Object::toString)
.map(
s -> {
- String tmp = org.apache.commons.lang3.StringUtils.remove(s, "'");
+ String tmp = s.replace("'", "");
if (org.apache.commons.lang3.StringUtils.isEmpty(tmp)) {
return tmp;
} else if (tmp.startsWith("array")) {
diff --git
a/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogIT.java
b/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogIT.java
index 92d9d5ca8a..bb692ac687 100644
---
a/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogIT.java
+++
b/spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogIT.java
@@ -1190,7 +1190,7 @@ public abstract class SparkIcebergCatalogIT extends
SparkCommonIT {
tableName);
}
- private SparkMetadataColumnInfo[] getIcebergMetadataColumns() {
+ protected SparkMetadataColumnInfo[] getIcebergMetadataColumns() {
return new SparkMetadataColumnInfo[] {
new SparkMetadataColumnInfo("_spec_id", DataTypes.IntegerType, false),
new SparkMetadataColumnInfo(
@@ -1200,7 +1200,9 @@ public abstract class SparkIcebergCatalogIT extends
SparkCommonIT {
true),
new SparkMetadataColumnInfo("_file", DataTypes.StringType, false),
new SparkMetadataColumnInfo("_pos", DataTypes.LongType, false),
- new SparkMetadataColumnInfo("_deleted", DataTypes.BooleanType, false)
+ new SparkMetadataColumnInfo("_deleted", DataTypes.BooleanType, false),
+ new SparkMetadataColumnInfo("_row_id", DataTypes.LongType, true),
+ new SparkMetadataColumnInfo("_last_updated_sequence_number",
DataTypes.LongType, true)
};
}
diff --git a/spark-connector/v3.3/spark/build.gradle.kts
b/spark-connector/v3.3/spark/build.gradle.kts
index a8c47ea5a4..2fbd6218c2 100644
--- a/spark-connector/v3.3/spark/build.gradle.kts
+++ b/spark-connector/v3.3/spark/build.gradle.kts
@@ -30,7 +30,7 @@ repositories {
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark33.get()
val sparkMajorVersion: String = sparkVersion.substringBeforeLast(".")
-val icebergVersion: String = libs.versions.iceberg4connector.get()
+val icebergVersion: String = libs.versions.iceberg4spark33.get()
val paimonVersion: String = libs.versions.paimon.get()
val kyuubiVersion: String = libs.versions.kyuubi4spark.get()
val scalaJava8CompatVersion: String = libs.versions.scala.java.compat.get()
diff --git
a/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogHiveBackendIT33.java
b/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogHiveBackendIT33.java
index 3c5fd19081..df88b21199 100644
---
a/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogHiveBackendIT33.java
+++
b/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogHiveBackendIT33.java
@@ -19,9 +19,23 @@
package org.apache.gravitino.spark.connector.integration.test.iceberg;
import
org.apache.gravitino.spark.connector.iceberg.GravitinoIcebergCatalogSpark33;
+import
org.apache.gravitino.spark.connector.integration.test.util.SparkMetadataColumnInfo;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructField;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIf;
+/**
+ * Spark 3.3 integration tests for the Iceberg Hive catalog backend.
+ *
+ * <p>Spark 3.3 pins Iceberg to 1.8.x ({@code iceberg4spark33}); Gravitino's
lakehouse Iceberg stack
+ * is 1.11. {@link DisabledIf} skips embedded mode to avoid classpath
conflicts. Deploy-mode tests
+ * run a 1.8.x client against Gravitino with Iceberg 1.11 on the server and
cover basic catalog CRUD
+ * only; they do not validate Iceberg 1.11 REST changes such as {@code
file-scan-tasks} scan
+ * planning or the configurable multi-level namespace separator contract.
+ */
+@DisabledIf("org.apache.gravitino.integration.test.util.ITUtils#isEmbedded")
public class SparkIcebergCatalogHiveBackendIT33 extends
SparkIcebergCatalogHiveBackendIT {
@Test
void testCatalogClassName() {
@@ -38,4 +52,23 @@ public class SparkIcebergCatalogHiveBackendIT33 extends
SparkIcebergCatalogHiveB
// Spark 3.3 does not support function operations
return false;
}
+
+ /**
+ * Spark 3.3 uses Iceberg 1.8.x; its {@code SparkTable#metadataColumns()}
exposes five columns,
+ * not the seven row-lineage columns added in Iceberg 1.11.
+ */
+ @Override
+ protected SparkMetadataColumnInfo[] getIcebergMetadataColumns() {
+ return new SparkMetadataColumnInfo[] {
+ new SparkMetadataColumnInfo("_spec_id", DataTypes.IntegerType, false),
+ new SparkMetadataColumnInfo(
+ "_partition",
+ DataTypes.createStructType(
+ new StructField[] {DataTypes.createStructField("name",
DataTypes.StringType, true)}),
+ true),
+ new SparkMetadataColumnInfo("_file", DataTypes.StringType, false),
+ new SparkMetadataColumnInfo("_pos", DataTypes.LongType, false),
+ new SparkMetadataColumnInfo("_deleted", DataTypes.BooleanType, false)
+ };
+ }
}
diff --git
a/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogRestBackendIT33.java
b/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogRestBackendIT33.java
index 255627f84a..788b1009f3 100644
---
a/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogRestBackendIT33.java
+++
b/spark-connector/v3.3/spark/src/test/java/org/apache/gravitino/spark/connector/integration/test/iceberg/SparkIcebergCatalogRestBackendIT33.java
@@ -19,8 +19,20 @@
package org.apache.gravitino.spark.connector.integration.test.iceberg;
+import
org.apache.gravitino.spark.connector.integration.test.util.SparkMetadataColumnInfo;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructField;
import org.junit.jupiter.api.condition.DisabledIf;
+/**
+ * Spark 3.3 integration tests for the Iceberg REST catalog backend.
+ *
+ * <p>Spark 3.3 pins Iceberg to 1.8.x ({@code iceberg4spark33}); Gravitino's
lakehouse Iceberg stack
+ * is 1.11. {@link DisabledIf} skips embedded mode to avoid classpath
conflicts. Deploy-mode tests
+ * run a 1.8.x client against Gravitino's 1.11 REST server and cover basic
catalog CRUD only; they
+ * do not exercise Iceberg 1.11 REST changes such as {@code file-scan-tasks}
scan planning or the
+ * configurable multi-level namespace separator contract.
+ */
@DisabledIf("org.apache.gravitino.integration.test.util.ITUtils#isEmbedded")
public class SparkIcebergCatalogRestBackendIT33 extends
SparkIcebergCatalogRestBackendIT {
@Override
@@ -28,4 +40,23 @@ public class SparkIcebergCatalogRestBackendIT33 extends
SparkIcebergCatalogRestB
// Spark 3.3 does not support function operations
return false;
}
+
+ /**
+ * Spark 3.3 uses Iceberg 1.8.x; its {@code SparkTable#metadataColumns()}
exposes five columns,
+ * not the seven row-lineage columns added in Iceberg 1.11.
+ */
+ @Override
+ protected SparkMetadataColumnInfo[] getIcebergMetadataColumns() {
+ return new SparkMetadataColumnInfo[] {
+ new SparkMetadataColumnInfo("_spec_id", DataTypes.IntegerType, false),
+ new SparkMetadataColumnInfo(
+ "_partition",
+ DataTypes.createStructType(
+ new StructField[] {DataTypes.createStructField("name",
DataTypes.StringType, true)}),
+ true),
+ new SparkMetadataColumnInfo("_file", DataTypes.StringType, false),
+ new SparkMetadataColumnInfo("_pos", DataTypes.LongType, false),
+ new SparkMetadataColumnInfo("_deleted", DataTypes.BooleanType, false)
+ };
+ }
}
diff --git a/spark-connector/v3.4/spark/build.gradle.kts
b/spark-connector/v3.4/spark/build.gradle.kts
index a732aeb757..d271ba3f94 100644
--- a/spark-connector/v3.4/spark/build.gradle.kts
+++ b/spark-connector/v3.4/spark/build.gradle.kts
@@ -30,7 +30,7 @@ repositories {
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark34.get()
val sparkMajorVersion: String = sparkVersion.substringBeforeLast(".")
-val icebergVersion: String = libs.versions.iceberg4connector.get()
+val icebergVersion: String = libs.versions.iceberg4spark34.get()
val paimonVersion: String = libs.versions.paimon.get()
val kyuubiVersion: String = libs.versions.kyuubi4spark.get()
val scalaJava8CompatVersion: String = libs.versions.scala.java.compat.get()
diff --git a/spark-connector/v3.5/spark/build.gradle.kts
b/spark-connector/v3.5/spark/build.gradle.kts
index edcff1e07e..c17a21d394 100644
--- a/spark-connector/v3.5/spark/build.gradle.kts
+++ b/spark-connector/v3.5/spark/build.gradle.kts
@@ -30,7 +30,7 @@ repositories {
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
val sparkVersion: String = libs.versions.spark35.get()
val sparkMajorVersion: String = sparkVersion.substringBeforeLast(".")
-val icebergVersion: String = libs.versions.iceberg4connector.get()
+val icebergVersion: String = libs.versions.iceberg4spark35.get()
val paimonVersion: String = libs.versions.paimon.get()
val kyuubiVersion: String = libs.versions.kyuubi4spark.get()
val scalaJava8CompatVersion: String = libs.versions.scala.java.compat.get()
