This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new e27ff4e70 [#4393] improvement(docs): Refine authentication information
about Iceberg rest catalog and Add IT for Kerberos Hive backend. (#4395)
e27ff4e70 is described below
commit e27ff4e70da1fffd27317cb928e1d93ba8661bd6
Author: Qi Yu <[email protected]>
AuthorDate: Thu Aug 15 09:29:43 2024 +0800
[#4393] improvement(docs): Refine authentication information about Iceberg
rest catalog and Add IT for Kerberos Hive backend. (#4395)
### What changes were proposed in this pull request?
Add document about authentication for Iceberg rest catalog.
### Why are the changes needed?
To make the information consistent.
Fix: #4393
### Does this PR introduce _any_ user-facing change?
N/A.
### How was this patch tested?
N/A.
---
docs/iceberg-rest-service.md | 47 +++---
docs/lakehouse-iceberg-catalog.md | 45 +++---
.../iceberg/common/ops/IcebergTableOps.java | 18 +++
.../iceberg/common/ops/IcebergTableOpsManager.java | 7 +-
.../iceberg/common/utils/IcebergCatalogUtil.java | 7 +-
.../integration/test/IcebergRESTHiveCatalogIT.java | 2 +-
.../test/IcebergRestKerberosHiveCatalogIT.java | 160 +++++++++++++++++++++
.../util/IcebergRESTServerManagerForDeploy.java | 43 +++++-
8 files changed, 284 insertions(+), 45 deletions(-)
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index 234104b87..70c5b6812 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -83,6 +83,21 @@ You can also specify filter parameters by setting
configuration entries in the s
Gravitino Iceberg REST server supports OAuth2 and HTTPS, please refer to
[Security](./security.md) for more details.
+#### Backend authentication
+
+For JDBC backend, you can use the `gravitino.iceberg-rest.jdbc.user` and
`gravitino.iceberg-rest.jdbc.password` to authenticate the JDBC connection. For
Hive backend, you can use the `gravitino.iceberg-rest.authentication.type` to
specify the authentication type, and use the
`gravitino.iceberg-rest.authentication.kerberos.principal` and
`gravitino.iceberg-rest.authentication.kerberos.keytab-uri` to authenticate the
Kerberos connection.
+The detailed configuration items are as follows:
+
+| Configuration item |
Description
| Default value | Required
| Since Version |
+|---------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|------------------------------------------------------------------------------------|---------------|
+| `gravitino.iceberg-rest.authentication.type` |
The type of authentication for Iceberg rest catalog backend. This configuration
only applicable for for Hive backend, and only supports `Kerberos`, `simple`
currently. As for JDBC backend, only username/password authentication was
supported now. | `simple` | No
| 0.6.0 |
+| `gravitino.iceberg-rest.authentication.impersonation-enable` |
Whether to enable impersonation for the Iceberg catalog
| `false` | No
| 0.6.0 |
+| `gravitino.iceberg-rest.authentication.kerberos.principal` |
The principal of the Kerberos authentication
| (none) | required if the value of
`gravitino.iceberg-rest.authentication.type` is Kerberos. | 0.6.0 |
+| `gravitino.iceberg-rest.authentication.kerberos.keytab-uri` |
The URI of The keytab for the Kerberos authentication.
| (none) | required if the value of
`gravitino.iceberg-rest.authentication.type` is Kerberos. | 0.6.0 |
+| `gravitino.iceberg-rest.authentication.kerberos.check-interval-sec` |
The check interval of Kerberos credential for Iceberg catalog.
| 60 | No
| 0.6.0 |
+| `gravitino.iceberg-rest.authentication.kerberos.keytab-fetch-timeout-sec` |
The fetch timeout of retrieving Kerberos keytab from
`authentication.kerberos.keytab-uri`.
| 60 | No
| 0.6.0 |
+
+
### Storage
Gravitino Iceberg REST server supports S3 and HDFS for storage.
@@ -121,25 +136,25 @@ The Gravitino Iceberg REST catalog service uses the
memory catalog backend by de
#### Hive backend configuration
-| Configuration item | Description
| Default value
| Required | Since Version |
-|-----------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|----------|---------------|
-| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the
Gravitino Iceberg REST catalog service. Use the value **`hive`** for a Hive
catalog. | `memory`
| Yes | 0.2.0 |
-| `gravitino.iceberg-rest.uri` | The Hive metadata address,
such as `thrift://127.0.0.1:9083`.
| (none)
| Yes | 0.2.0 |
-| `gravitino.iceberg-rest.warehouse` | The warehouse directory of
the Hive catalog, such as `/user/hive/warehouse-hive/`.
| (none)
| Yes | 0.2.0 |
-| `gravitino.iceberg-rest.catalog-backend-name` | The catalog backend name
passed to underlying Iceberg catalog backend. Catalog name in JDBC backend is
used to isolate namespace and tables. | `hive` for Hive backend, `jdbc` for
JDBC backend, `memory` for memory backend | No | 0.5.2 |
+| Configuration item |
Description
| Default value
| Required |
Since Version |
+|---------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|----------|---------------|
+| `gravitino.iceberg-rest.catalog-backend` |
The Catalog backend of the Gravitino Iceberg REST catalog service. Use the
value **`hive`** for a Hive catalog. | `memory`
| Yes |
0.2.0 |
+| `gravitino.iceberg-rest.uri` |
The Hive metadata address, such as `thrift://127.0.0.1:9083`.
| (none)
| Yes |
0.2.0 |
+| `gravitino.iceberg-rest.warehouse` |
The warehouse directory of the Hive catalog, such as
`/user/hive/warehouse-hive/`.
| (none)
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.catalog-backend-name` |
The catalog backend name passed to underlying Iceberg catalog backend. Catalog
name in JDBC backend is used to isolate namespace and tables. | `hive` for Hive
backend, `jdbc` for JDBC backend, `memory` for memory backend | No |
0.5.2 |
#### JDBC backend configuration
-| Configuration item | Description
| Default value | Required | Since Version |
-|-----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|-------------------------|----------|---------------|
-| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the
Gravitino Iceberg REST catalog service. Use the value **`jdbc`** for a JDBC
catalog. | `memory` | Yes | 0.2.0
|
-| `gravitino.iceberg-rest.uri` | The JDBC connection address,
such as `jdbc:postgresql://127.0.0.1:5432` for Postgres, or
`jdbc:mysql://127.0.0.1:3306/` for mysql. | (none) | Yes
| 0.2.0 |
-| `gravitino.iceberg-rest.warehouse ` | The warehouse directory of
JDBC catalog. Set the HDFS prefix if using HDFS, such as
`hdfs://127.0.0.1:9000/user/hive/warehouse-jdbc` | (none) |
Yes | 0.2.0 |
-| `gravitino.iceberg-rest.catalog-backend-name` | The catalog name passed to
underlying Iceberg catalog backend. Catalog name in JDBC backend is used to
isolate namespace and tables. | `jdbc` for JDBC backend | No | 0.5.2
|
-| `gravitino.iceberg-rest.jdbc.user` | The username of the JDBC
connection.
| (none) | Yes | 0.2.0
|
-| `gravitino.iceberg-rest.jdbc.password` | The password of the JDBC
connection.
| (none) | Yes | 0.2.0
|
-| `gravitino.iceberg-rest.jdbc-initialize` | Whether to initialize the
meta tables when creating the JDBC catalog.
| `true` | No | 0.2.0
|
-| `gravitino.iceberg-rest.jdbc-driver` | `com.mysql.jdbc.Driver` or
`com.mysql.cj.jdbc.Driver` for MySQL, `org.postgresql.Driver` for PostgreSQL.
| (none) | Yes | 0.3.0 |
+| Configuration item |
Description
| Default value
| Required | Since Version |
+|---------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|---------------|
+| `gravitino.iceberg-rest.catalog-backend` |
The Catalog backend of the Gravitino Iceberg REST catalog service. Use the
value **`jdbc`** for a JDBC catalog. | `memory`
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.uri` |
The JDBC connection address, such as `jdbc:postgresql://127.0.0.1:5432` for
Postgres, or `jdbc:mysql://127.0.0.1:3306/` for mysql. | (none)
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.warehouse ` |
The warehouse directory of JDBC catalog. Set the HDFS prefix if using HDFS,
such as `hdfs://127.0.0.1:9000/user/hive/warehouse-jdbc` | (none)
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.catalog-backend-name` |
The catalog name passed to underlying Iceberg catalog backend. Catalog name in
JDBC backend is used to isolate namespace and tables. | `jdbc` for JDBC
backend | No | 0.5.2 |
+| `gravitino.iceberg-rest.jdbc.user` |
The username of the JDBC connection.
| (none)
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.jdbc.password` |
The password of the JDBC connection.
| (none)
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.jdbc-initialize` |
Whether to initialize the meta tables when creating the JDBC catalog.
| `true`
| No | 0.2.0 |
+| `gravitino.iceberg-rest.jdbc-driver` |
`com.mysql.jdbc.Driver` or `com.mysql.cj.jdbc.Driver` for MySQL,
`org.postgresql.Driver` for PostgreSQL. | (none)
| Yes | 0.3.0 |
If you have a JDBC Iceberg catalog prior, you must set `catalog-backend-name`
to keep consistent with your Jdbc Iceberg catalog name to operate the prior
namespace and tables.
diff --git a/docs/lakehouse-iceberg-catalog.md
b/docs/lakehouse-iceberg-catalog.md
index 1b330d30f..f20487fa1 100644
--- a/docs/lakehouse-iceberg-catalog.md
+++ b/docs/lakehouse-iceberg-catalog.md
@@ -25,7 +25,7 @@ Builds with Apache Iceberg `1.5.2`. The Apache Iceberg table
format version is `
### Catalog capabilities
-- Works as a catalog proxy, supporting `HiveCatalog`, `JdbcCatalog` and
`RESTCatalog`.
+- Works as a catalog proxy, supporting `Hive`, `JDBC` and `REST` as catalog
backend.
- Supports DDL operations for Iceberg schemas and tables.
- Doesn't support snapshot or table management operations.
- Supports S3 and HDFS storage.
@@ -33,30 +33,24 @@ Builds with Apache Iceberg `1.5.2`. The Apache Iceberg
table format version is `
### Catalog properties
-| Property name | Description
| Default value | Required
| Since Version |
-|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-------------------------------------------------------------|---------------|
-| `catalog-backend` | Catalog backend of
Gravitino Iceberg catalog. Supports `hive` or `jdbc` or `rest`.
| (none) | Yes
| 0.2.0 |
-| `uri` | The URI configuration
of the Iceberg catalog. `thrift://127.0.0.1:9083` or
`jdbc:postgresql://127.0.0.1:5432/db_name` or
`jdbc:mysql://127.0.0.1:3306/metastore_db` or `http://127.0.0.1:9001`. | (none)
| Yes |
0.2.0 |
-| `warehouse` | Warehouse directory of
catalog. `file:///user/hive/warehouse-hive/` for local fs or
`hdfs://namespace/hdfs/path` for HDFS.
| (none) | Yes
| 0.2.0 |
-| `catalog-backend-name` | The catalog name passed
to underlying Iceberg catalog backend. Catalog name in JDBC backend is used to
isolate namespace and tables.
| Gravitino catalog name | No
| 0.5.2 |
-| `authentication.type` | The type of
authentication for Iceberg catalog backend, currently Gravitino only supports
`Kerberos`, `simple`.
| `simple` | No
| 0.6.0 |
-| `authentication.impersonation-enable` | Whether to enable
impersonation for the Iceberg catalog
| `false` | No
| 0.6.0 |
-| `authentication.kerberos.principal` | The principal of the
Kerberos authentication
| (none) | required if the value of
`authentication.type` is Kerberos. | 0.6.0 |
-| `authentication.kerberos.keytab-uri` | The URI of The keytab
for the Kerberos authentication.
| (none) | required if the value of
`authentication.type` is Kerberos. | 0.6.0 |
-| `authentication.kerberos.check-interval-sec` | The check interval of
Kerberos credential for Iceberg catalog.
| 60 | No
| 0.6.0 |
-| `authentication.kerberos.keytab-fetch-timeout-sec` | The fetch timeout of
retrieving Kerberos keytab from `authentication.kerberos.keytab-uri`.
| 60 | No
| 0.6.0 |
+| Property name | Description
| Default value
| Required | Since Version
|
+|----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-------------------------------------------------------------|---------------|
+| `catalog-backend` | Catalog backend of
Gravitino Iceberg catalog. Supports `hive` or `jdbc` or `rest`.
| (none)
| Yes | 0.2.0
|
+| `uri` | The URI configuration
of the Iceberg catalog. `thrift://127.0.0.1:9083` or
`jdbc:postgresql://127.0.0.1:5432/db_name` or
`jdbc:mysql://127.0.0.1:3306/metastore_db` or `http://127.0.0.1:9001`.
| (none) | Yes
| 0.2.0 |
+| `warehouse` | Warehouse directory of
catalog. `file:///user/hive/warehouse-hive/` for local fs or
`hdfs://namespace/hdfs/path` for HDFS.
|
(none) | Yes
| 0.2.0 |
+| `catalog-backend-name` | The catalog name passed
to underlying Iceberg catalog backend. Catalog name in JDBC backend is used to
isolate namespace and tables.
| Gravitino catalog
name | No | 0.5.2
|
-Any properties not defined by Gravitino with `gravitino.bypass.` prefix will
pass to Iceberg catalog properties and HDFS configuration. For example, if
specify `gravitino.bypass.list-all-tables`, `list-all-tables` will pass to
Iceberg catalog properties.
+Any property not defined by Gravitino with `gravitino.bypass.` prefix will
pass to Iceberg catalog properties and HDFS configuration. For example, if
specify `gravitino.bypass.list-all-tables`, `list-all-tables` will pass to
Iceberg catalog properties.
-When you use the Gravitino with Trino. You can pass the Trino Iceberg
connector configuration using prefix `trino.bypass.`. For example, using
`trino.bypass.iceberg.table-statistics-enabled` to pass the
`iceberg.table-statistics-enabled` to the Gravitino Iceberg catalog in Trino
runtime.
+If you are using the Gravitino with Trino, you can pass the Trino Iceberg
connector configuration using prefix `trino.bypass.`. For example, using
`trino.bypass.iceberg.table-statistics-enabled` to pass the
`iceberg.table-statistics-enabled` to the Gravitino Iceberg catalog in Trino
runtime.
-When you use the Gravitino with Spark. You can pass the Spark Iceberg
connector configuration using prefix `spark.bypass.`. For example, using
`spark.bypass.io-impl` to pass the `io-impl` to the Spark Iceberg connector in
Spark runtime.
+If you are using the Gravitino with Spark, you can pass the Spark Iceberg
connector configuration using prefix `spark.bypass.`. For example, using
`spark.bypass.io-impl` to pass the `io-impl` to the Spark Iceberg connector in
Spark runtime.
-#### JDBC catalog
+#### JDBC backend
-If you are using JDBC catalog, you must provide `jdbc-user`, `jdbc-password`
and `jdbc-driver` to catalog properties.
+If you are using JDBC backend, you must provide properties like `jdbc-user`,
`jdbc-password` and `jdbc-driver`.
| Property name | Description
| Default value | Required |
Since Version |
|-------------------|---------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
@@ -68,7 +62,7 @@ If you are using JDBC catalog, you must provide `jdbc-user`,
`jdbc-password` and
If you have a JDBC Iceberg catalog prior, you must set `catalog-backend-name`
to keep consistent with your Jdbc Iceberg catalog name to operate the prior
namespace and tables.
:::caution
-You must download the corresponding JDBC driver to the
`catalogs/lakehouse-iceberg/libs` directory.
+You must download the corresponding JDBC driver and place it to the
`catalogs/lakehouse-iceberg/libs` directory If you are using JDBC backend.
:::
#### S3
@@ -89,6 +83,19 @@ For other Iceberg s3 properties not managed by Gravitino
like `s3.sse.type`, you
Please set `gravitino.iceberg-rest.warehouse` to
`s3://{bucket_name}/${prefix_name}` for JDBC catalog backend,
`s3a://{bucket_name}/${prefix_name}` for Hive catalog backend.
:::
+#### Catalog backend security
+
+Users can use the following properties to configure the security of the
catalog backend if needed. For example, if you are using a Kerberos Hive
catalog backend, you must set `authentication.type` to `Kerberos` and provide
`authentication.kerberos.principal` and `authentication.kerberos.keytab-uri`.
+
+| Property name | Description
| Default value
| Required | Since Version
|
+|----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-------------------------------------------------------------|---------------|
+| `authentication.type` | The type of
authentication for Iceberg catalog backend. This configuration only applicable
for for Hive backend, and only supports `Kerberos`, `simple` currently. As for
JDBC backend, only username/password authentication was supported now. |
`simple` | No
| 0.6.0 |
+| `authentication.impersonation-enable` | Whether to enable
impersonation for the Iceberg catalog
| `false`
| No | 0.6.0
|
+| `authentication.kerberos.principal` | The principal of the
Kerberos authentication
| (none)
| required if the value of `authentication.type` is Kerberos. | 0.6.0
|
+| `authentication.kerberos.keytab-uri` | The URI of The keytab
for the Kerberos authentication.
| (none)
| required if the value of `authentication.type` is Kerberos. | 0.6.0
|
+| `authentication.kerberos.check-interval-sec` | The check interval of
Kerberos credential for Iceberg catalog.
| 60
| No | 0.6.0
|
+| `authentication.kerberos.keytab-fetch-timeout-sec` | The fetch timeout of
retrieving Kerberos keytab from `authentication.kerberos.keytab-uri`.
| 60
| No | 0.6.0
|
+
### Catalog operations
Please refer to [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#catalog-operations)
for more details.
diff --git
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOps.java
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOps.java
index 56ca6e505..0720c0abb 100644
---
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOps.java
+++
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOps.java
@@ -36,6 +36,8 @@ import org.apache.gravitino.iceberg.common.IcebergConfig;
import org.apache.gravitino.iceberg.common.utils.IcebergCatalogUtil;
import org.apache.gravitino.utils.IsolatedClassLoader;
import org.apache.gravitino.utils.MapUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.iceberg.Transaction;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.catalog.Namespace;
@@ -66,6 +68,7 @@ public class IcebergTableOps implements AutoCloseable {
private final IcebergCatalogBackend catalogBackend;
private String catalogUri = null;
private Map<String, String> catalogConfigToClients;
+ private Map<String, String> catalogPropertiesMap;
private static final Set<String> catalogPropertiesToClientKeys =
ImmutableSet.of(
IcebergConstants.IO_IMPL,
@@ -91,6 +94,8 @@ public class IcebergTableOps implements AutoCloseable {
MapUtils.getFilteredMap(
icebergConfig.getIcebergCatalogProperties(),
key -> catalogPropertiesToClientKeys.contains(key));
+
+ this.catalogPropertiesMap = icebergConfig.getIcebergCatalogProperties();
}
public IcebergTableOps() {
@@ -139,6 +144,19 @@ public class IcebergTableOps implements AutoCloseable {
return CatalogHandlers.registerTable(catalog, namespace, request);
}
+ /**
+ * Reload hadoop configuration, this is useful when the hadoop configuration
UserGroupInformation
+ * is shared by multiple threads. UserGroupInformation#authenticationMethod
was first initialized
+ * in KerberosClient, however, when switching to iceberg-rest thead,
+ * UserGroupInformation#authenticationMethod will be reset to the default
value; we need to
+ * reinitialize it again.
+ */
+ public void reloadHadoopConf() {
+ Configuration configuration = new Configuration();
+ this.catalogPropertiesMap.forEach(configuration::set);
+ UserGroupInformation.setConfiguration(configuration);
+ }
+
public LoadTableResponse createTable(Namespace namespace, CreateTableRequest
request) {
request.validate();
if (request.stageCreate()) {
diff --git
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOpsManager.java
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOpsManager.java
index 7b6270be6..57792b89f 100644
---
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOpsManager.java
+++
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergTableOpsManager.java
@@ -54,7 +54,12 @@ public class IcebergTableOpsManager implements AutoCloseable
{
*/
public IcebergTableOps getOps(String rawPrefix) {
String catalogName = getCatalogName(rawPrefix);
- return icebergTableOpsCache.get(catalogName, k ->
provider.getIcebergTableOps(catalogName));
+ IcebergTableOps tableOps =
+ icebergTableOpsCache.get(catalogName, k ->
provider.getIcebergTableOps(catalogName));
+ // Reload conf to reset UserGroupInformation or icebergTableOps will
always use
+ // Simple auth.
+ tableOps.reloadHadoopConf();
+ return tableOps;
}
private String getCatalogName(String rawPrefix) {
diff --git
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/utils/IcebergCatalogUtil.java
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/utils/IcebergCatalogUtil.java
index 8f171bea6..f6e8b4e36 100644
---
a/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/utils/IcebergCatalogUtil.java
+++
b/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/utils/IcebergCatalogUtil.java
@@ -99,8 +99,11 @@ public class IcebergCatalogUtil {
Map<String, String> properties, Configuration conf) {
try {
KerberosClient kerberosClient = new KerberosClient(properties, conf);
- File keytabFile =
-
kerberosClient.saveKeyTabFileFromUri(Long.valueOf(properties.get("catalog_uuid")));
+
+ // For Iceberg rest server, we haven't set the catalog_uuid, so we set
it to 0 as there is
+ // only one catalog in the rest server, so it's okay to set it to 0.
+ String catalogUUID = properties.getOrDefault("catalog_uuid", "0");
+ File keytabFile =
kerberosClient.saveKeyTabFileFromUri(Long.valueOf(catalogUUID));
kerberosClient.login(keytabFile.getAbsolutePath());
return kerberosClient;
} catch (IOException e) {
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRESTHiveCatalogIT.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRESTHiveCatalogIT.java
index 837b40961..2284a9a82 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRESTHiveCatalogIT.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRESTHiveCatalogIT.java
@@ -32,7 +32,7 @@ import org.junit.jupiter.api.TestInstance.Lifecycle;
@Tag("gravitino-docker-test")
@TestInstance(Lifecycle.PER_CLASS)
public class IcebergRESTHiveCatalogIT extends IcebergRESTServiceIT {
- private static final ContainerSuite containerSuite =
ContainerSuite.getInstance();
+ protected static final ContainerSuite containerSuite =
ContainerSuite.getInstance();
public IcebergRESTHiveCatalogIT() {
catalogType = IcebergCatalogBackend.HIVE;
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRestKerberosHiveCatalogIT.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRestKerberosHiveCatalogIT.java
new file mode 100644
index 000000000..e647c5959
--- /dev/null
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/IcebergRestKerberosHiveCatalogIT.java
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.iceberg.integration.test;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.commons.io.FileUtils;
+import org.apache.gravitino.iceberg.common.IcebergCatalogBackend;
+import org.apache.gravitino.iceberg.common.IcebergConfig;
+import org.apache.gravitino.integration.test.container.HiveContainer;
+import org.apache.gravitino.integration.test.util.GravitinoITUtils;
+import org.apache.gravitino.integration.test.util.ITUtils;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.TestInstance.Lifecycle;
+import org.junit.jupiter.api.condition.EnabledIf;
+
+@Tag("gravitino-docker-test")
+@TestInstance(Lifecycle.PER_CLASS)
+@EnabledIf("isEmbedded")
+public class IcebergRestKerberosHiveCatalogIT extends IcebergRESTHiveCatalogIT
{
+
+ private static final String HIVE_METASTORE_CLIENT_PRINCIPAL =
"cli@HADOOPKRB";
+ private static final String HIVE_METASTORE_CLIENT_KEYTAB = "/client.keytab";
+
+ private static String tempDir;
+
+ public IcebergRestKerberosHiveCatalogIT() {
+ super();
+ }
+
+ void initEnv() {
+ containerSuite.startKerberosHiveContainer();
+ try {
+
+ // Init kerberos configurations;
+ File baseDir = new File(System.getProperty("java.io.tmpdir"));
+ File file = Files.createTempDirectory(baseDir.toPath(), "test").toFile();
+ file.deleteOnExit();
+ tempDir = file.getAbsolutePath();
+
+ HiveContainer kerberosHiveContainer =
containerSuite.getKerberosHiveContainer();
+ kerberosHiveContainer
+ .getContainer()
+ .copyFileFromContainer("/etc/admin.keytab", tempDir +
HIVE_METASTORE_CLIENT_KEYTAB);
+
+ String tmpKrb5Path = tempDir + "/krb5.conf_tmp";
+ String krb5Path = tempDir + "/krb5.conf";
+
kerberosHiveContainer.getContainer().copyFileFromContainer("/etc/krb5.conf",
tmpKrb5Path);
+
+ // Modify the krb5.conf and change the kdc and admin_server to the
container IP
+ String ip =
containerSuite.getKerberosHiveContainer().getContainerIpAddress();
+ String content = FileUtils.readFileToString(new File(tmpKrb5Path),
StandardCharsets.UTF_8);
+ content = content.replace("kdc = localhost:88", "kdc = " + ip + ":88");
+ content = content.replace("admin_server = localhost", "admin_server = "
+ ip + ":749");
+ FileUtils.write(new File(krb5Path), content, StandardCharsets.UTF_8);
+
+ LOG.info("Kerberos kdc config:\n{}, path: {}", content, krb5Path);
+ System.setProperty("java.security.krb5.conf", krb5Path);
+ System.setProperty("sun.security.krb5.debug", "true");
+ System.setProperty("java.security.krb5.realm", "HADOOPKRB");
+ System.setProperty("java.security.krb5.kdc", ip);
+
+ sun.security.krb5.Config.refresh();
+ resetDefaultRealm();
+
+ // Give cli@HADOOPKRB permission to access the hdfs
+ containerSuite
+ .getKerberosHiveContainer()
+ .executeInContainer("hadoop", "fs", "-chown", "-R", "cli",
"/user/hive/");
+
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ void resetDefaultRealm() {
+ try {
+ String kerberosNameClass =
"org.apache.hadoop.security.authentication.util.KerberosName";
+ Class<?> cl = Class.forName(kerberosNameClass);
+ cl.getMethod("resetDefaultRealm").invoke(null);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ Map<String, String> getCatalogConfig() {
+ Map<String, String> configMap = new HashMap<>();
+
+ configMap.put("gravitino.iceberg-rest.authentication.type", "kerberos");
+ configMap.put(
+ "gravitino.iceberg-rest.authentication.kerberos.principal",
+ HIVE_METASTORE_CLIENT_PRINCIPAL);
+ configMap.put(
+ "gravitino.iceberg-rest.authentication.kerberos.keytab-uri",
+ tempDir + HIVE_METASTORE_CLIENT_KEYTAB);
+ configMap.put("gravitino.iceberg-rest.hive.metastore.sasl.enabled",
"true");
+ configMap.put(
+ "gravitino.iceberg-rest.hive.metastore.kerberos.principal",
+ "hive/_HOST@HADOOPKRB"
+ .replace("_HOST",
containerSuite.getKerberosHiveContainer().getHostName()));
+
+ configMap.put("gravitino.iceberg-rest.hadoop.security.authentication",
"kerberos");
+
+ configMap.put(
+ "gravitino.iceberg-rest.dfs.namenode.kerberos.principal",
+ "hdfs/_HOST@HADOOPKRB"
+ .replace("_HOST",
containerSuite.getKerberosHiveContainer().getHostName()));
+
+ configMap.put(
+ IcebergConfig.ICEBERG_CONFIG_PREFIX +
IcebergConfig.CATALOG_BACKEND.getKey(),
+ IcebergCatalogBackend.HIVE.toString().toLowerCase());
+
+ configMap.put(
+ IcebergConfig.ICEBERG_CONFIG_PREFIX +
IcebergConfig.CATALOG_URI.getKey(),
+ String.format(
+ "thrift://%s:%d",
+ containerSuite.getKerberosHiveContainer().getContainerIpAddress(),
+ HiveContainer.HIVE_METASTORE_PORT));
+
+ configMap.put(
+ IcebergConfig.ICEBERG_CONFIG_PREFIX +
IcebergConfig.CATALOG_WAREHOUSE.getKey(),
+ GravitinoITUtils.genRandomName(
+ String.format(
+ "hdfs://%s:%d/user/hive/warehouse-hive",
+
containerSuite.getKerberosHiveContainer().getContainerIpAddress(),
+ HiveContainer.HDFS_DEFAULTFS_PORT)));
+ return configMap;
+ }
+
+ private static boolean isEmbedded() {
+ String mode =
+ System.getProperty(ITUtils.TEST_MODE) == null
+ ? ITUtils.EMBEDDED_TEST_MODE
+ : System.getProperty(ITUtils.TEST_MODE);
+
+ return Objects.equals(mode, ITUtils.EMBEDDED_TEST_MODE);
+ }
+}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/util/IcebergRESTServerManagerForDeploy.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/util/IcebergRESTServerManagerForDeploy.java
index dda2c593e..135713223 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/util/IcebergRESTServerManagerForDeploy.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/integration/test/util/IcebergRESTServerManagerForDeploy.java
@@ -18,10 +18,14 @@
*/
package org.apache.gravitino.iceberg.integration.test.util;
+import java.io.File;
+import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Optional;
+import java.util.UUID;
import java.util.concurrent.Future;
+import org.apache.commons.io.FileUtils;
import org.apache.gravitino.integration.test.util.CommandExecutor;
import org.apache.gravitino.integration.test.util.JdbcDriverDownloader;
import org.apache.gravitino.integration.test.util.ProcessData;
@@ -51,12 +55,39 @@ public class IcebergRESTServerManagerForDeploy extends
IcebergRESTServerManager
SQLITE_DRIVER_DOWNLOAD_URL,
Paths.get(icebergRESTServerHome.toString(), "iceberg-rest-server",
"libs").toString());
- String cmd = String.format("%s/bin/%s start",
icebergRESTServerHome.toString(), SCRIPT_NAME);
- CommandExecutor.executeCommandLocalHost(
- cmd,
- false,
- ProcessData.TypesOfData.OUTPUT,
- ImmutableMap.of("GRAVITINO_HOME", icebergRESTServerHome.toString()));
+ String gravitinoRestStartShell = icebergRESTServerHome.toString() +
"/bin/" + SCRIPT_NAME;
+ String krb5Path = System.getProperty("java.security.krb5.conf");
+ if (krb5Path != null) {
+ LOG.info("java.security.krb5.conf: {}", krb5Path);
+ String modifiedGravitinoStartShell =
+ String.format(
+ "%s/bin/gravitino-iceberg-rest-server_%s.sh",
+ icebergRESTServerHome.toString(), UUID.randomUUID());
+ // Replace '/etc/krb5.conf' with the one in the test
+ try {
+ String content =
+ FileUtils.readFileToString(new File(gravitinoRestStartShell),
StandardCharsets.UTF_8);
+ content =
+ content.replace(
+ "#JAVA_OPTS+=\" -Djava.security.krb5.conf=/etc/krb5.conf\"",
+ String.format("JAVA_OPTS+=\" -Djava.security.krb5.conf=%s\"",
krb5Path));
+ File tmp = new File(modifiedGravitinoStartShell);
+ FileUtils.write(tmp, content, StandardCharsets.UTF_8);
+ tmp.setExecutable(true);
+ LOG.info("modifiedGravitinoStartShell content: \n{}", content);
+ CommandExecutor.executeCommandLocalHost(
+ modifiedGravitinoStartShell + " start", false,
ProcessData.TypesOfData.OUTPUT);
+ } catch (Exception e) {
+ LOG.error("Can replace /etc/krb5.conf with real kerberos
configuration", e);
+ }
+ } else {
+ String cmd = String.format("%s/bin/%s start",
icebergRESTServerHome.toString(), SCRIPT_NAME);
+ CommandExecutor.executeCommandLocalHost(
+ cmd,
+ false,
+ ProcessData.TypesOfData.OUTPUT,
+ ImmutableMap.of("GRAVITINO_HOME", icebergRESTServerHome.toString()));
+ }
return Optional.empty();
}
