FANNG1 commented on code in PR #8358:
URL: https://github.com/apache/gravitino/pull/8358#discussion_r2313006878
##########
docs/iceberg-rest-service.md:
##########
@@ -81,6 +88,116 @@ Please note that, it only takes affect in `gravitino.conf`,
you don't need to sp
The filter in `customFilters` should be a standard javax servlet filter.
You can also specify filter parameters by setting configuration entries in the
style `gravitino.iceberg-rest.<class name of filter>.param.<param
name>=<value>`.
+### Catalog backend configuration
+
+:::info
+The Gravitino Iceberg REST catalog service uses the memory catalog backend by
default. You can specify a Hive or JDBC catalog backend for production
environment.
+:::
+
+#### Hive backend configuration
+
+| Configuration item |
Description
| Default value
| Required |
Since Version |
+|---------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|----------|---------------|
+| `gravitino.iceberg-rest.catalog-backend` |
The Catalog backend of the Gravitino Iceberg REST catalog service. Use the
value **`hive`** for a Hive catalog. | `memory`
| Yes |
0.2.0 |
+| `gravitino.iceberg-rest.uri` |
The Hive metadata address, such as `thrift://127.0.0.1:9083`.
| (none)
| Yes |
0.2.0 |
+| `gravitino.iceberg-rest.warehouse` |
The warehouse directory of the Hive catalog, such as
`/user/hive/warehouse-hive/`.
| (none)
| Yes | 0.2.0 |
+| `gravitino.iceberg-rest.catalog-backend-name` |
The catalog backend name passed to underlying Iceberg catalog backend. Catalog
name in JDBC backend is used to isolate namespace and tables. | `hive` for Hive
backend, `jdbc` for JDBC backend, `memory` for memory backend | No |
0.5.2 |
+
+#### JDBC backend configuration
+
+| Configuration item | Description
| Default value | Required | Since Version |
+|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|---------------|
+| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the
Gravitino Iceberg REST catalog service. Use the value **`jdbc`** for a JDBC
catalog. | `memory` | Yes | 0.2.0
|
+| `gravitino.iceberg-rest.uri` | The JDBC connection address,
such as `jdbc:postgresql://127.0.0.1:5432` for Postgres, or
`jdbc:mysql://127.0.0.1:3306/` for mysql. | (none) | Yes
| 0.2.0 |
+| `gravitino.iceberg-rest.warehouse ` | The warehouse directory of
JDBC catalog. Set the HDFS prefix if using HDFS, such as
`hdfs://127.0.0.1:9000/user/hive/warehouse-jdbc` | (none) |
Yes | 0.2.0 |
+| `gravitino.iceberg-rest.catalog-backend-name` | The catalog name passed to
underlying Iceberg catalog backend. Catalog name in JDBC backend is used to
isolate namespace and tables. | `jdbc` for JDBC backend | No | 0.5.2
|
+| `gravitino.iceberg-rest.jdbc-user` | The username of the JDBC
connection.
| (none) | No | 0.2.0
|
+| `gravitino.iceberg-rest.jdbc-password` | The password of the JDBC
connection.
| (none) | No | 0.2.0
|
+| `gravitino.iceberg-rest.jdbc-initialize` | Whether to initialize the
meta tables when creating the JDBC catalog.
| `true` | No | 0.2.0
|
+| `gravitino.iceberg-rest.jdbc-driver` | `com.mysql.jdbc.Driver` or
`com.mysql.cj.jdbc.Driver` for MySQL, `org.postgresql.Driver` for PostgreSQL.
| (none) | Yes | 0.3.0
|
+
+If you have a JDBC Iceberg catalog prior, you must set `catalog-backend-name`
to keep consistent with your Jdbc Iceberg catalog name to operate the prior
namespace and tables.
+
+:::caution
+You must download the corresponding JDBC driver to the
`iceberg-rest-server/libs` directory.
+If you are using multiple JDBC catalog backends, setting `jdbc-initialize` to
true may not take effect for RDMS like `Mysql`, you should create Iceberg meta
tables explicitly.
+:::
+
+#### Custom backend configuration
+
+| Configuration item | Description
| Default value | Required | Since Version |
+|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `gravitino.iceberg-rest.catalog-backend` | The Catalog backend of the
Gravitino Iceberg REST catalog service. Use the value **`custom`** for a Custom
catalog. | `memory` | Yes | 0.2.0 |
+| `gravitino.iceberg-rest.catalog-backend-impl` | The fully-qualified class
name of a custom catalog implementation, only worked if `catalog-backend` is
`custom`. | (none) | No | 0.7.0-incubating |
+
+If you want to use a custom Iceberg Catalog as `catalog-backend`, you can add
a corresponding jar file to the classpath and load a custom Iceberg Catalog
implementation by specifying the `catalog-backend-impl` property.
+
+### Multi catalog support
+
+The Gravitino Iceberg REST server supports multiple catalogs, and you could
use `static-config-provider` to manage catalog config in the file, or use
`dynamic-config-provider` to manage catalog config by Gravitino server.
+
+| Configuration item | Description
|
Default value | Required | Since Version |
+|--------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|------------------|
+| `gravitino.iceberg-rest.catalog-config-provider` | The className of catalog
configuration provider, Gravitino provides build-in `static-config-provider`
and `dynamic-config-provider`, you could also develop a custom class that
implements `apache.gravitino.iceberg.service.provider.IcebergConfigProvider`
and add the corresponding jar file to the Iceberg REST service classpath
directory. | `static-config-provider` | No | 0.7.0-incubating |
+
+#### Static catalog configuration provider
+
+The static catalog configuration provider retrieves the catalog configuration
from the configuration file of the Gravitino Iceberg REST server. You could
configure the default catalog with `gravitino.iceberg-rest.<param
name>=<value>`. For others, use `gravitino.iceberg-rest.catalog.<catalog
name>.<param name>=<value>` to config the catalog with `catalog name`.
+
+For instance, you could configure three different catalogs, the default
catalog and `hive_backend` and `jdbc_backend` catalogs separately.
+
+```text
+gravitino.iceberg-rest.catalog-backend = jdbc
+gravitino.iceberg-rest.uri = jdbc:postgresql://127.0.0.1:5432
+gravitino.iceberg-rest.warehouse =
hdfs://127.0.0.1:9000/user/hive/warehouse-postgresql
+...
+gravitino.iceberg-rest.catalog.hive_backend.catalog-backend = hive
+gravitino.iceberg-rest.catalog.hive_backend.uri = thrift://127.0.0.1:9084
+gravitino.iceberg-rest.catalog.hive_backend.warehouse =
/user/hive/warehouse-hive/
+...
+gravitino.iceberg-rest.catalog.jdbc_backend.catalog-backend = jdbc
+gravitino.iceberg-rest.catalog.jdbc_backend.uri = jdbc:mysql://127.0.0.1:3306/
+gravitino.iceberg-rest.catalog.jdbc_backend.warehouse =
hdfs://127.0.0.1:9000/user/hive/warehouse-mysql
+...
+```
+
+You can access different catalogs by setting the `warehouse` to the specific
catalog name in the Iceberg REST client configuration. The default catalog will
be used if you do not specify a `warehouse`. For instance, consider the case of
SparkSQL.
+
+```shell
+./bin/spark-sql -v \
+...
+--conf spark.sql.catalog.default_rest_catalog.type=rest \
+--conf
spark.sql.catalog.default_rest_catalog.uri=http://127.0.0.1:9001/iceberg/ \
+...
+--conf spark.sql.catalog.hive_backend_catalog.type=rest \
+--conf
spark.sql.catalog.hive_backend_catalog.uri=http://127.0.0.1:9001/iceberg/ \
+--conf spark.sql.catalog.hive_backend_catalog.warehouse=hive_backend \
+...
+--conf spark.sql.catalog.jdbc_backend_catalog.type=rest \
+--conf
spark.sql.catalog.jdbc_backend_catalog.uri=http://127.0.0.1:9001/iceberg/ \
+--conf spark.sql.catalog.jdbc_backend_catalog.warehouse=jdbc_backend \
+...
+```
+
+#### Dynamic catalog configuration provider
+
+The dynamic catalog configuration provider retrieves the catalog configuration
from the Gravitino server.
+
+| Configuration item | Description
| Default value |
Required | Since Version |
+|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
+| `gravitino.iceberg-rest.gravitino-uri` | The uri of
Gravitino server address, only worked if `catalog-config-provider` is
`dynamic-config-provider`. | (none)
| No | 0.7.0-incubating |
+| `gravitino.iceberg-rest.gravitino-metalake` | The metalake
name that `dynamic-config-provider` used to request to Gravitino, only worked
if `catalog-config-provider` is `dynamic-config-provider`. | (none) | No
| 0.7.0-incubating |
+| `gravitino.iceberg-rest.catalog-cache-eviction-interval-ms` | Catalog cache
eviction interval.
| 3600000 | No
| 0.7.0-incubating |
+
+```text
+gravitino.iceberg-rest.catalog-cache-eviction-interval-ms = 300000
+gravitino.iceberg-rest.catalog-config-provider = dynamic-config-provider
+gravitino.iceberg-rest.gravitino-uri = http://127.0.0.1:8090
+gravitino.iceberg-rest.gravitino-metalake = test
+```
+
+You can access different catalogs by setting the `warehouse` to the catalog
name in Gravitino server in the client side.
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
