jerryshao commented on code in PR #12355:
URL: https://github.com/apache/gravitino/pull/12355#discussion_r3710780986
##########
docs/gravitino-server-config.md:
##########
@@ -8,576 +8,659 @@ license: "This software is licensed under the Apache
License version 2."
## Introduction
-Apache Gravitino supports several configurations:
+The Apache Gravitino server reads `conf/gravitino.conf` at startup. Almost
every property has a
+default, so the server starts with an empty file, and most deployments change
only a handful of
+them. The exception is `gravitino.authorization.serviceAdmins`, which you must
set once you turn
+authorization on.
-1. **Gravitino server configuration**: Used to start up the Gravitino server.
-2. **Gravitino catalog properties configuration**: Used to make default values
for different catalogs.
-3. **Some other configurations**: Includes HDFS and other configurations.
+This page covers the server itself. Catalog properties, which configure an
individual catalog
+rather than the server, are covered further down. Properties for the auxiliary
services live with
+those services: see [Iceberg REST Catalog Service](iceberg-rest-service.md) and
+[Security](security/security.md).
-## Gravitino Server Configurations
+## Quick Start
-Customize the Gravitino server by editing the configuration file
`gravitino.conf` in the `conf` directory. The default values are sufficient for
most use cases.
-We strongly recommend that you read the following sections to understand the
configuration file, so you can change the default values to suit your specific
situation and usage scenario.
+### Development
-The `gravitino.conf` file lists the configuration items in the following
table. It groups those items into the following categories:
-
-### HTTP Server Configuration
-
-| Configuration item | Description
| Default value
| Required | Since version |
-|------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------|------------------|
-| `gravitino.server.webserver.host` | The host of the
Gravitino server.
| `0.0.0.0`
| No | 0.1.0 |
-| `gravitino.server.webserver.httpPort` | The port on which the
Gravitino server listens for incoming connections.
| `8090`
| No | 0.1.0 |
-| `gravitino.server.webserver.minThreads` | The minimum number of
threads in the thread pool used by the Jetty webserver. `minThreads` is 8 if
the value is less than 8.
| `Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100),
8)` | No | 0.2.0 |
-| `gravitino.server.webserver.maxThreads` | The maximum number of
threads in the thread pool used by the Jetty webserver. `maxThreads` is 8 if
the value is less than 8, and `maxThreads` must be great or equal to
`minThreads`. | `Math.max(Runtime.getRuntime().availableProcessors() * 4, 400)`
| No | 0.1.0 |
-| `gravitino.server.webserver.threadPoolWorkQueueSize` | The size of the queue
in the thread pool used by the Jetty webserver.
| `100`
| No | 0.1.0 |
-| `gravitino.server.webserver.stopTimeout` | Time in milliseconds
to gracefully shut down the Jetty webserver, for more, see
`org.eclipse.jetty.server.Server#setStopTimeout`.
| `30000`
| No | 0.2.0 |
-| `gravitino.server.webserver.idleTimeout` | The timeout in
milliseconds of idle connections.
| `30000`
| No | 0.2.0 |
-| `gravitino.server.webserver.requestHeaderSize` | Maximum size of HTTP
requests.
| `131072`
| No | 0.1.0 |
-| `gravitino.server.webserver.responseHeaderSize` | Maximum size of HTTP
responses.
| `131072`
| No | 0.1.0 |
-| `gravitino.server.shutdown.timeout` | Time in milliseconds
to gracefully shut down of the Gravitino webserver.
| `3000`
| No | 0.2.0 |
-| `gravitino.server.webserver.customFilters` | Comma-separated list
of filter class names to apply to the API.
| (none)
| No | 0.4.0 |
-| `gravitino.server.rest.extensionPackages` | Comma-separated list
of REST API packages to expand
| (none)
| No | 0.6.0-incubating |
-| `gravitino.server.visibleConfigs` | List of configs that
are visible in the config servlet
| (none)
| No | 0.9.0-incubating |
-
-The filter in the customFilters should be a standard javax servlet filter.
-Specify filter parameters by setting configuration entries of the form
`gravitino.server.webserver.<class name of filter>.param.<param name>=<value>`.
-
-### Storage Configuration
-
-#### Storage Backend Configuration
-
-Gravitino only supports JDBC database backend, and the default implementation
is H2 database as it's an embedded database, has no external dependencies and
is very suitable for local development or tests.
-If you are going to use H2 in the production environment, Gravitino will not
guarantee the data consistency and durability. It's highly recommended using
MySQL as the backend database.
-
-The following table lists the storage configuration items:
-
-| Configuration item | Description
| Default value
| Required | Since
version |
-|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------|-------------------------------------------------|------------------|
-| `gravitino.entity.store` | Which entity storage
implementation to use. Only`relational` storage is supported.
|
`relational` | No
| 0.1.0 |
-| `gravitino.entity.store.maxTransactionSkewTimeMs` | The maximum skew time
of transactions in milliseconds.
| `2000`
| No | 0.3.0
|
-| `gravitino.entity.store.deleteAfterTimeMs` | The maximum time in
milliseconds that deleted and old-version data is kept. Set to at least 10
minutes and no longer than 30 days.
|
`604800000`(7 days) | No
| 0.5.0 |
-| `gravitino.entity.store.versionRetentionCount` | The Count of versions
allowed to be retained, including the current version, used to delete old
versions data. Set to at least 1 and no greater than 10.
| `1`
| No |
0.5.0 |
-| `gravitino.entityChangeLog.pollIntervalSecs` | The interval in seconds
for polling the entity change log. The poller invalidates stale local caches
(e.g. the catalog cache) across HA nodes by consuming change log records. Must
be positive. | `3`
| No |
1.3.0 |
-| `gravitino.entityChangeLog.retentionSecs` | The retention time in
seconds for entity change log rows. Expired rows are pruned periodically. Set
to `0` to disable automatic cleanup. Must be non-negative.
| `86400`(1
day) | No |
1.3.0 |
-| `gravitino.entityChangeLog.cleanupIntervalSecs` | The interval in seconds
for pruning expired entity change log rows. Must be positive.
| `3600`(1
hour) | No |
1.3.0 |
-| `gravitino.entity.store.relational` | Detailed implementation
of Relational storage. `H2`, `MySQL` and `PostgreSQL` is supported, and the
implementation is `JDBCBackend`.
|
`JDBCBackend` | No
| 0.5.0 |
-| `gravitino.entity.store.relational.jdbcUrl` | The database url that
the `JDBCBackend` needs to connect to. If you use `MySQL` or `PostgreSQL`, you
should firstly initialize the database tables yourself by executing the ddl
scripts in the `${GRAVITINO_HOME}/scripts/{DATABASE_TYPE}/` directory. |
`jdbc:h2` | No
| 0.5.0 |
-| `gravitino.entity.store.relational.jdbcDriver` | The jdbc driver name
that the `JDBCBackend` needs to use. You should place the driver Jar package in
the `${GRAVITINO_HOME}/libs/` directory.
|
`org.h2.Driver` | Yes if the jdbc connection url is not
`jdbc:h2` | 0.5.0 |
-| `gravitino.entity.store.relational.jdbcUser` | The username that the
`JDBCBackend` needs to use when connecting the database. It is required for
`MySQL`.
|
`gravitino` | Yes if the jdbc connection url is not
`jdbc:h2` | 0.5.0 |
-| `gravitino.entity.store.relational.jdbcPassword` | The password that the
`JDBCBackend` needs to use when connecting the database. It is required for
`MySQL`.
|
`gravitino` | Yes if the jdbc connection url is not
`jdbc:h2` | 0.5.0 |
-| `gravitino.entity.store.relational.storagePath` | The storage path for
embedded JDBC storage implementation. It supports both absolute and relative
path, if the value is a relative path, the final path is
`${GRAVITINO_HOME}/${PATH_YOU_HAVA_SET}`, default value is
`${GRAVITINO_HOME}/data/jdbc` | `${GRAVITINO_HOME}/data/jdbc` | No
| 0.6.0-incubating |
-| `gravitino.entity.store.relational.maxConnections` | The maximum number of
connections for the JDBC Backend connection pool
| `100`
| No |
0.9.0-incubating |
-| `gravitino.entity.store.relational.maxWaitMillis` | The maximum wait time
in milliseconds for a connection from the JDBC Backend connection pool
| `1000`
| No |
0.9.0-incubating |
-
-
-:::caution
-We strongly recommend that you change the default value of
`gravitino.entity.store.relational.storagePath`, as it's under the deployment
directory and future version upgrades may remove it.
-:::
-
-#### Create JDBC Backend Schema and Table
-
-For H2 database, All tables needed by Gravitino are created automatically when
the Gravitino server starts up. For MySQL, you should firstly initialize the
database tables yourself by executing the ddl scripts in the
`${GRAVITINO_HOME}/scripts/mysql/` directory.
-
-### Storage Cache Configuration
-
-To enable storage caching, modify the following settings in the
`${GRAVITINO_HOME}/conf/gravitino.conf` file:
+The defaults are already right for local work. The server listens on
`0.0.0.0:8090` and keeps its
+metadata in an embedded H2 database, so no configuration is needed:
+```shell
+${GRAVITINO_HOME}/bin/gravitino.sh start
```
-# Whether to enable the cache
-gravitino.cache.enabled=true
-# Specify the cache implementation (no need to use the fully qualified class
name)
-gravitino.cache.implementation=caffeine
+One property is still worth setting. By default, H2 writes its database files
to
+`${GRAVITINO_HOME}/data/jdbc`, which is inside the unpacked distribution.
Upgrading Gravitino
+means unpacking a new distribution, so the metadata sits in the very directory
you are about to
+replace or abandon. Move it somewhere the upgrade does not touch:
-# Number of lock segments for cache concurrency optimization
-gravitino.cache.lockSegments=16
+```text
+# conf/gravitino.conf
+gravitino.entity.store.relational.storagePath = /var/lib/gravitino/data/jdbc
```
-| Configuration Key | Description
| Default Value | Required | Since Version |
-|----------------------------------|--------------------------------------------|------------------------|----------|---------------|
-| `gravitino.cache.enabled` | Whether to enable caching
| `true` | Yes | 1.0.0 |
-| `gravitino.cache.implementation` | Specifies the cache implementation
| `caffeine` | Yes | 1.0.0 |
-| `gravitino.cache.maxEntries` | Maximum number of entries allowed in
cache | `10000` | No | 1.0.0 |
-| `gravitino.cache.expireTimeInMs` | Cache expiration time (in milliseconds)
| `3600000` (about 1 hr) | No | 1.0.0 |
-| `gravitino.cache.enableStats` | Whether to enable cache statistics
logging | `false` | No | 1.0.0 |
-| `gravitino.cache.enableWeigher` | Whether to enable weight-based eviction
| `true` | No | 1.0.0 |
-| `gravitino.cache.lockSegments` | Number of lock segments.
| `16` | No | 1.0.0 |
-
-- `gravitino.cache.enableWeigher`: When enabled, eviction is based on weight
and `maxEntries` will be ignored.
-- `gravitino.cache.expireTimeInMs`: Controls the cache TTL in milliseconds.
-- If `gravitino.cache.enableStats` is enabled, Gravitino will log cache
statistics (hit count, miss count, load failures, etc.) every 5 minutes at the
Info level.
-
-#### Eviction Strategies
-
-Gravitino supports multiple eviction strategies including capacity-based,
weight-based, and time-based (TTL) eviction. The following describes how they
work with Caffeine:
-
-##### Capacity-based eviction
-
-When `gravitino.cache.enableWeigher` is **disabled**, Gravitino limits the
number of cached entries using `gravitino.cache.maxEntries` and employs
Caffeine’s W-TinyLFU eviction policy to remove the least-used entries when the
cache is full.
-
-##### Weight-based eviction
-
-When `gravitino.cache.enableWeigher` is **enabled**, Gravitino uses a
combination of `maximumWeight` and a custom weigher to control the total weight
of the cache:
-
-- Each entity type has a default weight (e.g., Metalake > Catalog > Schema);
-- Entries are evicted based on the combined weight limit (`maximumWeight`);
-- If a single cache item exceeds the total weight limit, it will not be cached;
-- When this strategy is active, `maxEntries` will be ignored.
-
-##### Time-based eviction
-
-All cache entries are subject to a TTL (Time-To-Live) expiration policy. By
default, the TTL is `3600000ms` (1 hour) and can be adjusted via the
`gravitino.cache.expireTimeInMs` setting:
+Nothing here is authenticated. The default `simple` authenticator takes
whatever username the
+client sends, the server speaks plain HTTP, and authorization is off, so every
caller can do
+everything. Together with H2, for which Gravitino makes no consistency or
durability guarantee,
+that makes this configuration unfit for anything but local work.
+
+### Production
+
+A production server keeps metadata in MySQL or PostgreSQL, authenticates its
callers, enforces
+authorization, and writes an audit log:
+
+```text
+# conf/gravitino.conf
+# Entity store
+gravitino.entity.store.relational.jdbcUrl =
jdbc:mysql://{db_host}:3306/{database}
+gravitino.entity.store.relational.jdbcDriver = com.mysql.cj.jdbc.Driver
+gravitino.entity.store.relational.jdbcUser = {username}
+gravitino.entity.store.relational.jdbcPassword = {password}
+
+# Transport
+gravitino.server.webserver.enableHttps = true
+gravitino.server.webserver.keyStorePath = /etc/gravitino/tls/server.jks
+gravitino.server.webserver.keyStorePassword = {keystore_password}
+gravitino.server.webserver.managerPassword = {manager_password}
+
+# Authentication
+gravitino.authenticators = oauth
+gravitino.authenticator.oauth.jwksUri = {jwks_uri}
+gravitino.authenticator.oauth.tokenValidatorClass =
org.apache.gravitino.server.authentication.JwksTokenValidator
+gravitino.authenticator.oauth.serviceAudience = {audience}
+gravitino.authenticator.oauth.principalFields =
preferred_username,email,sub
+
+# Authorization
+gravitino.authorization.enable = true
+gravitino.authorization.serviceAdmins = {admin_user}
+
+# Audit log
+gravitino.audit.enabled = true
+
+# Entity cache
+gravitino.cache.enabled = true
+gravitino.cache.implementation = caffeine
+gravitino.cache.lockSegments = 16
+gravitino.cache.enableStats = true
+```
-- TTL starts at the time of entry creation; once it exceeds the configured
duration, the entry expires automatically;
-- TTL can work in conjunction with both capacity and weight-based eviction;
-- Expired entries will also trigger asynchronous cleanup mechanisms for
resource release and logging.
+Only `gravitino.cache.enableStats` changes behavior here; it logs hit count,
miss count, and load
+failures every five minutes, which is what makes a cache problem visible in
production. The three
+lines above it restate defaults, and are spelled out so the cache
configuration is reviewable in
+one place rather than inferred from its absence. Raise `lockSegments` above
the default if the
+server runs hot enough for cache lock contention to show up in profiles.
-### Job Configuration
+Four things this block depends on:
-The following table lists the job configuration items:
-
-| Configuration Item | Description
| Default Value
| Required | Since Version |
-|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|----------|---------------|
-| `gravitino.job.stagingDir` | Directory for managing staging
files when running jobs.
|
`/tmp/gravitino/jobs/staging` | No | 1.0.0 |
-| `gravitino.job.executor` | The executor to run jobs. By
default it is `local`; users can implement their own executor and set it here.
| `local`
| No | 1.0.0 |
-| `gravitino.job.stagingDirKeepTimeInMs` | The time in milliseconds to keep
the staging files of the finished job in the job staging directory. The minimum
recommended value is 10 minutes if you are not testing. | `604800000` (7 days)
| No | 1.0.0 |
-| `gravitino.job.statusPullIntervalInMs` | The interval in milliseconds to
pull the job status from the job executor. The minimum recommended value is 1
minute if you are not testing. | `300000` (5
minutes) | No | 1.0.0 |
-
-### Tree Lock Configuration
-
-The Gravitino server uses a tree lock to ensure data consistency. The tree
lock is an in-memory lock; Gravitino currently supports only in-memory locks.
The configuration items are as follows:
+**The database schema is not created for you.** Initialize it and put the JDBC
driver jar in
+`${GRAVITINO_HOME}/libs/` before the first start. See
+[Relational Backend Storage](how-to-use-relational-backend-storage.md).
-| Configuration item | Description
| Default value | Required | Since Version |
-|--------------------------------------|---------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.lock.maxNodes` | The maximum number of tree lock nodes
to keep in memory | 100000 | No | 0.5.0 |
-| `gravitino.lock.minNodes` | The minimum number of tree lock nodes
to keep in memory | 1000 | No | 0.5.0 |
-| `gravitino.lock.cleanIntervalInSecs` | The interval in seconds to clean up
the stale tree lock nodes | 60 | No | 0.5.0 |
+**HTTPS replaces HTTP rather than joining it.** A server with `enableHttps`
set no longer serves
+plain HTTP, so clients and the Web UI must move to `httpsPort`, which defaults
to `8433`. See
+[HTTPS](security/how-to-use-https.md).
-### Catalog Configuration
+**The authenticator is one line, its provider is not.** The block above
validates JWTs against a
+JWKS endpoint, which is the common case for an external identity provider.
Static sign keys,
+Kerberos, and the OIDC login flow for the Web UI each take a different set of
properties. See
+[How to Authenticate](security/how-to-authenticate.md), or
+[How to Use the Built-in IdP](security/how-to-use-built-in-idp.md) to keep
users and groups in
+Gravitino's own metadata store instead of an external provider.
-| Configuration item | Description
| Default value | Required | Since version |
-|----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.catalog.cache.evictionIntervalMs` | The interval in
milliseconds to evict the catalog cache; default 3600000ms(1h).
| `3600000` | No |
0.1.0 |
-| `gravitino.catalog.classloader.isolated` | Whether to use an
isolated classloader for catalog. If `true`, an isolated classloader loads all
catalog-related libraries and configurations, not the AppClassLoader. The
default value is `true`.
| `true` | No
| 0.1.0 |
-| `gravitino.catalog.credential.backfillToProperties` | For backward
compatibility only: if `true`, the server exposes hidden catalog credentials
(such as jdbc-user and jdbc-password) in the catalog properties response so
that connectors that do not support credential vending can still read them.
**Enabling this is a security risk** — credentials are visible to anyone who
can read catalog properties. Disable once all connectors are upgraded. |
`false` | No | 1.3.0 |
+**`gravitino.authorization.serviceAdmins` has no default.** It is the one
property here that
+Gravitino will not fill in for you, and enabling authorization without it
fails at startup. What
+those admins and everyone else may then do is the subject of
+[Access Control](security/access-control.md).
-### Schema configuration
+Give the JVM more than the 1 GB it takes by default:
-| Configuration item | Description
| Default value | Required | Since version |
-|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.schema.separator` | The separator used to represent a
hierarchical (multi-level) schema in schema names at the API boundary, e.g. `:`
for `A:B:C`. It only takes effect for catalogs that support hierarchical
schemas (currently the Iceberg catalog). The value must not be blank and must
not contain `.` or the internal physical separator (ASCII-1, `\u0001`). See
[Hierarchical schema](./lakehouse-iceberg-catalog.md#hierarchical-schema). |
`:` | No | 1.3.0 |
+```shell
+export GRAVITINO_MEM="-Xms4g -Xmx4g -XX:MaxMetaspaceSize=1g"
+```
-### Auxiliary Service Configuration
+### Docker
-| Configuration item | Description
| Default value | Since Version |
-|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------|---------------|---------------|
-| `gravitino.auxService.names ` | The auxiliary service name of the Gravitino
Iceberg REST server. Use **`iceberg-rest`** for the Gravitino Iceberg REST
server. | (none) | 0.2.0 |
+The Gravitino image does not simply run the server against the configuration
file you give it. At
+startup the entrypoint rewrites `conf/gravitino.conf`, applying its own
defaults over roughly two
+dozen properties and then applying any supported environment variables.
Configure the container
+through environment variables:
-Refer to [Iceberg REST catalog service](iceberg-rest-service.md) for
configuration details.
+```shell
+docker run --rm -d \
+ -p 8090:8090 \
+ -e
GRAVITINO_ENTITY_STORE_RELATIONAL_JDBC_URL="jdbc:postgresql://{db_host}:5432/{database}"
\
+ -e GRAVITINO_ENTITY_STORE_RELATIONAL_JDBC_DRIVER="org.postgresql.Driver" \
+ apache/gravitino:{tag}
+```
-### Event Listener Configuration
+To supply a configuration file instead, for example from a Kubernetes
ConfigMap, disable the
+rewrite with `SKIP_CONFIG_REWRITE=true`. See
+[Container Configuration](#container-configuration) for what the rewrite does
and which variables
+it recognizes.
+
+### Running More Than One Server
+
+Servers behind a load balancer share the entity store but keep local caches.
Each server polls the
+entity change log and invalidates entries that another server has modified.
The defaults are safe:
Review Comment:
Fixed, please check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
