jerryshao commented on code in PR #12291:
URL: https://github.com/apache/gravitino/pull/12291#discussion_r3688500877
##########
docs/gravitino-server-config.md:
##########
@@ -8,587 +8,662 @@ license: "This software is licensed under the Apache
License version 2."
## Introduction
-Apache Gravitino supports several configurations:
+The Apache Gravitino server reads `conf/gravitino.conf` at startup. Almost
every property has a
+default, so the server starts with an empty file, and most deployments change
only a handful of
+them. The exception is `gravitino.authorization.serviceAdmins`, which you must
set once you turn
+authorization on.
-1. **Gravitino server configuration**: Used to start up the Gravitino server.
-2. **Gravitino catalog properties configuration**: Used to make default values
for different catalogs.
-3. **Some other configurations**: Includes HDFS and other configurations.
+This page covers the server itself. Catalog properties, which configure an
individual catalog
+rather than the server, are covered further down. Properties for the auxiliary
services live with
+those services: see [Iceberg REST Catalog Service](iceberg-rest-service.md) and
+[Security](security/security.md).
-## Gravitino Server Configurations
+## Quick Start
-Customize the Gravitino server by editing the configuration file
`gravitino.conf` in the `conf` directory. The default values are sufficient for
most use cases.
-We strongly recommend that you read the following sections to understand the
configuration file, so you can change the default values to suit your specific
situation and usage scenario.
+### Development
-The `gravitino.conf` file lists the configuration items in the following
table. It groups those items into the following categories:
-
-### HTTP Server Configuration
-
-| Configuration item | Description
| Default value
| Required | Since version |
-|------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------|------------------|
-| `gravitino.server.webserver.host` | The host of the
Gravitino server.
| `0.0.0.0`
| No | 0.1.0 |
-| `gravitino.server.webserver.httpPort` | The port on which the
Gravitino server listens for incoming connections.
| `8090`
| No | 0.1.0 |
-| `gravitino.server.webserver.minThreads` | The minimum number of
threads in the thread pool used by the Jetty webserver. `minThreads` is 8 if
the value is less than 8.
| `Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100),
8)` | No | 0.2.0 |
-| `gravitino.server.webserver.maxThreads` | The maximum number of
threads in the thread pool used by the Jetty webserver. `maxThreads` is 8 if
the value is less than 8, and `maxThreads` must be great or equal to
`minThreads`. | `Math.max(Runtime.getRuntime().availableProcessors() * 4, 400)`
| No | 0.1.0 |
-| `gravitino.server.webserver.threadPoolWorkQueueSize` | The size of the queue
in the thread pool used by the Jetty webserver.
| `100`
| No | 0.1.0 |
-| `gravitino.server.webserver.stopTimeout` | Time in milliseconds
to gracefully shut down the Jetty webserver, for more, see
`org.eclipse.jetty.server.Server#setStopTimeout`.
| `30000`
| No | 0.2.0 |
-| `gravitino.server.webserver.idleTimeout` | The timeout in
milliseconds of idle connections.
| `30000`
| No | 0.2.0 |
-| `gravitino.server.webserver.requestHeaderSize` | Maximum size of HTTP
requests.
| `131072`
| No | 0.1.0 |
-| `gravitino.server.webserver.responseHeaderSize` | Maximum size of HTTP
responses.
| `131072`
| No | 0.1.0 |
-| `gravitino.server.shutdown.timeout` | Time in milliseconds
to gracefully shut down of the Gravitino webserver.
| `3000`
| No | 0.2.0 |
-| `gravitino.server.webserver.customFilters` | Comma-separated list
of filter class names to apply to the API.
| (none)
| No | 0.4.0 |
-| `gravitino.server.rest.extensionPackages` | Comma-separated list
of REST API packages to expand
| (none)
| No | 0.6.0-incubating |
-| `gravitino.server.visibleConfigs` | List of configs that
are visible in the config servlet
| (none)
| No | 0.9.0-incubating |
-
-The filter in the customFilters should be a standard javax servlet filter.
-Specify filter parameters by setting configuration entries of the form
`gravitino.server.webserver.<class name of filter>.param.<param name>=<value>`.
-
-### Storage Configuration
-
-#### Storage Backend Configuration
-
-Gravitino only supports JDBC database backend, and the default implementation
is H2 database as it's an embedded database, has no external dependencies and
is very suitable for local development or tests.
-If you are going to use H2 in the production environment, Gravitino will not
guarantee the data consistency and durability. It's highly recommended using
MySQL as the backend database.
-
-The following table lists the storage configuration items:
-
-| Configuration item | Description
| Default value
| Required | Since
version |
-|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------|-------------------------------------------------|------------------|
-| `gravitino.entity.store` | Which entity storage
implementation to use. Only`relational` storage is supported.
|
`relational` | No
| 0.1.0 |
-| `gravitino.entity.store.maxTransactionSkewTimeMs` | The maximum skew time
of transactions in milliseconds.
| `2000`
| No | 0.3.0
|
-| `gravitino.entity.store.deleteAfterTimeMs` | The maximum time in
milliseconds that deleted and old-version data is kept. Set to at least 10
minutes and no longer than 30 days.
|
`604800000`(7 days) | No
| 0.5.0 |
-| `gravitino.entity.store.versionRetentionCount` | The Count of versions
allowed to be retained, including the current version, used to delete old
versions data. Set to at least 1 and no greater than 10.
| `1`
| No |
0.5.0 |
-| `gravitino.entityChangeLog.pollIntervalSecs` | The interval in seconds
for polling the entity change log. The poller invalidates stale local caches
(e.g. the catalog cache) across HA nodes by consuming change log records. Must
be positive. | `3`
| No |
1.3.0 |
-| `gravitino.entityChangeLog.listenerMaxRetries` | The number of times the
poller retries a change log batch for a failing listener before applying
`gravitino.entityChangeLog.listenerFailureAction`. Must be non-negative.
| `10`
| No |
2.0.0 |
-| `gravitino.entityChangeLog.listenerFailureAction` | What the poller does
once a listener exhausted its retries. `EXIT` stops this server, because its
local caches are known to be stale and no longer safe to serve from; `SKIP`
drops the batch for that listener and keeps serving. |
`EXIT` | No
| 2.0.0 |
-| `gravitino.entityChangeLog.retentionSecs` | The retention time in
seconds for entity change log rows. A dedicated cleaner removes rows older than
this period using database time. Set to `0` to disable cleanup, otherwise at
least 10x `pollIntervalSecs`. |
`2592000`(30 days) | No
| 1.3.0 |
-| `gravitino.entityChangeLog.cleanupIntervalSecs` | The interval in seconds
for running the dedicated entity change log cleaner. Must be positive.
| `86400`(1
day) | No |
1.3.0 |
-| `gravitino.entity.store.relational` | Detailed implementation
of Relational storage. `H2`, `MySQL` and `PostgreSQL` is supported, and the
implementation is `JDBCBackend`.
|
`JDBCBackend` | No
| 0.5.0 |
-| `gravitino.entity.store.relational.jdbcUrl` | The database url that
the `JDBCBackend` needs to connect to. If you use `MySQL` or `PostgreSQL`, you
should firstly initialize the database tables yourself by executing the ddl
scripts in the `${GRAVITINO_HOME}/scripts/{DATABASE_TYPE}/` directory. |
`jdbc:h2` | No
| 0.5.0 |
-| `gravitino.entity.store.relational.jdbcDriver` | The jdbc driver name
that the `JDBCBackend` needs to use. You should place the driver Jar package in
the `${GRAVITINO_HOME}/libs/` directory.
|
`org.h2.Driver` | Yes if the jdbc connection url is not
`jdbc:h2` | 0.5.0 |
-| `gravitino.entity.store.relational.jdbcUser` | The username that the
`JDBCBackend` needs to use when connecting the database. It is required for
`MySQL`.
|
`gravitino` | Yes if the jdbc connection url is not
`jdbc:h2` | 0.5.0 |
-| `gravitino.entity.store.relational.jdbcPassword` | The password that the
`JDBCBackend` needs to use when connecting the database. It is required for
`MySQL`.
|
`gravitino` | Yes if the jdbc connection url is not
`jdbc:h2` | 0.5.0 |
-| `gravitino.entity.store.relational.storagePath` | The storage path for
embedded JDBC storage implementation. It supports both absolute and relative
path, if the value is a relative path, the final path is
`${GRAVITINO_HOME}/${PATH_YOU_HAVA_SET}`, default value is
`${GRAVITINO_HOME}/data/jdbc` | `${GRAVITINO_HOME}/data/jdbc` | No
| 0.6.0-incubating |
-| `gravitino.entity.store.relational.maxConnections` | The maximum number of
connections for the JDBC Backend connection pool
| `100`
| No |
0.9.0-incubating |
-| `gravitino.entity.store.relational.maxWaitMillis` | The maximum wait time
in milliseconds for a connection from the JDBC Backend connection pool
| `1000`
| No |
0.9.0-incubating |
-
-
-:::caution
-We strongly recommend that you change the default value of
`gravitino.entity.store.relational.storagePath`, as it's under the deployment
directory and future version upgrades may remove it.
-:::
-
-#### Create JDBC Backend Schema and Table
-
-For H2 database, All tables needed by Gravitino are created automatically when
the Gravitino server starts up. For MySQL, you should firstly initialize the
database tables yourself by executing the ddl scripts in the
`${GRAVITINO_HOME}/scripts/mysql/` directory.
-
-### Storage Cache Configuration
-
-To enable storage caching, modify the following settings in the
`${GRAVITINO_HOME}/conf/gravitino.conf` file:
+The defaults are already right for local work. The server listens on
`0.0.0.0:8090` and keeps its
+metadata in an embedded H2 database, so no configuration is needed:
+```shell
+${GRAVITINO_HOME}/bin/gravitino.sh start
```
-# Whether to enable the cache
-gravitino.cache.enabled=true
-# Specify the cache implementation (no need to use the fully qualified class
name)
-gravitino.cache.implementation=caffeine
+One property is still worth setting. By default, H2 writes its database files
to
+`${GRAVITINO_HOME}/data/jdbc`, which is inside the unpacked distribution.
Upgrading Gravitino
+means unpacking a new distribution, so the metadata sits in the very directory
you are about to
+replace or abandon. Move it somewhere the upgrade does not touch:
-# Number of lock segments for cache concurrency optimization
-gravitino.cache.lockSegments=16
+```text
+# conf/gravitino.conf
+gravitino.entity.store.relational.storagePath = /var/lib/gravitino/data/jdbc
```
-| Configuration Key | Description
| Default Value | Required | Since Version |
-|----------------------------------|--------------------------------------------|------------------------|----------|---------------|
-| `gravitino.cache.enabled` | Whether to enable caching
| `true` | Yes | 1.0.0 |
-| `gravitino.cache.implementation` | Specifies the cache implementation
| `caffeine` | Yes | 1.0.0 |
-| `gravitino.cache.maxEntries` | Maximum number of entries allowed in
cache | `10000` | No | 1.0.0 |
-| `gravitino.cache.expireTimeInMs` | Cache expiration time (in milliseconds)
| `3600000` (about 1 hr) | No | 1.0.0 |
-| `gravitino.cache.enableStats` | Whether to enable cache statistics
logging | `false` | No | 1.0.0 |
-| `gravitino.cache.enableWeigher` | Whether to enable weight-based eviction
| `true` | No | 1.0.0 |
-| `gravitino.cache.lockSegments` | Number of lock segments.
| `16` | No | 1.0.0 |
-
-- `gravitino.cache.enableWeigher`: When enabled, eviction is based on weight
and `maxEntries` will be ignored.
-- `gravitino.cache.expireTimeInMs`: Controls the cache TTL in milliseconds.
-- If `gravitino.cache.enableStats` is enabled, Gravitino will log cache
statistics (hit count, miss count, load failures, etc.) every 5 minutes at the
Info level.
-
-#### Eviction Strategies
-
-Gravitino supports multiple eviction strategies including capacity-based,
weight-based, and time-based (TTL) eviction. The following describes how they
work with Caffeine:
-
-##### Capacity-based eviction
-
-When `gravitino.cache.enableWeigher` is **disabled**, Gravitino limits the
number of cached entries using `gravitino.cache.maxEntries` and employs
Caffeine’s W-TinyLFU eviction policy to remove the least-used entries when the
cache is full.
-
-##### Weight-based eviction
-
-When `gravitino.cache.enableWeigher` is **enabled**, Gravitino uses a
combination of `maximumWeight` and a custom weigher to control the total weight
of the cache:
-
-- Each entity type has a default weight (e.g., Metalake > Catalog > Schema);
-- Entries are evicted based on the combined weight limit (`maximumWeight`);
-- If a single cache item exceeds the total weight limit, it will not be cached;
-- When this strategy is active, `maxEntries` will be ignored.
-
-##### Time-based eviction
-
-All cache entries are subject to a TTL (Time-To-Live) expiration policy. By
default, the TTL is `3600000ms` (1 hour) and can be adjusted via the
`gravitino.cache.expireTimeInMs` setting:
-
-- TTL starts at the time of entry creation; once it exceeds the configured
duration, the entry expires automatically;
-- TTL can work in conjunction with both capacity and weight-based eviction;
-- Expired entries will also trigger asynchronous cleanup mechanisms for
resource release and logging.
-
-### Job Configuration
-
-The following table lists the job configuration items:
-
-| Configuration Item | Description
| Default Value
| Required | Since Version |
-|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|----------|---------------|
-| `gravitino.job.stagingDir` | Directory for managing staging
files when running jobs.
|
`/tmp/gravitino/jobs/staging` | No | 1.0.0 |
-| `gravitino.job.executor` | The executor to run jobs. By
default it is `local`; users can implement their own executor and set it here.
| `local`
| No | 1.0.0 |
-| `gravitino.job.stagingDirKeepTimeInMs` | The time in milliseconds to keep
the staging files of the finished job in the job staging directory. The minimum
recommended value is 10 minutes if you are not testing. | `604800000` (7 days)
| No | 1.0.0 |
-| `gravitino.job.statusPullIntervalInMs` | The interval in milliseconds to
pull the job status from the job executor. The minimum recommended value is 1
minute if you are not testing. | `300000` (5
minutes) | No | 1.0.0 |
-
-### Tree Lock Configuration
-
-The Gravitino server uses a tree lock to ensure data consistency. The tree
lock is an in-memory lock; Gravitino currently supports only in-memory locks.
The configuration items are as follows:
-
-| Configuration item | Description
| Default value | Required | Since Version |
-|--------------------------------------|---------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.lock.maxNodes` | The maximum number of tree lock nodes
to keep in memory | 100000 | No | 0.5.0 |
-| `gravitino.lock.minNodes` | The minimum number of tree lock nodes
to keep in memory | 1000 | No | 0.5.0 |
-| `gravitino.lock.cleanIntervalInSecs` | The interval in seconds to clean up
the stale tree lock nodes | 60 | No | 0.5.0 |
-
-### Catalog Configuration
-
-| Configuration item | Description
| Default value | Required | Since version |
-|----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.catalog.cache.evictionIntervalMs` | The interval in
milliseconds to evict the catalog cache; default 3600000ms(1h).
| `3600000` | No |
0.1.0 |
-| `gravitino.catalog.classloader.isolated` | Whether to use an
isolated classloader for catalog. If `true`, an isolated classloader loads all
catalog-related libraries and configurations, not the AppClassLoader. The
default value is `true`.
| `true` | No
| 0.1.0 |
-| `gravitino.catalog.classloader.sharing.enabled` | Whether to share
one ClassLoader across catalogs that have identical isolation-relevant
properties. When `true` (default), such catalogs reuse a single ClassLoader,
reducing Metaspace usage; when `false`, each catalog gets its own ClassLoader
as in releases before 2.0.0.
| `true`
| No | 2.0.0 |
-| `gravitino.catalog.credential.backfillToProperties` | For backward
compatibility only: if `true`, the server exposes hidden catalog credentials
(such as jdbc-user and jdbc-password) in the catalog properties response so
that connectors that do not support credential vending can still read them.
**Enabling this is a security risk** — credentials are visible to anyone who
can read catalog properties. Disable once all connectors are upgraded. |
`false` | No | 1.3.0 |
-
-### Schema configuration
-
-| Configuration item | Description
| Default value | Required | Since version |
-|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.schema.separator` | The separator used to represent a
hierarchical (multi-level) schema in schema names at the API boundary, e.g. `:`
for `A:B:C`. It only takes effect for catalogs that support hierarchical
schemas (currently the Iceberg catalog). The value must not be blank and must
not contain `.` or the internal physical separator (ASCII-1, `\u0001`). See
[Hierarchical schema](./lakehouse-iceberg-catalog.md#hierarchical-schema). |
`:` | No | 1.3.0 |
-
-### Auxiliary Service Configuration
-
-| Configuration item | Description
| Default value | Since Version |
-|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------|---------------|---------------|
-| `gravitino.auxService.names ` | The auxiliary service name of the Gravitino
Iceberg REST server. Use **`iceberg-rest`** for the Gravitino Iceberg REST
server. | (none) | 0.2.0 |
-
-Refer to [Iceberg REST catalog service](iceberg-rest-service.md) for
configuration details.
-
-### Event Listener Configuration
-
-Gravitino provides event listener mechanism to allow users to capture the
events which are provided by Gravitino server to integrate some custom
operations.
-
-To leverage the event listener, you must implement the `EventListenerPlugin`
interface and place the JAR file in the classpath of the Gravitino server.
Then, add configurations to gravitino.conf to enable the event listener.
-
-| Property name | Description
| Default
value | Required | Since Version |
-|----------------------------------------|--------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
-| `gravitino.eventListener.names` | The name of the event listener, For
multiple listeners, separate names with a comma, like "audit,sync" | (none)
| Yes | 0.5.0 |
-| `gravitino.eventListener.{name}.class` | The class name of the event
listener, replace `{name}` with the actual listener name. |
(none) | Yes | 0.5.0 |
-| `gravitino.eventListener.{name}.{key}` | Custom properties that will be
passed to the event listener plugin. |
(none) | Yes | 0.5.0 |
+Nothing here is authenticated. The default `simple` authenticator takes
whatever username the
+client sends, the server speaks plain HTTP, and authorization is off, so every
caller can do
+everything. Together with H2, for which Gravitino makes no consistency or
durability guarantee,
+that makes this configuration unfit for anything but local work.
+
+### Production
+
+A production server keeps metadata in MySQL or PostgreSQL, authenticates its
callers, enforces
+authorization, and writes an audit log:
+
+```text
+# conf/gravitino.conf
+# Entity store
+gravitino.entity.store.relational.jdbcUrl =
jdbc:mysql://{db_host}:3306/{database}
+gravitino.entity.store.relational.jdbcDriver = com.mysql.cj.jdbc.Driver
+gravitino.entity.store.relational.jdbcUser = {username}
+gravitino.entity.store.relational.jdbcPassword = {password}
+
+# Transport
+gravitino.server.webserver.enableHttps = true
+gravitino.server.webserver.keyStorePath = /etc/gravitino/tls/server.jks
+gravitino.server.webserver.keyStorePassword = {keystore_password}
+gravitino.server.webserver.managerPassword = {manager_password}
+
+# Authentication
+gravitino.authenticators = oauth
+gravitino.authenticator.oauth.jwksUri = {jwks_uri}
+gravitino.authenticator.oauth.tokenValidatorClass =
org.apache.gravitino.server.authentication.JwksTokenValidator
+gravitino.authenticator.oauth.serviceAudience = {audience}
+gravitino.authenticator.oauth.principalFields =
preferred_username,email,sub
+
+# Authorization
+gravitino.authorization.enable = true
+gravitino.authorization.serviceAdmins = {admin_user}
+
+# Audit log
+gravitino.audit.enabled = true
+
+# Entity cache
+gravitino.cache.enabled = true
+gravitino.cache.implementation = caffeine
+gravitino.cache.lockSegments = 16
+gravitino.cache.enableStats = true
+```
-#### Event
+Only `gravitino.cache.enableStats` changes behavior here; it logs hit count,
miss count, and load
+failures every five minutes, which is what makes a cache problem visible in
production. The three
+lines above it restate defaults, and are spelled out so the cache
configuration is reviewable in
+one place rather than inferred from its absence. Raise `lockSegments` above
the default if the
+server runs hot enough for cache lock contention to show up in profiles.
-Gravitino triggers a pre-event before the operation, a post-event after the
completion of the operation and a failure event after the operation failed.
+Four things this block depends on:
-##### Post-event
+**The database schema is not created for you.** Initialize it and put the JDBC
driver jar in
+`${GRAVITINO_HOME}/libs/` before the first start. See
+[Relational Backend Storage](how-to-use-relational-backend-storage.md).
-| Operation type | Post-event
| Since Version |
-|-----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
-| table operation | `CreateTableEvent`,
`AlterTableEvent`, `DropTableEvent`, `LoadTableEvent`, `ListTableEvent`,
`PurgeTableEvent`, `CreateTableFailureEvent`, `AlterTableFailureEvent`,
`DropTableFailureEvent`, `LoadTableFailureEvent`, `ListTableFailureEvent`,
`PurgeTableFailureEvent`
| 0.5.0 |
-| fileset operation | `CreateFilesetEvent`,
`AlterFilesetEvent`, `DropFilesetEvent`, `LoadFilesetEvent`,
`ListFilesetEvent`, `GetFileLocationEvent`, `ListFilesEvent`,
`CreateFilesetFailureEvent`, `AlterFilesetFailureEvent`,
`DropFilesetFailureEvent`, `LoadFilesetFailureEvent`,
`ListFilesetFailureEvent`, `GetFileLocationFailureEvent`,
`ListFilesFailureEvent`
| 0.5.0 |
-| topic operation | `CreateTopicEvent`,
`AlterTopicEvent`, `DropTopicEvent`, `LoadTopicEvent`, `ListTopicEvent`,
`CreateTopicFailureEvent`, `AlterTopicFailureEvent`, `DropTopicFailureEvent`,
`LoadTopicFailureEvent`, `ListTopicFailureEvent`
| 0.5.0
|
-| schema operation | `CreateSchemaEvent`,
`AlterSchemaEvent`, `DropSchemaEvent`, `LoadSchemaEvent`, `ListSchemaEvent`,
`CreateSchemaFailureEvent`, `AlterSchemaFailureEvent`,
`DropSchemaFailureEvent`, `LoadSchemaFailureEvent`, `ListSchemaFailureEvent`
| 0.5.0 |
-| catalog operation | `CreateCatalogEvent`,
`AlterCatalogEvent`, `DropCatalogEvent`, `LoadCatalogEvent`,
`ListCatalogEvent`, `EnableCatalogEvent`, `DisableCatalogEvent`,
`CreateCatalogFailureEvent`, `AlterCatalogFailureEvent`,
`DropCatalogFailureEvent`, `LoadCatalogFailureEvent`,
`ListCatalogFailureEvent`, `EnableCatalogFailureEvent`,
`DisableCatalogFailureEvent`
| 0.5.0 |
-| metalake operation | `CreateMetalakeEvent`,
`AlterMetalakeEvent`, `DropMetalakeEvent`, `LoadMetalakeEvent`,
`ListMetalakeEvent`, `EnableMetalakeEvent`, `DisableMetalakeEvent`,
`CreateMetalakeFailureEvent`, `AlterMetalakeFailureEvent`,
`DropMetalakeFailureEvent`, `LoadMetalakeFailureEvent`,
`ListMetalakeFailureEvent`, `EnableMetalakeFailureEvent`,
`DisableMetalakeFailureEvent`
|
0.5.0 |
-| partition operation | `AddPartitionEvent`,
`GetPartitionEvent`, `DropPartitionEvent`, `PurgePartitionEvent`,
`ListPartitionEvent`, `ListPartitionNamesEvent`, `PartitionExistsEvent`,
`AddPartitionFailureEvent`, `GetPartitionFailureEvent`,
`DropPartitionFailureEvent`, `PurgePartitionFailureEvent`,
`ListPartitionFailureEvent`, `ListPartitionNamesFailureEvent`,
`PartitionExistsFailureEvent`
|
0.6.0-incubating |
-| Iceberg REST server table operation | `IcebergCreateTableEvent`,
`IcebergUpdateTableEvent`, `IcebergDropTableEvent`, `IcebergLoadTableEvent`,
`IcebergListTableEvent`, `IcebergTableExistsEvent`, `IcebergRenameTableEvent`,
`IcebergRegisterTableEvent`, `IcebergLoadTableCredentialEvent`,
`IcebergPlanTableScanEvent`, `IcebergCreateTableFailureEvent`,
`IcebergUpdateTableFailureEvent`, `IcebergDropTableFailureEvent`,
`IcebergLoadTableFailureEvent`, `IcebergListTableFailureEvent`,
`IcebergTableExistsFailureEvent`, `IcebergRenameTableFailureEvent`,
`IcebergRegisterTableFailureEvent`, `IcebergLoadTableCredentialFailureEvent`,
`IcebergPlanTableScanFailureEvent` | 0.7.0-incubating |
-| Iceberg REST server namespace operation | `IcebergCreateNamespaceEvent`,
`IcebergUpdateNamespaceEvent`, `IcebergDropNamespaceEvent`,
`IcebergLoadNamespaceEvent`, `IcebergListNamespacesEvent`,
`IcebergNamespaceExistsEvent`, `IcebergCreateNamespaceFailureEvent`,
`IcebergUpdateNamespaceFailureEvent`, `IcebergDropNamespaceFailureEvent`,
`IcebergLoadNamespaceFailureEvent`, `IcebergListNamespacesFailureEvent`,
`IcebergNamespaceExistsFailureEvent`
| 0.8.0-incubating |
-| Iceberg REST server view operation | `IcebergCreateViewEvent`,
`IcebergReplaceViewEvent`, `IcebergDropViewEvent`, `IcebergLoadViewEvent`,
`IcebergListViewEvent`, `IcebergViewExistsEvent`, `IcebergRenameViewEvent`,
`IcebergCreateViewFailureEvent`, `IcebergReplaceViewFailureEvent`,
`IcebergDropViewFailureEvent`, `IcebergLoadViewFailureEvent`,
`IcebergListViewFailureEvent`, `IcebergViewExistsFailureEvent`,
`IcebergRenameViewFailureEvent`
| 0.8.0-incubating |
-| tag operation | `ListTagsEvent`,
`ListTagsInfoEvent`, `CreateTagEvent`, `GetTagEvent`, `AlterTagEvent`,
`DeleteTagEvent`, `ListMetadataObjectsForTagEvent`,
`ListTagsForMetadataObjectEvent`, `ListTagsInfoForMetadataObjectEvent`,
`AssociateTagsForMetadataObjectEvent`, `GetTagForMetadataObjectEvent`,
`ListTagsFailureEvent`, `ListTagInfoFailureEvent`, `CreateTagFailureEvent`,
`GetTagFailureEvent`, `AlterTagFailureEvent`, `DeleteTagFailureEvent`,
`ListMetadataObjectsForTagFailureEvent`,
`ListTagsForMetadataObjectFailureEvent`,
`ListTagsInfoForMetadataObjectFailureEvent`,
`AssociateTagsForMetadataObjectFailureEvent`,
`GetTagForMetadataObjectFailureEvent` | 0.9.0-incubating |
-| model operation | `DeleteModelEvent`,
`DeleteModelVersionEvent`, `GetModelEvent`, `GetModelVersionEvent`,
`GetModelVersionUriEvent`, `LinkModelVersionEvent`, `ListModelEvent`,
`ListModelVersionsEvent`, `ListModelVersionInfosEvent`,
`RegisterAndLinkModelEvent`, `RegisterModelEvent`, `AlterModelEvent`,
`AlterModelVersionEvent`, `DeleteModelFailureEvent`,
`DeleteModelVersionFailureEvent`, `GetModelFailureEvent`,
`GetModelVersionFailureEvent`, `GetModelVersionUriFailureEvent`,
`LinkModelVersionFailureEvent`, `ListModelFailureEvent`,
`ListModelVersionFailureEvent`, `ListModelVersionInfosFailureEvent`,
`RegisterAndLinkModelFailureEvent`, `RegisterModelFailureEvent`,
`AlterModelFailureEvent`, `AlterModelVersionFailureEvent` | 0.9.0-incubating |
-| user operation | `AddUserEvent`, `RemoveUserEvent`,
`RemoveUserByExternalIdEvent`, `GetUserEvent`, `GetUserByExternalIdEvent`,
`EnableUserEvent`, `DisableUserEvent`, `ListUserNamesEvent`, `ListUsersEvent`,
`GrantUserRolesEvent`, `RevokeUserRolesEvent`, `AddUserFailureEvent`,
`RemoveUserFailureEvent`, `RemoveUserByExternalIdFailureEvent`,
`GetUserFailureEvent`, `GetUserByExternalIdFailureEvent`,
`EnableUserFailureEvent`, `DisableUserFailureEvent`,
`GrantUserRolesFailureEvent`, `ListUserNamesFailureEvent`,
`ListUsersFailureEvent`, `RevokeUserRolesFailureEvent`
|
0.9.0-incubating |
-| group operation | `AddGroupEvent`,
`RemoveGroupEvent`, `RemoveGroupByExternalIdEvent`, `GetGroupEvent`,
`GetGroupByExternalIdEvent`, `ListGroupNamesEvent`, `ListGroupsEvent`,
`GrantGroupRolesEvent`, `RevokeGroupRolesEvent`, `AddGroupFailureEvent`,
`RemoveGroupFailureEvent`, `RemoveGroupByExternalIdFailureEvent`,
`GetGroupFailureEvent`, `GetGroupByExternalIdFailureEvent`,
`GrantGroupRolesFailureEvent`, `ListGroupNamesFailureEvent`,
`ListGroupsFailureEvent`, `RevokeGroupRolesFailureEvent`
|
0.9.0-incubating |
-| role operation | `CreateRoleEvent`,
`DeleteRoleEvent`, `GetRoleEvent`, `GrantPrivilegesEvent`,
`ListRoleNamesEvent`, `RevokePrivilegesEvent`, `OverridePrivilegesEvent`,
`CreateRoleFailureEvent`, `DeleteRoleFailureEvent`, `GetRoleFailureEvent`,
`GrantPrivilegesFailureEvent`, `ListRoleNamesFailureEvent`,
`RevokePrivilegesFailureEvent`, `OverridePrivilegesFailureEvent`
| 0.9.0-incubating |
-| owner operation | `SetOwnerEvent`, `GetOwnerEvent`,
`SetOwnerFailureEvent`, `GetOwnerFailureEvent`
| 1.0.0 |
-| Gravitino server job template operation | `RegisterJobTemplateEvent`,
`GetJobTemplateEvent`, `ListJobTemplatesEvent`, `AlterJobTemplateEvent`,
`DeleteJobTemplateEvent`, `RegisterJobTemplateFailureEvent`,
`GetJobTemplateFailureEvent`, `ListJobTemplatesFailureEvent`,
`AlterJobTemplateFailureEvent`, `DeleteJobTemplateFailureEvent`
| 1.0.1 |
-| Gravitino server job operation | `RunJobEvent`, `GetJobEvent`,
`ListJobsEvent`, `CancelJobEvent`, `RunJobFailureEvent`, `GetJobFailureEvent`,
`ListJobsFailureEvent`, `CancelJobFailureEvent`
| 1.0.1 |
-| Gravitino server statistics operation | `ListStatisticsEvent`,
`UpdateStatisticsEvent`, `DropStatisticsEvent`, `ListPartitionStatisticsEvent`,
`UpdatePartitionStatisticsEvent`, `DropPartitionStatisticsEvent`,
`ListStatisticsFailureEvent`, `UpdateStatisticsFailureEvent`,
`DropStatisticsFailureEvent`, `ListPartitionStatisticsFailureEvent`,
`UpdatePartitionStatisticsFailureEvent`, `DropPartitionStatisticsFailureEvent`
| 1.1.0 |
-| policy operation | `CreatePolicyEvent`,
`AlterPolicyEvent`, `DeletePolicyEvent`, `GetPolicyEvent`, `ListPoliciesEvent`,
`ListPolicyInfosEvent`, `EnablePolicyEvent`, `DisablePolicyEvent`,
`GetPolicyForMetadataObjectEvent`, `AssociatePoliciesForMetadataObjectEvent`,
`ListPolicyInfosForMetadataObjectEvent`, `ListMetadataObjectsForPolicyEvent`,
`CreatePolicyFailureEvent`, `AlterPolicyFailureEvent`,
`DeletePolicyFailureEvent`, `GetPolicyFailureEvent`,
`ListPoliciesFailureEvent`, `ListPolicyInfosFailureEvent`,
`EnablePolicyFailureEvent`, `DisablePolicyFailureEvent`,
`GetPolicyForMetadataObjectFailureEvent`,
`AssociatePoliciesForMetadataObjectFailureEvent`,
`ListPolicyInfosForMetadataObjectFailureEvent`,
`ListMetadataObjectsForPolicyFailureEvent` | 1.1.0 |
-| function operation | `RegisterFunctionEvent`,
`GetFunctionEvent`, `AlterFunctionEvent`, `DropFunctionEvent`,
`ListFunctionEvent`, `ListFunctionInfosEvent`, `RegisterFunctionFailureEvent`,
`GetFunctionFailureEvent`, `AlterFunctionFailureEvent`,
`DropFunctionFailureEvent`, `ListFunctionFailureEvent`
| 1.3.0 |
+**HTTPS replaces HTTP rather than joining it.** A server with `enableHttps`
set no longer serves
+plain HTTP, so clients and the Web UI must move to `httpsPort`, which defaults
to `8433`. See
+[HTTPS](security/how-to-use-https.md).
-##### Pre-event
+**The authenticator is one line, its provider is not.** The block above
validates JWTs against a
+JWKS endpoint, which is the common case for an external identity provider.
Static sign keys,
+Kerberos, and the OIDC login flow for the Web UI each take a different set of
properties. See
+[How to Authenticate](security/how-to-authenticate.md), or
+[How to Use the Built-in IdP](security/how-to-use-built-in-idp.md) to keep
users and groups in
+Gravitino's own metadata store instead of an external provider.
-| Operation type | Pre-event
| Since Version |
-|-----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
-| Iceberg REST server table operation | `IcebergCreateTablePreEvent`,
`IcebergUpdateTablePreEvent`, `IcebergDropTablePreEvent`,
`IcebergLoadTablePreEvent`, `IcebergListTablePreEvent`,
`IcebergTableExistsPreEvent`, `IcebergRenameTablePreEvent`,
`IcebergRegisterTablePreEvent`, `IcebergLoadTableCredentialPreEvent`,
`IcebergPlanTableScanPreEvent` | 0.7.0-incubating |
-| Iceberg REST server namespace operation | `IcebergCreateNamespacePreEvent`,
`IcebergUpdateNamespacePreEvent`, `IcebergDropNamespacePreEvent`,
`IcebergLoadNamespacePreEvent`, `IcebergListNamespacesPreEvent`,
`IcebergNamespaceExistsPreEvent`
|
0.8.0-incubating |
-| Iceberg REST server view operation | `IcebergCreateViewPreEvent`,
`IcebergReplaceViewPreEvent`, `IcebergDropViewPreEvent`,
`IcebergLoadViewPreEvent`, `IcebergListViewPreEvent`,
`IcebergViewExistsPreEvent`, `IcebergRenameViewPreEvent`
| 0.8.0-incubating |
-| Gravitino server table operation | `CreateTablePreEvent`,
`AlterTablePreEvent`, `DropTablePreEvent`, `PurgeTablePreEvent`,
`LoadTablePreEvent`, `ListTablePreEvent`
|
0.8.0-incubating |
-| Gravitino server schema operation | `CreateSchemaPreEvent`,
`AlterSchemaPreEvent`, `DropSchemaPreEvent`, `LoadSchemaPreEvent`,
`ListSchemaPreEvent`
|
0.8.0-incubating |
-| Gravitino server catalog operation | `CreateCatalogPreEvent`,
`AlterCatalogPreEvent`, `DropCatalogPreEvent`, `LoadCatalogPreEvent`,
`ListCatalogPreEvent`, `EnableCatalogPreEvent`, `DisableCatalogPreEvent`
| 0.8.0-incubating
|
-| Gravitino server metalake operation | `CreateMetalakePreEvent`,
`AlterMetalakePreEvent`, `DropMetalakePreEvent`, `LoadMetalakePreEvent`,
`ListMetalakePreEvent`, `EnableMetalakePreEvent`, `DisableMetalakePreEvent`
| 0.8.0-incubating |
-| Gravitino server partition operation | `AddPartitionPreEvent`,
`DropPartitionPreEvent`, `GetPartitionPreEvent`,
`PurgePartitionPreEvent`,`ListPartitionPreEvent`,`ListPartitionNamesPreEvent`
| 0.8.0-incubating |
-| Gravitino server fileset operation | `CreateFilesetPreEvent`,
`AlterFilesetPreEvent`, `DropFilesetPreEvent`,
`LoadFilesetPreEvent`,`ListFilesetPreEvent`,`GetFileLocationPreEvent`,
`ListFilesPreEvent`
| 0.8.0-incubating |
-| Gravitino server model operation | `DeleteModelPreEvent`,
`DeleteModelVersionPreEvent`, `RegisterAndLinkModelPreEvent`,
`GetModelPreEvent`, `GetModelVersionPreEvent`, `GetModelVersionUriPreEvent`,
`LinkModelVersionPreEvent`, `ListModelPreEvent`, `ListModelVersionPreEvent`,
`ListModelVersionInfosPreEvent`, `RegisterModelPreEvent`, `AlterModelPreEvent`,
`AlterModelVersionPreEvent` | 0.9.0-incubating |
-| Gravitino server tag operation | `ListTagsPreEvent`,
`ListTagsInfoPreEvent`, `CreateTagPreEvent`, `GetTagPreEvent`,
`AlterTagPreEvent`, `DeleteTagPreEvent`, `ListMetadataObjectsForTagPreEvent`,
`ListTagsForMetadataObjectPreEvent`, `ListTagsInfoForMetadataObjectPreEvent`,
`AssociateTagsForMetadataObjectPreEvent`, `GetTagForMetadataObjectPreEvent` |
0.9.0-incubating |
-| Gravitino server user operation | `AddUserPreEvent`,
`RemoveUserPreEvent`, `RemoveUserByExternalIdPreEvent`, `GetUserPreEvent`,
`GetUserByExternalIdPreEvent`, `EnableUserPreEvent`, `DisableUserPreEvent`,
`ListUserNamesPreEvent`, `ListUsersPreEvent`, `GrantUserRolesPreEvent`,
`RevokeUserRolesPreEvent` |
0.9.0-incubating |
-| Gravitino server group operation | `AddGroupPreEvent`,
`RemoveGroupPreEvent`, `RemoveGroupByExternalIdPreEvent`, `GetGroupPreEvent`,
`GetGroupByExternalIdPreEvent`, `ListGroupNamesPreEvent`, `ListGroupsPreEvent`,
`GrantGroupRolesPreEvent`, `RevokeGroupRolesPreEvent`
| 0.9.0-incubating |
-| Gravitino server role operation | `CreateRolePreEvent`,
`DeleteRolePreEvent`, `GetRolePreEvent`, `GrantPrivilegesPreEvent`,
`ListRoleNamesPreEvent`, `RevokePrivilegesPreEvent`,
`OverridePrivilegesPreEvent`
| 0.9.0-incubating |
-| Gravitino server owner operation | `SetOwnerPreEvent`,
`GetOwnerPreEvent`
| 1.0.0 |
-| Gravitino server job template operation | `RegisterJobTemplatePreEvent`,
`GetJobTemplatePreEvent`, `ListJobTemplatesPreEvent`,
`AlterJobTemplatePreEvent`, `DeleteJobTemplatePreEvent`
| 1.0.1
|
-| Gravitino server job operation | `RunJobPreEvent`,
`GetJobPreEvent`, `ListJobsPreEvent`, `CancelJobPreEvent`
| 1.0.1 |
-| Gravitino server statistics operation | `ListStatisticsPreEvent`,
`UpdateStatisticsPreEvent`, `DropStatisticsPreEvent`,
`ListPartitionStatisticsPreEvent`, `UpdatePartitionStatisticsPreEvent`,
`DropPartitionStatisticsPreEvent`
| 1.1.0 |
-| policy operation | `CreatePolicyPreEvent`,
`AlterPolicyPreEvent`, `DeletePolicyPreEvent`, `GetPolicyPreEvent`,
`ListPoliciesPreEvent`, `ListPolicyInfosPreEvent`, `EnablePolicyPreEvent`,
`DisablePolicyPreEvent`, `GetPolicyForMetadataObjectPreEvent`,
`AssociatePoliciesForMetadataObjectPreEvent`,
`ListPolicyInfosForMetadataObjectPreEvent`,
`ListMetadataObjectsForPolicyPreEvent` | 1.1.0 |
-| function operation | `RegisterFunctionPreEvent`,
`GetFunctionPreEvent`, `AlterFunctionPreEvent`, `DropFunctionPreEvent`,
`ListFunctionPreEvent`
| 1.3.0 |
+**`gravitino.authorization.serviceAdmins` has no default.** It is the one
property here that
+Gravitino will not fill in for you, and enabling authorization without it
fails at startup. What
+those admins and everyone else may then do is the subject of
+[Access Control](security/access-control.md).
-#### Event Listener Plugin
+Give the JVM more than the 1 GB it takes by default:
-The `EventListenerPlugin` defines an interface for event listeners that manage
the lifecycle and state of a plugin. This includes handling its initialization,
startup, and shutdown processes, as well as handing events triggered by various
operations.
+```shell
+export GRAVITINO_MEM="-Xms4g -Xmx4g -XX:MaxMetaspaceSize=1g"
+```
-The plugin provides several operational modes for how to process event,
supporting both synchronous and asynchronous processing approaches.
+### Docker
-- **SYNC**: Events are processed synchronously, immediately following the
associated operation. This mode ensures events are processed before the
operation's result is returned to the client, but it may delay the main process
if event processing takes too long.
+The Gravitino image does not simply run the server against the configuration
file you give it. At
+startup the entrypoint rewrites `conf/gravitino.conf`, applying its own
defaults over roughly two
+dozen properties and then applying any supported environment variables.
Configure the container
+through environment variables:
-- **ASYNC_SHARED**: This mode employs a shared queue and dispatcher for
asynchronous event processing. It prevents the main process from being blocked,
though there's a risk events might be dropped if not promptly consumed. Sharing
a dispatcher can lead to poor isolation in case of slow listeners.
-
-- **ASYNC_ISOLATED**: Events are processed asynchronously, with each listener
having its own dedicated queue and dispatcher thread. This approach offers
better isolation but at the expense of multiple queues and dispatchers.
+```shell
+docker run --rm -d \
+ -p 8090:8090 \
+ -e
GRAVITINO_ENTITY_STORE_RELATIONAL_JDBC_URL="jdbc:postgresql://{db_host}:5432/{database}"
\
+ -e GRAVITINO_ENTITY_STORE_RELATIONAL_JDBC_DRIVER="org.postgresql.Driver" \
+ apache/gravitino:{tag}
+```
-When processing pre-event, you could throw a `ForbiddenException` to skip the
following executions. For more details, refer to the definition of the plugin.
+To supply a configuration file instead, for example from a Kubernetes
ConfigMap, disable the
+rewrite with `SKIP_CONFIG_REWRITE=true`. See
+[Container Configuration](#container-configuration) for what the rewrite does
and which variables
+it recognizes.
+
+### Running More Than One Server
+
+Servers behind a load balancer share the entity store but keep local caches.
Each server polls the
+entity change log and invalidates entries that another server has modified.
The defaults are safe:
+a three second poll, and a server that cannot keep its caches current exits
rather than serving
+metadata it knows to be stale. Point the load balancer's health check at `GET
/health/ready` so a
+server that has lost its database stops receiving traffic.
+
+## Server Configuration
+
+Every property in this section belongs in
`${GRAVITINO_HOME}/conf/gravitino.conf`, one
+`property = value` pair per line. The server reads the file once, at startup,
so a change
+takes effect on the next restart. A Default Value of `(empty)` means the
property exists with an
+empty string or list; `(none)` means it has no default at all.
+
+### Serving Requests
+
+#### HTTP Server
+
+| Configuration Item | Description
| Default Value |
+|------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
Review Comment:
Hi Mark, why do we remove the `Since Version` column in the new doc? My
feeling is that `Since Version` is useful to let users know which version we
supported this configuration with. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
