This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new baa1e9cfa [#4171] feat(docs): Add Paimon Catalog doc (#4192)
baa1e9cfa is described below
commit baa1e9cfaa6234845fd33cc51a0b14c5816f4562
Author: cai can <[email protected]>
AuthorDate: Thu Jul 25 17:23:02 2024 +0800
[#4171] feat(docs): Add Paimon Catalog doc (#4192)
### What changes were proposed in this pull request?
Add Paimon Catalog doc
### Why are the changes needed?
Fix: https://github.com/apache/gravitino/issues/4171
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No testing required.
---------
Co-authored-by: caican <[email protected]>
---
docs/lakehouse-paimon-catalog.md | 149 +++++++++++++++++++++++++++++++++++++++
1 file changed, 149 insertions(+)
diff --git a/docs/lakehouse-paimon-catalog.md b/docs/lakehouse-paimon-catalog.md
new file mode 100644
index 000000000..7265073a5
--- /dev/null
+++ b/docs/lakehouse-paimon-catalog.md
@@ -0,0 +1,149 @@
+---
+title: "Paimon catalog"
+slug: /lakehouse-Paimon-catalog
+keywords:
+ - lakehouse
+ - Paimon
+ - metadata
+license: "This software is licensed under the Apache License version 2."
+---
+
+## Introduction
+
+Apache Gravitino provides the ability to manage Apache Paimon metadata.
+
+### Requirements
+
+:::info
+Builds with Apache Paimon `0.8.0`.
+:::
+
+## Catalog
+
+### Catalog capabilities
+
+- Works as a catalog proxy, supporting `FilesystemCatalog`.
+- Supports DDL operations for Paimon schemas and tables.
+
+- Doesn't support `JdbcCatalog` and `HiveCatalog` catalog backend now.
+- Doesn't support alterTable now.
+- Doesn't support alterSchema.
+
+### Catalog properties
+
+| Property name | Description
| Default value | Required
| Since Version |
+|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-----------------------------------------------------------------|---------------|
+| `catalog-backend` | Catalog backend of
Gravitino Paimon catalog. Only supports `filesystem` now.
| (none) | Yes
| 0.6.0 |
+| `uri` | The URI configuration
of the Paimon catalog. `thrift://127.0.0.1:9083` or
`jdbc:postgresql://127.0.0.1:5432/db_name` or
`jdbc:mysql://127.0.0.1:3306/metastore_db`. It is optional for
`FilesystemCatalog`. | (none) | required if the value of
`catalog-backend` is not `filesystem`. | 0.6.0 |
+| `warehouse` | Warehouse directory of
catalog. `file:///user/hive/warehouse-paimon/` for local fs or
`hdfs://namespace/hdfs/path` for HDFS.
| (none) | Yes
| 0.6.0 |
+| `authentication.type` | The type of
authentication for Paimon catalog backend, currently Gravitino only supports
`Kerberos` and `simple`.
| `simple` | No
| 0.6.0 |
+| `authentication.kerberos.principal` | The principal of the
Kerberos authentication.
| (none) | required if the value of
`authentication.type` is Kerberos. | 0.6.0 |
+| `authentication.kerberos.keytab-uri` | The URI of The keytab
for the Kerberos authentication.
| (none) | required if the value of
`authentication.type` is Kerberos. | 0.6.0 |
+| `authentication.kerberos.check-interval-sec` | The check interval of
Kerberos credential for Paimon catalog.
| 60 | No
| 0.6.0 |
+| `authentication.kerberos.keytab-fetch-timeout-sec` | The fetch timeout of
retrieving Kerberos keytab from `authentication.kerberos.keytab-uri`.
| 60 | No
| 0.6.0 |
+
+
+Any properties not defined by Gravitino with `gravitino.bypass.` prefix will
pass to Paimon catalog properties and HDFS configuration. For example, if
specify `gravitino.bypass.table.type`, `table.type` will pass to Paimon catalog
properties.
+
+### Catalog operations
+
+Please refer to [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#catalog-operations)
for more details.
+
+## Schema
+
+### Schema capabilities
+
+- Supporting createSchema, dropSchema, loadSchema and listSchema.
+- Supporting cascade drop schema.
+
+- Doesn't support alterSchema.
+
+### Schema properties
+
+- Doesn't support specify location and store any schema properties when
createSchema for FilesystemCatalog now.
+- Doesn't return any schema properties when loadSchema for FilesystemCatalog
now.
+- Doesn't support store schema comment for FilesystemCatalog now.
+
+### Schema operations
+
+Please refer to [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations)
for more details.
+
+## Table
+
+### Table capabilities
+
+- Supporting createTable, dropTable, loadTable and listTable.
+```
+dropTable will delete the table location directly, similar with purgeTable.
+```
+- Supporting Column default value through table properties, such as
`fields.{columnName}.default-value`.
+
+- Doesn't support alterTable now.
+
+#### Table partitions
+
+- Doesn't support table partition now.
+
+### Table sort orders
+
+- Doesn't support table sort orders.
+
+### Table distributions
+
+- Only supporting `NoneDistribution` now.
+
+### Table indexes
+
+- Doesn't support table indexes.
+
+### Table column types
+
+| Gravitino Type | Apache Paimon Type |
+|-------------------------------|--------------------------------|
+| `Sturct` | `Row` |
+| `Map` | `Map` |
+| `Array` | `Array` |
+| `Boolean` | `Boolean` |
+| `Byte` | `TinyInt` |
+| `Short` | `SmallInt` |
+| `Integer` | `Int` |
+| `Long` | `BigInt` |
+| `Float` | `Float` |
+| `Double` | `Double` |
+| `Decimal` | `Decimal` |
+| `String` | `VarChar(Integer.MAX_VALUE)` |
+| `VarChar` | `VarChar` |
+| `FixedChar` | `Char` |
+| `Date` | `Date` |
+| `Time` | `Time` |
+| `TimestampType withZone` | `LocalZonedTimestamp` |
+| `TimestampType withoutZone` | `Timestamp` |
+| `Binary` | `Binary` |
+| `Fixed` | `VarBinary` |
+
+:::info
+Gravitino doesn't support Paimon `MultisetType` type.
+:::
+
+### Table properties
+
+You can pass [Paimon table
properties](https://paimon.apache.org/docs/0.8/maintenance/configurations/) to
Gravitino when creating a Paimon table.
+
+The Gravitino server doesn't allow passing the following reserved fields.
+
+| Configuration item | Description
|
+|---------------------------------|----------------------------------------------------------|
+| `comment` | The table comment.
|
+| `creator` | The table creator.
|
+
+### Table operations
+
+Please refer to [Manage Relational Metadata Using
Gravitino](./manage-relational-metadata-using-gravitino.md#table-operations)
for more details.
+
+## HDFS configuration
+
+You can place `core-site.xml` and `hdfs-site.xml` in the
`catalogs/lakehouse-paimon/conf` directory to automatically load as the default
HDFS configuration.
+
+:::caution
+When reading and writing to HDFS, the Gravitino server can only operate as the
specified Kerberos user and doesn't support proxying to other Kerberos users
now.
+:::
