This is an automated email from the ASF dual-hosted git repository.
fanng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new eee54966e9 [MINOR] docs: Add paimon spark connector doc (#6328)
eee54966e9 is described below
commit eee54966e973a4851cf2fdb88835d41cf71f99eb
Author: cai can <[email protected]>
AuthorDate: Mon Jan 20 11:54:34 2025 +0800
[MINOR] docs: Add paimon spark connector doc (#6328)
### What changes were proposed in this pull request?
Add paimon spark connector doc
---------
Co-authored-by: caican <[email protected]>
---
docs/spark-connector/spark-catalog-paimon.md | 87 ++++++++++++++++++++++++++++
1 file changed, 87 insertions(+)
diff --git a/docs/spark-connector/spark-catalog-paimon.md
b/docs/spark-connector/spark-catalog-paimon.md
new file mode 100644
index 0000000000..2cae203e5d
--- /dev/null
+++ b/docs/spark-connector/spark-catalog-paimon.md
@@ -0,0 +1,87 @@
+---
+title: "Spark connector Paimon catalog"
+slug: /spark-connector/spark-catalog-paimon
+keyword: spark connector paimon catalog
+license: "This software is licensed under the Apache License version 2."
+---
+
+The Apache Gravitino Spark connector offers the capability to read and write
Paimon tables, with the metadata managed by the Gravitino server. To enable the
use of the Paimon catalog within the Spark connector now, you must set download
[Paimon Spark runtime
jar](https://paimon.apache.org/docs/0.8/spark/quick-start/#preparation) to
Spark classpath.
+
+## Capabilities
+
+### Support DDL and DML operations:
+
+- `CREATE NAMESPACE`
+- `DROP NAMESPACE`
+- `LIST NAMESPACE`
+- `LOAD NAMESPACE`
+ - It can not return any user-specified configs now, as we only support
FilesystemCatalog in spark-connector now.
+- `CREATE TABLE`
+ - Doesn't support distribution and sort orders.
+- `DROP TABLE`
+- `ALTER TABLE`
+- `LIST TABLE`
+- `DESRICE TABLE`
+- `SELECT`
+- `INSERT INTO & OVERWRITE`
+- `Schema Evolution`
+- `PARTITION MANAGEMENT`, such as `LIST PARTITIONS`, `ALTER TABLE ... DROP
PARTITION ...`
+
+:::info
+Only supports Paimon FilesystemCatalog on HDFS now.
+:::
+
+#### Not supported operations:
+
+- `ALTER NAMESPACE`
+ - Paimon does not support alter namespace.
+- Row Level operations, such as `MERGE INTO`, `DELETE`, `UPDATE`, `TRUNCATE`
+- Metadata tables, such as
`{paimon_catalog}.{paimon_database}.{paimon_table}$snapshots`
+- Other Paimon extension SQLs, such as `Tag`
+- Call Statements
+- View
+- Time Travel
+- Hive and Jdbc backend, and Object Storage for FilesystemCatalog
+
+## SQL example
+
+```sql
+-- Suppose paimon_catalog is the Paimon catalog name managed by Gravitino
+USE paimon_catalog;
+
+CREATE DATABASE IF NOT EXISTS mydatabase;
+USE mydatabase;
+
+CREATE TABLE IF NOT EXISTS employee (
+ id bigint,
+ name string,
+ department string,
+ hire_date timestamp
+) PARTITIONED BY (name);
+
+SHOW TABLES;
+DESC TABLE EXTENDED employee;
+
+INSERT INTO employee
+VALUES
+(1, 'Alice', 'Engineering', TIMESTAMP '2021-01-01 09:00:00'),
+(2, 'Bob', 'Marketing', TIMESTAMP '2021-02-01 10:30:00'),
+(3, 'Charlie', 'Sales', TIMESTAMP '2021-03-01 08:45:00');
+
+SELECT * FROM employee WHERE name = 'Alice';
+
+SHOW PARTITIONS employee;
+ALTER TABLE employee DROP PARTITION (`name`='Alice');
+```
+
+## Catalog properties
+
+Gravitino spark connector will transform below property names which are
defined in catalog properties to Spark Paimon connector configuration.
+
+| Gravitino catalog property name | Spark Paimon connector configuration |
Description
| Since Version |
+|---------------------------------|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
+| `catalog-backend` | `metastore` |
Catalog backend type
| 0.8.0-incubating |
+| `uri` | `uri` |
Catalog backend uri
| 0.8.0-incubating |
+| `warehouse` | `warehouse` |
Catalog backend warehouse
| 0.8.0-incubating |
+
+Gravitino catalog property names with the prefix `spark.bypass.` are passed to
Spark Paimon connector. For example, using `spark.bypass.client-pool-size` to
pass the `client-pool-size` to the Spark Paimon connector.
