This is an automated email from the ASF dual-hosted git repository. XiaoHongbo-Hope pushed a commit to branch release-2.1 in repository https://gitbox.apache.org/repos/asf/paimon.git
commit 01b9ba46852688025fbe939be19f26fcecbaee47 Author: Jingsong Lee <[email protected]> AuthorDate: Fri Sep 11 12:40:02 2026 +0800 [docs] Reorganize ecosystem guides and add SVG diagrams (#9728) --- docs/docs/ecosystem/amoro.md | 48 +++- docs/docs/ecosystem/connecting-engines.md | 91 +++++++ docs/docs/ecosystem/doris.md | 255 ++++++++---------- docs/docs/ecosystem/hive.mdx | 369 ++++++++++----------------- docs/docs/ecosystem/index.md | 88 ++++--- docs/docs/ecosystem/starrocks.md | 200 +++++---------- docs/docs/ecosystem/trino.md | 328 ++++++++++-------------- docs/sidebars.js | 15 +- docs/static/img/amoro-paimon.png | Bin 164164 -> 0 bytes docs/static/img/amoro-paimon.svg | 64 +++++ docs/static/img/ecosystem-catalog-access.svg | 49 ++++ docs/static/img/ecosystem-overview.svg | 56 ++++ 12 files changed, 808 insertions(+), 755 deletions(-) diff --git a/docs/docs/ecosystem/amoro.md b/docs/docs/ecosystem/amoro.md index e9c9d3d600..0580413821 100644 --- a/docs/docs/ecosystem/amoro.md +++ b/docs/docs/ecosystem/amoro.md @@ -22,21 +22,51 @@ specific language governing permissions and limitations under the License. --> -# Apache Amoro With Paimon +# Amoro -**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and -**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)** features for a Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture. -**[AMS](https://amoro.apache.org/docs/latest/#architecture)(Amoro Management Service)** provides Lakehouse management features, like self-optimizing, data expiration, etc. It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services like HMS(Hive Metastore). +Use Apache Amoro to inspect Paimon tables through Amoro Management Service (AMS). Register an +existing Paimon catalog so operators can browse table metadata in one place. +## Supported Workflows -# Table Format +The [Amoro 0.8.1 Paimon guide](https://amoro.apache.org/docs/0.8.1/paimon-format/) describes table +metadata inspection and Spark SQL access through the Terminal. Check the Paimon guide for your +deployed Amoro version before configuring the integration. -Apache Amoro supports all catalog types supported by paimon, including common catalog: Hadoop, Hive, Glue, JDBC, Nessie and other third-party catalog. -Amoro supports all storage types supported by Paimon, including common store: Hadoop, S3, GCS, ECS, OSS, and so on. + - +| Task | Where it runs | +| --- | --- | +| Browse schemas, options, files, snapshots, DDL, and compaction information | AMS table views | +| Query or operate on tables with SQL | Amoro Terminal, using a configured Spark runtime | +| Ingest data | A writer such as Flink or Spark | -In the future, Paimon automatic optimization strategy will be supported, and users can achieve the best balance experience by cooperating with Amoro automatic optimization +## Connect a Paimon Catalog +1. Deploy AMS using the [Amoro deployment guide](https://amoro.apache.org/docs/0.8.1/deployment/). +2. Prepare the Paimon catalog settings and make its metastore and storage accessible to AMS. + See [Connecting Engines](./connecting-engines) for the information to collect. +3. Install the filesystem dependencies required by the Amoro distribution. For S3 or OSS, + the Paimon guide describes adding the corresponding jars to Amoro's `lib` directory; + match their versions to the Paimon dependency used by that distribution. +4. Register the catalog using [Managing Catalogs](https://amoro.apache.org/docs/0.8.1/managing-catalogs/). +5. Open an existing table and check its schema, options, files, and snapshot history. +Use the catalog and filesystem implementations supported by the Paimon integration in that +Amoro release. Catalog support for another table format does not establish Paimon support. +## Query with the Terminal + +The Terminal can run Spark locally inside AMS or use a configured Kyuubi service. Verify the +execution backend's Paimon connector, catalog settings, and storage access, especially when +using an external backend. See the [Amoro Terminal guide](https://amoro.apache.org/docs/0.8.1/using-kyuubi/) +and [Paimon Spark Quick Start](../spark/quick-start). + +## Table Maintenance + +The ability to display compaction information does not establish support for running Paimon +self-optimization. Verify maintenance support in your deployed Amoro version before assigning +it table maintenance work. + +For Paimon-managed maintenance, see [Compaction](../primary-key-table/compaction), +[Snapshot Management](../maintenance/manage-snapshots), and [Flink Actions](../flink/action-jars). diff --git a/docs/docs/ecosystem/connecting-engines.md b/docs/docs/ecosystem/connecting-engines.md new file mode 100644 index 0000000000..2f01f6238e --- /dev/null +++ b/docs/docs/ecosystem/connecting-engines.md @@ -0,0 +1,91 @@ +--- +title: "Connecting Engines" +sidebar_position: 1 +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Connecting Engines + +An engine needs to discover a Paimon table and read its files. Use the same catalog backend +and table location as the writer, then configure storage access in the engine that runs the query. +The SQL catalog name is local to that engine and can differ between engines. + + + +## Identify the Catalog and Storage + +Collect these settings from the job or engine that created the table: + +| Setting | What to check | +| --- | --- | +| Catalog backend | Filesystem, Hive metastore, REST, or another backend supported by the reader. | +| Catalog endpoint | The metastore or service URI, with the authentication needed to connect. A filesystem catalog has no separate metastore service. | +| Warehouse or table location | The shared storage URI. A filesystem catalog uses the warehouse root; a Hive external table points to one table directory. REST catalogs can use a warehouse identifier. | +| Storage access | Filesystem libraries, endpoint configuration, and credentials available to the engine processes that access the files. | +| Table features | Primary keys, bucket mode, file format, types, and features that the reader must understand. | + +See [Catalog](../concepts/catalog) for Paimon's catalog backends and +[Filesystems](../maintenance/filesystems) for Paimon storage dependencies. External engines +can use their own filesystem implementations and configuration names. + +## Configure the Reader + +1. Install the connector or enable the engine's built-in Paimon integration. +2. Configure a catalog using the engine's own property names. +3. Configure access to the catalog service and the warehouse storage on the relevant nodes. +4. Query an existing table by its fully qualified name before adding optional read settings. + +Catalog properties are not interchangeable. For example, a Hive-backed Paimon catalog uses +`metastore = hive` in Flink, while StarRocks and Doris use `paimon.catalog.type = hive` and +`paimon.catalog.type = hms`, respectively. Follow the matching guide: +[Hive](./hive#installation), [Trino](./trino#configure-paimon-catalog), +[StarRocks](./starrocks#create-paimon-catalog), or [Doris](./doris#create-paimon-catalog). + +For a local experiment, a `file:` warehouse is sufficient when all processes share that path. +For a distributed cluster, use shared storage that every participating process can access. + +## Check the Read Semantics + +Query a small, known table first. Compare values as well as row counts, especially for +primary-key tables with updates and deletes. + +| Read path | What the result represents | +| --- | --- | +| Ordinary batch query | A table snapshot, subject to the engine's metadata caching and supported read mode. | +| Time travel | A selected historical snapshot or tag; syntax and support depend on the engine. The referenced state must still be retained. | +| Read-optimized query | Compacted data from a primary-key table; recent changes may be absent until full compaction completes. | +| Streaming query | A snapshot and/or subsequent changes, according to the engine and scan configuration. | + +See [Table Mode](../primary-key-table/table-mode), +[System Tables](../concepts/system-tables), and +[Snapshot Retention](../maintenance/manage-snapshots) for the underlying behavior. + +## Troubleshooting + +| Symptom | Check first | +| --- | --- | +| Catalog exists but the table is missing | Confirm the backend, warehouse, database, and table registration match the writer. | +| Tables can be listed but a query cannot open files | Check storage endpoints, credentials, and filesystem dependencies on the nodes performing the read. | +| Class-loading or method-not-found errors | Check the engine/connector version pair and remove conflicting connector jars. | +| New data or schema is not visible | Check whether the writer committed a snapshot, then inspect engine metadata caches and any time-travel or read-optimized settings. | +| Updates or deletes produce unexpected rows | Confirm support for the table's merge engine and deletion vectors; use a Paimon-aware reader rather than scanning the underlying Parquet or ORC files directly. | + +Continue with the engine guide for exact SQL and version-specific limitations. diff --git a/docs/docs/ecosystem/doris.md b/docs/docs/ecosystem/doris.md index 878bba7257..8661daf337 100644 --- a/docs/docs/ecosystem/doris.md +++ b/docs/docs/ecosystem/doris.md @@ -24,189 +24,148 @@ under the License. # Doris -This documentation is a guide for using Paimon in Doris. - -> More details can be found in [Apache Doris Website](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog) +Query existing Paimon tables through a Doris external catalog. Use +[Connecting Engines](./connecting-engines) to identify the catalog backend and warehouse first. ## Version -Paimon currently supports Apache Doris 2.0.6 and above. +Select a Doris release that supports your catalog backend and Paimon table features. +The [Doris Paimon catalog documentation](https://doris.apache.org/docs/3.x/lakehouse/catalogs/paimon-catalog/) +provides version-specific settings. The REST catalog example below requires Doris 3.1 or later. + +This integration reads Paimon tables. An external catalog exposes existing tables without copying +them into Doris; it does not enable writes to Paimon. + +## Prerequisites + +Prepare an existing Paimon table and configure metastore and storage access for the Doris +processes that perform metadata and data reads. The examples below use placeholder hosts and +credentials; replace them with your deployment's settings. ## Create Paimon Catalog -Use `CREATE CATALOG` statement in Apache Doris to create Paimon Catalog. +Choose one catalog definition for the backend used by the writer. HDFS examples assume the +cluster's Hadoop access is already configured. -Doris support multi types of Paimon Catalogs. Here are some examples: +### Filesystem Catalog on HDFS ```sql --- HDFS based Paimon Catalog -CREATE CATALOG `paimon_hdfs` PROPERTIES ( - "type" = "paimon", - "warehouse" = "hdfs://172.21.0.1:8020/user/paimon", - "hadoop.username" = "hadoop" +CREATE CATALOG paimon_hdfs PROPERTIES ( + 'type' = 'paimon', + 'paimon.catalog.type' = 'filesystem', + 'warehouse' = 'hdfs://namenode:8020/warehouse/paimon', + 'hadoop.username' = 'hadoop' ); +``` + +### Filesystem Catalog on OSS --- Aliyun OSS based Paimon Catalog -CREATE CATALOG `paimon_oss` PROPERTIES ( - "type" = "paimon", - "warehouse" = "oss://paimon-bucket/paimonoss", - "oss.endpoint" = "oss-cn-beijing.aliyuncs.com", - "oss.access_key" = "ak", - "oss.secret_key" = "sk" +```sql +CREATE CATALOG paimon_oss PROPERTIES ( + 'type' = 'paimon', + 'paimon.catalog.type' = 'filesystem', + 'warehouse' = 'oss://paimon-bucket/warehouse', + 'oss.endpoint' = 'oss-cn-beijing.aliyuncs.com', + 'oss.access_key' = '<access-key-id>', + 'oss.secret_key' = '<access-key-secret>' ); +``` --- Hive Metastore based Paimon Catalog -CREATE CATALOG `paimon_hms` PROPERTIES ( - "type" = "paimon", - "paimon.catalog.type" = "hms", - "warehouse" = "hdfs://172.21.0.1:8020/user/zhangdong/paimon2", - "hive.metastore.uris" = "thrift://172.21.0.44:7004", - "hadoop.username" = "hadoop" +### Hive Metastore Catalog + +```sql +CREATE CATALOG paimon_hms PROPERTIES ( + 'type' = 'paimon', + 'paimon.catalog.type' = 'hms', + 'warehouse' = 'hdfs://namenode:8020/warehouse/paimon', + 'hive.metastore.uris' = 'thrift://metastore:9083', + 'hadoop.username' = 'hadoop' ); +``` + +### DLF 1.0 Catalog --- Integrate with Aliyun DLF 1.0 +```sql CREATE CATALOG paimon_dlf PROPERTIES ( 'type' = 'paimon', 'paimon.catalog.type' = 'dlf', - 'warehouse' = 'oss://paimon-bucket/paimonoss/', + 'warehouse' = 'oss://paimon-bucket/warehouse', 'dlf.proxy.mode' = 'DLF_ONLY', - 'dlf.uid' = 'xxxxx', + 'dlf.uid' = '<account-id>', 'dlf.region' = 'cn-beijing', - 'dlf.access_key' = 'ak', - 'dlf.secret_key' = 'sk' + 'dlf.access_key' = '<access-key-id>', + 'dlf.secret_key' = '<access-key-secret>' ); +``` + +### DLF REST Catalog + +For Doris 3.1+, configure the REST backend explicitly. Here, `warehouse` is the DLF catalog +name, rather than an object-storage path. --- Integrate with Aliyun DLF 3.0 Paimon Rest --- Apache Doris supported since version 3.1.0 +```sql CREATE CATALOG dlf_paimon_rest PROPERTIES ( 'type' = 'paimon', + 'paimon.catalog.type' = 'rest', 'uri' = 'http://cn-beijing-vpc.dlf.aliyuncs.com', - 'warehouse' = 'catalog_name', + 'warehouse' = '<catalog-name>', 'paimon.rest.token.provider' = 'dlf', - 'paimon.rest.dlf.access-key-id' = 'ak', - 'paimon.rest.dlf.access-key-secret' = 'sk' + 'paimon.rest.dlf.access-key-id' = '<access-key-id>', + 'paimon.rest.dlf.access-key-secret' = '<access-key-secret>' ); ``` -See [Apache Doris Website](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog#examples) for more examples. +See the [Doris catalog examples](https://doris.apache.org/docs/3.x/lakehouse/catalogs/paimon-catalog/#examples) +for storage authentication and differences between Doris releases. ## Access Paimon Catalog -1. Query Paimon table with full qualified name +Query a fully qualified table name: - ```sql - SELECT * FROM paimon_hdfs.paimon_db.paimon_table; - ``` +```sql +SELECT * FROM paimon_hdfs.paimon_db.paimon_table LIMIT 10; +``` -2. Switch to Paimon Catalog and query +Alternatively, select the catalog and database for the session: - ```sql - SWITCH paimon_hdfs; - USE paimon_db; - SELECT * FROM paimon_table; - ``` +```sql +SWITCH paimon_hdfs; +USE paimon_db; +SELECT * FROM paimon_table LIMIT 10; +``` ## Query Optimization -- Read optimized for Primary Key Table - - Doris can utilize the [Read optimized](https://paimon.apache.org/docs/0.8/primary-key-table/read-optimized/) feature for Primary Key Table(release in Paimon 0.6), by reading base data files using native Parquet/ORC reader and delta file using JNI. - -- Deletion Vectors - - Doris(2.1.4+) natively supports [Deletion Vectors](https://paimon.apache.org/docs/0.8/primary-key-table/deletion-vectors/)(released in Paimon 0.8). - -## Doris to Paimon type mapping - -<table class="table table-bordered"> - <thead> - <tr> - <th class="text-left" style="width: 10%">Doris Data Type</th> - <th class="text-left" style="width: 10%">Paimon Data Type</th> - <th class="text-left" style="width: 5%">Atomic Type</th> - </tr> - </thead> - <tbody> - <tr> - <td><code>Boolean</code></td> - <td><code>BooleanType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>TinyInt</code></td> - <td><code>TinyIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>SmallInt</code></td> - <td><code>SmallIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Int</code></td> - <td><code>IntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>BigInt</code></td> - <td><code>BigIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Float</code></td> - <td><code>FloatType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Double</code></td> - <td><code>DoubleType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VarChar</code></td> - <td><code>VarCharType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Char</code></td> - <td><code>CharType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Binary</code></td> - <td><code>VarBinaryType, BinaryType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Decimal(precision, scale)</code></td> - <td><code>DecimalType(precision, scale)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Datetime</code></td> - <td><code>TimestampType,LocalZonedTimestampType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Date</code></td> - <td><code>DateType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>Array</code></td> - <td><code>ArrayType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>Map</code></td> - <td><code>MapType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>Struct</code></td> - <td><code>RowType</code></td> - <td>false</td> - </tr> - </tbody> -</table> - +Primary-key tables can require merging multiple row versions. Review the reader capabilities in +your Doris release before selecting a [table mode](../primary-key-table/table-mode). + +- **Read optimized:** reading compacted base data can reduce merge work, but a + [read-optimized view](../concepts/system-tables#read-optimized-table) can omit recent changes. +- **Deletion vectors:** readers must apply deletion vectors to suppress obsolete rows. See + [Merge On Write](../primary-key-table/table-mode#merge-on-write) for the Paimon configuration + and read semantics, and check Doris support before enabling it. + +## Type Mapping {#doris-to-paimon-type-mapping} + +This is a summary of common **Paimon-to-Doris read mappings**. Check the +[Doris type mapping](https://doris.apache.org/docs/3.x/lakehouse/catalogs/paimon-catalog/#column-type-mapping) +for the installed version; binary and timestamp mappings can depend on release and catalog options. + +| Paimon type | Doris type | +| --- | --- | +| `BOOLEAN` | `BOOLEAN` | +| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | Corresponding integer type | +| `FLOAT`, `DOUBLE` | `FLOAT`, `DOUBLE` | +| `DECIMAL(p, s)` | `DECIMAL(p, s)` | +| `CHAR`, `VARCHAR` | `STRING` | +| `BINARY`, `VARBINARY` | See the version-specific binary mapping | +| `DATE` | `DATE` | +| `TIMESTAMP`, `TIMESTAMP WITH LOCAL TIME ZONE` | See the version-specific timestamp mapping and precision limits | +| `ARRAY` | `ARRAY` | +| `MAP` | `MAP` | +| `ROW` | `STRUCT` | + +## Next Steps + +Use the Doris guide for time-travel and system-table syntax supported by your release. For +connection or visibility problems, follow [Connecting Engines](./connecting-engines#troubleshooting). diff --git a/docs/docs/ecosystem/hive.mdx b/docs/docs/ecosystem/hive.mdx index da316cc778..aaed893462 100644 --- a/docs/docs/ecosystem/hive.mdx +++ b/docs/docs/ecosystem/hive.mdx @@ -27,295 +27,206 @@ under the License. # Hive -This documentation is a guide for using Paimon in Hive. +Use the Paimon storage handler to read tables from Hive, create tables, and insert records. +A Hive-backed Paimon catalog and the Hive query engine are separate integrations: +[using Hive metastore](../concepts/catalog#hive-catalog) does not require running queries in Hive. ## Version -Paimon currently supports Hive 3.1, 2.3, 2.2, 2.1 and 2.1-cdh-6.3. +This branch contains connectors for Hive 3.1, 2.3, 2.2, 2.1, and 2.1-cdh-6.3. +Choose the jar matching your Hive distribution and Paimon release. ## Execution Engine -Paimon currently supports MR and Tez execution engine for Hive Read, and MR execution engine for Hive Write. -Note If you use beeline, please restart the hive cluster. +| Operation | Hive execution engine | Limitations | +| --- | --- | --- | +| Read | MapReduce (MR) or Tez | Configure catalog and filesystem access before querying. | +| Write | MapReduce (MR) | `INSERT INTO` only; `INSERT OVERWRITE` is not supported. | + +Prefer append tables for Hive inserts. Writing primary-key tables from Hive can produce many +small files. For continuous ingestion or more extensive write operations, use +[Flink](../flink/sql-write) or [Spark](../spark/sql-write). ## Installation -Download the jar file with corresponding version. +### Download the Connector <Stable> - - -| | Jar | -|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Hive 3.1 | [paimon-hive-connector-3.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/paimon-hive-connector-3.1-@@VERSION@@.jar) | -| Hive 2.3 | [paimon-hive-connector-2.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/paimon-hive-connector-2.3-@@VERSION@@.jar) | -| Hive 2.2 | [paimon-hive-connector-2.2-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/paimon-hive-connector-2.2-@@VERSION@@.jar) | -| Hive 2.1 | [paimon-hive-connector-2.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/paimon-hive-connector-2.1-@@VERSION@@.jar) | -| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar) | - - +| Hive version | Connector | +| --- | --- | +| 3.1 | [paimon-hive-connector-3.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/paimon-hive-connector-3.1-@@VERSION@@.jar) | +| 2.3 | [paimon-hive-connector-2.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/paimon-hive-connector-2.3-@@VERSION@@.jar) | +| 2.2 | [paimon-hive-connector-2.2-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/paimon-hive-connector-2.2-@@VERSION@@.jar) | +| 2.1 | [paimon-hive-connector-2.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/paimon-hive-connector-2.1-@@VERSION@@.jar) | +| 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar) | </Stable> <Unstable> +| Hive version | Connector | +| --- | --- | +| 3.1 | [paimon-hive-connector-3.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/) | +| 2.3 | [paimon-hive-connector-2.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/) | +| 2.2 | [paimon-hive-connector-2.2-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/) | +| 2.1 | [paimon-hive-connector-2.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/) | +| 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/) | +</Unstable> -| | Jar | -|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Hive 3.1 | [paimon-hive-connector-3.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/) | -| Hive 2.3 | [paimon-hive-connector-2.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/) | -| Hive 2.2 | [paimon-hive-connector-2.2-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/) | -| Hive 2.1 | [paimon-hive-connector-2.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/) | -| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/) | - - +Snapshot links point to artifact directories. Select a published timestamped jar for your version. -</Unstable> +### Install the Jar -You can also manually build bundled jar from the source code. +Copy the matching connector jar into the Hive installation's `auxlib` directory so the storage +handler is available to Hive and its execution jobs. Restart existing Hive services after changing +jars. When using Beeline, restart the HiveServer2 service used by the session. -To build from source code, [clone the git repository](@@GITHUB_REPO@@). +For a temporary Hive CLI session, you can use `ADD JAR`: -Build bundled jar with the following command. -`mvn clean install -DskipTests` +```sql +ADD JAR /path/to/paimon-hive-connector-3.1-@@VERSION@@.jar; +``` -You can find Hive connector jar in `./paimon-hive/paimon-hive-connector-<hive-version>/target/paimon-hive-connector-<hive-version>-@@VERSION@@.jar`. +`ADD JAR` can cause class-loading failures for joins with the MR engine, including +`KryoException: unable to find class`. Use `auxlib` for a persistent installation. -There are several ways to add this jar to Hive. +### Build from Source -* You can create an `auxlib` folder under the root directory of Hive, and copy `paimon-hive-connector-@@VERSION@@.jar` into `auxlib`. -* You can also copy this jar to a path accessible by Hive, then use `add jar /path/to/paimon-hive-connector-@@VERSION@@.jar` to enable paimon support in Hive. Note that this method is not recommended. If you're using the MR execution engine and running a join statement, you may be faced with the exception `org.apache.hive.com.esotericsoftware.kryo.kryoexception: unable to find class`. +From a checkout of the [Paimon repository](@@GITHUB_REPO@@), build only the required connector +and its dependencies. For Hive 3.1: -NOTE: +```bash +mvn -pl paimon-hive/paimon-hive-connector-3.1 -am -DskipTests package +``` -* If you are using HDFS : - * Make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set. - * You can set `paimon.hadoop-load-default-config` =`false` to disable loading the default value from `core-default.xml`、`hdfs-default.xml`, which may lead smaller size for split. -* With hive cbo, it may lead to some incorrect query results, such as to query `struct` type with `not null` predicate, you can disable the cbo by `set hive.cbo.enable=false;` command. +The bundled jar is under +`paimon-hive/paimon-hive-connector-3.1/target/paimon-hive-connector-3.1-@@VERSION@@.jar`. +Replace `3.1` with the connector module matching your Hive distribution. -## Hive SQL: access Paimon Tables already in Hive metastore +## Access Existing Tables {#hive-sql-access-paimon-tables-already-in-hive-metastore} -Run the following Hive SQL in Hive CLI to access the created table. +If the table is already registered in the Hive metastore used by your Hive session, query it +directly. For example, assume `default.test_table` contains columns `a` and `b`: ```sql --- Assume that paimon-hive-connector-<hive-version>-@@VERSION@@.jar is already in auxlib directory. --- List tables in Hive --- (you might need to switch to "default" database if you're not there by default) - +USE default; SHOW TABLES; - -/* -OK -test_table -*/ - --- Read records from test_table - SELECT a, b FROM test_table ORDER BY a; +``` -/* -OK -1 Table -2 Store -*/ +If the table is not registered there, use an external table as described below. --- Insert records into test table --- Limitations: --- Only support INSERT INTO, not support INSERT OVERWRITE --- It is recommended to write to a non primary key table --- Writing to a primary key table may result in a large number of small files +## Register an External Table {#hive-sql-access-paimon-tables-by-external-table} -INSERT INTO test_table VALUES (3, 'Paimon'); +Point the external table at the existing **table directory**, rather than the warehouse root. +The storage handler loads the schema from Paimon; do not repeat column definitions. -SELECT a, b FROM test_table ORDER BY a; +```sql +CREATE EXTERNAL TABLE external_test_table +STORED BY 'org.apache.paimon.hive.PaimonStorageHandler' +LOCATION 'hdfs://namenode:8020/warehouse/paimon/default.db/test_table'; -/* -OK -1 Table -2 Store -3 Paimon -*/ +SELECT a, b FROM external_test_table ORDER BY a; +``` --- time travel - -SET paimon.scan.snapshot-id=1; -SELECT a, b FROM test_table ORDER BY a; -/* -OK -1 Table -2 Store -3 Paimon -*/ -SET paimon.scan.snapshot-id=null; +Alternatively, use `paimon_location` in `TBLPROPERTIES`. This avoids Hive accessing the Paimon +location through its own filesystem during table creation, which is useful for object storage. +Paimon's filesystem still needs the required libraries and credentials. +```sql +CREATE EXTERNAL TABLE external_s3_table +STORED BY 'org.apache.paimon.hive.PaimonStorageHandler' +TBLPROPERTIES ( + 'paimon_location' = 's3://paimon-bucket/warehouse/default.db/test_table' +); ``` -## Hive SQL: create new Paimon Tables +## Create a Table {#hive-sql-create-new-paimon-tables} -You can create new paimon tables in Hive. Run the following Hive SQL in Hive CLI. +Set the warehouse to your shared storage location, then create a table with the Paimon storage handler: ```sql --- Assume that paimon-hive-connector-@@VERSION@@.jar is already in auxlib directory. --- Let's create a new paimon table. +SET hive.metastore.warehouse.dir=hdfs://namenode:8020/warehouse/paimon; -SET hive.metastore.warehouse.dir=warehouse_path; - -CREATE TABLE hive_test_table( +CREATE TABLE hive_test_table ( a INT COMMENT 'The a field', b STRING COMMENT 'The b field' ) STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'; ``` -## Hive SQL: access Paimon Tables by External Table +## Insert Records -To access existing paimon table, you can also register them as external tables in Hive. Run the following Hive SQL in Hive CLI. +Use the MR execution engine. This example continues from the table created above: ```sql --- Assume that paimon-hive-connector-@@VERSION@@.jar is already in auxlib directory. --- Let's use the test_table created in the above section. --- To create an external table, you don't need to specify any column or table properties. --- Pointing the location to the path of table is enough. - -CREATE EXTERNAL TABLE external_test_table -STORED BY 'org.apache.paimon.hive.PaimonStorageHandler' -LOCATION '/path/to/table/store/warehouse/default.db/test_table'; - --- In addition to the way setting location above, you can also place the location setting in TBProperties --- to avoid Hive accessing Paimon's location through its own file system when creating tables. --- This method is effective in scenarios using Object storage,such as s3. - -CREATE EXTERNAL TABLE external_test_table -STORED BY 'org.apache.paimon.hive.PaimonStorageHandler' -TBLPROPERTIES ( - 'paimon_location' ='s3://xxxxx/path/to/table/store/warehouse/default.db/test_table' -); - --- Read records from external_test_table - -SELECT a, b FROM external_test_table ORDER BY a; - -/* -OK -1 Table -2 Store -*/ +SET hive.execution.engine=mr; +INSERT INTO hive_test_table VALUES (1, 'Paimon'), (2, 'Hive'); +SELECT a, b FROM hive_test_table ORDER BY a; +``` --- Insert records into test table +The same `INSERT INTO` syntax can write to a registered external Paimon table, subject to the +[write limitations](#execution-engine). -INSERT INTO external_test_table VALUES (3, 'Paimon'); +## Time Travel -SELECT a, b FROM external_test_table ORDER BY a; +Set `paimon.scan.snapshot-id` to an existing, retained snapshot. The setting applies to the +session's subsequent Paimon scans, so clear it before resuming ordinary queries. -/* -OK -1 Table -2 Store -3 Paimon -*/ +```sql +-- Use 1 only if snapshot 1 still exists for this table. +SET paimon.scan.snapshot-id=1; +SELECT a, b FROM hive_test_table ORDER BY a; +-- Resume reading the latest snapshot. +SET paimon.scan.snapshot-id=null; ``` -## Hive Type Conversion - -This section lists all supported type conversion between Hive and Paimon. -All Hive's data types are available in package `org.apache.hadoop.hive.serde2.typeinfo`. - -<table className="table table-bordered"> - <thead> - <tr> - <th className="text-left" style={{width: "10%"}}>Hive Data Type</th> - <th className="text-left" style={{width: "10%"}}>Paimon Data Type</th> - <th className="text-left" style={{width: "5%"}}>Atomic Type</th> - </tr> - </thead> - <tbody> - <tr> - <td><code>StructTypeInfo</code></td> - <td><code>RowType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>MapTypeInfo</code></td> - <td><code>MapType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>ListTypeInfo</code></td> - <td><code>ArrayType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("boolean")</code></td> - <td><code>BooleanType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("tinyint")</code></td> - <td><code>TinyIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("smallint")</code></td> - <td><code>SmallIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("int")</code></td> - <td><code>IntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("bigint")</code></td> - <td><code>BigIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("float")</code></td> - <td><code>FloatType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("double")</code></td> - <td><code>DoubleType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>CharTypeInfo(length)</code></td> - <td><code>CharType(length)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("string")</code></td> - <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VarcharTypeInfo(length)</code></td> - <td><code>VarCharType(length), length is less than VarCharType.MAX_LENGTH</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("date")</code></td> - <td><code>DateType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("timestamp")</code></td> - <td><code>TimestampType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DecimalTypeInfo(precision, scale)</code></td> - <td><code>DecimalType(precision, scale)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>PrimitiveTypeInfo("binary")</code></td> - <td><code>VarBinaryType</code>, <code>BinaryType</code></td> - <td>true</td> - </tr> - </tbody> -</table> +The rows returned depend on what was committed in that snapshot. See +[Snapshot Management](../maintenance/manage-snapshots) for retention behavior. + +## Type Mapping {#hive-type-conversion} + +When creating a Paimon table in Hive, the connector converts Hive SQL types as follows. +Nested types are converted recursively. + +| Hive type | Paimon type | +| --- | --- | +| `BOOLEAN` | `BOOLEAN` | +| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | Corresponding integer type | +| `FLOAT`, `DOUBLE` | `FLOAT`, `DOUBLE` | +| `DECIMAL(p, s)` | `DECIMAL(p, s)` | +| `CHAR(n)`, `VARCHAR(n)` | `CHAR(n)`, `VARCHAR(n)` | +| `STRING` | `STRING` (`VARCHAR` with maximum length) | +| `BINARY` | `VARBINARY` with maximum length | +| `DATE` | `DATE` | +| `TIMESTAMP` | `TIMESTAMP` | +| `TIMESTAMP WITH LOCAL TIME ZONE` (Hive 3) | `TIMESTAMP WITH LOCAL TIME ZONE` | +| `ARRAY`, `MAP`, `STRUCT` | `ARRAY`, `MAP`, `ROW` | + +The reverse mapping, when reading an existing Paimon table, is not always symmetric: + +| Paimon type | Hive representation | +| --- | --- | +| `BINARY`, `VARBINARY` | `BINARY` | +| `CHAR(n)` or `VARCHAR(n)` beyond Hive's length limit | `STRING` | +| `TIME` | `STRING` | +| `TIMESTAMP WITH LOCAL TIME ZONE` | Hive 3 local-time-zone timestamp; ordinary `TIMESTAMP` in Hive 2 | +| `MULTISET<T>` | `MAP<T, INT>` | + +See [Paimon Data Types](../concepts/data-types) for the Paimon type system. + +## Troubleshooting + +- **HDFS configuration:** set `HADOOP_HOME` or `HADOOP_CONF_DIR` so the connector can load the + Hadoop configuration. `SET paimon.hadoop-load-default-config=false;` skips loading + `core-default.xml` and `hdfs-default.xml`, which can reduce serialized split size; explicitly + provide the settings your filesystem requires. +- **Incorrect results with Hive CBO:** if predicates on complex types produce incorrect results + (for example, `IS NOT NULL` on a struct), retry with `SET hive.cbo.enable=false;`. +- **Class-loading failures:** verify the matching connector is installed in `auxlib` and restart + Hive services. See [installation](#installation) for the limitation of `ADD JAR`. + +For shared catalog and storage checks, see [Connecting Engines](./connecting-engines#troubleshooting). diff --git a/docs/docs/ecosystem/index.md b/docs/docs/ecosystem/index.md index be5017b1f6..4428f80cb5 100644 --- a/docs/docs/ecosystem/index.md +++ b/docs/docs/ecosystem/index.md @@ -22,58 +22,70 @@ specific language governing permissions and limitations under the License. --> -# Overview +# Ecosystem -## Compatibility Matrix +Use Paimon tables across ingestion pipelines, SQL engines, and lakehouse management tools. +Start with an integration below, then follow [Connecting Engines](./connecting-engines) to +configure catalog discovery and storage access. -| Engine | Version | Batch Read | Batch Write | Create Table | Alter Table | Streaming Write | Streaming Read | Batch Overwrite | DELETE & UPDATE | MERGE INTO | Time Travel | -|:-------------------------------------------------------------------------------:|:-------------:|:-----------:|:-----------:|:-------------:|:-------------:|:----------------:|:----------------:|:---------------:|:---------------:|:----------:|:-----------:| -| Flink | 1.16 - 1.20 | ✅ | ✅ | ✅ | ✅(1.17+) | ✅ | ✅ | ✅ | ✅(1.17+) | ❌ | ✅ | -| Spark | 3.2 - 4.1 | ✅ | ✅ | ✅ | ✅ | ✅(3.3+) | ✅(3.3+) | ✅ | ✅ | ✅ | ✅(3.3+) | -| Hive | 2.1 - 3.1 | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | -| Trino | 420 - 440 | ✅ | ✅(427+) | ✅(427+) | ✅(427+) | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | -| Presto | 0.236 - 0.280 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| [StarRocks](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/) | 3.1+ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | -| [Doris](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog) | 2.0.6+ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | + -## Streaming Engines +## Choose an Integration -### Flink Streaming +| What you want to do | Start here | +| --- | --- | +| Ingest CDC events or build a streaming pipeline | [Flink Quick Start](../flink/quick-start), [CDC Ingestion](../cdc-ingestion/) | +| Run batch transformations or Spark SQL | [Spark Quick Start](../spark/quick-start) | +| Process streams with Spark micro-batches | [Spark Structured Streaming](../spark/structured-streaming) | +| Query Paimon from an OLAP engine | [StarRocks](./starrocks), [Doris](./doris) | +| Query or write tables with distributed SQL | [Trino](./trino) | +| Access tables from Hive | [Hive](./hive) | +| Inspect tables in a lakehouse management service | [Amoro](./amoro) | -Flink is the most comprehensive streaming computing engine that is widely used for data CDC ingestion and the -construction of streaming pipelines. +## Compatibility Matrix -Recommended version is Flink 1.17.2. +For bundled connectors, match the engine version to the connector artifact for your Paimon +release. The Flink, Spark, and Hive versions below describe connector modules in this branch; +artifact availability depends on the release. Externally maintained integrations have their own +release cycle, embedded Paimon version, and feature limits. + +| Integration | Version selection | Access to Paimon tables | +| --- | --- | --- | +| [Flink](../flink/installation) | 1.16–1.20 and 2.0–2.2 | Batch and streaming reads/writes; [DDL](../flink/sql-ddl) and [row changes](../flink/sql-write) have version-specific requirements. | +| [Spark](../spark/quick-start) | 3.2–3.5, 4.0, and 4.1; match the Scala binary version | Batch reads/writes, [DDL](../spark/sql-ddl), and [row changes](../spark/sql-write); [streaming](../spark/structured-streaming) requires Spark 3.3+. | +| [Hive](./hive#version) | 2.1, 2.2, 2.3, 3.1, and 2.1-cdh-6.3 | Batch reads, table creation, and `INSERT INTO`; writes require MapReduce. | +| [Trino](./trino#version) | Match the independently released Paimon connector to Trino | Batch reads; supported connectors also provide DDL, inserts, and time travel. See the guide's table-layout limits. | +| [Presto](https://github.com/apache/paimon-presto) | Follow the separate connector's version requirements | See the connector repository for installation and supported operations. | +| [StarRocks](./starrocks#version) | Paimon catalogs available from 3.1 | Query existing tables through an external catalog; check the engine release for individual features. | +| [Doris](./doris#version) | Select a release with the required catalog and reader features | Query existing tables through an external catalog; REST catalog access requires Doris 3.1+. | + +A connector's ability to read a table also depends on its data types, file format, merge engine, +and enabled features, such as deletion vectors. Check the relevant engine guide before enabling +a new table feature in a warehouse shared by several engines. -### Spark Streaming +## Streaming Engines -You can also use Spark Streaming to build a streaming pipeline. Spark's schema evolution capability will be better -implemented, but you must accept the mechanism of mini-batch. +Use [Flink](../flink/) for continuous ingestion, change processing, and lookup joins. +Use [Spark Structured Streaming](../spark/structured-streaming) for micro-batch pipelines. +Configure the table's [changelog producer](../primary-key-table/changelog-producer) for the +changes that downstream readers need. ## Batch Engines -### Spark Batch - -Spark Batch is the most widely used batch computing engine. - -Recommended version is Spark 3.5.8. - -### Flink Batch - -Flink Batch is also available, which can make your pipeline more integrated with streaming and batch unified. +Use [Spark SQL](../spark/sql-query) or [Flink batch SQL](../flink/sql-query) to read a snapshot +and run transformations. Consult the write guides for [Spark](../spark/sql-write) and +[Flink](../flink/sql-write) before using overwrite, `DELETE`, `UPDATE`, or `MERGE INTO`: +SQL support and table requirements differ by engine and version. ## OLAP Engines -### StarRocks - -StarRocks is the most recommended OLAP engine with the most advanced integration. - -Recommended version is StarRocks 3.2.6. - -### Other OLAP - -You can also use Doris and Trino and Presto, or, you can just use Spark, Flink and Hive to query Paimon tables. +[StarRocks](./starrocks) and [Doris](./doris) query Paimon through their own external catalogs. +[Trino](./trino) and [Presto](https://github.com/apache/paimon-presto) use separately distributed +connectors. Configure access to both the catalog and the underlying files, and choose the +[table read mode](../primary-key-table/table-mode) for your freshness requirements. ## Download -[Download Link](../project/download#engine-jars) +Use the [engine downloads](../project/download#engine-jars) for Paimon artifacts and each +integration guide for installation. For Trino and Presto, use the connector project's release +instructions; its version does not necessarily match this documentation's Paimon version. diff --git a/docs/docs/ecosystem/starrocks.md b/docs/docs/ecosystem/starrocks.md index a155beaeea..3c3a243a49 100644 --- a/docs/docs/ecosystem/starrocks.md +++ b/docs/docs/ecosystem/starrocks.md @@ -24,159 +24,99 @@ under the License. # StarRocks -This documentation is a guide for using Paimon in StarRocks. +Query existing Paimon tables through a StarRocks external catalog. Start with +[Connecting Engines](./connecting-engines) if you need to identify the catalog and warehouse. ## Version -Paimon currently supports StarRocks 3.1 and above. Recommended version is StarRocks 3.2.6 or above. +Paimon catalogs are available in StarRocks 3.1 and later. Individual features depend on the +StarRocks release. Use the [StarRocks Paimon catalog documentation](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/) +for the supported catalog backends, storage configuration, and release-specific limitations. + +This integration reads Paimon tables. Creating an external catalog does not create or copy the +tables, and the catalog does not support inserting, updating, or deleting Paimon data. + +## Prerequisites + +Prepare an existing Paimon database and table. Ensure the StarRocks processes that access metadata +and data can reach the metastore and warehouse. Configure authentication and filesystem access +using the StarRocks documentation for your storage system. ## Create Paimon Catalog -Paimon catalogs are registered by executing a `CREATE EXTERNAL CATALOG` SQL in StarRocks. -For example, you can use the following SQL to create a Paimon catalog named paimon_catalog. +Choose the backend used by the writer. These examples assume HDFS access is already configured; +replace the host names and warehouse path with your deployment values. + +### Filesystem Catalog ```sql -CREATE EXTERNAL CATALOG paimon_catalog PROPERTIES( +CREATE EXTERNAL CATALOG paimon_catalog PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", - "paimon.catalog.warehouse" = "oss://<your_bucket>/user/warehouse/" + "paimon.catalog.warehouse" = "hdfs://namenode:8020/warehouse/paimon" ); ``` -More catalog types and configures can be seen in [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/). +### Hive Metastore Catalog -## Query -Suppose there already exists a database named `test_db` and a table named `test_tbl` in `paimon_catalog`, -you can query this table using the following SQL: ```sql -SELECT * FROM paimon_catalog.test_db.test_tbl; +CREATE EXTERNAL CATALOG paimon_hms PROPERTIES ( + "type" = "paimon", + "paimon.catalog.type" = "hive", + "paimon.catalog.warehouse" = "hdfs://namenode:8020/warehouse/paimon", + "hive.metastore.uris" = "thrift://metastore:9083" +); ``` -## Query System Tables +For object storage, use the warehouse URI and authentication properties from the +[StarRocks catalog examples](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/#examples). + +## Query -You can access all kinds of Paimon system tables by StarRocks. For example, you can read the `ro` -(read-optimized) system table to improve reading performance of primary-key tables. +Assume `test_db.test_tbl` already exists in the configured warehouse: ```sql -SELECT * FROM paimon_catalog.test_db.test_tbl$ro; +SELECT * FROM paimon_catalog.test_db.test_tbl LIMIT 10; ``` -For another example, you can query partition files of the table using the following SQL: +## Query System Tables + +On StarRocks versions supporting Paimon system tables, append the system-table suffix to the +table name. Quote the complete table identifier containing `$`: ```sql -SELECT * FROM paimon_catalog.test_db.partition_tbl$partitions; -/* -+-----------+--------------+--------------------+------------+----------------------------+ -| partition | record_count | file_size_in_bytes | file_count | last_update_time | -+-----------+--------------+--------------------+------------+----------------------------+ -| [1] | 1 | 645 | 1 | 2024-01-01 00:00:00.000000 | -+-----------+--------------+--------------------+------------+----------------------------+ -*/ +SELECT * FROM paimon_catalog.test_db.`test_tbl$partitions`; +SELECT * FROM paimon_catalog.test_db.`test_tbl$ro`; ``` -## StarRocks to Paimon type mapping - -This section lists all supported type conversion between StarRocks and Paimon. -All StarRocks's data types can be found in this doc [StarRocks Data type overview](https://docs.starrocks.io/docs/sql-reference/data-types/). - -<table class="table table-bordered"> - <thead> - <tr> - <th class="text-left" style="width: 10%">StarRocks Data Type</th> - <th class="text-left" style="width: 10%">Paimon Data Type</th> - <th class="text-left" style="width: 5%">Atomic Type</th> - </tr> - </thead> - <tbody> - <tr> - <td><code>STRUCT</code></td> - <td><code>RowType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>MAP</code></td> - <td><code>MapType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>ARRAY</code></td> - <td><code>ArrayType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>BOOLEAN</code></td> - <td><code>BooleanType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>TINYINT</code></td> - <td><code>TinyIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>SMALLINT</code></td> - <td><code>SmallIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>INT</code></td> - <td><code>IntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>BIGINT</code></td> - <td><code>BigIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>FLOAT</code></td> - <td><code>FloatType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DOUBLE</code></td> - <td><code>DoubleType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>CHAR(length)</code></td> - <td><code>CharType(length)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VARCHAR(MAX_VARCHAR_LENGTH)</code></td> - <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VARCHAR(length)</code></td> - <td><code>VarCharType(length), length is less than VarCharType.MAX_LENGTH</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DATE</code></td> - <td><code>DateType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DATETIME</code></td> - <td><code>TimestampType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DECIMAL(precision, scale)</code></td> - <td><code>DecimalType(precision, scale)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VARBINARY(length)</code></td> - <td><code>VarBinaryType(length)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DATETIME</code></td> - <td><code>LocalZonedTimestampType</code></td> - <td>true</td> - </tr> - </tbody> -</table> +The [`$ro` table](../concepts/system-tables#read-optimized-table) reads compacted data from a +primary-key table. Its results can lag behind the latest snapshot until a full compaction +completes. Use the ordinary table query when you need the latest state supported by the reader. +See [MOR Read Optimized](../primary-key-table/table-mode#mor-read-optimized) for maintenance settings. + +## Type Mapping {#starrocks-to-paimon-type-mapping} + +The following table summarizes common Paimon types as exposed by StarRocks. It describes the +**read direction**; it does not imply support for creating these types in Paimon from StarRocks. +Refer to the [upstream type mapping](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/#paimon-to-starrocks-data-types) +for your release, including precision and type limits. + +| Paimon type | StarRocks type | +| --- | --- | +| `BOOLEAN` | `BOOLEAN` | +| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | Corresponding integer type | +| `FLOAT`, `DOUBLE` | `FLOAT`, `DOUBLE` | +| `DECIMAL(p, s)` | `DECIMAL(p, s)` | +| `CHAR(n)` | `CHAR(n)` | +| `VARCHAR(n)`, `STRING` | `VARCHAR` | +| `BINARY(n)`, `VARBINARY(n)` | `VARBINARY` | +| `DATE` | `DATE` | +| `TIMESTAMP`, `TIMESTAMP WITH LOCAL TIME ZONE` | `DATETIME` | +| `ARRAY` | `ARRAY` | +| `MAP` | `MAP` | +| `ROW` | `STRUCT` | + +## Next Steps + +- Review [query performance](../primary-key-table/query-performance) and [table modes](../primary-key-table/table-mode). +- Diagnose catalog and storage issues with [Connecting Engines](./connecting-engines#troubleshooting). diff --git a/docs/docs/ecosystem/trino.md b/docs/docs/ecosystem/trino.md index 00eb9b72f9..961ba443b9 100644 --- a/docs/docs/ecosystem/trino.md +++ b/docs/docs/ecosystem/trino.md @@ -24,106 +24,140 @@ under the License. # Trino -This documentation is a guide for using Paimon in Trino. +Use the separately distributed Paimon connector to query tables from Trino. The examples on this +page describe the **Trino 440 connector**. For other versions, follow the matching revision of +[Apache Paimon Trino](https://github.com/apache/paimon-trino). ## Version -Paimon currently supports Trino 440. +Match the Trino server version, connector artifact, and Java runtime. The connector repository +has its own release cycle and Paimon dependency; its default branch can target a different Trino +and Java version from the examples here. -## Filesystem +The write examples below require a connector with write support and a supported table layout. +They do not imply that every historical Trino connector supports these operations. -From version 0.8, Paimon share Trino filesystem for all actions, which means, you should -config Trino filesystem before using trino-paimon. You can find information about how to config -filesystems for Trino on Trino official website. +## Installation {#preparing-paimon-jar-file} -## Preparing Paimon Jar File +### Obtain the Plugin -[Download](../project/download) +Use the [download page](../project/download#engine-jars) to locate the Trino 440 plugin archive. +For other versions, use the [connector repository](https://github.com/apache/paimon-trino). +The plugin is a distribution containing its dependencies, rather than a single jar to place on +the general Trino classpath. -You can also manually build a bundled jar from the source code. However, there are a few preliminary steps that need to be taken before compiling: - -- To build from the source code, [clone the git repository](@@TRINO_GITHUB_REPO@@). -- Install JDK21 locally, and configure JDK21 as a global environment variable; - -Then,you can build bundled jar with the following command: +To build from source, check out a connector revision matching the Trino server and use the Java +version required by that revision's `pom.xml`. Run its documented build; for revisions using the +Maven build, the command is: ```bash mvn clean install -DskipTests ``` -You can find Trino connector jar in `./paimon-trino-<trino-version>/target/paimon-trino-<trino-version>-@@VERSION@@-plugin.tar.gz`. +Locate the plugin archive in that revision's build output. Artifact names and module paths +vary across revisions. Use the connector artifact's actual version rather than substituting +the Paimon version shown by this documentation. + +### Install Paimon Connector -We use [hadoop-apache](https://mvnrepository.com/artifact/io.trino.hadoop/hadoop-apache) as a dependency for Hadoop, -and the default Hadoop dependency typically supports both Hadoop 2 and Hadoop 3. -If you encounter an unsupported scenario, you can specify the corresponding Apache Hadoop version. +For a Trino 440 distribution named +`paimon-trino-440-<connector-version>-plugin.tar.gz`, extract it under the Trino plugin directory +on the coordinator and every worker: -For example, if you want to use Hadoop 3.3.5-1, you can use the following command to build the jar: ```bash -mvn clean install -DskipTests -Dhadoop.apache.version=3.3.5-1 +tar -zxf paimon-trino-440-<connector-version>-plugin.tar.gz -C "${TRINO_HOME}/plugin" ``` -## Configure Paimon Catalog +For the Trino 440 connector running on JDK 21, include these entries in `etc/jvm.config`: -### Install Paimon Connector -```bash -tar -zxf paimon-trino-<trino-version>-@@VERSION@@-plugin.tar.gz -C ${TRINO_HOME}/plugin +```text +--add-opens=java.base/sun.nio.ch=ALL-UNNAMED +--add-opens=java.base/java.nio=ALL-UNNAMED ``` -> NOTE: For JDK 21, when Deploying Trino, should add jvm options: `--add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED` +Restart Trino after installing the plugin and completing the catalog configuration. -### Configure +## Configure Paimon Catalog -Catalogs are registered by creating a catalog properties file in the etc/catalog directory. For example, create etc/catalog/paimon.properties with the following contents to mount the paimon connector as the paimon catalog: +Create `etc/catalog/paimon.properties` on the Trino nodes. The file name registers the SQL +catalog as `paimon`: ```properties connector.name=paimon -warehouse=file:/tmp/warehouse +warehouse=hdfs://namenode:8020/warehouse/paimon ``` -If you are using HDFS, choose one of the following ways to configure your HDFS: +This example uses a filesystem catalog and assumes HDFS access is configured. For a local +single-node experiment, `warehouse=file:/tmp/warehouse` is sufficient. Use shared storage for +a distributed deployment. -- set environment variable HADOOP_HOME. -- set environment variable HADOOP_CONF_DIR. -- configure `hadoop-conf-dir` in the properties. +### Filesystem -If you are using a Hadoop filesystem, you can still use trino-hdfs and trino-hive to config it. -For example, if you use oss as a storage, you can write in `paimon.properties` according to [Trino Reference](https://trino.io/docs/current/connector/hive.html#hdfs-configuration): +The Trino 440 connector uses Trino's filesystem integration. Configure storage access before +querying a table. For HDFS, provide `HADOOP_HOME`, `HADOOP_CONF_DIR`, or the connector's +`hadoop-conf-dir` property as appropriate for the deployment. + +For Hadoop-backed object storage, supply the filesystem configuration through: ```properties hive.config.resources=/path/to/core-site.xml ``` -Then, config core-site.xml according to [Jindo Reference](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto) +Make the configuration file and required filesystem libraries available on all nodes that access +the warehouse. Follow the filesystem instructions for your **Trino and connector versions**; +properties from newer Trino releases may differ. + +Connector revisions using `io.trino.hadoop:hadoop-apache` can override that dependency when a +specific Hadoop distribution is required. For example, if supported by the selected revision: -## Kerberos +```bash +mvn clean install -DskipTests -Dhadoop.apache.version=3.3.5-1 +``` -You can configure kerberos keytab file when using KERBEROS authentication in the properties. +### Kerberos + +For the Trino 440 connector's Kerberos login, set these catalog properties: ```properties -security.kerberos.login.principal=hadoop-user [email protected] security.kerberos.login.keytab=/etc/trino/hdfs.keytab ``` -Keytab files must be distributed to every node in the cluster that runs Trino. +Distribute the keytab to every Trino node that needs it and configure access for the service user. + +### Temporary Directory {#tmp-dir} + +Paimon extracts jars for code generation into the JVM temporary directory. Choose a writable +location that is not removed by periodic cleanup while Trino is running. Set this JVM option +in `etc/jvm.config` on each node: + +```text +-Djava.io.tmpdir=/path/to/trino-tmp +``` ## Create Schema +The following examples form one sequence in the `paimon` catalog: + ```sql CREATE SCHEMA paimon.test_db; ``` ## Create Table +Create a primary-key table with a fixed bucket count. Include the partition column in the +primary key: + ```sql CREATE TABLE paimon.test_db.orders ( order_key bigint, - orders_tatus varchar, + order_status varchar, total_price decimal(18,4), order_date date ) WITH ( file_format = 'ORC', - primary_key = ARRAY['order_key','order_date'], + primary_key = ARRAY['order_key', 'order_date'], partitioned_by = ARRAY['order_date'], bucket = '2', bucket_key = 'order_key', @@ -131,25 +165,15 @@ WITH ( ); ``` -## Add Column +## Insert -```sql -CREATE TABLE paimon.test_db.orders ( - order_key bigint, - orders_tatus varchar, - total_price decimal(18,4), - order_date date -) -WITH ( - file_format = 'ORC', - primary_key = ARRAY['order_key','order_date'], - partitioned_by = ARRAY['order_date'], - bucket = '2', - bucket_key = 'order_key', - changelog_producer = 'input' -); +The Trino 440 connector supports inserts into primary-key tables with fixed buckets and +non-primary-key tables with `bucket = -1`. -ALTER TABLE paimon.test_db.orders ADD COLUMN shipping_address varchar; +```sql +INSERT INTO paimon.test_db.orders + (order_key, order_status, total_price, order_date) +VALUES (1, 'NEW', DECIMAL '19.9900', DATE '2024-01-01'); ``` ## Query @@ -158,153 +182,63 @@ ALTER TABLE paimon.test_db.orders ADD COLUMN shipping_address varchar; SELECT * FROM paimon.test_db.orders; ``` -## Query with Time Traveling - -```sql --- read the snapshot from specified timestamp -SELECT * FROM t FOR TIMESTAMP AS OF TIMESTAMP '2023-01-01 00:00:00 Asia/Shanghai'; - --- read the snapshot with id 1L (use snapshot id as version) -SELECT * FROM t FOR VERSION AS OF 1; +## Add Column --- read tag 'my-tag' -SELECT * FROM t FOR VERSION AS OF 'my-tag'; +Continue with the existing table; there is no need to create it again: +```sql +ALTER TABLE paimon.test_db.orders ADD COLUMN shipping_address varchar; ``` -:::warning +## Query with Time Traveling -If tag's name is a number and equals to a snapshot id, the VERSION AS OF syntax will consider tag first. For example, if -you have a tag named '1' based on snapshot 2, the statement `SELECT * FROM paimon.test_db.orders FOR VERSION AS OF '1'` actually queries snapshot 2 -instead of snapshot 1. +Use a retained snapshot or tag. The timestamp and identifiers below are examples; replace them +with values from the table's history. -::: +```sql +-- Select the snapshot at the specified timestamp. +SELECT * FROM paimon.test_db.orders +FOR TIMESTAMP AS OF TIMESTAMP '2024-01-01 00:00:00 Asia/Shanghai'; -## Insert +-- Select snapshot 1 if it is still retained. +SELECT * FROM paimon.test_db.orders FOR VERSION AS OF 1; -```sql -INSERT INTO paimon.test_db.orders VALUES (.....); +-- Select an existing tag. +SELECT * FROM paimon.test_db.orders FOR VERSION AS OF 'my-tag'; ``` -Supports: -- primary key table with fixed bucket. -- non-primary-key table with bucket -1. - -## Trino to Paimon type mapping - -This section lists all supported type conversion between Trino and Paimon. -All Trino's data types are available in package `io.trino.spi.type`. - -<table class="table table-bordered"> - <thead> - <tr> - <th class="text-left" style="width: 10%">Trino Data Type</th> - <th class="text-left" style="width: 10%">Paimon Data Type</th> - <th class="text-left" style="width: 5%">Atomic Type</th> - </tr> - </thead> - <tbody> - <tr> - <td><code>RowType</code></td> - <td><code>RowType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>MapType</code></td> - <td><code>MapType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>ArrayType</code></td> - <td><code>ArrayType</code></td> - <td>false</td> - </tr> - <tr> - <td><code>BooleanType</code></td> - <td><code>BooleanType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>TinyintType</code></td> - <td><code>TinyIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>SmallintType</code></td> - <td><code>SmallIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>IntegerType</code></td> - <td><code>IntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>BigintType</code></td> - <td><code>BigIntType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>RealType</code></td> - <td><code>FloatType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DoubleType</code></td> - <td><code>DoubleType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>CharType(length)</code></td> - <td><code>CharType(length)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td> - <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VarCharType(length)</code></td> - <td><code>VarCharType(length), length is less than VarCharType.MAX_LENGTH</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DateType</code></td> - <td><code>DateType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>TimestampType</code></td> - <td><code>TimestampType</code></td> - <td>true</td> - </tr> - <tr> - <td><code>DecimalType(precision, scale)</code></td> - <td><code>DecimalType(precision, scale)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>VarBinaryType(length)</code></td> - <td><code>VarBinaryType(length)</code></td> - <td>true</td> - </tr> - <tr> - <td><code>TimestampWithTimeZoneType</code></td> - <td><code>LocalZonedTimestampType</code></td> - <td>true</td> - </tr> - </tbody> -</table> - -## Tmp Dir - -Paimon will unzip some jars to the tmp directory for codegen. By default, Trino will use `'/tmp'` as the temporary -directory, but `'/tmp'` may be periodically deleted. - -You can configure this environment variable when Trino starts: -```shell --Djava.io.tmpdir=/path/to/other/tmpdir -``` +:::warning Numeric tag names + +A numeric tag name takes precedence over a matching snapshot ID when supplied as a string. +If tag `'1'` points to snapshot 2, `FOR VERSION AS OF '1'` reads snapshot 2. Prefer descriptive +tag names to avoid ambiguity. + +::: -Let Paimon use a secure temporary directory. +See [Snapshot Management](../maintenance/manage-snapshots) and [Tags](../maintenance/manage-tags) +for retention and tag creation. + +## Type Mapping {#trino-to-paimon-type-mapping} + +Common read mappings for the connector are summarized below using SQL type names. Check the +matching connector revision for precision limits and unsupported types. + +| Paimon type | Trino type | +| --- | --- | +| `BOOLEAN` | `BOOLEAN` | +| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT` | +| `FLOAT`, `DOUBLE` | `REAL`, `DOUBLE` | +| `DECIMAL(p, s)` | `DECIMAL(p, s)` | +| `CHAR(n)` | `CHAR(n)` | +| `VARCHAR(n)`, `STRING` | `VARCHAR(n)`, `VARCHAR` | +| `VARBINARY` | `VARBINARY` | +| `DATE` | `DATE` | +| `TIMESTAMP` | `TIMESTAMP` | +| `TIMESTAMP WITH LOCAL TIME ZONE` | `TIMESTAMP WITH TIME ZONE` | +| `ARRAY`, `MAP`, `ROW` | `ARRAY`, `MAP`, `ROW` | + +## Next Steps + +Use [Connecting Engines](./connecting-engines#troubleshooting) for catalog and storage checks. +Report connector-specific problems to [Apache Paimon Trino](https://github.com/apache/paimon-trino/issues), +including the Trino, Java, and connector versions and the table options. diff --git a/docs/sidebars.js b/docs/sidebars.js index a96ae758ae..ceac74a000 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -486,10 +486,17 @@ const sidebars = { "id": "ecosystem/index" }, "items": [ - "ecosystem/starrocks", - "ecosystem/doris", - "ecosystem/hive", - "ecosystem/trino", + "ecosystem/connecting-engines", + { + type: "category", + "label": "Query Engines", + "items": [ + "ecosystem/starrocks", + "ecosystem/doris", + "ecosystem/hive", + "ecosystem/trino" + ] + }, "ecosystem/amoro" ] }, diff --git a/docs/static/img/amoro-paimon.png b/docs/static/img/amoro-paimon.png deleted file mode 100644 index 02b8555699..0000000000 Binary files a/docs/static/img/amoro-paimon.png and /dev/null differ diff --git a/docs/static/img/amoro-paimon.svg b/docs/static/img/amoro-paimon.svg new file mode 100644 index 0000000000..aef7f15bd7 --- /dev/null +++ b/docs/static/img/amoro-paimon.svg @@ -0,0 +1,64 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="960" height="518" viewBox="0 0 960 518" role="img" aria-labelledby="title desc"> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<title id="title">Inspect Paimon tables with Amoro</title> +<desc id="desc">Register a Paimon catalog with Amoro Management Service to inspect schemas, options, files, snapshots, DDL, and compaction information. Terminal SQL runs in Spark, locally inside AMS or through Kyuubi, with access to the same catalog and storage.</desc> +<defs> + <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#526277"/></marker> +</defs> +<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d"> +<rect x="1" y="1" width="958" height="516" rx="12" fill="#fff" stroke="#d7dfeb" stroke-width="1.5"/> +<text x="28" y="44" font-size="27" fill="#526277" font-weight="700">Inspect Paimon tables with Amoro</text> +<text x="28" y="77" font-size="18" fill="#526277">Catalog registration makes existing table metadata available in AMS.</text> +<rect x="28" y="124" width="258" height="166" rx="8" fill="#e5f5ef" stroke="#10705d" stroke-width="1.5"/> +<text x="46" y="157" font-size="21" fill="#10705d" font-weight="700">Paimon catalog</text> +<text x="46" y="189" font-size="18" fill="#526277">Catalog configuration</text> +<text x="46" y="215" font-size="18" fill="#526277">Metastore / service access</text> +<text x="46" y="241" font-size="18" fill="#526277">Warehouse storage</text> +<rect x="370" y="124" width="242" height="166" rx="8" fill="#f1ebfa" stroke="#7051a3" stroke-width="1.5"/> +<text x="388" y="157" font-size="21" fill="#7051a3" font-weight="700">AMS</text> +<text x="388" y="189" font-size="18" fill="#526277">Registered catalog</text> +<text x="388" y="215" font-size="18" fill="#526277">Paimon integration</text> +<text x="388" y="241" font-size="18" fill="#526277">Storage dependencies</text> +<rect x="696" y="124" width="236" height="166" rx="8" fill="#f1ebfa" stroke="#7051a3" stroke-width="1.5"/> +<text x="714" y="157" font-size="21" fill="#7051a3" font-weight="700">Table views</text> +<text x="714" y="189" font-size="18" fill="#526277">Schema · options · DDL</text> +<text x="714" y="215" font-size="18" fill="#526277">Files · snapshots</text> +<text x="714" y="241" font-size="18" fill="#526277">Compaction information</text> +<path d="M 290 206 H 366" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="306" y="190" font-size="16" fill="#526277">register</text> +<path d="M 616 206 H 692" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="631" y="190" font-size="16" fill="#526277">browse</text> +<path d="M 28 327 H 932" stroke="#d7dfeb" stroke-width="1.5"/> +<rect x="28" y="360" width="258" height="112" rx="8" fill="#eaf2ff" stroke="#2463b4" stroke-width="1.5"/> +<text x="46" y="393" font-size="21" fill="#2463b4" font-weight="700">Amoro Terminal</text> +<text x="46" y="425" font-size="18" fill="#526277">Submit Spark SQL</text> +<rect x="370" y="360" width="242" height="112" rx="8" fill="#eaf2ff" stroke="#2463b4" stroke-width="1.5"/> +<text x="388" y="393" font-size="21" fill="#2463b4" font-weight="700">Spark runtime</text> +<text x="388" y="425" font-size="18" fill="#526277">Local in AMS / Kyuubi</text> +<rect x="696" y="360" width="236" height="112" rx="8" fill="#e5f5ef" stroke="#10705d" stroke-width="1.5"/> +<text x="714" y="393" font-size="21" fill="#10705d" font-weight="700">Paimon tables</text> +<text x="714" y="425" font-size="18" fill="#526277">Shared catalog + files</text> +<path d="M 290 416 H 366" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="304" y="399" font-size="16" fill="#526277">submit</text> +<path d="M 616 416 H 692" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="631" y="399" font-size="16" fill="#526277">access</text> +<text x="28" y="500" font-size="18" fill="#526277">Verify the Terminal backend, connector, and storage access.</text> +</g> +</svg> diff --git a/docs/static/img/ecosystem-catalog-access.svg b/docs/static/img/ecosystem-catalog-access.svg new file mode 100644 index 0000000000..6ca4e21e00 --- /dev/null +++ b/docs/static/img/ecosystem-catalog-access.svg @@ -0,0 +1,49 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="960" height="506" viewBox="0 0 960 506" role="img" aria-labelledby="title desc"> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<title id="title">Connect to metadata and storage</title> +<desc id="desc">An engine first resolves a table using a catalog backend. It then reads the table metadata and data files from shared storage. A filesystem catalog resolves metadata directly from the warehouse.</desc> +<defs> + <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#526277"/></marker> +</defs> +<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d"> +<rect x="1" y="1" width="958" height="504" rx="12" fill="#fff" stroke="#d7dfeb" stroke-width="1.5"/> +<text x="28" y="44" font-size="27" fill="#526277" font-weight="700">Connect to metadata and storage</text> +<text x="28" y="77" font-size="18" fill="#526277">A working catalog connection is only one part of a successful query.</text> +<rect x="28" y="162" width="268" height="164" rx="8" fill="#eaf2ff" stroke="#2463b4" stroke-width="1.5"/> +<text x="46" y="195" font-size="21" fill="#2463b4" font-weight="700">Engine + connector</text> +<text x="46" y="227" font-size="18" fill="#526277">Local SQL catalog name</text> +<text x="46" y="253" font-size="18" fill="#526277">Engine-specific properties</text> +<text x="46" y="279" font-size="18" fill="#526277">Query execution</text> +<rect x="474" y="118" width="458" height="132" rx="8" fill="#f1ebfa" stroke="#7051a3" stroke-width="1.5"/> +<text x="492" y="151" font-size="21" fill="#7051a3" font-weight="700">Catalog backend</text> +<text x="492" y="183" font-size="18" fill="#526277">Resolve a database and table</text> +<text x="492" y="209" font-size="18" fill="#526277">Hive metastore / REST / filesystem</text> +<rect x="474" y="316" width="458" height="132" rx="8" fill="#e5f5ef" stroke="#10705d" stroke-width="1.5"/> +<text x="492" y="349" font-size="21" fill="#10705d" font-weight="700">Warehouse storage</text> +<text x="492" y="381" font-size="18" fill="#526277">Paimon snapshots, manifests, and data files</text> +<text x="492" y="407" font-size="18" fill="#526277">HDFS or object storage</text> +<path d="M 300 203 H 372 V 184 H 470" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="312" y="159" font-size="16" fill="#526277">1. Resolve table</text> +<path d="M 300 281 H 372 V 382 H 470" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="311" y="411" font-size="16" fill="#526277">2. Read files</text> +<text x="491" y="278" font-size="15" fill="#526277">Filesystem catalogs resolve metadata in the warehouse.</text> +<text x="28" y="483" font-size="18" fill="#526277">Configure both access paths on the engine processes that use them.</text> +</g> +</svg> diff --git a/docs/static/img/ecosystem-overview.svg b/docs/static/img/ecosystem-overview.svg new file mode 100644 index 0000000000..859dec09ef --- /dev/null +++ b/docs/static/img/ecosystem-overview.svg @@ -0,0 +1,56 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="960" height="490" viewBox="0 0 960 490" role="img" aria-labelledby="title desc"> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<title id="title">One table, several integrations</title> +<desc id="desc">Flink and Spark read and write Paimon tables. StarRocks, Doris, Hive, Trino, and Presto query the same tables. Amoro connects to the catalog to inspect table metadata.</desc> +<defs> + <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#526277"/></marker> +</defs> +<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d"> +<rect x="1" y="1" width="958" height="488" rx="12" fill="#fff" stroke="#d7dfeb" stroke-width="1.5"/> +<text x="28" y="44" font-size="27" fill="#526277" font-weight="700">One table, several integrations</text> +<text x="28" y="77" font-size="18" fill="#526277">Choose an integration for ingestion, SQL queries, or table management.</text> +<rect x="28" y="148" width="236" height="150" rx="8" fill="#eaf2ff" stroke="#2463b4" stroke-width="1.5"/> +<text x="46" y="181" font-size="21" fill="#2463b4" font-weight="700">Write and transform</text> +<text x="46" y="213" font-size="18" fill="#526277">Flink</text> +<text x="46" y="239" font-size="18" fill="#526277">Spark</text> +<text x="46" y="265" font-size="18" fill="#526277">Batch and streaming</text> +<rect x="350" y="148" width="260" height="150" rx="8" fill="#e5f5ef" stroke="#10705d" stroke-width="1.5"/> +<text x="368" y="181" font-size="21" fill="#10705d" font-weight="700">Paimon tables</text> +<text x="368" y="213" font-size="18" fill="#526277">Schemas and snapshots</text> +<text x="368" y="239" font-size="18" fill="#526277">Data and changelog files</text> +<text x="368" y="265" font-size="18" fill="#526277">Shared storage</text> +<rect x="696" y="122" width="236" height="202" rx="8" fill="#eaf2ff" stroke="#2463b4" stroke-width="1.5"/> +<text x="714" y="155" font-size="21" fill="#2463b4" font-weight="700">Query</text> +<text x="714" y="187" font-size="18" fill="#526277">StarRocks · Doris</text> +<text x="714" y="213" font-size="18" fill="#526277">Hive · Trino · Presto</text> +<text x="714" y="239" font-size="18" fill="#526277">Flink · Spark</text> +<text x="714" y="265" font-size="18" fill="#526277">Engine-specific readers</text> +<path d="M 268 224 H 346" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)" marker-start="url(#arrow)"/> +<text x="274" y="207" font-size="14" fill="#526277">read / write</text> +<path d="M 614 224 H 692" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> +<text x="633" y="207" font-size="16" fill="#526277">read</text> +<path d="M 480 302 V 388" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)" stroke-dasharray="6 5"/> +<text x="496" y="351" font-size="16" fill="#526277">inspect metadata</text> +<rect x="28" y="392" width="904" height="76" rx="8" fill="#f1ebfa" stroke="#7051a3" stroke-width="1.5"/> +<text x="46" y="425" font-size="21" fill="#7051a3" font-weight="700">Amoro · Table management</text> +<text x="46" y="448" font-size="18" fill="#526277">Register the catalog; browse schemas, files, snapshots, and table options.</text> +<text x="28" y="357" font-size="16" fill="#526277">Operations and table features vary by connector version.</text> +</g> +</svg>
