This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 86192edc39 [docs] Reorganize ecosystem guides and add SVG diagrams
(#9728)
86192edc39 is described below
commit 86192edc395c3bb82fa9daeeca3d7d441648968a
Author: Jingsong Lee <[email protected]>
AuthorDate: Fri Sep 11 12:40:02 2026 +0800
[docs] Reorganize ecosystem guides and add SVG diagrams (#9728)
---
docs/docs/ecosystem/amoro.md | 48 +++-
docs/docs/ecosystem/connecting-engines.md | 91 +++++++
docs/docs/ecosystem/doris.md | 255 ++++++++----------
docs/docs/ecosystem/hive.mdx | 369 ++++++++++-----------------
docs/docs/ecosystem/index.md | 88 ++++---
docs/docs/ecosystem/starrocks.md | 200 +++++----------
docs/docs/ecosystem/trino.md | 328 ++++++++++--------------
docs/sidebars.js | 15 +-
docs/static/img/amoro-paimon.png | Bin 164164 -> 0 bytes
docs/static/img/amoro-paimon.svg | 64 +++++
docs/static/img/ecosystem-catalog-access.svg | 49 ++++
docs/static/img/ecosystem-overview.svg | 56 ++++
12 files changed, 808 insertions(+), 755 deletions(-)
diff --git a/docs/docs/ecosystem/amoro.md b/docs/docs/ecosystem/amoro.md
index e9c9d3d600..0580413821 100644
--- a/docs/docs/ecosystem/amoro.md
+++ b/docs/docs/ecosystem/amoro.md
@@ -22,21 +22,51 @@ specific language governing permissions and limitations
under the License.
-->
-# Apache Amoro With Paimon
+# Amoro
-**[Apache Amoro(incubating)](https://amoro.apache.org)** is a Lakehouse
management system built on open data lake formats. Working with compute engines
including Flink, Spark, and Trino, Amoro brings pluggable and
-**[Table Maintenance](https://amoro.apache.org/docs/latest/self-optimizing/)**
features for a Lakehouse to provide out-of-the-box data warehouse experience,
and helps data platforms or products easily build infra-decoupled,
stream-and-batch-fused and lake-native architecture.
-**[AMS](https://amoro.apache.org/docs/latest/#architecture)(Amoro Management
Service)** provides Lakehouse management features, like self-optimizing, data
expiration, etc. It also provides a unified catalog service for all compute
engines, which can also be combined with existing metadata services like
HMS(Hive Metastore).
+Use Apache Amoro to inspect Paimon tables through Amoro Management Service
(AMS). Register an
+existing Paimon catalog so operators can browse table metadata in one place.
+## Supported Workflows
-# Table Format
+The [Amoro 0.8.1 Paimon
guide](https://amoro.apache.org/docs/0.8.1/paimon-format/) describes table
+metadata inspection and Spark SQL access through the Terminal. Check the
Paimon guide for your
+deployed Amoro version before configuring the integration.
-Apache Amoro supports all catalog types supported by paimon, including common
catalog: Hadoop, Hive, Glue, JDBC, Nessie and other third-party catalog.
-Amoro supports all storage types supported by Paimon, including common store:
Hadoop, S3, GCS, ECS, OSS, and so on.
+
-
+| Task | Where it runs |
+| --- | --- |
+| Browse schemas, options, files, snapshots, DDL, and compaction information |
AMS table views |
+| Query or operate on tables with SQL | Amoro Terminal, using a configured
Spark runtime |
+| Ingest data | A writer such as Flink or Spark |
-In the future, Paimon automatic optimization strategy will be supported, and
users can achieve the best balance experience by cooperating with Amoro
automatic optimization
+## Connect a Paimon Catalog
+1. Deploy AMS using the [Amoro deployment
guide](https://amoro.apache.org/docs/0.8.1/deployment/).
+2. Prepare the Paimon catalog settings and make its metastore and storage
accessible to AMS.
+ See [Connecting Engines](./connecting-engines) for the information to
collect.
+3. Install the filesystem dependencies required by the Amoro distribution. For
S3 or OSS,
+ the Paimon guide describes adding the corresponding jars to Amoro's `lib`
directory;
+ match their versions to the Paimon dependency used by that distribution.
+4. Register the catalog using [Managing
Catalogs](https://amoro.apache.org/docs/0.8.1/managing-catalogs/).
+5. Open an existing table and check its schema, options, files, and snapshot
history.
+Use the catalog and filesystem implementations supported by the Paimon
integration in that
+Amoro release. Catalog support for another table format does not establish
Paimon support.
+## Query with the Terminal
+
+The Terminal can run Spark locally inside AMS or use a configured Kyuubi
service. Verify the
+execution backend's Paimon connector, catalog settings, and storage access,
especially when
+using an external backend. See the [Amoro Terminal
guide](https://amoro.apache.org/docs/0.8.1/using-kyuubi/)
+and [Paimon Spark Quick Start](../spark/quick-start).
+
+## Table Maintenance
+
+The ability to display compaction information does not establish support for
running Paimon
+self-optimization. Verify maintenance support in your deployed Amoro version
before assigning
+it table maintenance work.
+
+For Paimon-managed maintenance, see
[Compaction](../primary-key-table/compaction),
+[Snapshot Management](../maintenance/manage-snapshots), and [Flink
Actions](../flink/action-jars).
diff --git a/docs/docs/ecosystem/connecting-engines.md
b/docs/docs/ecosystem/connecting-engines.md
new file mode 100644
index 0000000000..2f01f6238e
--- /dev/null
+++ b/docs/docs/ecosystem/connecting-engines.md
@@ -0,0 +1,91 @@
+---
+title: "Connecting Engines"
+sidebar_position: 1
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Connecting Engines
+
+An engine needs to discover a Paimon table and read its files. Use the same
catalog backend
+and table location as the writer, then configure storage access in the engine
that runs the query.
+The SQL catalog name is local to that engine and can differ between engines.
+
+
+
+## Identify the Catalog and Storage
+
+Collect these settings from the job or engine that created the table:
+
+| Setting | What to check |
+| --- | --- |
+| Catalog backend | Filesystem, Hive metastore, REST, or another backend
supported by the reader. |
+| Catalog endpoint | The metastore or service URI, with the authentication
needed to connect. A filesystem catalog has no separate metastore service. |
+| Warehouse or table location | The shared storage URI. A filesystem catalog
uses the warehouse root; a Hive external table points to one table directory.
REST catalogs can use a warehouse identifier. |
+| Storage access | Filesystem libraries, endpoint configuration, and
credentials available to the engine processes that access the files. |
+| Table features | Primary keys, bucket mode, file format, types, and features
that the reader must understand. |
+
+See [Catalog](../concepts/catalog) for Paimon's catalog backends and
+[Filesystems](../maintenance/filesystems) for Paimon storage dependencies.
External engines
+can use their own filesystem implementations and configuration names.
+
+## Configure the Reader
+
+1. Install the connector or enable the engine's built-in Paimon integration.
+2. Configure a catalog using the engine's own property names.
+3. Configure access to the catalog service and the warehouse storage on the
relevant nodes.
+4. Query an existing table by its fully qualified name before adding optional
read settings.
+
+Catalog properties are not interchangeable. For example, a Hive-backed Paimon
catalog uses
+`metastore = hive` in Flink, while StarRocks and Doris use
`paimon.catalog.type = hive` and
+`paimon.catalog.type = hms`, respectively. Follow the matching guide:
+[Hive](./hive#installation), [Trino](./trino#configure-paimon-catalog),
+[StarRocks](./starrocks#create-paimon-catalog), or
[Doris](./doris#create-paimon-catalog).
+
+For a local experiment, a `file:` warehouse is sufficient when all processes
share that path.
+For a distributed cluster, use shared storage that every participating process
can access.
+
+## Check the Read Semantics
+
+Query a small, known table first. Compare values as well as row counts,
especially for
+primary-key tables with updates and deletes.
+
+| Read path | What the result represents |
+| --- | --- |
+| Ordinary batch query | A table snapshot, subject to the engine's metadata
caching and supported read mode. |
+| Time travel | A selected historical snapshot or tag; syntax and support
depend on the engine. The referenced state must still be retained. |
+| Read-optimized query | Compacted data from a primary-key table; recent
changes may be absent until full compaction completes. |
+| Streaming query | A snapshot and/or subsequent changes, according to the
engine and scan configuration. |
+
+See [Table Mode](../primary-key-table/table-mode),
+[System Tables](../concepts/system-tables), and
+[Snapshot Retention](../maintenance/manage-snapshots) for the underlying
behavior.
+
+## Troubleshooting
+
+| Symptom | Check first |
+| --- | --- |
+| Catalog exists but the table is missing | Confirm the backend, warehouse,
database, and table registration match the writer. |
+| Tables can be listed but a query cannot open files | Check storage
endpoints, credentials, and filesystem dependencies on the nodes performing the
read. |
+| Class-loading or method-not-found errors | Check the engine/connector
version pair and remove conflicting connector jars. |
+| New data or schema is not visible | Check whether the writer committed a
snapshot, then inspect engine metadata caches and any time-travel or
read-optimized settings. |
+| Updates or deletes produce unexpected rows | Confirm support for the table's
merge engine and deletion vectors; use a Paimon-aware reader rather than
scanning the underlying Parquet or ORC files directly. |
+
+Continue with the engine guide for exact SQL and version-specific limitations.
diff --git a/docs/docs/ecosystem/doris.md b/docs/docs/ecosystem/doris.md
index 878bba7257..8661daf337 100644
--- a/docs/docs/ecosystem/doris.md
+++ b/docs/docs/ecosystem/doris.md
@@ -24,189 +24,148 @@ under the License.
# Doris
-This documentation is a guide for using Paimon in Doris.
-
-> More details can be found in [Apache Doris
Website](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog)
+Query existing Paimon tables through a Doris external catalog. Use
+[Connecting Engines](./connecting-engines) to identify the catalog backend and
warehouse first.
## Version
-Paimon currently supports Apache Doris 2.0.6 and above.
+Select a Doris release that supports your catalog backend and Paimon table
features.
+The [Doris Paimon catalog
documentation](https://doris.apache.org/docs/3.x/lakehouse/catalogs/paimon-catalog/)
+provides version-specific settings. The REST catalog example below requires
Doris 3.1 or later.
+
+This integration reads Paimon tables. An external catalog exposes existing
tables without copying
+them into Doris; it does not enable writes to Paimon.
+
+## Prerequisites
+
+Prepare an existing Paimon table and configure metastore and storage access
for the Doris
+processes that perform metadata and data reads. The examples below use
placeholder hosts and
+credentials; replace them with your deployment's settings.
## Create Paimon Catalog
-Use `CREATE CATALOG` statement in Apache Doris to create Paimon Catalog.
+Choose one catalog definition for the backend used by the writer. HDFS
examples assume the
+cluster's Hadoop access is already configured.
-Doris support multi types of Paimon Catalogs. Here are some examples:
+### Filesystem Catalog on HDFS
```sql
--- HDFS based Paimon Catalog
-CREATE CATALOG `paimon_hdfs` PROPERTIES (
- "type" = "paimon",
- "warehouse" = "hdfs://172.21.0.1:8020/user/paimon",
- "hadoop.username" = "hadoop"
+CREATE CATALOG paimon_hdfs PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 'hdfs://namenode:8020/warehouse/paimon',
+ 'hadoop.username' = 'hadoop'
);
+```
+
+### Filesystem Catalog on OSS
--- Aliyun OSS based Paimon Catalog
-CREATE CATALOG `paimon_oss` PROPERTIES (
- "type" = "paimon",
- "warehouse" = "oss://paimon-bucket/paimonoss",
- "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
- "oss.access_key" = "ak",
- "oss.secret_key" = "sk"
+```sql
+CREATE CATALOG paimon_oss PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 'oss://paimon-bucket/warehouse',
+ 'oss.endpoint' = 'oss-cn-beijing.aliyuncs.com',
+ 'oss.access_key' = '<access-key-id>',
+ 'oss.secret_key' = '<access-key-secret>'
);
+```
--- Hive Metastore based Paimon Catalog
-CREATE CATALOG `paimon_hms` PROPERTIES (
- "type" = "paimon",
- "paimon.catalog.type" = "hms",
- "warehouse" = "hdfs://172.21.0.1:8020/user/zhangdong/paimon2",
- "hive.metastore.uris" = "thrift://172.21.0.44:7004",
- "hadoop.username" = "hadoop"
+### Hive Metastore Catalog
+
+```sql
+CREATE CATALOG paimon_hms PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'warehouse' = 'hdfs://namenode:8020/warehouse/paimon',
+ 'hive.metastore.uris' = 'thrift://metastore:9083',
+ 'hadoop.username' = 'hadoop'
);
+```
+
+### DLF 1.0 Catalog
--- Integrate with Aliyun DLF 1.0
+```sql
CREATE CATALOG paimon_dlf PROPERTIES (
'type' = 'paimon',
'paimon.catalog.type' = 'dlf',
- 'warehouse' = 'oss://paimon-bucket/paimonoss/',
+ 'warehouse' = 'oss://paimon-bucket/warehouse',
'dlf.proxy.mode' = 'DLF_ONLY',
- 'dlf.uid' = 'xxxxx',
+ 'dlf.uid' = '<account-id>',
'dlf.region' = 'cn-beijing',
- 'dlf.access_key' = 'ak',
- 'dlf.secret_key' = 'sk'
+ 'dlf.access_key' = '<access-key-id>',
+ 'dlf.secret_key' = '<access-key-secret>'
);
+```
+
+### DLF REST Catalog
+
+For Doris 3.1+, configure the REST backend explicitly. Here, `warehouse` is
the DLF catalog
+name, rather than an object-storage path.
--- Integrate with Aliyun DLF 3.0 Paimon Rest
--- Apache Doris supported since version 3.1.0
+```sql
CREATE CATALOG dlf_paimon_rest PROPERTIES (
'type' = 'paimon',
+ 'paimon.catalog.type' = 'rest',
'uri' = 'http://cn-beijing-vpc.dlf.aliyuncs.com',
- 'warehouse' = 'catalog_name',
+ 'warehouse' = '<catalog-name>',
'paimon.rest.token.provider' = 'dlf',
- 'paimon.rest.dlf.access-key-id' = 'ak',
- 'paimon.rest.dlf.access-key-secret' = 'sk'
+ 'paimon.rest.dlf.access-key-id' = '<access-key-id>',
+ 'paimon.rest.dlf.access-key-secret' = '<access-key-secret>'
);
```
-See [Apache Doris
Website](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog#examples)
for more examples.
+See the [Doris catalog
examples](https://doris.apache.org/docs/3.x/lakehouse/catalogs/paimon-catalog/#examples)
+for storage authentication and differences between Doris releases.
## Access Paimon Catalog
-1. Query Paimon table with full qualified name
+Query a fully qualified table name:
- ```sql
- SELECT * FROM paimon_hdfs.paimon_db.paimon_table;
- ```
+```sql
+SELECT * FROM paimon_hdfs.paimon_db.paimon_table LIMIT 10;
+```
-2. Switch to Paimon Catalog and query
+Alternatively, select the catalog and database for the session:
- ```sql
- SWITCH paimon_hdfs;
- USE paimon_db;
- SELECT * FROM paimon_table;
- ```
+```sql
+SWITCH paimon_hdfs;
+USE paimon_db;
+SELECT * FROM paimon_table LIMIT 10;
+```
## Query Optimization
-- Read optimized for Primary Key Table
-
- Doris can utilize the [Read
optimized](https://paimon.apache.org/docs/0.8/primary-key-table/read-optimized/)
feature for Primary Key Table(release in Paimon 0.6), by reading base data
files using native Parquet/ORC reader and delta file using JNI.
-
-- Deletion Vectors
-
- Doris(2.1.4+) natively supports [Deletion
Vectors](https://paimon.apache.org/docs/0.8/primary-key-table/deletion-vectors/)(released
in Paimon 0.8).
-
-## Doris to Paimon type mapping
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 10%">Doris Data Type</th>
- <th class="text-left" style="width: 10%">Paimon Data Type</th>
- <th class="text-left" style="width: 5%">Atomic Type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>Boolean</code></td>
- <td><code>BooleanType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>TinyInt</code></td>
- <td><code>TinyIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>SmallInt</code></td>
- <td><code>SmallIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Int</code></td>
- <td><code>IntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>BigInt</code></td>
- <td><code>BigIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Float</code></td>
- <td><code>FloatType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Double</code></td>
- <td><code>DoubleType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VarChar</code></td>
- <td><code>VarCharType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Char</code></td>
- <td><code>CharType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Binary</code></td>
- <td><code>VarBinaryType, BinaryType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Decimal(precision, scale)</code></td>
- <td><code>DecimalType(precision, scale)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Datetime</code></td>
- <td><code>TimestampType,LocalZonedTimestampType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Date</code></td>
- <td><code>DateType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>Array</code></td>
- <td><code>ArrayType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>Map</code></td>
- <td><code>MapType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>Struct</code></td>
- <td><code>RowType</code></td>
- <td>false</td>
- </tr>
- </tbody>
-</table>
-
+Primary-key tables can require merging multiple row versions. Review the
reader capabilities in
+your Doris release before selecting a [table
mode](../primary-key-table/table-mode).
+
+- **Read optimized:** reading compacted base data can reduce merge work, but a
+ [read-optimized view](../concepts/system-tables#read-optimized-table) can
omit recent changes.
+- **Deletion vectors:** readers must apply deletion vectors to suppress
obsolete rows. See
+ [Merge On Write](../primary-key-table/table-mode#merge-on-write) for the
Paimon configuration
+ and read semantics, and check Doris support before enabling it.
+
+## Type Mapping {#doris-to-paimon-type-mapping}
+
+This is a summary of common **Paimon-to-Doris read mappings**. Check the
+[Doris type
mapping](https://doris.apache.org/docs/3.x/lakehouse/catalogs/paimon-catalog/#column-type-mapping)
+for the installed version; binary and timestamp mappings can depend on release
and catalog options.
+
+| Paimon type | Doris type |
+| --- | --- |
+| `BOOLEAN` | `BOOLEAN` |
+| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | Corresponding integer type |
+| `FLOAT`, `DOUBLE` | `FLOAT`, `DOUBLE` |
+| `DECIMAL(p, s)` | `DECIMAL(p, s)` |
+| `CHAR`, `VARCHAR` | `STRING` |
+| `BINARY`, `VARBINARY` | See the version-specific binary mapping |
+| `DATE` | `DATE` |
+| `TIMESTAMP`, `TIMESTAMP WITH LOCAL TIME ZONE` | See the version-specific
timestamp mapping and precision limits |
+| `ARRAY` | `ARRAY` |
+| `MAP` | `MAP` |
+| `ROW` | `STRUCT` |
+
+## Next Steps
+
+Use the Doris guide for time-travel and system-table syntax supported by your
release. For
+connection or visibility problems, follow [Connecting
Engines](./connecting-engines#troubleshooting).
diff --git a/docs/docs/ecosystem/hive.mdx b/docs/docs/ecosystem/hive.mdx
index da316cc778..aaed893462 100644
--- a/docs/docs/ecosystem/hive.mdx
+++ b/docs/docs/ecosystem/hive.mdx
@@ -27,295 +27,206 @@ under the License.
# Hive
-This documentation is a guide for using Paimon in Hive.
+Use the Paimon storage handler to read tables from Hive, create tables, and
insert records.
+A Hive-backed Paimon catalog and the Hive query engine are separate
integrations:
+[using Hive metastore](../concepts/catalog#hive-catalog) does not require
running queries in Hive.
## Version
-Paimon currently supports Hive 3.1, 2.3, 2.2, 2.1 and 2.1-cdh-6.3.
+This branch contains connectors for Hive 3.1, 2.3, 2.2, 2.1, and 2.1-cdh-6.3.
+Choose the jar matching your Hive distribution and Paimon release.
## Execution Engine
-Paimon currently supports MR and Tez execution engine for Hive Read, and MR
execution engine for Hive Write.
-Note If you use beeline, please restart the hive cluster.
+| Operation | Hive execution engine | Limitations |
+| --- | --- | --- |
+| Read | MapReduce (MR) or Tez | Configure catalog and filesystem access
before querying. |
+| Write | MapReduce (MR) | `INSERT INTO` only; `INSERT OVERWRITE` is not
supported. |
+
+Prefer append tables for Hive inserts. Writing primary-key tables from Hive
can produce many
+small files. For continuous ingestion or more extensive write operations, use
+[Flink](../flink/sql-write) or [Spark](../spark/sql-write).
## Installation
-Download the jar file with corresponding version.
+### Download the Connector
<Stable>
-
-
-| | Jar
|
-|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Hive 3.1 |
[paimon-hive-connector-3.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/paimon-hive-connector-3.1-@@VERSION@@.jar)
|
-| Hive 2.3 |
[paimon-hive-connector-2.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/paimon-hive-connector-2.3-@@VERSION@@.jar)
|
-| Hive 2.2 |
[paimon-hive-connector-2.2-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/paimon-hive-connector-2.2-@@VERSION@@.jar)
|
-| Hive 2.1 |
[paimon-hive-connector-2.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/paimon-hive-connector-2.1-@@VERSION@@.jar)
|
-| Hive 2.1-cdh-6.3 |
[paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar)
|
-
-
+| Hive version | Connector |
+| --- | --- |
+| 3.1 |
[paimon-hive-connector-3.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/paimon-hive-connector-3.1-@@VERSION@@.jar)
|
+| 2.3 |
[paimon-hive-connector-2.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/paimon-hive-connector-2.3-@@VERSION@@.jar)
|
+| 2.2 |
[paimon-hive-connector-2.2-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/paimon-hive-connector-2.2-@@VERSION@@.jar)
|
+| 2.1 |
[paimon-hive-connector-2.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/paimon-hive-connector-2.1-@@VERSION@@.jar)
|
+| 2.1-cdh-6.3 |
[paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar)
|
</Stable>
<Unstable>
+| Hive version | Connector |
+| --- | --- |
+| 3.1 |
[paimon-hive-connector-3.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/)
|
+| 2.3 |
[paimon-hive-connector-2.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/)
|
+| 2.2 |
[paimon-hive-connector-2.2-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/)
|
+| 2.1 |
[paimon-hive-connector-2.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/)
|
+| 2.1-cdh-6.3 |
[paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/)
|
+</Unstable>
-| | Jar
|
-|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Hive 3.1 |
[paimon-hive-connector-3.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/)
|
-| Hive 2.3 |
[paimon-hive-connector-2.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/)
|
-| Hive 2.2 |
[paimon-hive-connector-2.2-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/)
|
-| Hive 2.1 |
[paimon-hive-connector-2.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/)
|
-| Hive 2.1-cdh-6.3 |
[paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/)
|
-
-
+Snapshot links point to artifact directories. Select a published timestamped
jar for your version.
-</Unstable>
+### Install the Jar
-You can also manually build bundled jar from the source code.
+Copy the matching connector jar into the Hive installation's `auxlib`
directory so the storage
+handler is available to Hive and its execution jobs. Restart existing Hive
services after changing
+jars. When using Beeline, restart the HiveServer2 service used by the session.
-To build from source code, [clone the git repository](@@GITHUB_REPO@@).
+For a temporary Hive CLI session, you can use `ADD JAR`:
-Build bundled jar with the following command.
-`mvn clean install -DskipTests`
+```sql
+ADD JAR /path/to/paimon-hive-connector-3.1-@@VERSION@@.jar;
+```
-You can find Hive connector jar in
`./paimon-hive/paimon-hive-connector-<hive-version>/target/paimon-hive-connector-<hive-version>-@@VERSION@@.jar`.
+`ADD JAR` can cause class-loading failures for joins with the MR engine,
including
+`KryoException: unable to find class`. Use `auxlib` for a persistent
installation.
-There are several ways to add this jar to Hive.
+### Build from Source
-* You can create an `auxlib` folder under the root directory of Hive, and copy
`paimon-hive-connector-@@VERSION@@.jar` into `auxlib`.
-* You can also copy this jar to a path accessible by Hive, then use `add jar
/path/to/paimon-hive-connector-@@VERSION@@.jar` to enable paimon support in
Hive. Note that this method is not recommended. If you're using the MR
execution engine and running a join statement, you may be faced with the
exception `org.apache.hive.com.esotericsoftware.kryo.kryoexception: unable to
find class`.
+From a checkout of the [Paimon repository](@@GITHUB_REPO@@), build only the
required connector
+and its dependencies. For Hive 3.1:
-NOTE:
+```bash
+mvn -pl paimon-hive/paimon-hive-connector-3.1 -am -DskipTests package
+```
-* If you are using HDFS :
- * Make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR`
is set.
- * You can set `paimon.hadoop-load-default-config` =`false` to disable
loading the default value from `core-default.xml`、`hdfs-default.xml`, which may
lead smaller size for split.
-* With hive cbo, it may lead to some incorrect query results, such as to query
`struct` type with `not null` predicate, you can disable the cbo by `set
hive.cbo.enable=false;` command.
+The bundled jar is under
+`paimon-hive/paimon-hive-connector-3.1/target/paimon-hive-connector-3.1-@@VERSION@@.jar`.
+Replace `3.1` with the connector module matching your Hive distribution.
-## Hive SQL: access Paimon Tables already in Hive metastore
+## Access Existing Tables
{#hive-sql-access-paimon-tables-already-in-hive-metastore}
-Run the following Hive SQL in Hive CLI to access the created table.
+If the table is already registered in the Hive metastore used by your Hive
session, query it
+directly. For example, assume `default.test_table` contains columns `a` and
`b`:
```sql
--- Assume that paimon-hive-connector-<hive-version>-@@VERSION@@.jar is already
in auxlib directory.
--- List tables in Hive
--- (you might need to switch to "default" database if you're not there by
default)
-
+USE default;
SHOW TABLES;
-
-/*
-OK
-test_table
-*/
-
--- Read records from test_table
-
SELECT a, b FROM test_table ORDER BY a;
+```
-/*
-OK
-1 Table
-2 Store
-*/
+If the table is not registered there, use an external table as described below.
--- Insert records into test table
--- Limitations:
--- Only support INSERT INTO, not support INSERT OVERWRITE
--- It is recommended to write to a non primary key table
--- Writing to a primary key table may result in a large number of small files
+## Register an External Table
{#hive-sql-access-paimon-tables-by-external-table}
-INSERT INTO test_table VALUES (3, 'Paimon');
+Point the external table at the existing **table directory**, rather than the
warehouse root.
+The storage handler loads the schema from Paimon; do not repeat column
definitions.
-SELECT a, b FROM test_table ORDER BY a;
+```sql
+CREATE EXTERNAL TABLE external_test_table
+STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
+LOCATION 'hdfs://namenode:8020/warehouse/paimon/default.db/test_table';
-/*
-OK
-1 Table
-2 Store
-3 Paimon
-*/
+SELECT a, b FROM external_test_table ORDER BY a;
+```
--- time travel
-
-SET paimon.scan.snapshot-id=1;
-SELECT a, b FROM test_table ORDER BY a;
-/*
-OK
-1 Table
-2 Store
-3 Paimon
-*/
-SET paimon.scan.snapshot-id=null;
+Alternatively, use `paimon_location` in `TBLPROPERTIES`. This avoids Hive
accessing the Paimon
+location through its own filesystem during table creation, which is useful for
object storage.
+Paimon's filesystem still needs the required libraries and credentials.
+```sql
+CREATE EXTERNAL TABLE external_s3_table
+STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
+TBLPROPERTIES (
+ 'paimon_location' = 's3://paimon-bucket/warehouse/default.db/test_table'
+);
```
-## Hive SQL: create new Paimon Tables
+## Create a Table {#hive-sql-create-new-paimon-tables}
-You can create new paimon tables in Hive. Run the following Hive SQL in Hive
CLI.
+Set the warehouse to your shared storage location, then create a table with
the Paimon storage handler:
```sql
--- Assume that paimon-hive-connector-@@VERSION@@.jar is already in auxlib
directory.
--- Let's create a new paimon table.
+SET hive.metastore.warehouse.dir=hdfs://namenode:8020/warehouse/paimon;
-SET hive.metastore.warehouse.dir=warehouse_path;
-
-CREATE TABLE hive_test_table(
+CREATE TABLE hive_test_table (
a INT COMMENT 'The a field',
b STRING COMMENT 'The b field'
)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';
```
-## Hive SQL: access Paimon Tables by External Table
+## Insert Records
-To access existing paimon table, you can also register them as external tables
in Hive. Run the following Hive SQL in Hive CLI.
+Use the MR execution engine. This example continues from the table created
above:
```sql
--- Assume that paimon-hive-connector-@@VERSION@@.jar is already in auxlib
directory.
--- Let's use the test_table created in the above section.
--- To create an external table, you don't need to specify any column or table
properties.
--- Pointing the location to the path of table is enough.
-
-CREATE EXTERNAL TABLE external_test_table
-STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
-LOCATION '/path/to/table/store/warehouse/default.db/test_table';
-
--- In addition to the way setting location above, you can also place the
location setting in TBProperties
--- to avoid Hive accessing Paimon's location through its own file system when
creating tables.
--- This method is effective in scenarios using Object storage,such as s3.
-
-CREATE EXTERNAL TABLE external_test_table
-STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'
-TBLPROPERTIES (
- 'paimon_location'
='s3://xxxxx/path/to/table/store/warehouse/default.db/test_table'
-);
-
--- Read records from external_test_table
-
-SELECT a, b FROM external_test_table ORDER BY a;
-
-/*
-OK
-1 Table
-2 Store
-*/
+SET hive.execution.engine=mr;
+INSERT INTO hive_test_table VALUES (1, 'Paimon'), (2, 'Hive');
+SELECT a, b FROM hive_test_table ORDER BY a;
+```
--- Insert records into test table
+The same `INSERT INTO` syntax can write to a registered external Paimon table,
subject to the
+[write limitations](#execution-engine).
-INSERT INTO external_test_table VALUES (3, 'Paimon');
+## Time Travel
-SELECT a, b FROM external_test_table ORDER BY a;
+Set `paimon.scan.snapshot-id` to an existing, retained snapshot. The setting
applies to the
+session's subsequent Paimon scans, so clear it before resuming ordinary
queries.
-/*
-OK
-1 Table
-2 Store
-3 Paimon
-*/
+```sql
+-- Use 1 only if snapshot 1 still exists for this table.
+SET paimon.scan.snapshot-id=1;
+SELECT a, b FROM hive_test_table ORDER BY a;
+-- Resume reading the latest snapshot.
+SET paimon.scan.snapshot-id=null;
```
-## Hive Type Conversion
-
-This section lists all supported type conversion between Hive and Paimon.
-All Hive's data types are available in package
`org.apache.hadoop.hive.serde2.typeinfo`.
-
-<table className="table table-bordered">
- <thead>
- <tr>
- <th className="text-left" style={{width: "10%"}}>Hive Data Type</th>
- <th className="text-left" style={{width: "10%"}}>Paimon Data Type</th>
- <th className="text-left" style={{width: "5%"}}>Atomic Type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>StructTypeInfo</code></td>
- <td><code>RowType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>MapTypeInfo</code></td>
- <td><code>MapType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>ListTypeInfo</code></td>
- <td><code>ArrayType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("boolean")</code></td>
- <td><code>BooleanType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("tinyint")</code></td>
- <td><code>TinyIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("smallint")</code></td>
- <td><code>SmallIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("int")</code></td>
- <td><code>IntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("bigint")</code></td>
- <td><code>BigIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("float")</code></td>
- <td><code>FloatType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("double")</code></td>
- <td><code>DoubleType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>CharTypeInfo(length)</code></td>
- <td><code>CharType(length)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("string")</code></td>
- <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VarcharTypeInfo(length)</code></td>
- <td><code>VarCharType(length), length is less than
VarCharType.MAX_LENGTH</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("date")</code></td>
- <td><code>DateType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("timestamp")</code></td>
- <td><code>TimestampType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DecimalTypeInfo(precision, scale)</code></td>
- <td><code>DecimalType(precision, scale)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>PrimitiveTypeInfo("binary")</code></td>
- <td><code>VarBinaryType</code>, <code>BinaryType</code></td>
- <td>true</td>
- </tr>
- </tbody>
-</table>
+The rows returned depend on what was committed in that snapshot. See
+[Snapshot Management](../maintenance/manage-snapshots) for retention behavior.
+
+## Type Mapping {#hive-type-conversion}
+
+When creating a Paimon table in Hive, the connector converts Hive SQL types as
follows.
+Nested types are converted recursively.
+
+| Hive type | Paimon type |
+| --- | --- |
+| `BOOLEAN` | `BOOLEAN` |
+| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | Corresponding integer type |
+| `FLOAT`, `DOUBLE` | `FLOAT`, `DOUBLE` |
+| `DECIMAL(p, s)` | `DECIMAL(p, s)` |
+| `CHAR(n)`, `VARCHAR(n)` | `CHAR(n)`, `VARCHAR(n)` |
+| `STRING` | `STRING` (`VARCHAR` with maximum length) |
+| `BINARY` | `VARBINARY` with maximum length |
+| `DATE` | `DATE` |
+| `TIMESTAMP` | `TIMESTAMP` |
+| `TIMESTAMP WITH LOCAL TIME ZONE` (Hive 3) | `TIMESTAMP WITH LOCAL TIME ZONE`
|
+| `ARRAY`, `MAP`, `STRUCT` | `ARRAY`, `MAP`, `ROW` |
+
+The reverse mapping, when reading an existing Paimon table, is not always
symmetric:
+
+| Paimon type | Hive representation |
+| --- | --- |
+| `BINARY`, `VARBINARY` | `BINARY` |
+| `CHAR(n)` or `VARCHAR(n)` beyond Hive's length limit | `STRING` |
+| `TIME` | `STRING` |
+| `TIMESTAMP WITH LOCAL TIME ZONE` | Hive 3 local-time-zone timestamp;
ordinary `TIMESTAMP` in Hive 2 |
+| `MULTISET<T>` | `MAP<T, INT>` |
+
+See [Paimon Data Types](../concepts/data-types) for the Paimon type system.
+
+## Troubleshooting
+
+- **HDFS configuration:** set `HADOOP_HOME` or `HADOOP_CONF_DIR` so the
connector can load the
+ Hadoop configuration. `SET paimon.hadoop-load-default-config=false;` skips
loading
+ `core-default.xml` and `hdfs-default.xml`, which can reduce serialized split
size; explicitly
+ provide the settings your filesystem requires.
+- **Incorrect results with Hive CBO:** if predicates on complex types produce
incorrect results
+ (for example, `IS NOT NULL` on a struct), retry with `SET
hive.cbo.enable=false;`.
+- **Class-loading failures:** verify the matching connector is installed in
`auxlib` and restart
+ Hive services. See [installation](#installation) for the limitation of `ADD
JAR`.
+
+For shared catalog and storage checks, see [Connecting
Engines](./connecting-engines#troubleshooting).
diff --git a/docs/docs/ecosystem/index.md b/docs/docs/ecosystem/index.md
index be5017b1f6..4428f80cb5 100644
--- a/docs/docs/ecosystem/index.md
+++ b/docs/docs/ecosystem/index.md
@@ -22,58 +22,70 @@ specific language governing permissions and limitations
under the License.
-->
-# Overview
+# Ecosystem
-## Compatibility Matrix
+Use Paimon tables across ingestion pipelines, SQL engines, and lakehouse
management tools.
+Start with an integration below, then follow [Connecting
Engines](./connecting-engines) to
+configure catalog discovery and storage access.
-| Engine
| Version | Batch Read | Batch Write | Create Table | Alter Table
| Streaming Write | Streaming Read | Batch Overwrite | DELETE & UPDATE |
MERGE INTO | Time Travel |
-|:-------------------------------------------------------------------------------:|:-------------:|:-----------:|:-----------:|:-------------:|:-------------:|:----------------:|:----------------:|:---------------:|:---------------:|:----------:|:-----------:|
-| Flink
| 1.16 - 1.20 | ✅ | ✅ | ✅ | ✅(1.17+) |
✅ | ✅ | ✅ | ✅(1.17+) | ❌
| ✅ |
-| Spark
| 3.2 - 4.1 | ✅ | ✅ | ✅ | ✅ |
✅(3.3+) | ✅(3.3+) | ✅ | ✅ | ✅
| ✅(3.3+) |
-| Hive
| 2.1 - 3.1 | ✅ | ✅ | ✅ | ❌ |
❌ | ❌ | ❌ | ❌ | ❌
| ✅ |
-| Trino
| 420 - 440 | ✅ | ✅(427+) | ✅(427+) | ✅(427+) |
❌ | ❌ | ❌ | ❌ | ❌
| ✅ |
-| Presto
| 0.236 - 0.280 | ✅ | ❌ | ✅ | ✅ |
❌ | ❌ | ❌ | ❌ | ❌
| ❌ |
-|
[StarRocks](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/)
| 3.1+ | ✅ | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ❌ | ❌ |
✅ |
-| [Doris](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog)
| 2.0.6+ | ✅ | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ❌ | ❌
| ✅ |
+
-## Streaming Engines
+## Choose an Integration
-### Flink Streaming
+| What you want to do | Start here |
+| --- | --- |
+| Ingest CDC events or build a streaming pipeline | [Flink Quick
Start](../flink/quick-start), [CDC Ingestion](../cdc-ingestion/) |
+| Run batch transformations or Spark SQL | [Spark Quick
Start](../spark/quick-start) |
+| Process streams with Spark micro-batches | [Spark Structured
Streaming](../spark/structured-streaming) |
+| Query Paimon from an OLAP engine | [StarRocks](./starrocks),
[Doris](./doris) |
+| Query or write tables with distributed SQL | [Trino](./trino) |
+| Access tables from Hive | [Hive](./hive) |
+| Inspect tables in a lakehouse management service | [Amoro](./amoro) |
-Flink is the most comprehensive streaming computing engine that is widely used
for data CDC ingestion and the
-construction of streaming pipelines.
+## Compatibility Matrix
-Recommended version is Flink 1.17.2.
+For bundled connectors, match the engine version to the connector artifact for
your Paimon
+release. The Flink, Spark, and Hive versions below describe connector modules
in this branch;
+artifact availability depends on the release. Externally maintained
integrations have their own
+release cycle, embedded Paimon version, and feature limits.
+
+| Integration | Version selection | Access to Paimon tables |
+| --- | --- | --- |
+| [Flink](../flink/installation) | 1.16–1.20 and 2.0–2.2 | Batch and streaming
reads/writes; [DDL](../flink/sql-ddl) and [row changes](../flink/sql-write)
have version-specific requirements. |
+| [Spark](../spark/quick-start) | 3.2–3.5, 4.0, and 4.1; match the Scala
binary version | Batch reads/writes, [DDL](../spark/sql-ddl), and [row
changes](../spark/sql-write); [streaming](../spark/structured-streaming)
requires Spark 3.3+. |
+| [Hive](./hive#version) | 2.1, 2.2, 2.3, 3.1, and 2.1-cdh-6.3 | Batch reads,
table creation, and `INSERT INTO`; writes require MapReduce. |
+| [Trino](./trino#version) | Match the independently released Paimon connector
to Trino | Batch reads; supported connectors also provide DDL, inserts, and
time travel. See the guide's table-layout limits. |
+| [Presto](https://github.com/apache/paimon-presto) | Follow the separate
connector's version requirements | See the connector repository for
installation and supported operations. |
+| [StarRocks](./starrocks#version) | Paimon catalogs available from 3.1 |
Query existing tables through an external catalog; check the engine release for
individual features. |
+| [Doris](./doris#version) | Select a release with the required catalog and
reader features | Query existing tables through an external catalog; REST
catalog access requires Doris 3.1+. |
+
+A connector's ability to read a table also depends on its data types, file
format, merge engine,
+and enabled features, such as deletion vectors. Check the relevant engine
guide before enabling
+a new table feature in a warehouse shared by several engines.
-### Spark Streaming
+## Streaming Engines
-You can also use Spark Streaming to build a streaming pipeline. Spark's schema
evolution capability will be better
-implemented, but you must accept the mechanism of mini-batch.
+Use [Flink](../flink/) for continuous ingestion, change processing, and lookup
joins.
+Use [Spark Structured Streaming](../spark/structured-streaming) for
micro-batch pipelines.
+Configure the table's [changelog
producer](../primary-key-table/changelog-producer) for the
+changes that downstream readers need.
## Batch Engines
-### Spark Batch
-
-Spark Batch is the most widely used batch computing engine.
-
-Recommended version is Spark 3.5.8.
-
-### Flink Batch
-
-Flink Batch is also available, which can make your pipeline more integrated
with streaming and batch unified.
+Use [Spark SQL](../spark/sql-query) or [Flink batch SQL](../flink/sql-query)
to read a snapshot
+and run transformations. Consult the write guides for
[Spark](../spark/sql-write) and
+[Flink](../flink/sql-write) before using overwrite, `DELETE`, `UPDATE`, or
`MERGE INTO`:
+SQL support and table requirements differ by engine and version.
## OLAP Engines
-### StarRocks
-
-StarRocks is the most recommended OLAP engine with the most advanced
integration.
-
-Recommended version is StarRocks 3.2.6.
-
-### Other OLAP
-
-You can also use Doris and Trino and Presto, or, you can just use Spark, Flink
and Hive to query Paimon tables.
+[StarRocks](./starrocks) and [Doris](./doris) query Paimon through their own
external catalogs.
+[Trino](./trino) and [Presto](https://github.com/apache/paimon-presto) use
separately distributed
+connectors. Configure access to both the catalog and the underlying files, and
choose the
+[table read mode](../primary-key-table/table-mode) for your freshness
requirements.
## Download
-[Download Link](../project/download#engine-jars)
+Use the [engine downloads](../project/download#engine-jars) for Paimon
artifacts and each
+integration guide for installation. For Trino and Presto, use the connector
project's release
+instructions; its version does not necessarily match this documentation's
Paimon version.
diff --git a/docs/docs/ecosystem/starrocks.md b/docs/docs/ecosystem/starrocks.md
index a155beaeea..3c3a243a49 100644
--- a/docs/docs/ecosystem/starrocks.md
+++ b/docs/docs/ecosystem/starrocks.md
@@ -24,159 +24,99 @@ under the License.
# StarRocks
-This documentation is a guide for using Paimon in StarRocks.
+Query existing Paimon tables through a StarRocks external catalog. Start with
+[Connecting Engines](./connecting-engines) if you need to identify the catalog
and warehouse.
## Version
-Paimon currently supports StarRocks 3.1 and above. Recommended version is
StarRocks 3.2.6 or above.
+Paimon catalogs are available in StarRocks 3.1 and later. Individual features
depend on the
+StarRocks release. Use the [StarRocks Paimon catalog
documentation](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/)
+for the supported catalog backends, storage configuration, and
release-specific limitations.
+
+This integration reads Paimon tables. Creating an external catalog does not
create or copy the
+tables, and the catalog does not support inserting, updating, or deleting
Paimon data.
+
+## Prerequisites
+
+Prepare an existing Paimon database and table. Ensure the StarRocks processes
that access metadata
+and data can reach the metastore and warehouse. Configure authentication and
filesystem access
+using the StarRocks documentation for your storage system.
## Create Paimon Catalog
-Paimon catalogs are registered by executing a `CREATE EXTERNAL CATALOG` SQL in
StarRocks.
-For example, you can use the following SQL to create a Paimon catalog named
paimon_catalog.
+Choose the backend used by the writer. These examples assume HDFS access is
already configured;
+replace the host names and warehouse path with your deployment values.
+
+### Filesystem Catalog
```sql
-CREATE EXTERNAL CATALOG paimon_catalog PROPERTIES(
+CREATE EXTERNAL CATALOG paimon_catalog PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "filesystem",
- "paimon.catalog.warehouse" = "oss://<your_bucket>/user/warehouse/"
+ "paimon.catalog.warehouse" = "hdfs://namenode:8020/warehouse/paimon"
);
```
-More catalog types and configures can be seen in [Paimon
catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/).
+### Hive Metastore Catalog
-## Query
-Suppose there already exists a database named `test_db` and a table named
`test_tbl` in `paimon_catalog`,
-you can query this table using the following SQL:
```sql
-SELECT * FROM paimon_catalog.test_db.test_tbl;
+CREATE EXTERNAL CATALOG paimon_hms PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hive",
+ "paimon.catalog.warehouse" = "hdfs://namenode:8020/warehouse/paimon",
+ "hive.metastore.uris" = "thrift://metastore:9083"
+);
```
-## Query System Tables
+For object storage, use the warehouse URI and authentication properties from
the
+[StarRocks catalog
examples](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/#examples).
+
+## Query
-You can access all kinds of Paimon system tables by StarRocks. For example,
you can read the `ro`
-(read-optimized) system table to improve reading performance of primary-key
tables.
+Assume `test_db.test_tbl` already exists in the configured warehouse:
```sql
-SELECT * FROM paimon_catalog.test_db.test_tbl$ro;
+SELECT * FROM paimon_catalog.test_db.test_tbl LIMIT 10;
```
-For another example, you can query partition files of the table using the
following SQL:
+## Query System Tables
+
+On StarRocks versions supporting Paimon system tables, append the system-table
suffix to the
+table name. Quote the complete table identifier containing `$`:
```sql
-SELECT * FROM paimon_catalog.test_db.partition_tbl$partitions;
-/*
-+-----------+--------------+--------------------+------------+----------------------------+
-| partition | record_count | file_size_in_bytes | file_count |
last_update_time |
-+-----------+--------------+--------------------+------------+----------------------------+
-| [1] | 1 | 645 | 1 | 2024-01-01
00:00:00.000000 |
-+-----------+--------------+--------------------+------------+----------------------------+
-*/
+SELECT * FROM paimon_catalog.test_db.`test_tbl$partitions`;
+SELECT * FROM paimon_catalog.test_db.`test_tbl$ro`;
```
-## StarRocks to Paimon type mapping
-
-This section lists all supported type conversion between StarRocks and Paimon.
-All StarRocks's data types can be found in this doc [StarRocks Data type
overview](https://docs.starrocks.io/docs/sql-reference/data-types/).
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 10%">StarRocks Data Type</th>
- <th class="text-left" style="width: 10%">Paimon Data Type</th>
- <th class="text-left" style="width: 5%">Atomic Type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>STRUCT</code></td>
- <td><code>RowType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>MAP</code></td>
- <td><code>MapType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>ARRAY</code></td>
- <td><code>ArrayType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>BOOLEAN</code></td>
- <td><code>BooleanType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>TINYINT</code></td>
- <td><code>TinyIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>SMALLINT</code></td>
- <td><code>SmallIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>INT</code></td>
- <td><code>IntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>BIGINT</code></td>
- <td><code>BigIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>FLOAT</code></td>
- <td><code>FloatType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DOUBLE</code></td>
- <td><code>DoubleType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>CHAR(length)</code></td>
- <td><code>CharType(length)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VARCHAR(MAX_VARCHAR_LENGTH)</code></td>
- <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VARCHAR(length)</code></td>
- <td><code>VarCharType(length), length is less than
VarCharType.MAX_LENGTH</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DATE</code></td>
- <td><code>DateType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DATETIME</code></td>
- <td><code>TimestampType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DECIMAL(precision, scale)</code></td>
- <td><code>DecimalType(precision, scale)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VARBINARY(length)</code></td>
- <td><code>VarBinaryType(length)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DATETIME</code></td>
- <td><code>LocalZonedTimestampType</code></td>
- <td>true</td>
- </tr>
- </tbody>
-</table>
+The [`$ro` table](../concepts/system-tables#read-optimized-table) reads
compacted data from a
+primary-key table. Its results can lag behind the latest snapshot until a full
compaction
+completes. Use the ordinary table query when you need the latest state
supported by the reader.
+See [MOR Read Optimized](../primary-key-table/table-mode#mor-read-optimized)
for maintenance settings.
+
+## Type Mapping {#starrocks-to-paimon-type-mapping}
+
+The following table summarizes common Paimon types as exposed by StarRocks. It
describes the
+**read direction**; it does not imply support for creating these types in
Paimon from StarRocks.
+Refer to the [upstream type
mapping](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/#paimon-to-starrocks-data-types)
+for your release, including precision and type limits.
+
+| Paimon type | StarRocks type |
+| --- | --- |
+| `BOOLEAN` | `BOOLEAN` |
+| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | Corresponding integer type |
+| `FLOAT`, `DOUBLE` | `FLOAT`, `DOUBLE` |
+| `DECIMAL(p, s)` | `DECIMAL(p, s)` |
+| `CHAR(n)` | `CHAR(n)` |
+| `VARCHAR(n)`, `STRING` | `VARCHAR` |
+| `BINARY(n)`, `VARBINARY(n)` | `VARBINARY` |
+| `DATE` | `DATE` |
+| `TIMESTAMP`, `TIMESTAMP WITH LOCAL TIME ZONE` | `DATETIME` |
+| `ARRAY` | `ARRAY` |
+| `MAP` | `MAP` |
+| `ROW` | `STRUCT` |
+
+## Next Steps
+
+- Review [query performance](../primary-key-table/query-performance) and
[table modes](../primary-key-table/table-mode).
+- Diagnose catalog and storage issues with [Connecting
Engines](./connecting-engines#troubleshooting).
diff --git a/docs/docs/ecosystem/trino.md b/docs/docs/ecosystem/trino.md
index 00eb9b72f9..961ba443b9 100644
--- a/docs/docs/ecosystem/trino.md
+++ b/docs/docs/ecosystem/trino.md
@@ -24,106 +24,140 @@ under the License.
# Trino
-This documentation is a guide for using Paimon in Trino.
+Use the separately distributed Paimon connector to query tables from Trino.
The examples on this
+page describe the **Trino 440 connector**. For other versions, follow the
matching revision of
+[Apache Paimon Trino](https://github.com/apache/paimon-trino).
## Version
-Paimon currently supports Trino 440.
+Match the Trino server version, connector artifact, and Java runtime. The
connector repository
+has its own release cycle and Paimon dependency; its default branch can target
a different Trino
+and Java version from the examples here.
-## Filesystem
+The write examples below require a connector with write support and a
supported table layout.
+They do not imply that every historical Trino connector supports these
operations.
-From version 0.8, Paimon share Trino filesystem for all actions, which means,
you should
-config Trino filesystem before using trino-paimon. You can find information
about how to config
-filesystems for Trino on Trino official website.
+## Installation {#preparing-paimon-jar-file}
-## Preparing Paimon Jar File
+### Obtain the Plugin
-[Download](../project/download)
+Use the [download page](../project/download#engine-jars) to locate the Trino
440 plugin archive.
+For other versions, use the [connector
repository](https://github.com/apache/paimon-trino).
+The plugin is a distribution containing its dependencies, rather than a single
jar to place on
+the general Trino classpath.
-You can also manually build a bundled jar from the source code. However, there
are a few preliminary steps that need to be taken before compiling:
-
-- To build from the source code, [clone the git
repository](@@TRINO_GITHUB_REPO@@).
-- Install JDK21 locally, and configure JDK21 as a global environment variable;
-
-Then,you can build bundled jar with the following command:
+To build from source, check out a connector revision matching the Trino server
and use the Java
+version required by that revision's `pom.xml`. Run its documented build; for
revisions using the
+Maven build, the command is:
```bash
mvn clean install -DskipTests
```
-You can find Trino connector jar in
`./paimon-trino-<trino-version>/target/paimon-trino-<trino-version>-@@VERSION@@-plugin.tar.gz`.
+Locate the plugin archive in that revision's build output. Artifact names and
module paths
+vary across revisions. Use the connector artifact's actual version rather than
substituting
+the Paimon version shown by this documentation.
+
+### Install Paimon Connector
-We use
[hadoop-apache](https://mvnrepository.com/artifact/io.trino.hadoop/hadoop-apache)
as a dependency for Hadoop,
-and the default Hadoop dependency typically supports both Hadoop 2 and Hadoop
3.
-If you encounter an unsupported scenario, you can specify the corresponding
Apache Hadoop version.
+For a Trino 440 distribution named
+`paimon-trino-440-<connector-version>-plugin.tar.gz`, extract it under the
Trino plugin directory
+on the coordinator and every worker:
-For example, if you want to use Hadoop 3.3.5-1, you can use the following
command to build the jar:
```bash
-mvn clean install -DskipTests -Dhadoop.apache.version=3.3.5-1
+tar -zxf paimon-trino-440-<connector-version>-plugin.tar.gz -C
"${TRINO_HOME}/plugin"
```
-## Configure Paimon Catalog
+For the Trino 440 connector running on JDK 21, include these entries in
`etc/jvm.config`:
-### Install Paimon Connector
-```bash
-tar -zxf paimon-trino-<trino-version>-@@VERSION@@-plugin.tar.gz -C
${TRINO_HOME}/plugin
+```text
+--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+--add-opens=java.base/java.nio=ALL-UNNAMED
```
-> NOTE: For JDK 21, when Deploying Trino, should add jvm options:
`--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED`
+Restart Trino after installing the plugin and completing the catalog
configuration.
-### Configure
+## Configure Paimon Catalog
-Catalogs are registered by creating a catalog properties file in the
etc/catalog directory. For example, create etc/catalog/paimon.properties with
the following contents to mount the paimon connector as the paimon catalog:
+Create `etc/catalog/paimon.properties` on the Trino nodes. The file name
registers the SQL
+catalog as `paimon`:
```properties
connector.name=paimon
-warehouse=file:/tmp/warehouse
+warehouse=hdfs://namenode:8020/warehouse/paimon
```
-If you are using HDFS, choose one of the following ways to configure your HDFS:
+This example uses a filesystem catalog and assumes HDFS access is configured.
For a local
+single-node experiment, `warehouse=file:/tmp/warehouse` is sufficient. Use
shared storage for
+a distributed deployment.
-- set environment variable HADOOP_HOME.
-- set environment variable HADOOP_CONF_DIR.
-- configure `hadoop-conf-dir` in the properties.
+### Filesystem
-If you are using a Hadoop filesystem, you can still use trino-hdfs and
trino-hive to config it.
-For example, if you use oss as a storage, you can write in `paimon.properties`
according to [Trino
Reference](https://trino.io/docs/current/connector/hive.html#hdfs-configuration):
+The Trino 440 connector uses Trino's filesystem integration. Configure storage
access before
+querying a table. For HDFS, provide `HADOOP_HOME`, `HADOOP_CONF_DIR`, or the
connector's
+`hadoop-conf-dir` property as appropriate for the deployment.
+
+For Hadoop-backed object storage, supply the filesystem configuration through:
```properties
hive.config.resources=/path/to/core-site.xml
```
-Then, config core-site.xml according to [Jindo
Reference](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto)
+Make the configuration file and required filesystem libraries available on all
nodes that access
+the warehouse. Follow the filesystem instructions for your **Trino and
connector versions**;
+properties from newer Trino releases may differ.
+
+Connector revisions using `io.trino.hadoop:hadoop-apache` can override that
dependency when a
+specific Hadoop distribution is required. For example, if supported by the
selected revision:
-## Kerberos
+```bash
+mvn clean install -DskipTests -Dhadoop.apache.version=3.3.5-1
+```
-You can configure kerberos keytab file when using KERBEROS authentication in
the properties.
+### Kerberos
+
+For the Trino 440 connector's Kerberos login, set these catalog properties:
```properties
-security.kerberos.login.principal=hadoop-user
[email protected]
security.kerberos.login.keytab=/etc/trino/hdfs.keytab
```
-Keytab files must be distributed to every node in the cluster that runs Trino.
+Distribute the keytab to every Trino node that needs it and configure access
for the service user.
+
+### Temporary Directory {#tmp-dir}
+
+Paimon extracts jars for code generation into the JVM temporary directory.
Choose a writable
+location that is not removed by periodic cleanup while Trino is running. Set
this JVM option
+in `etc/jvm.config` on each node:
+
+```text
+-Djava.io.tmpdir=/path/to/trino-tmp
+```
## Create Schema
+The following examples form one sequence in the `paimon` catalog:
+
```sql
CREATE SCHEMA paimon.test_db;
```
## Create Table
+Create a primary-key table with a fixed bucket count. Include the partition
column in the
+primary key:
+
```sql
CREATE TABLE paimon.test_db.orders (
order_key bigint,
- orders_tatus varchar,
+ order_status varchar,
total_price decimal(18,4),
order_date date
)
WITH (
file_format = 'ORC',
- primary_key = ARRAY['order_key','order_date'],
+ primary_key = ARRAY['order_key', 'order_date'],
partitioned_by = ARRAY['order_date'],
bucket = '2',
bucket_key = 'order_key',
@@ -131,25 +165,15 @@ WITH (
);
```
-## Add Column
+## Insert
-```sql
-CREATE TABLE paimon.test_db.orders (
- order_key bigint,
- orders_tatus varchar,
- total_price decimal(18,4),
- order_date date
-)
-WITH (
- file_format = 'ORC',
- primary_key = ARRAY['order_key','order_date'],
- partitioned_by = ARRAY['order_date'],
- bucket = '2',
- bucket_key = 'order_key',
- changelog_producer = 'input'
-);
+The Trino 440 connector supports inserts into primary-key tables with fixed
buckets and
+non-primary-key tables with `bucket = -1`.
-ALTER TABLE paimon.test_db.orders ADD COLUMN shipping_address varchar;
+```sql
+INSERT INTO paimon.test_db.orders
+ (order_key, order_status, total_price, order_date)
+VALUES (1, 'NEW', DECIMAL '19.9900', DATE '2024-01-01');
```
## Query
@@ -158,153 +182,63 @@ ALTER TABLE paimon.test_db.orders ADD COLUMN
shipping_address varchar;
SELECT * FROM paimon.test_db.orders;
```
-## Query with Time Traveling
-
-```sql
--- read the snapshot from specified timestamp
-SELECT * FROM t FOR TIMESTAMP AS OF TIMESTAMP '2023-01-01 00:00:00
Asia/Shanghai';
-
--- read the snapshot with id 1L (use snapshot id as version)
-SELECT * FROM t FOR VERSION AS OF 1;
+## Add Column
--- read tag 'my-tag'
-SELECT * FROM t FOR VERSION AS OF 'my-tag';
+Continue with the existing table; there is no need to create it again:
+```sql
+ALTER TABLE paimon.test_db.orders ADD COLUMN shipping_address varchar;
```
-:::warning
+## Query with Time Traveling
-If tag's name is a number and equals to a snapshot id, the VERSION AS OF
syntax will consider tag first. For example, if
-you have a tag named '1' based on snapshot 2, the statement `SELECT * FROM
paimon.test_db.orders FOR VERSION AS OF '1'` actually queries snapshot 2
-instead of snapshot 1.
+Use a retained snapshot or tag. The timestamp and identifiers below are
examples; replace them
+with values from the table's history.
-:::
+```sql
+-- Select the snapshot at the specified timestamp.
+SELECT * FROM paimon.test_db.orders
+FOR TIMESTAMP AS OF TIMESTAMP '2024-01-01 00:00:00 Asia/Shanghai';
-## Insert
+-- Select snapshot 1 if it is still retained.
+SELECT * FROM paimon.test_db.orders FOR VERSION AS OF 1;
-```sql
-INSERT INTO paimon.test_db.orders VALUES (.....);
+-- Select an existing tag.
+SELECT * FROM paimon.test_db.orders FOR VERSION AS OF 'my-tag';
```
-Supports:
-- primary key table with fixed bucket.
-- non-primary-key table with bucket -1.
-
-## Trino to Paimon type mapping
-
-This section lists all supported type conversion between Trino and Paimon.
-All Trino's data types are available in package `io.trino.spi.type`.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 10%">Trino Data Type</th>
- <th class="text-left" style="width: 10%">Paimon Data Type</th>
- <th class="text-left" style="width: 5%">Atomic Type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>RowType</code></td>
- <td><code>RowType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>MapType</code></td>
- <td><code>MapType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>ArrayType</code></td>
- <td><code>ArrayType</code></td>
- <td>false</td>
- </tr>
- <tr>
- <td><code>BooleanType</code></td>
- <td><code>BooleanType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>TinyintType</code></td>
- <td><code>TinyIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>SmallintType</code></td>
- <td><code>SmallIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>IntegerType</code></td>
- <td><code>IntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>BigintType</code></td>
- <td><code>BigIntType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>RealType</code></td>
- <td><code>FloatType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DoubleType</code></td>
- <td><code>DoubleType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>CharType(length)</code></td>
- <td><code>CharType(length)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td>
- <td><code>VarCharType(VarCharType.MAX_LENGTH)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VarCharType(length)</code></td>
- <td><code>VarCharType(length), length is less than
VarCharType.MAX_LENGTH</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DateType</code></td>
- <td><code>DateType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>TimestampType</code></td>
- <td><code>TimestampType</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>DecimalType(precision, scale)</code></td>
- <td><code>DecimalType(precision, scale)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>VarBinaryType(length)</code></td>
- <td><code>VarBinaryType(length)</code></td>
- <td>true</td>
- </tr>
- <tr>
- <td><code>TimestampWithTimeZoneType</code></td>
- <td><code>LocalZonedTimestampType</code></td>
- <td>true</td>
- </tr>
- </tbody>
-</table>
-
-## Tmp Dir
-
-Paimon will unzip some jars to the tmp directory for codegen. By default,
Trino will use `'/tmp'` as the temporary
-directory, but `'/tmp'` may be periodically deleted.
-
-You can configure this environment variable when Trino starts:
-```shell
--Djava.io.tmpdir=/path/to/other/tmpdir
-```
+:::warning Numeric tag names
+
+A numeric tag name takes precedence over a matching snapshot ID when supplied
as a string.
+If tag `'1'` points to snapshot 2, `FOR VERSION AS OF '1'` reads snapshot 2.
Prefer descriptive
+tag names to avoid ambiguity.
+
+:::
-Let Paimon use a secure temporary directory.
+See [Snapshot Management](../maintenance/manage-snapshots) and
[Tags](../maintenance/manage-tags)
+for retention and tag creation.
+
+## Type Mapping {#trino-to-paimon-type-mapping}
+
+Common read mappings for the connector are summarized below using SQL type
names. Check the
+matching connector revision for precision limits and unsupported types.
+
+| Paimon type | Trino type |
+| --- | --- |
+| `BOOLEAN` | `BOOLEAN` |
+| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | `TINYINT`, `SMALLINT`, `INTEGER`,
`BIGINT` |
+| `FLOAT`, `DOUBLE` | `REAL`, `DOUBLE` |
+| `DECIMAL(p, s)` | `DECIMAL(p, s)` |
+| `CHAR(n)` | `CHAR(n)` |
+| `VARCHAR(n)`, `STRING` | `VARCHAR(n)`, `VARCHAR` |
+| `VARBINARY` | `VARBINARY` |
+| `DATE` | `DATE` |
+| `TIMESTAMP` | `TIMESTAMP` |
+| `TIMESTAMP WITH LOCAL TIME ZONE` | `TIMESTAMP WITH TIME ZONE` |
+| `ARRAY`, `MAP`, `ROW` | `ARRAY`, `MAP`, `ROW` |
+
+## Next Steps
+
+Use [Connecting Engines](./connecting-engines#troubleshooting) for catalog and
storage checks.
+Report connector-specific problems to [Apache Paimon
Trino](https://github.com/apache/paimon-trino/issues),
+including the Trino, Java, and connector versions and the table options.
diff --git a/docs/sidebars.js b/docs/sidebars.js
index a96ae758ae..ceac74a000 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -486,10 +486,17 @@ const sidebars = {
"id": "ecosystem/index"
},
"items": [
- "ecosystem/starrocks",
- "ecosystem/doris",
- "ecosystem/hive",
- "ecosystem/trino",
+ "ecosystem/connecting-engines",
+ {
+ type: "category",
+ "label": "Query Engines",
+ "items": [
+ "ecosystem/starrocks",
+ "ecosystem/doris",
+ "ecosystem/hive",
+ "ecosystem/trino"
+ ]
+ },
"ecosystem/amoro"
]
},
diff --git a/docs/static/img/amoro-paimon.png b/docs/static/img/amoro-paimon.png
deleted file mode 100644
index 02b8555699..0000000000
Binary files a/docs/static/img/amoro-paimon.png and /dev/null differ
diff --git a/docs/static/img/amoro-paimon.svg b/docs/static/img/amoro-paimon.svg
new file mode 100644
index 0000000000..aef7f15bd7
--- /dev/null
+++ b/docs/static/img/amoro-paimon.svg
@@ -0,0 +1,64 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="960" height="518" viewBox="0 0
960 518" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Inspect Paimon tables with Amoro</title>
+<desc id="desc">Register a Paimon catalog with Amoro Management Service to
inspect schemas, options, files, snapshots, DDL, and compaction information.
Terminal SQL runs in Spark, locally inside AMS or through Kyuubi, with access
to the same catalog and storage.</desc>
+<defs>
+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker>
+</defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="958" height="516" rx="12" fill="#fff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="44" font-size="27" fill="#526277" font-weight="700">Inspect
Paimon tables with Amoro</text>
+<text x="28" y="77" font-size="18" fill="#526277">Catalog registration makes
existing table metadata available in AMS.</text>
+<rect x="28" y="124" width="258" height="166" rx="8" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="46" y="157" font-size="21" fill="#10705d" font-weight="700">Paimon
catalog</text>
+<text x="46" y="189" font-size="18" fill="#526277">Catalog configuration</text>
+<text x="46" y="215" font-size="18" fill="#526277">Metastore / service
access</text>
+<text x="46" y="241" font-size="18" fill="#526277">Warehouse storage</text>
+<rect x="370" y="124" width="242" height="166" rx="8" fill="#f1ebfa"
stroke="#7051a3" stroke-width="1.5"/>
+<text x="388" y="157" font-size="21" fill="#7051a3"
font-weight="700">AMS</text>
+<text x="388" y="189" font-size="18" fill="#526277">Registered catalog</text>
+<text x="388" y="215" font-size="18" fill="#526277">Paimon integration</text>
+<text x="388" y="241" font-size="18" fill="#526277">Storage dependencies</text>
+<rect x="696" y="124" width="236" height="166" rx="8" fill="#f1ebfa"
stroke="#7051a3" stroke-width="1.5"/>
+<text x="714" y="157" font-size="21" fill="#7051a3" font-weight="700">Table
views</text>
+<text x="714" y="189" font-size="18" fill="#526277">Schema · options ·
DDL</text>
+<text x="714" y="215" font-size="18" fill="#526277">Files · snapshots</text>
+<text x="714" y="241" font-size="18" fill="#526277">Compaction
information</text>
+<path d="M 290 206 H 366" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="306" y="190" font-size="16" fill="#526277">register</text>
+<path d="M 616 206 H 692" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="631" y="190" font-size="16" fill="#526277">browse</text>
+<path d="M 28 327 H 932" stroke="#d7dfeb" stroke-width="1.5"/>
+<rect x="28" y="360" width="258" height="112" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="46" y="393" font-size="21" fill="#2463b4" font-weight="700">Amoro
Terminal</text>
+<text x="46" y="425" font-size="18" fill="#526277">Submit Spark SQL</text>
+<rect x="370" y="360" width="242" height="112" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="388" y="393" font-size="21" fill="#2463b4" font-weight="700">Spark
runtime</text>
+<text x="388" y="425" font-size="18" fill="#526277">Local in AMS /
Kyuubi</text>
+<rect x="696" y="360" width="236" height="112" rx="8" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="714" y="393" font-size="21" fill="#10705d" font-weight="700">Paimon
tables</text>
+<text x="714" y="425" font-size="18" fill="#526277">Shared catalog +
files</text>
+<path d="M 290 416 H 366" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="304" y="399" font-size="16" fill="#526277">submit</text>
+<path d="M 616 416 H 692" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="631" y="399" font-size="16" fill="#526277">access</text>
+<text x="28" y="500" font-size="18" fill="#526277">Verify the Terminal
backend, connector, and storage access.</text>
+</g>
+</svg>
diff --git a/docs/static/img/ecosystem-catalog-access.svg
b/docs/static/img/ecosystem-catalog-access.svg
new file mode 100644
index 0000000000..6ca4e21e00
--- /dev/null
+++ b/docs/static/img/ecosystem-catalog-access.svg
@@ -0,0 +1,49 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="960" height="506" viewBox="0 0
960 506" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Connect to metadata and storage</title>
+<desc id="desc">An engine first resolves a table using a catalog backend. It
then reads the table metadata and data files from shared storage. A filesystem
catalog resolves metadata directly from the warehouse.</desc>
+<defs>
+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker>
+</defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="958" height="504" rx="12" fill="#fff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="44" font-size="27" fill="#526277" font-weight="700">Connect to
metadata and storage</text>
+<text x="28" y="77" font-size="18" fill="#526277">A working catalog connection
is only one part of a successful query.</text>
+<rect x="28" y="162" width="268" height="164" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="46" y="195" font-size="21" fill="#2463b4" font-weight="700">Engine +
connector</text>
+<text x="46" y="227" font-size="18" fill="#526277">Local SQL catalog
name</text>
+<text x="46" y="253" font-size="18" fill="#526277">Engine-specific
properties</text>
+<text x="46" y="279" font-size="18" fill="#526277">Query execution</text>
+<rect x="474" y="118" width="458" height="132" rx="8" fill="#f1ebfa"
stroke="#7051a3" stroke-width="1.5"/>
+<text x="492" y="151" font-size="21" fill="#7051a3" font-weight="700">Catalog
backend</text>
+<text x="492" y="183" font-size="18" fill="#526277">Resolve a database and
table</text>
+<text x="492" y="209" font-size="18" fill="#526277">Hive metastore / REST /
filesystem</text>
+<rect x="474" y="316" width="458" height="132" rx="8" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="492" y="349" font-size="21" fill="#10705d"
font-weight="700">Warehouse storage</text>
+<text x="492" y="381" font-size="18" fill="#526277">Paimon snapshots,
manifests, and data files</text>
+<text x="492" y="407" font-size="18" fill="#526277">HDFS or object
storage</text>
+<path d="M 300 203 H 372 V 184 H 470" fill="none" stroke="#526277"
stroke-width="2" marker-end="url(#arrow)"/>
+<text x="312" y="159" font-size="16" fill="#526277">1. Resolve table</text>
+<path d="M 300 281 H 372 V 382 H 470" fill="none" stroke="#526277"
stroke-width="2" marker-end="url(#arrow)"/>
+<text x="311" y="411" font-size="16" fill="#526277">2. Read files</text>
+<text x="491" y="278" font-size="15" fill="#526277">Filesystem catalogs
resolve metadata in the warehouse.</text>
+<text x="28" y="483" font-size="18" fill="#526277">Configure both access paths
on the engine processes that use them.</text>
+</g>
+</svg>
diff --git a/docs/static/img/ecosystem-overview.svg
b/docs/static/img/ecosystem-overview.svg
new file mode 100644
index 0000000000..859dec09ef
--- /dev/null
+++ b/docs/static/img/ecosystem-overview.svg
@@ -0,0 +1,56 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="960" height="490" viewBox="0 0
960 490" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">One table, several integrations</title>
+<desc id="desc">Flink and Spark read and write Paimon tables. StarRocks,
Doris, Hive, Trino, and Presto query the same tables. Amoro connects to the
catalog to inspect table metadata.</desc>
+<defs>
+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker>
+</defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="958" height="488" rx="12" fill="#fff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="44" font-size="27" fill="#526277" font-weight="700">One table,
several integrations</text>
+<text x="28" y="77" font-size="18" fill="#526277">Choose an integration for
ingestion, SQL queries, or table management.</text>
+<rect x="28" y="148" width="236" height="150" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="46" y="181" font-size="21" fill="#2463b4" font-weight="700">Write and
transform</text>
+<text x="46" y="213" font-size="18" fill="#526277">Flink</text>
+<text x="46" y="239" font-size="18" fill="#526277">Spark</text>
+<text x="46" y="265" font-size="18" fill="#526277">Batch and streaming</text>
+<rect x="350" y="148" width="260" height="150" rx="8" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="368" y="181" font-size="21" fill="#10705d" font-weight="700">Paimon
tables</text>
+<text x="368" y="213" font-size="18" fill="#526277">Schemas and
snapshots</text>
+<text x="368" y="239" font-size="18" fill="#526277">Data and changelog
files</text>
+<text x="368" y="265" font-size="18" fill="#526277">Shared storage</text>
+<rect x="696" y="122" width="236" height="202" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="714" y="155" font-size="21" fill="#2463b4"
font-weight="700">Query</text>
+<text x="714" y="187" font-size="18" fill="#526277">StarRocks · Doris</text>
+<text x="714" y="213" font-size="18" fill="#526277">Hive · Trino ·
Presto</text>
+<text x="714" y="239" font-size="18" fill="#526277">Flink · Spark</text>
+<text x="714" y="265" font-size="18" fill="#526277">Engine-specific
readers</text>
+<path d="M 268 224 H 346" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)" marker-start="url(#arrow)"/>
+<text x="274" y="207" font-size="14" fill="#526277">read / write</text>
+<path d="M 614 224 H 692" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<text x="633" y="207" font-size="16" fill="#526277">read</text>
+<path d="M 480 302 V 388" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)" stroke-dasharray="6 5"/>
+<text x="496" y="351" font-size="16" fill="#526277">inspect metadata</text>
+<rect x="28" y="392" width="904" height="76" rx="8" fill="#f1ebfa"
stroke="#7051a3" stroke-width="1.5"/>
+<text x="46" y="425" font-size="21" fill="#7051a3" font-weight="700">Amoro ·
Table management</text>
+<text x="46" y="448" font-size="18" fill="#526277">Register the catalog;
browse schemas, files, snapshots, and table options.</text>
+<text x="28" y="357" font-size="16" fill="#526277">Operations and table
features vary by connector version.</text>
+</g>
+</svg>