This is an automated email from the ASF dual-hosted git repository. XiaoHongbo-Hope pushed a commit to branch release-2.1 in repository https://gitbox.apache.org/repos/asf/paimon.git
commit bf0a7a62161117da5dbe1e3a4efc43b1f468f0eb Author: Jingsong Lee <[email protected]> AuthorDate: Fri Sep 11 13:39:52 2026 +0800 [docs] Reorganize migration guides and add SVG diagrams (#9733) --- docs/docs/migration/clone-to-paimon.md | 174 ++++++++++++++----- docs/docs/migration/index.md | 39 +++++ docs/docs/migration/migration-from-hive.mdx | 228 ++++++++++++++++++------- docs/docs/migration/upsert-to-partitioned.mdx | 222 ++++++++++++------------ docs/sidebars.js | 4 +- docs/static/img/migration-clone-overwrite.svg | 52 ++++++ docs/static/img/migration-strategies.svg | 60 +++++++ docs/static/img/migration-tag-to-partition.svg | 60 +++++++ 8 files changed, 626 insertions(+), 213 deletions(-) diff --git a/docs/docs/migration/clone-to-paimon.md b/docs/docs/migration/clone-to-paimon.md index b8383875f3..a75075a938 100644 --- a/docs/docs/migration/clone-to-paimon.md +++ b/docs/docs/migration/clone-to-paimon.md @@ -1,6 +1,6 @@ --- -title: "Clone To Paimon" -sidebar_position: 5 +title: "Clone to Paimon" +sidebar_position: 2 --- <!-- @@ -22,65 +22,149 @@ specific language governing permissions and limitations under the License. --> -# Clone To Paimon +# Clone to Paimon -Clone supports cloning tables to Paimon tables. +Clone copies data into separate Paimon tables while keeping source tables and files available. +Use it to validate a Paimon copy before switching workloads, or to refresh selected target +partitions. Unlike [migration](./migration-from-hive), clone does not move source files. -1. Clone is `OVERWRITE` semantic that will overwrite the partitions of the target table according to the data. -2. Clone is reentrant, but it requires existing tables to contain all fields from the source table and have the - same partition fields. +The Flink `clone` action supports Hive and Paimon sources. This guide starts with Hive tables +in ORC, Parquet, or Avro format, which become Paimon [append tables](../append-table/). -Currently, clone supports clone Hive tables in Hive Catalog to Paimon Catalog, supports Parquet, ORC, Avro formats, -target table will be append table. +## Before You Start + +- Install the matching [Flink action jar](../flink/action-jars) and set `FLINK_HOME`. +- Configure source and target catalogs separately. Hive sources require a Hive catalog; the + examples use a filesystem catalog with a separate warehouse for the target. +- Ensure the job can read the source and write the target. Keep source data stable during the + copy when you need a consistent validation baseline. +- Use distinct source and target locations. For an existing Hive-clone target, use an append + table with `bucket = -1`, compatible file format, all source fields with compatible types, + and the same partition fields. + +## Understand Overwrite Behavior + +With the default target option `dynamic-partition-overwrite = true`, clone replaces target +partitions represented in the copied data. Other target partitions remain unchanged. For a +non-partitioned target, the copied data replaces the table contents. + + + +Re-running clone refreshes the copied partitions; it does not append a second copy. It also +does not synchronize source deletions for partitions absent from the copied data. If an +existing target sets `dynamic-partition-overwrite = false`, overwrite can replace the entire +target table even when the source is filtered. Check this option before cloning a subset. ## Clone Hive Table +The following example copies `default.hivetable` to `analytics.hivetable_copy`: + ```bash -<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \ -clone \ ---database default \ ---table hivetable \ ---catalog_conf metastore=hive \ ---catalog_conf uri=thrift://localhost:9088 \ ---target_database test \ ---target_table test_table \ ---target_catalog_conf warehouse=my_warehouse \ ---parallelism 10 \ ---where <filter_spec> +"$FLINK_HOME/bin/flink" run /path/to/paimon-flink-action-@@VERSION@@.jar \ + clone \ + --clone_from hive \ + --database default \ + --table hivetable \ + --catalog_conf metastore=hive \ + --catalog_conf uri=thrift://localhost:9083 \ + --target_database analytics \ + --target_table hivetable_copy \ + --target_catalog_conf warehouse=hdfs:///paimon-warehouse \ + --parallelism 10 ``` -You can use filter spec to specify the filtering condition for the partition. +To copy only selected partitions, add a quoted partition predicate. For a source partitioned +by the string column `dt`, append this argument to the command: + +```bash +--where "dt = '2026-09-10'" +``` + +`--where` selects partitions, not arbitrary rows within a partition. Omit it to copy all +partitions. Table include/exclude lists are not accepted when cloning a single table. ## Clone Hive Database +Omit `--table` and supply `--target_database`. Source table names are preserved in the target +database, which is created if needed. + ```bash -<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \ -clone \ ---database default \ ---catalog_conf metastore=hive \ ---catalog_conf uri=thrift://localhost:9088 \ ---target_database test \ ---parallelism 10 \ ---target_catalog_conf warehouse=my_warehouse ---included_tables <included_tables_spec> \ ---excluded_tables <excluded_tables_spec> +"$FLINK_HOME/bin/flink" run /path/to/paimon-flink-action-@@VERSION@@.jar \ + clone \ + --clone_from hive \ + --database default \ + --catalog_conf metastore=hive \ + --catalog_conf uri=thrift://localhost:9083 \ + --target_database analytics \ + --target_catalog_conf warehouse=hdfs:///paimon-warehouse \ + --parallelism 10 \ + --included_tables default.orders,default.customers ``` -"--included_tables" and "--excluded_tables" are optional parameters, which are used to specify the tables that need or don't need to be cloned. -The format is `<database1>.<table1>,<database2>.<table2>,<database3>.<table3>`. -"--excluded_tables" has higher priority than "--included_tables" if you specified both. + +Omit `--included_tables` to consider all tables in the source database. Use +`--excluded_tables` to remove tables from that selection. ## Clone Hive Catalog +Omit source and target database/table arguments to preserve database and table names across +the catalog. The following example limits the selection to two source tables: + ```bash -<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \ -clone \ ---catalog_conf metastore=hive \ ---catalog_conf uri=thrift://localhost:9088 \ ---parallelism 10 \ ---target_catalog_conf warehouse=my_warehouse \ ---included_tables <included_tables_spec> \ ---excluded_tables <excluded_tables_spec> +"$FLINK_HOME/bin/flink" run /path/to/paimon-flink-action-@@VERSION@@.jar \ + clone \ + --clone_from hive \ + --catalog_conf metastore=hive \ + --catalog_conf uri=thrift://localhost:9083 \ + --target_catalog_conf warehouse=hdfs:///paimon-warehouse \ + --parallelism 10 \ + --included_tables sales.orders,crm.customers ``` -"--included_tables" and "--excluded_tables" are optional parameters, which are used to specify the tables that need or don't need to be cloned. -The format is `<database1>.<table1>,<database2>.<table2>,<database3>.<table3>`. -"--excluded_tables" has higher priority than "--included_tables" if you specified both. + +## Selection and Retry Options + +| Option | Behavior | +| --- | --- | +| `--included_tables db.table,db.other` | Selects fully qualified source table names for database or catalog cloning. Omit to consider all tables in scope. | +| `--excluded_tables db.table,db.other` | Removes tables from the selection. Exclusion wins when a table is also included. | +| `--where "dt = '2026-09-10'"` | Restricts copied data using a partition predicate. | +| `--clone_if_exists false` | Skips targets that already exist. The default is `true`, which clones into existing compatible targets. | +| `--meta_only true` | Clones the schema without copying data. The default is `false`. | +| `--parallelism 10` | Sets the clone job parallelism. | + +A multi-table clone does not commit all tables atomically. After a failure, inspect target +tables and rerun the required scope. `--clone_if_exists false` is useful for skipping existing +tables, but is not a resume mechanism: an existing table may have been created before its +data was copied. + +## Clone a Paimon Source + +Use `--clone_from paimon` and configure the source Paimon catalog. For example, to copy between +two filesystem catalogs: + +```bash +"$FLINK_HOME/bin/flink" run /path/to/paimon-flink-action-@@VERSION@@.jar \ + clone \ + --clone_from paimon \ + --catalog_conf warehouse=hdfs:///source-paimon-warehouse \ + --database sales \ + --table orders \ + --target_catalog_conf warehouse=hdfs:///target-paimon-warehouse \ + --target_database analytics \ + --target_table orders_copy \ + --parallelism 10 +``` + +Paimon sources can retain primary keys in the newly created target. The Hive append-table +restriction above applies to Hive sources. For Paimon sources, repeated +`--target_table_conf key=value` arguments can override options when creating the target; +these overrides are not supported for Hive sources. + +## Verify the Clone + +1. Confirm that the Flink job completed and that every selected target exists. +2. Connect an engine to the target catalog and inspect the schema and partition keys. +3. Compare row counts and aggregates with the stable source data, using the same partition + predicate when cloning a subset. Confirm that unrelated target partitions remain as expected. +4. Verify that source readers still work, then switch workloads when the target is ready. + +For SQL invocation instead of the action jar, see the [Flink `clone` procedure](../flink/procedures/table-operations#clone). diff --git a/docs/docs/migration/index.md b/docs/docs/migration/index.md index bc01176fbc..4068865371 100644 --- a/docs/docs/migration/index.md +++ b/docs/docs/migration/index.md @@ -21,3 +21,42 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Migration + +Choose a migration path based on whether you need to replace a Hive table, keep a separate +copy, or expose historical Paimon data to existing Hive queries. + + + +## Choose a Guide + +| Goal | Guide | Effect on the source | Result | +| --- | --- | --- | --- | +| Convert a Hive table or database to Paimon | [Migrate from Hive](./migration-from-hive) | Moves data files; removes the original Hive table by default | Paimon append tables in a Hive catalog | +| Copy tables while keeping the source available | [Clone to Paimon](./clone-to-paimon) | Keeps source tables and data | Separate Paimon tables; Hive sources become append tables | +| Query daily views of an updating Paimon table using Hive partition filters | [Expose Tags as Hive Partitions](./upsert-to-partitioned) | Keeps the Paimon table and its write path | Hive partition values that select tags or preview snapshots | + +Migration and clone import existing data. Tag-to-partition changes how Hive reads a Paimon +table; it does not migrate Hive files or physically repartition the Paimon table. + +## Before Moving Data + +1. **Choose the target table model.** Hive migration and Hive clone create + [append tables](../append-table/). If the target needs primary-key updates or a different + schema or partition layout, plan a data rewrite into a table with that model. +2. **Prepare the runtime and catalogs.** Configure access to the Hive metastore and storage. + Use the engine setup and connector requirements linked from each guide. +3. **Define the validation scope.** Record source schemas, partitions, row counts, and key + aggregates before the operation. For a consistent comparison, keep source data stable + while it is being moved or copied. +4. **Plan the switch.** In-place migration is not atomic and requires a backup. Clone lets + you validate a separate target before switching readers and writers. + +## Related Guides + +- [Flink Procedures](../flink/procedures/table-operations#migrate_table) and + [Spark Migration Procedures](../spark/procedures/migration): engine-specific arguments. +- [COPY INTO](../spark/copy-into): import files with Spark SQL. +- [Manage Tags](../maintenance/manage-tags): create and retain historical views. +- [Hive](../ecosystem/hive): install the connector used to query Paimon from Hive. diff --git a/docs/docs/migration/migration-from-hive.mdx b/docs/docs/migration/migration-from-hive.mdx index f893d6dfc3..fd95902973 100644 --- a/docs/docs/migration/migration-from-hive.mdx +++ b/docs/docs/migration/migration-from-hive.mdx @@ -1,5 +1,5 @@ --- -title: "Migration From Hive" +title: "Migrate from Hive" sidebar_position: 1 --- @@ -27,114 +27,222 @@ under the License. # Hive Table Migration -Apache Hive supports ORC, Parquet file formats that could be migrated to Paimon. -When migrating data to a paimon table, the origin table will be permanently disappeared. So please back up your data if you -still need the original table. The migrated table will be [append table](../append-table/). +Migrate existing Hive data files into Paimon [append tables](../append-table/) using a Paimon +Hive catalog. The migrator moves files and registers them in Paimon metadata. It supports +ORC, Parquet, and Avro source files, for both partitioned and non-partitioned tables. -Now, we can use paimon hive catalog with Migrate Table Procedure to totally migrate a table from hive to paimon. -At the same time, you can use paimon hive catalog with Migrate Database Procedure to fully synchronize all tables in the database to paimon. +By default, the original Hive table is replaced by a Paimon table with the same name. Use +[Clone to Paimon](./clone-to-paimon) when you need a separate copy and a readable source table. -* Migrate Table Procedure: Paimon table does not exist, use the procedure upgrade hive table to paimon table. Hive table will disappear after action done. -* Migrate Database Procedure: Paimon table does not exist, use the procedure upgrade all hive tables in database to paimon table. All hive tables will disappear after action done. +:::warning Migration is not atomic -These two actions now support file format of hive "orc" and "parquet" and "avro". +Back up the source data and Hive metadata before migrating, and stop writes to the source +tables. An interruption can leave files or metadata partially migrated. Do not rely on +`delete_origin => false` as a backup: it keeps the Hive metadata, but files are still moved +out of the source location. -<span style={{color: "red"}}> **We highly recommend to back up hive table data before migrating, because migrating action is not atomic. If been interrupted while migrating, you may lose your data.** </span> +::: + +## Before You Start + +- Configure a Paimon catalog with `metastore = hive` and access to the source Hive metastore. + See [Catalog](../concepts/catalog), [Flink Procedures](../flink/procedures), or + [Spark Catalogs](../spark/catalogs) for setup. +- Give the migration runtime access to source and target storage, including the ability to + move files and update Hive metadata. This is a file-move workflow, not a cross-storage copy. +- Use a target append table with no primary key and `bucket = -1`. For an existing target, + check that the source data schema is compatible and partition names and types match. +- Record source row counts, partition values, and representative query results for validation. + The examples below use an ORC source table named `default.hivetable`; adjust identifiers, + paths, and `file.format` for your source. + +## Choose the Target Behavior + +| Procedure arguments | Target | Source after success | +| --- | --- | --- | +| Omit `target_table` | Replaces the original table under its original name | Original Hive table is removed | +| Set a different `target_table`; omit `delete_origin` or set it to `true` | Creates or imports files into the named Paimon table | Original Hive table is removed | +| Set a different `target_table` and `delete_origin => false` | Creates or imports files into the named Paimon table | Hive metadata remains, but its files have moved | + +For an existing target, migration adds imported files; it does not use clone's partition +overwrite behavior. Check the existing target data before importing to avoid duplicates. +`target_table` and `delete_origin` are SQL procedure options; the Flink migration action shown +below uses the default replacement behavior. ## Migrate Hive Table -<Tabs groupId="migrate-table"> +Choose one of the following entry points. The SQL examples select the Paimon Hive catalog +before calling the procedure. +<Tabs groupId="migration-engine"> <TabItem value="flink-sql" label="Flink SQL"> ```sql -CREATE CATALOG PAIMON WITH ( - 'type'='paimon', - 'metastore' = 'hive', - 'uri' = 'thrift://localhost:9083', - 'warehouse'='/path/to/warehouse/'); +CREATE CATALOG paimon_hive WITH ( + 'type' = 'paimon', + 'metastore' = 'hive', + 'uri' = 'thrift://localhost:9083', + 'warehouse' = 'hdfs:///warehouse' +); -USE CATALOG PAIMON; +USE CATALOG paimon_hive; +-- Flink 1.19 and later: replace default.hivetable with a Paimon table. CALL sys.migrate_table( connector => 'hive', source_table => 'default.hivetable', - -- You can specify the target table, and if the target table already exists - -- the file will be migrated directly to it - -- target_table => 'default.paimontarget', - -- You can specify delete_origin is false, this won't delete hivetable - -- delete_origin => false, - options => 'file.format=orc'); + options => 'file.format=orc' +); +``` + +For Flink 1.18, use positional arguments instead of the call above: + +```sql +CALL sys.migrate_table('hive', 'default.hivetable', 'file.format=orc'); +``` + +To import into a different target with Flink 1.19 or later, use this call **instead of** the +replacement call. The original Hive table is removed by default. + +```sql +CALL sys.migrate_table( + connector => 'hive', + source_table => 'default.hivetable', + target_table => 'default.paimon_target', + options => 'file.format=orc' +); ``` </TabItem> +<TabItem value="spark-sql" label="Spark SQL"> -<TabItem value="flink-action" label="Flink Action"> +Configure a [Spark Hive catalog](../spark/catalogs) named `paimon_hive` with the following +Spark startup options: ```bash -<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \ -migrate_table \ ---warehouse /path/to/warehouse \ ---catalog_conf uri=thrift://localhost:9083 \ ---catalog_conf metastore=hive \ ---source_type hive \ ---table default.hive_or_paimon +--conf spark.sql.catalog.paimon_hive=org.apache.paimon.spark.SparkCatalog \ +--conf spark.sql.catalog.paimon_hive.metastore=hive \ +--conf spark.sql.catalog.paimon_hive.uri=thrift://localhost:9083 \ +--conf spark.sql.catalog.paimon_hive.warehouse=hdfs:///warehouse \ +--conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions +``` + +Then run: + +```sql +USE paimon_hive.default; + +CALL sys.migrate_table( + source_type => 'hive', + table => 'default.hivetable', + options => 'file.format=orc' +); ``` +Spark uses `source_type` and `table` for these arguments. See +[Spark Migration Procedures](../spark/procedures/migration#migrate_table) for optional target +and parallelism arguments. + </TabItem> +<TabItem value="flink-action" label="Flink Action"> +Set `FLINK_HOME` to your Flink installation and use the matching +[Flink action jar](../flink/action-jars). + +```bash +"$FLINK_HOME/bin/flink" run /path/to/paimon-flink-action-@@VERSION@@.jar \ + migrate_table \ + --warehouse hdfs:///warehouse \ + --catalog_conf metastore=hive \ + --catalog_conf uri=thrift://localhost:9083 \ + --source_type hive \ + --table default.hivetable \ + --options file.format=orc +``` + +</TabItem> </Tabs> -After invoke, "hivetable" will totally convert to paimon format. Writing and reading the table by old "hive way" will fail. +After success, read and write the target through Paimon. Existing jobs that expect the +original Hive storage format must be updated. Hive queries need the +[Paimon Hive connector](../ecosystem/hive). ## Migrate Hive Database -<Tabs groupId="migrate-database"> +Database migration attempts each table in the source database and replaces successfully +migrated tables under their original names. It is not a transaction across all tables: a +failure can leave a mixture of Hive and Paimon tables. Review every source table's format and +requirements first. Use individual table migration for a selected subset. +<Tabs groupId="migration-engine"> <TabItem value="flink-sql" label="Flink SQL"> -```sql -CREATE CATALOG PAIMON WITH ( - 'type'='paimon', - 'metastore' = 'hive', - 'uri' = 'thrift://localhost:9083', - 'warehouse'='/path/to/warehouse/'); +Using the Paimon Hive catalog configured above: -USE CATALOG PAIMON; +```sql +USE CATALOG paimon_hive; +-- Flink 1.19 and later CALL sys.migrate_database( connector => 'hive', source_database => 'default', - options => 'file.format=orc'); + options => 'file.format=orc' +); +``` + +For Flink 1.18, use this call instead: + +```sql +CALL sys.migrate_database('hive', 'default', 'file.format=orc'); ``` </TabItem> +<TabItem value="spark-sql" label="Spark SQL"> -<TabItem value="flink-action" label="Flink Action"> +```sql +USE paimon_hive.default; -```bash -<FLINK_HOME>/bin/flink run \ -/path/to/paimon-flink-action-@@VERSION@@.jar \ -migrate_database \ ---warehouse <warehouse-path> \ ---source_type hive \ ---database <database> \ -[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]] \ -[--options <paimon-table-conf [,paimon-table-conf ...]> ] +CALL sys.migrate_database( + source_type => 'hive', + database => 'default', + options => 'file.format=orc' +); ``` -Example: +</TabItem> +<TabItem value="flink-action" label="Flink Action"> + ```bash -<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar migrate_table \ ---warehouse /path/to/warehouse \ ---catalog_conf uri=thrift://localhost:9083 \ ---catalog_conf metastore=hive \ ---source_type hive \ ---database default +"$FLINK_HOME/bin/flink" run /path/to/paimon-flink-action-@@VERSION@@.jar \ + migrate_database \ + --warehouse hdfs:///warehouse \ + --catalog_conf metastore=hive \ + --catalog_conf uri=thrift://localhost:9083 \ + --source_type hive \ + --database default \ + --options file.format=orc ``` </TabItem> - </Tabs> -After invoke, all tables in "default" database will totally convert to paimon format. Writing and reading the table by old "hive way" will fail. +## Verify the Migration + +1. Check the procedure output and logs. For a database, inspect both success and failure + counts; completion of the call does not mean every table migrated. +2. Inspect the target schema, partition keys, and table options. Confirm that it is a Paimon + append table with `bucket = -1`. +3. Query the target through Paimon and compare row counts and aggregates against the values + recorded before migration. Include null partition values and representative partitions. +4. Update reader and writer configurations, then verify them against the migrated table. + +For example, in the selected Paimon catalog: + +```sql +SHOW CREATE TABLE default.hivetable; +SELECT COUNT(*) FROM default.hivetable; +``` + +If migration fails, inspect the source and target metadata and file locations before retrying. +For database migration, identify the failed tables and handle them individually after resolving +the cause; do not assume another database-wide call will safely resume the previous run. diff --git a/docs/docs/migration/upsert-to-partitioned.mdx b/docs/docs/migration/upsert-to-partitioned.mdx index eb82a8deeb..0eff41cd25 100644 --- a/docs/docs/migration/upsert-to-partitioned.mdx +++ b/docs/docs/migration/upsert-to-partitioned.mdx @@ -1,11 +1,8 @@ --- -title: "Upsert To Partitioned" -sidebar_position: 1 +title: "Expose Tags as Hive Partitions" +sidebar_position: 3 --- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file @@ -25,169 +22,182 @@ specific language governing permissions and limitations under the License. --> -# Upsert To Partitioned +# Expose Tags as Hive Partitions {#upsert-to-partitioned} -:::warning +Keep a non-partitioned Paimon primary-key table updated from an upstream database, while +letting Hive batch jobs read historical views using a partition filter such as +`WHERE dt = '2023-10-16'`. -__Note:__ Only Hive Engine can be used to query these upsert-to-partitioned tables. +Set `metastore.tag-to-partition` to the Hive partition field name. Paimon exposes tag names +as partition values in the Hive metastore. Each partition reads a full table view at the tag's +snapshot; it does not contain only the rows changed on that date. -::: + -The [Tag Management](../maintenance/manage-tags) will maintain the manifests and data files of the snapshot. -A typical usage is creating tags daily, then you can maintain the historical data of each day for batch reading. +:::note Query engine -When using primary key tables, a non-partitioned approach is often used to maintain updates, in order to mirror and -synchronize tables from upstream database tables. This allows users to query the latest data. The tradition of Hive -data warehouses is not like this. Offline data warehouses require an immutable view every day to ensure the idempotence -of calculations. So we created a Tag mechanism to output these views. +The partition view described here is for the Hive engine and requires the +[Paimon Hive connector](../ecosystem/hive). Continue writing the underlying Paimon table with +Flink. The Hive partition field is a metadata mapping; do not add it as a physical partition +column to the Paimon table. -However, the traditional use of Hive data warehouses is more accustomed to using partitions to specify the query's Tag, -and is more accustomed to using Hive computing engines. +::: -So, we introduce `'metastore.tag-to-partition'` and `'metastore.tag-to-partition.preview'` to mapping a non-partitioned -primary key table to the partition table in Hive metastore, and mapping the partition field to the name of the Tag to be -fully compatible with Hive. +## Choose a View -## Example for Tag to Partition +| View | Configuration | What Hive reads | +| --- | --- | --- | +| Created tag | `metastore.tag-to-partition = dt` | The snapshot referenced by the named tag. Later upserts do not change that snapshot. | +| Preview before tag creation | Also set `metastore.tag-to-partition.preview = process-time` | The latest eligible retained snapshot for the requested period. Results can change as new snapshots arrive. | -**Step 1: Create table and tag in Flink SQL** +Tags retain the manifests and files needed to read their snapshots. Preview alone does not +provide that retention. Use [tag creation and retention](../maintenance/manage-tags) for +reproducible historical reads. -<Tabs groupId="create-table-and-tag-in-flink-sql-1"> +## Example for Tag to Partition + +### 1. Create the Paimon Table in Flink -<TabItem value="flink" label="Flink"> +Configure a Hive catalog and create a non-partitioned table with an explicit primary key. +Use a new table for this example. ```sql CREATE CATALOG my_hive WITH ( 'type' = 'paimon', 'metastore' = 'hive', - -- 'uri' = 'thrift://<hive-metastore-host-name>:<port>', default use 'hive.metastore.uris' in HiveConf - -- 'hive-conf-dir' = '...', this is recommended in the kerberos environment - -- 'hadoop-conf-dir' = '...', this is recommended in the kerberos environment - -- 'warehouse' = 'hdfs:///path/to/table/store/warehouse', default use 'hive.metastore.warehouse.dir' in HiveConf + 'uri' = 'thrift://localhost:9083', + 'warehouse' = 'hdfs:///warehouse' ); USE CATALOG my_hive; +CREATE DATABASE IF NOT EXISTS mydb; + +SET 'execution.runtime-mode' = 'batch'; +SET 'table.dml-sync' = 'true'; CREATE TABLE mydb.t ( pk INT, col1 STRING, - col2 STRING + col2 STRING, + PRIMARY KEY (pk) NOT ENFORCED ) WITH ( - 'bucket' = '-1', + 'bucket' = '2', 'metastore.tag-to-partition' = 'dt' ); -INSERT INTO t VALUES (1, '10', '100'), (2, '20', '200'); +INSERT INTO mydb.t VALUES (1, '10', '100'), (2, '20', '200'); --- create tag '2023-10-16' for snapshot 1 -CALL sys.create_tag('mydb.t', '2023-10-16', 1); +-- Pin the latest committed snapshot. This tag name is a label chosen for the example. +CALL sys.create_tag('mydb.t', '2023-10-16'); ``` -</TabItem> +Wait for the insert to finish before creating a tag. Here, `table.dml-sync` makes the SQL +client wait for each insert; omitting the snapshot argument selects the latest snapshot. -</Tabs> +### 2. Read the Tag in Hive -**Step 2: Query table in Hive with Partition Pruning** +Connect Hive to the same metastore, with the Paimon Hive connector installed: -<Tabs groupId="query-table-in-hive-with-partition-pruning-1"> +```sql +SHOW PARTITIONS mydb.t; +-- dt=2023-10-16 + +SELECT * FROM mydb.t WHERE dt = '2023-10-16' ORDER BY pk; +-- pk col1 col2 dt +-- 1 10 100 2023-10-16 +-- 2 20 200 2023-10-16 +``` + +### 3. Upsert and Create Another View -<TabItem value="hive" label="Hive"> +Back in the same Flink session, update an existing key and insert a new key: ```sql -SHOW PARTITIONS t; -/* -OK -dt=2023-10-16 -*/ - -SELECT * FROM t WHERE dt='2023-10-16'; -/* -OK -1 10 100 2023-10-16 -2 20 200 2023-10-16 -*/ +INSERT INTO mydb.t VALUES (1, '11', '110'), (3, '30', '300'); +CALL sys.create_tag('mydb.t', '2023-10-17'); ``` -</TabItem> +In Hive, the earlier tag still returns the original value for key `1`. The new tag returns +the updated value and all rows present at the new snapshot, including unchanged key `2`: -</Tabs> +```sql +SELECT * FROM mydb.t WHERE dt = '2023-10-16' ORDER BY pk; +-- 1 10 100 2023-10-16 +-- 2 20 200 2023-10-16 + +SELECT * FROM mydb.t WHERE dt = '2023-10-17' ORDER BY pk; +-- 1 11 110 2023-10-17 +-- 2 20 200 2023-10-17 +-- 3 30 300 2023-10-17 +``` -## Example for Tag Preview +Always select the tag you intend to read. A query across multiple tag partitions can return +multiple historical versions of the same primary key. -The above example can only query tags that have already been created, but Paimon is a real-time data lake, and you also -need to query the latest data. Therefore, Paimon provides a preview feature: +## Example for Tag Preview -**Step 1: Create table and tag in Flink SQL** +Preview exposes a period before a tag has been created for it. Use it when Hive readers need +to see data still arriving during that period. -<Tabs groupId="create-table-and-tag-in-flink-sql-2"> +### 1. Enable Preview in Flink -<TabItem value="flink" label="Flink"> +Using the catalog and database above, create a separate example table: ```sql -CREATE CATALOG my_hive WITH ( - 'type' = 'paimon', - 'metastore' = 'hive', - -- 'uri' = 'thrift://<hive-metastore-host-name>:<port>', default use 'hive.metastore.uris' in HiveConf - -- 'hive-conf-dir' = '...', this is recommended in the kerberos environment - -- 'hadoop-conf-dir' = '...', this is recommended in the kerberos environment - -- 'warehouse' = 'hdfs:///path/to/table/store/warehouse', default use 'hive.metastore.warehouse.dir' in HiveConf -); - -USE CATALOG my_hive; - -CREATE TABLE mydb.t ( +CREATE TABLE mydb.t_preview ( pk INT, col1 STRING, - col2 STRING + col2 STRING, + PRIMARY KEY (pk) NOT ENFORCED ) WITH ( - 'bucket' = '-1', + 'bucket' = '2', 'metastore.tag-to-partition' = 'dt', - -- preview tag creation mode process-time - -- paimon will create partitions early based on process-time - 'metastore.tag-to-partition.preview' = 'process-time' + 'metastore.tag-to-partition.preview' = 'process-time', + 'tag.creation-period' = 'daily' ); -INSERT INTO t VALUES (1, '10', '100'), (2, '20', '200'); +INSERT INTO mydb.t_preview VALUES (1, '10', '100'), (2, '20', '200'); +``` --- create tag '2023-10-16' for snapshot 1 -CALL sys.create_tag('mydb.t', '2023-10-16', 1); +After a commit, Paimon registers a preview partition based on processing time and the tag +creation period. Its actual value depends on when the job runs and the configured +`sink.process-time-zone`; inserting example data does not create a historical date partition. --- new data in '2023-10-17' -INSERT INTO t VALUES (3, '30', '300'), (4, '40', '400'); +### 2. Discover and Query the Preview in Hive --- haven't finished writing the data for '2023-10-17' yet, so there's no need to create a tag for now --- but the data is already visible for Hive +```sql +SHOW PARTITIONS mydb.t_preview; ``` -</TabItem> +Copy the partition value returned by `SHOW PARTITIONS` into the predicate below, replacing +`<preview-partition>`: -</Tabs> +```sql +SELECT * FROM mydb.t_preview +WHERE dt = '<preview-partition>' +ORDER BY pk; +``` -**Step 2: Query table in Hive with Partition Pruning** +Without a real tag of that name, Hive resolves the preview to the latest available snapshot +whose period is no later than the requested period, or an eligible retained tag if needed. +It fails if no suitable snapshot or tag remains. Further commits in the same period can +change the result. Once a tag with that name exists, reads use the tag's snapshot. -<Tabs groupId="query-table-in-hive-with-partition-pruning-2"> +## Retain Historical Views -<TabItem value="hive" label="Hive"> +Preview does not create permanent tags. To generate daily tags while keeping the current +period visible, configure the writing job with matching creation and preview modes: ```sql -SHOW PARTITIONS t; -/* -OK -dt=2023-10-16 -dt=2023-10-17 -*/ - -SELECT * FROM t WHERE dt='2023-10-17'; --- preview tag '2023-10-17' -/* -OK -1 10 100 2023-10-17 -2 20 200 2023-10-17 -3 30 300 2023-10-17 -4 40 400 2023-10-17 -*/ +-- Table options for a continuously written table +'metastore.tag-to-partition' = 'dt', +'metastore.tag-to-partition.preview' = 'process-time', +'tag.automatic-creation' = 'process-time', +'tag.creation-period' = 'daily', +'tag.num-retained-max' = '90' ``` -</TabItem> - -</Tabs> +Apply these as table options when creating the table, or through `ALTER TABLE` before starting +the writer. Choose retention for your batch workloads. Removing a tag also removes its Hive +partition; do not expire tags still needed by downstream jobs. See +[Manage Tags](../maintenance/manage-tags) for automatic creation, delay, and retention options. diff --git a/docs/sidebars.js b/docs/sidebars.js index 89b2926a90..788cb2c615 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -625,8 +625,8 @@ const sidebars = { }, "items": [ "migration/migration-from-hive", - "migration/upsert-to-partitioned", - "migration/clone-to-paimon" + "migration/clone-to-paimon", + "migration/upsert-to-partitioned" ] }, { diff --git a/docs/static/img/migration-clone-overwrite.svg b/docs/static/img/migration-clone-overwrite.svg new file mode 100644 index 0000000000..fadb194398 --- /dev/null +++ b/docs/static/img/migration-clone-overwrite.svg @@ -0,0 +1,52 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="490" viewBox="0 0 900 490" role="img" aria-labelledby="title desc"> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + <title id="title">Clone replaces the copied partitions</title> + <desc id="desc">With dynamic-partition-overwrite enabled, copying Hive partition dt=2026-09-10 replaces old rows in that target partition and retains dt=2026-09-09. The source table and files stay available. The diagram assumes data is copied for the selected partition.</desc> + <defs> + <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"> + <path d="M 0 0 L 10 5 L 0 10 z" fill="#526277"/> + </marker> + </defs> + <g font-family="Arial, Helvetica, sans-serif"> + <rect x="1" y="1" width="898" height="488" rx="12" fill="#ffffff" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="28" y="43" font-size="27" fill="#172b4d" font-weight="700" text-anchor="start">Clone replaces the copied partitions</text> + <text x="28" y="77" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Example: copy dt = 2026-09-10 with dynamic-partition-overwrite = true</text> + <rect x="28" y="104" width="844" height="79" rx="8" fill="#eaf2ff" stroke="#bfd2ed" stroke-width="1.5"/> + <text x="48" y="135" font-size="21" fill="#2463b4" font-weight="700" text-anchor="start">Hive source</text> + <text x="48" y="164" font-size="20" fill="#172b4d" font-weight="400" text-anchor="start">dt=2026-09-10 → new rows</text> + <text x="848" y="150" font-size="19" fill="#2463b4" font-weight="400" text-anchor="end">Source table and files stay available</text> + <text x="28" y="228" font-size="16" fill="#526277" font-weight="700" text-anchor="start">TARGET BEFORE</text> + <text x="539" y="228" font-size="16" fill="#526277" font-weight="700" text-anchor="start">TARGET AFTER</text> + <rect x="28" y="246" width="333" height="143" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <rect x="539" y="246" width="333" height="143" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <rect x="44" y="262" width="301" height="47" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="60" y="292" font-size="20" fill="#172b4d" font-weight="400" text-anchor="start">09-09 | retained rows</text> + <rect x="44" y="326" width="301" height="47" rx="8" fill="#fff7e6" stroke="#e4cb92" stroke-width="1.5"/> + <text x="60" y="356" font-size="20" fill="#946200" font-weight="400" text-anchor="start">09-10 | old rows</text> + <rect x="555" y="262" width="301" height="47" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="571" y="292" font-size="20" fill="#172b4d" font-weight="400" text-anchor="start">09-09 | retained rows</text> + <rect x="555" y="326" width="301" height="47" rx="8" fill="#e8f7f2" stroke="#b8dcd2" stroke-width="1.5"/> + <text x="571" y="356" font-size="20" fill="#087f6e" font-weight="700" text-anchor="start">09-10 | new rows</text> + <text x="450" y="303" font-size="20" fill="#2463b4" font-weight="700" text-anchor="middle">Clone</text> + <path d="M 375 324 H 525" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> + <text x="28" y="426" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Re-run: refresh the copied partition. Partitions absent from the copy remain unchanged.</text> + <text x="28" y="460" font-size="18" fill="#946200" font-weight="400" text-anchor="start">Check existing target options: disabling dynamic overwrite can replace the whole table.</text> + </g> +</svg> diff --git a/docs/static/img/migration-strategies.svg b/docs/static/img/migration-strategies.svg new file mode 100644 index 0000000000..2aa61d7e58 --- /dev/null +++ b/docs/static/img/migration-strategies.svg @@ -0,0 +1,60 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="520" viewBox="0 0 900 520" role="img" aria-labelledby="title desc"> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + <title id="title">Choose the path by what must stay</title> + <desc id="desc">Migrate moves Hive files into Paimon and replaces the Hive table by default. Clone copies data into a separate Paimon table and keeps the source. Tag-to-partition exposes snapshots of an existing Paimon table as Hive partitions without moving its data.</desc> + <defs> + <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"> + <path d="M 0 0 L 10 5 L 0 10 z" fill="#526277"/> + </marker> + </defs> + <g font-family="Arial, Helvetica, sans-serif"> + <rect x="1" y="1" width="898" height="518" rx="12" fill="#ffffff" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="28" y="43" font-size="27" fill="#172b4d" font-weight="700" text-anchor="start">Choose the path by what must stay</text> + <text x="28" y="77" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Source ownership and the reader interface determine the workflow</text> + <text x="28" y="126" font-size="16" fill="#946200" font-weight="700" text-anchor="start">MIGRATE</text> + <rect x="28" y="141" width="292" height="73" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="44" y="170" font-size="21" fill="#172b4d" font-weight="700" text-anchor="start">Hive table</text> + <text x="44" y="197" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Original table replaced</text> + <text x="454" y="162" font-size="19" fill="#946200" font-weight="700" text-anchor="middle">Move files</text> + <path d="M 334 181 H 573" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> + <rect x="588" y="141" width="284" height="73" rx="8" fill="#fff7e6" stroke="#e4cb92" stroke-width="1.5"/> + <text x="604" y="170" font-size="21" fill="#946200" font-weight="700" text-anchor="start">Paimon append table</text> + <text x="604" y="197" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Same table name by default</text> + <text x="28" y="252" font-size="16" fill="#2463b4" font-weight="700" text-anchor="start">CLONE</text> + <rect x="28" y="267" width="292" height="73" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="44" y="296" font-size="21" fill="#172b4d" font-weight="700" text-anchor="start">Hive / Paimon table</text> + <text x="44" y="323" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Source remains available</text> + <text x="454" y="288" font-size="19" fill="#2463b4" font-weight="700" text-anchor="middle">Copy data</text> + <path d="M 334 307 H 573" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> + <rect x="588" y="267" width="284" height="73" rx="8" fill="#eaf2ff" stroke="#bfd2ed" stroke-width="1.5"/> + <text x="604" y="296" font-size="21" fill="#2463b4" font-weight="700" text-anchor="start">Separate Paimon table</text> + <text x="604" y="323" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Validate before switching</text> + <text x="28" y="378" font-size="16" fill="#087f6e" font-weight="700" text-anchor="start">TAG TO PARTITION</text> + <rect x="28" y="393" width="292" height="73" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="44" y="422" font-size="21" fill="#172b4d" font-weight="700" text-anchor="start">Paimon upsert table</text> + <text x="44" y="449" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Keep updating this table</text> + <text x="454" y="414" font-size="19" fill="#087f6e" font-weight="700" text-anchor="middle">Expose views</text> + <path d="M 334 433 H 573" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> + <rect x="588" y="393" width="284" height="73" rx="8" fill="#e8f7f2" stroke="#b8dcd2" stroke-width="1.5"/> + <text x="604" y="422" font-size="21" fill="#087f6e" font-weight="700" text-anchor="start">Hive partition interface</text> + <text x="604" y="449" font-size="18" fill="#526277" font-weight="400" text-anchor="start">dt selects a tag or preview</text> + <text x="28" y="499" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Hive migration and Hive clone produce append tables; tag mapping keeps the Paimon table.</text> + </g> +</svg> diff --git a/docs/static/img/migration-tag-to-partition.svg b/docs/static/img/migration-tag-to-partition.svg new file mode 100644 index 0000000000..6d37cf590f --- /dev/null +++ b/docs/static/img/migration-tag-to-partition.svg @@ -0,0 +1,60 @@ +<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="662" viewBox="0 0 900 662" role="img" aria-labelledby="title desc"> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + <title id="title">One updating table, multiple Hive views</title> + <desc id="desc">A non-partitioned Paimon primary-key table exposes tag names as Hive dt partitions. The first tag retains key 1 with value 10 and key 2 with value 20. The second tag retains the update to key 1 with value 11 and the new key 3, as well as unchanged key 2. A preview partition reads an eligible snapshot until a real tag exists; preview results can change.</desc> + <defs> + <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"> + <path d="M 0 0 L 10 5 L 0 10 z" fill="#526277"/> + </marker> + </defs> + <g font-family="Arial, Helvetica, sans-serif"> + <rect x="1" y="1" width="898" height="660" rx="12" fill="#ffffff" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="28" y="43" font-size="27" fill="#172b4d" font-weight="700" text-anchor="start">One updating table, multiple Hive views</text> + <text x="28" y="77" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Hive partitions select full table snapshots, not the rows changed that day</text> + <rect x="28" y="106" width="844" height="60" rx="8" fill="#f5f7fb" stroke="#d7dfeb" stroke-width="1.5"/> + <text x="48" y="143" font-size="22" fill="#172b4d" font-weight="700" text-anchor="start">Upserts → non-partitioned Paimon table</text> + <text x="848" y="143" font-size="19" fill="#526277" font-weight="400" text-anchor="end">Primary key: pk</text> + <text x="28" y="206" font-size="16" fill="#526277" font-weight="700" text-anchor="start">PAIMON SNAPSHOT / TAG</text> + <text x="520" y="206" font-size="16" fill="#526277" font-weight="700" text-anchor="start">HIVE PARTITION VIEW</text> + <rect x="28" y="226" width="337" height="95" rx="8" fill="#eaf2ff" stroke="#bfd2ed" stroke-width="1.5"/> + <text x="44" y="258" font-size="21" fill="#2463b4" font-weight="700" text-anchor="start">Tag: 2023-10-16</text> + <text x="44" y="291" font-size="19" fill="#172b4d" font-weight="400" text-anchor="start">pk/value: 1/10, 2/20</text> + <path d="M 379 273 H 504" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> + <rect x="520" y="226" width="352" height="95" rx="8" fill="#e8f7f2" stroke="#b8dcd2" stroke-width="1.5"/> + <text x="536" y="258" font-size="22" fill="#087f6e" font-weight="700" text-anchor="start">dt=2023-10-16</text> + <text x="536" y="291" font-size="19" fill="#172b4d" font-weight="400" text-anchor="start">Fixed at the tagged snapshot</text> + <rect x="28" y="350" width="337" height="95" rx="8" fill="#eaf2ff" stroke="#bfd2ed" stroke-width="1.5"/> + <text x="44" y="382" font-size="21" fill="#2463b4" font-weight="700" text-anchor="start">Tag: 2023-10-17</text> + <text x="44" y="415" font-size="19" fill="#172b4d" font-weight="400" text-anchor="start">pk/value: 1/11, 2/20, 3/30</text> + <path d="M 379 397 H 504" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)"/> + <rect x="520" y="350" width="352" height="95" rx="8" fill="#e8f7f2" stroke="#b8dcd2" stroke-width="1.5"/> + <text x="536" y="382" font-size="22" fill="#087f6e" font-weight="700" text-anchor="start">dt=2023-10-17</text> + <text x="536" y="415" font-size="19" fill="#172b4d" font-weight="400" text-anchor="start">Fixed at the tagged snapshot</text> + <rect x="28" y="474" width="337" height="97" rx="8" fill="#fff7e6" stroke="#e4cb92" stroke-width="1.5" stroke-dasharray="7 5"/> + <text x="44" y="507" font-size="21" fill="#946200" font-weight="700" text-anchor="start">Eligible retained snapshot</text> + <text x="44" y="542" font-size="18" fill="#172b4d" font-weight="400" text-anchor="start">Period based on processing time</text> + <path d="M 379 522 H 504" fill="none" stroke="#526277" stroke-width="2" marker-end="url(#arrow)" stroke-dasharray="7 5"/> + <rect x="520" y="474" width="352" height="97" rx="8" fill="#fff7e6" stroke="#e4cb92" stroke-width="1.5" stroke-dasharray="7 5"/> + <text x="536" y="507" font-size="22" fill="#946200" font-weight="700" text-anchor="start">Preview partition</text> + <text x="536" y="542" font-size="19" fill="#172b4d" font-weight="400" text-anchor="start">Can change until a tag exists</text> + <text x="28" y="610" font-size="18" fill="#526277" font-weight="400" text-anchor="start">dt is a Hive metadata field. Paimon data is not physically repartitioned by tag date.</text> + <text x="28" y="638" font-size="18" fill="#526277" font-weight="400" text-anchor="start">Create and retain real tags for reproducible history; preview alone does not retain data.</text> + </g> +</svg>
