wzx140 opened a new pull request, #10978:
URL: https://github.com/apache/paimon/pull/10978

   ## Purpose
   
   Add the `entropy-inject-template` external path strategy so that multiple 
databases and tables can share hash prefixes ahead of the table context. Data 
files retain their fully expanded `externalPath` in metadata. Snapshot 
expiration uses these paths, orphan-file cleanup explicitly rejects this 
strategy, and table deletion removes manifest-referenced files before deleting 
metadata.
   
   ## Design: shared hash prefixes across Paimon tables
   
   The following is an English translation of the supplied design document. It 
preserves the proposal, including its operational assumptions and original 
compatibility-test results. Differences between the proposal and this 
implementation are listed separately below. S3 pre-partitioning estimates are 
assumptions reported in the source document, not guarantees established by this 
PR.
   
   ### Background
   
   #### Distributing lake-table requests across prefixes
   
   - **Iceberg:** The hash can precede the database and table names, for 
example `s3://data-bucket/1001/my_db/my_table/<file>.parquet`. After partitions 
have been provisioned for `s3://data-bucket/{0000~1111}`, files belonging to 
different tables can share the S3 partitions warmed or scaled for these 
prefixes.
   - **Paimon:** The existing layout is 
`s3://data-bucket/my_db/my_table/bucket-0/1001/<file>.parquet`. The hash 
follows the table, partition, and bucket directories. S3 partitions warmed or 
scaled for that hash prefix serve only files under this directory. Other tables 
cannot share them and require separate warming or scaling for their own paths.
   
   Iceberg supports shared S3 hash-prefix partitions across databases and 
tables to distribute requests and increase throughput. With 
`write.object-storage.enabled` enabled and `write.data.path` set to a shared 
root outside the table location, its built-in path generator inserts a hash 
after the root and then appends the last two components of the table location 
as table context.
   
   ```sql
   -- Spark SQL; iceberg_catalog and my_db are already configured.
   CREATE TABLE iceberg_catalog.my_db.my_table (
       id BIGINT,
       dt STRING
   )
   USING iceberg
   PARTITIONED BY (dt)
   LOCATION 's3://metadata-bucket/warehouse/my_db/my_table'
   TBLPROPERTIES (
       'write.object-storage.enabled' = 'true',
       'write.data.path' = 's3://data-bucket',
       'write.object-storage.partitioned-paths' = 'true'
   );
   
   -- Illustrative data-file path; the hash is only an example:
   -- 
s3://data-bucket/1001/1001/1001/10011001/my_db/my_table/dt=2026-11-02/<file>.parquet
   ```
   
   Paimon's existing `entropy-inject` strategy inserts the hash after the 
Paimon bucket directory. This design follows Iceberg's approach by moving the 
hash ahead of the table context, allowing multiple databases and tables to 
share prefix partitions.
   
   ```sql
   -- Existing Paimon behavior in Spark SQL;
   -- paimon_catalog and my_db are already configured.
   CREATE TABLE paimon_catalog.my_db.my_table (
       id BIGINT,
       dt STRING
   )
   USING paimon
   PARTITIONED BY (dt)
   TBLPROPERTIES (
       'bucket' = '-1',
       'data-file.external-paths' = 's3://data-bucket/my_db/my_table',
       'data-file.external-paths.strategy' = 'entropy-inject'
   );
   
   -- Illustrative data-file path; the hash is only an example:
   -- 
s3://data-bucket/my_db/my_table/dt=2026-11-02/bucket-0/1001/1001/1001/10011001/<file>.parquet
   ```
   
   ### Configuration and path layout
   
   Reuse `data-file.external-paths` as a path template containing `{hash}`.
   
   ```sql
   CREATE TABLE paimon_catalog.my_db.my_table (
       id BIGINT,
       dt STRING
   )
   USING paimon
   PARTITIONED BY (dt)
   TBLPROPERTIES (
       'bucket' = '-1',
       'data-file.external-paths' = 's3://bucket-a/{hash}/my_db.db/my_table',
       'data-file.external-paths.strategy' = 'entropy-inject-template'
   );
   ```
   
   1. At initialization, the provider randomly selects a starting point, then 
selects path templates in round-robin order.
   2. Compute the existing Paimon MurmurHash3 32-bit hash of the complete 
filename, take the low 20 bits, and arrange them into `4/4/4/8` segments. The 
partition path does not participate in hashing.
   3. Replace `{hash}` with the four-level hash directory.
   4. Append the existing relative bucket path and filename, and persist the 
fully expanded path as `externalPath`.
   
   ```text
   Template: s3://bucket-a/{hash}/my_db.db/my_table
   Expanded: s3://bucket-a/1001/1001/1001/10011001/my_db.db/my_table
   File:     
s3://bucket-a/1001/1001/1001/10011001/my_db.db/my_table/dt=2026-11-02/bucket-0/<file>.parquet
   ```
   
   ### Compatibility with older versions
   
   An unknown strategy does not cause every operation to fail. The source 
design reports a unit test on a checkout where `entropy-inject-template` did 
not yet exist: first write one real record with the existing `round-robin` 
strategy, then directly replace the strategy and template options in the 
persisted schema to simulate older code opening newer metadata. The test used 
`FileSystemCatalog`, `LocalFileIO`, and an append-only table without a primary 
key; production code was not modified.
   
   | Operation | Reported unit-test result |
   | --- | --- |
   | `catalog.getTable` | Succeeds: loads a table with an unknown strategy and 
reads the raw options. |
   | `latestSnapshot()` | Succeeds: reads existing snapshot metadata. |
   | Read: `newReadBuilder().newScan().plan()` | Throws an 
`IllegalArgumentException` mentioning the unknown strategy during read 
planning. |
   | Write: create a batch writer and attempt a write | Throws an 
`IllegalArgumentException` mentioning the unknown strategy; the write cannot 
complete. |
   | `newExpireSnapshots().expire()` | Throws an `IllegalArgumentException` 
mentioning the unknown strategy; expiration cannot complete. |
   | `LocalOrphanFilesClean.clean()` | Throws an `IllegalArgumentException` 
mentioning the unknown strategy; orphan cleanup cannot complete. |
   | `catalog.dropTable` | Succeeds, but only deletes files under the literal 
path `s3://bucket-a/{hash}/my_db.db/my_table`; it does not accidentally delete 
files from other tables. |
   
   ### `expire_snapshot`
   
   #### Existing behavior
   
   1. Determine expired snapshots according to the retention policy and derive 
deletion candidates from manifests.
   2. Exclude files still protected by retained references.
   3. Locate and delete files and extra files through the complete 
`externalPath` in file metadata, without scanning external directories.
   4. Attempt to reclaim empty bucket and partition directories under the table 
directory.
   
   #### Required changes
   
   The actual file-deletion logic does not need to change: it continues to use 
the complete `externalPath`. The configuration enum and path-provider factory 
must recognize the new strategy. Expiration creates an `ExternalPathProvider` 
while constructing `DataFilePathFactory`, so adding an enum value alone is 
insufficient. The writer persists the fully expanded path, and regression tests 
cover expiration deletion. The first version does not reclaim empty directories 
in the shared layout or introduce directory scans.
   
   ### `remove_orphan_files`
   
   #### Existing behavior
   
   1. Find partition and bucket directories under table-local directories and 
external paths, and collect their direct child files.
   2. Filter by `olderThan` to obtain candidate files.
   3. Collect snapshot, tag, and changelog references from valid branches, and 
exclude referenced files from the candidates.
   4. Delete the remaining files and reclaim empty directories.
   
   #### Revised behavior
   
   Not supported for now.
   
   ### Dropping a table (`DROP TABLE`)
   
   #### Existing behavior
   
   1. `AbstractCatalog.dropTable` collects all schemas from main and other 
branches.
   2. `getSchemaExternalPaths` extracts the raw external paths and global-index 
path.
   3. These paths are passed to `dropTableImpl(identifier, paths)`.
   4. Each catalog deletes metadata and directories according to its own 
semantics.
   
   Existing `DROP TABLE` behavior recursively deletes directories configured in 
the schemas rather than reading data-file lists from manifests and deleting 
files individually.
   
   These external paths are currently treated as table-exclusive directories. 
`FileSystemCatalog` deletes them recursively; managed tables in `HiveCatalog` 
and tables in `JdbcCatalog` also delete these directories. Hive external tables 
retain metadata-only deletion semantics.
   
   #### Revised behavior
   
   Traverse main and all other branches and inspect their historical schemas. 
Branches using other strategies retain their existing behavior; branches using 
`entropy-inject-template` use per-file deletion:
   
   1. Traverse retained snapshots, tags, and changelogs, and read their 
referenced base, delta, and changelog manifests. Collect recorded data files 
and extra files, including files appearing in both ADD and DELETE entries. 
Reading only the live files in the latest snapshot is insufficient.
   2. Deduplicate by complete file path across the table and delete files 
individually through `externalPath`. Treat already-missing files as cleaned up. 
Do not expand templates, scan hash directories, recursively delete data-file 
directories of template-strategy branches, or reclaim empty directories. These 
data directories must not enter the existing recursive directory-deletion list.
   3. For catalogs that permit data deletion, finish reading the file lists and 
deleting files before deleting table metadata. Abort on a read or deletion 
failure and retain manifests needed for retry. Files shared across branches of 
the same table are deleted only once. Hive external tables retain metadata-only 
deletion semantics.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to