wzx140 opened a new pull request, #10978:
URL: https://github.com/apache/paimon/pull/10978
## Purpose
Add the `entropy-inject-template` external path strategy so that multiple
databases and tables can share hash prefixes ahead of the table context. Data
files retain their fully expanded `externalPath` in metadata. Snapshot
expiration uses these paths, orphan-file cleanup explicitly rejects this
strategy, and table deletion removes manifest-referenced files before deleting
metadata.
## Design: shared hash prefixes across Paimon tables
The following is an English translation of the supplied design document. It
preserves the proposal, including its operational assumptions and original
compatibility-test results. Differences between the proposal and this
implementation are listed separately below. S3 pre-partitioning estimates are
assumptions reported in the source document, not guarantees established by this
PR.
### Background
#### Distributing lake-table requests across prefixes
- **Iceberg:** The hash can precede the database and table names, for
example `s3://data-bucket/1001/my_db/my_table/<file>.parquet`. After partitions
have been provisioned for `s3://data-bucket/{0000~1111}`, files belonging to
different tables can share the S3 partitions warmed or scaled for these
prefixes.
- **Paimon:** The existing layout is
`s3://data-bucket/my_db/my_table/bucket-0/1001/<file>.parquet`. The hash
follows the table, partition, and bucket directories. S3 partitions warmed or
scaled for that hash prefix serve only files under this directory. Other tables
cannot share them and require separate warming or scaling for their own paths.
Iceberg supports shared S3 hash-prefix partitions across databases and
tables to distribute requests and increase throughput. With
`write.object-storage.enabled` enabled and `write.data.path` set to a shared
root outside the table location, its built-in path generator inserts a hash
after the root and then appends the last two components of the table location
as table context.
```sql
-- Spark SQL; iceberg_catalog and my_db are already configured.
CREATE TABLE iceberg_catalog.my_db.my_table (
id BIGINT,
dt STRING
)
USING iceberg
PARTITIONED BY (dt)
LOCATION 's3://metadata-bucket/warehouse/my_db/my_table'
TBLPROPERTIES (
'write.object-storage.enabled' = 'true',
'write.data.path' = 's3://data-bucket',
'write.object-storage.partitioned-paths' = 'true'
);
-- Illustrative data-file path; the hash is only an example:
--
s3://data-bucket/1001/1001/1001/10011001/my_db/my_table/dt=2026-11-02/<file>.parquet
```
Paimon's existing `entropy-inject` strategy inserts the hash after the
Paimon bucket directory. This design follows Iceberg's approach by moving the
hash ahead of the table context, allowing multiple databases and tables to
share prefix partitions.
```sql
-- Existing Paimon behavior in Spark SQL;
-- paimon_catalog and my_db are already configured.
CREATE TABLE paimon_catalog.my_db.my_table (
id BIGINT,
dt STRING
)
USING paimon
PARTITIONED BY (dt)
TBLPROPERTIES (
'bucket' = '-1',
'data-file.external-paths' = 's3://data-bucket/my_db/my_table',
'data-file.external-paths.strategy' = 'entropy-inject'
);
-- Illustrative data-file path; the hash is only an example:
--
s3://data-bucket/my_db/my_table/dt=2026-11-02/bucket-0/1001/1001/1001/10011001/<file>.parquet
```
### Configuration and path layout
Reuse `data-file.external-paths` as a path template containing `{hash}`.
```sql
CREATE TABLE paimon_catalog.my_db.my_table (
id BIGINT,
dt STRING
)
USING paimon
PARTITIONED BY (dt)
TBLPROPERTIES (
'bucket' = '-1',
'data-file.external-paths' = 's3://bucket-a/{hash}/my_db.db/my_table',
'data-file.external-paths.strategy' = 'entropy-inject-template'
);
```
1. At initialization, the provider randomly selects a starting point, then
selects path templates in round-robin order.
2. Compute the existing Paimon MurmurHash3 32-bit hash of the complete
filename, take the low 20 bits, and arrange them into `4/4/4/8` segments. The
partition path does not participate in hashing.
3. Replace `{hash}` with the four-level hash directory.
4. Append the existing relative bucket path and filename, and persist the
fully expanded path as `externalPath`.
```text
Template: s3://bucket-a/{hash}/my_db.db/my_table
Expanded: s3://bucket-a/1001/1001/1001/10011001/my_db.db/my_table
File:
s3://bucket-a/1001/1001/1001/10011001/my_db.db/my_table/dt=2026-11-02/bucket-0/<file>.parquet
```
### Compatibility with older versions
An unknown strategy does not cause every operation to fail. The source
design reports a unit test on a checkout where `entropy-inject-template` did
not yet exist: first write one real record with the existing `round-robin`
strategy, then directly replace the strategy and template options in the
persisted schema to simulate older code opening newer metadata. The test used
`FileSystemCatalog`, `LocalFileIO`, and an append-only table without a primary
key; production code was not modified.
| Operation | Reported unit-test result |
| --- | --- |
| `catalog.getTable` | Succeeds: loads a table with an unknown strategy and
reads the raw options. |
| `latestSnapshot()` | Succeeds: reads existing snapshot metadata. |
| Read: `newReadBuilder().newScan().plan()` | Throws an
`IllegalArgumentException` mentioning the unknown strategy during read
planning. |
| Write: create a batch writer and attempt a write | Throws an
`IllegalArgumentException` mentioning the unknown strategy; the write cannot
complete. |
| `newExpireSnapshots().expire()` | Throws an `IllegalArgumentException`
mentioning the unknown strategy; expiration cannot complete. |
| `LocalOrphanFilesClean.clean()` | Throws an `IllegalArgumentException`
mentioning the unknown strategy; orphan cleanup cannot complete. |
| `catalog.dropTable` | Succeeds, but only deletes files under the literal
path `s3://bucket-a/{hash}/my_db.db/my_table`; it does not accidentally delete
files from other tables. |
### `expire_snapshot`
#### Existing behavior
1. Determine expired snapshots according to the retention policy and derive
deletion candidates from manifests.
2. Exclude files still protected by retained references.
3. Locate and delete files and extra files through the complete
`externalPath` in file metadata, without scanning external directories.
4. Attempt to reclaim empty bucket and partition directories under the table
directory.
#### Required changes
The actual file-deletion logic does not need to change: it continues to use
the complete `externalPath`. The configuration enum and path-provider factory
must recognize the new strategy. Expiration creates an `ExternalPathProvider`
while constructing `DataFilePathFactory`, so adding an enum value alone is
insufficient. The writer persists the fully expanded path, and regression tests
cover expiration deletion. The first version does not reclaim empty directories
in the shared layout or introduce directory scans.
### `remove_orphan_files`
#### Existing behavior
1. Find partition and bucket directories under table-local directories and
external paths, and collect their direct child files.
2. Filter by `olderThan` to obtain candidate files.
3. Collect snapshot, tag, and changelog references from valid branches, and
exclude referenced files from the candidates.
4. Delete the remaining files and reclaim empty directories.
#### Revised behavior
Not supported for now.
### Dropping a table (`DROP TABLE`)
#### Existing behavior
1. `AbstractCatalog.dropTable` collects all schemas from main and other
branches.
2. `getSchemaExternalPaths` extracts the raw external paths and global-index
path.
3. These paths are passed to `dropTableImpl(identifier, paths)`.
4. Each catalog deletes metadata and directories according to its own
semantics.
Existing `DROP TABLE` behavior recursively deletes directories configured in
the schemas rather than reading data-file lists from manifests and deleting
files individually.
These external paths are currently treated as table-exclusive directories.
`FileSystemCatalog` deletes them recursively; managed tables in `HiveCatalog`
and tables in `JdbcCatalog` also delete these directories. Hive external tables
retain metadata-only deletion semantics.
#### Revised behavior
Traverse main and all other branches and inspect their historical schemas.
Branches using other strategies retain their existing behavior; branches using
`entropy-inject-template` use per-file deletion:
1. Traverse retained snapshots, tags, and changelogs, and read their
referenced base, delta, and changelog manifests. Collect recorded data files
and extra files, including files appearing in both ADD and DELETE entries.
Reading only the live files in the latest snapshot is insufficient.
2. Deduplicate by complete file path across the table and delete files
individually through `externalPath`. Treat already-missing files as cleaned up.
Do not expand templates, scan hash directories, recursively delete data-file
directories of template-strategy branches, or reclaim empty directories. These
data directories must not enter the existing recursive directory-deletion list.
3. For catalogs that permit data deletion, finish reading the file lists and
deleting files before deleting table metadata. Abort on a read or deletion
failure and retain manifests needed for retry. Files shared across branches of
the same table are deleted only once. Hive external tables retain metadata-only
deletion semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]