Zerolzj commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-5552907234
Thanks for clarifying that the current Hive and Iceberg migrators can
perform metadata-only
migration, while generic files do not have an `add_files`-style API.
I would like to ask about the supported API boundary for a third-party
migrator. The target use case
is a fixed-bucket primary-key table whose immutable files already have a
registered Paimon file
format implementation. The migrator can provide partition and bucket
identity, schema ID, row count,
key statistics, sequence range, sort-order proof, file size, and a content
fingerprint. It wants to
create a normal Paimon snapshot without decoding and rewriting every row.
`DataFileMeta`, `CommitMessage`, and table commit interfaces are public, but
the community migration
implementations appear to construct the actual data increment through
classes such as
`FileMetaUtils`, `CommitMessageImpl`, and `DataIncrement`, which are not
marked as public APIs.
Could maintainers clarify the following?
1. Is using `FileMetaUtils.createCommitMessage` directly an intended and
compatibility-supported
extension point for third-party migrators?
2. If not, would the community consider a stable `ExistingFile`/`ImportFile`
builder or an
`add_files`-style procedure that validates and commits existing files?
3. Which validation inputs should be mandatory for a primary-key table so an
importer cannot create
a snapshot with incorrect bucket placement, key range, sequence
semantics, or sort order?
4. Is metadata-only registration expected to be limited to Paimon-owned
files, or is a
referenced-but-not-owned file model in scope? `externalPath` expresses
location, while normal
snapshot cleanup still owns the referenced data file and its `extraFiles`.
5. If borrowed-file ownership is intentionally out of scope, is a
Paimon-owned descriptor plus an
external lifecycle manager the recommended integration boundary?
This does not require a generic file parser in Paimon: format-specific
inspection and validation can
remain in the migrator. The requested public surface would only define the
metadata and commit
contract. I can provide a prototype and negative tests once the intended
boundary is clear.
References:
- Existing-file question: https://github.com/apache/paimon/issues/7173
- Multi-location and `add_files` discussion:
https://github.com/apache/paimon/issues/3627#issuecomment-2664980472
- `DataFileMeta`:
https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java
- `FileMetaUtils`:
https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/migrate/FileMetaUtils.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
