fengruotj opened a new issue, #11073: URL: https://github.com/apache/paimon/issues/11073
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Debugging Paimon data files often requires tracing a snapshot to its manifest lists, inspecting the file additions and removals recorded in manifests, and checking the schema associated with a snapshot or data file. The PyPaimon CLI already provides `paimon table snapshot` for the latest snapshot and `paimon table get` for the current table schema. These are useful starting points, but a dedicated metadata inspection workflow would make historical versions and manifest entries easier to investigate. See the existing [table metadata commands](https://paimon.apache.org/docs/master/pypaimon/cli-query/#table-get). Typical questions include: - Which files were added or removed by a particular commit or compaction? - Which manifest contains the metadata for a data file being investigated? - Which schema version does a snapshot reference, and which version was used to write a particular file? - What file sizes, record counts, sequence-number ranges, and column statistics are recorded in the metadata? Providing these capabilities through the CLI would reduce the need for custom Python scripts or an external query engine when investigating storage and metadata issues. ### Solution Add three read-only command groups: `manifest`, `snapshot`, and `schema`. The syntax below is a proposal and can be adjusted to follow community conventions. #### Manifest commands - List manifests referenced by the latest or a specified snapshot, distinguishing base, delta, and optional changelog manifest lists. - Show manifest summaries, including file name, size, schema ID, and added/deleted file counts. - Inspect an individual manifest and display its original `ADD`/`DELETE` entries, including partition, bucket, data-file path, file schema ID, file size, and record count. Expose additional metadata, such as level, sequence-number ranges, and statistics, when available. - Support filtering entries by partition or bucket and limiting the displayed results. ```bash # List manifests referenced by the latest snapshot paimon manifest list mydb.orders # Inspect the delta manifests for a specific commit paimon manifest list mydb.orders --snapshot-id 42 --type delta # Decode a manifest returned by the list command paimon manifest get mydb.orders --file manifest-example --format json ``` Manifest inspection should preserve file-level `ADD`/`DELETE` entries instead of silently reducing them to the snapshot's live file set. A `DELETE` entry removes a file reference; it does not represent a row-level delete or prove that the physical file has been deleted. #### Snapshot commands - List available snapshots in snapshot-ID order, with ID, schema ID, commit time, commit kind, commit user, and record counts. - Retrieve the latest snapshot or a specific snapshot by ID, including its manifest-list references and other available snapshot metadata. ```bash # Browse available snapshot history paimon snapshot list mydb.orders --limit 20 # Inspect the latest snapshot or a specific historical snapshot paimon snapshot get mydb.orders paimon snapshot get mydb.orders --snapshot-id 42 --format json ``` #### Schema commands - List available schema versions. - Retrieve the latest schema, a specific schema ID, or the schema referenced by a specified snapshot. - Display schema ID, field IDs, names, types, nullability, partition keys, primary keys, table options, and comments. ```bash # Browse schema history paimon schema list mydb.orders # Inspect the latest schema or a specific version paimon schema get mydb.orders paimon schema get mydb.orders --schema-id 3 --format json # Resolve the schema ID recorded in a historical snapshot paimon schema get mydb.orders --snapshot-id 42 --format json ``` `--schema-id` and `--snapshot-id` should be mutually exclusive. Snapshot-based lookup must use that snapshot's schema ID. Data files can reference older schema versions, which should remain visible in manifest inspection. #### Shared behavior - Reuse the existing `paimon.yaml` / `-c` catalog configuration and PyPaimon metadata readers; keep existing commands compatible. - Support readable table output and structured JSON output, with explicit IDs and file references for subsequent inspection. - Read metadata without scanning data-file contents or modifying the table. - Resolve the latest snapshot once per command and report the selected snapshot ID. - Return clear errors for unavailable snapshot/schema IDs, missing or unreadable manifests, and unsupported metadata versions. Never silently substitute the latest version for a requested historical version. - Handle empty tables and absent optional metadata explicitly. - Respect ordinary append, primary-key, and Data Evolution table semantics. Show mode-specific metadata, such as row-ID ranges and written columns, when available. The intended debugging workflow is: ```text Snapshot 42 ----------------------------> Schema 3 | +--> Base / delta manifest lists | +--> Manifest files | +--> ADD / DELETE entries | +--> Data-file path and statistics +--> File schema ID --> Schema version ``` ### Anything else? Related capabilities: - [PyPaimon CLI](https://paimon.apache.org/docs/master/pypaimon/cli/#basic-usage) - [Existing schema and snapshot inspection](https://paimon.apache.org/docs/master/pypaimon/cli-query/#table-get) - [PyPaimon system tables](https://paimon.apache.org/docs/master/pypaimon/system-tables/), including `$snapshots`, `$schemas`, `$manifests`, and `$files`. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
