fengruotj opened a new issue, #11073:
URL: https://github.com/apache/paimon/issues/11073

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   Debugging Paimon data files often requires tracing a snapshot to its 
manifest lists, inspecting the file additions and removals recorded in 
manifests, and checking the schema associated with a snapshot or data file. 
    
     The PyPaimon CLI already provides `paimon table snapshot` for the latest 
snapshot and `paimon table get` for the current table schema. These are useful 
starting points, but a dedicated metadata inspection workflow would make 
historical versions and manifest entries easier to investigate. See the 
existing [table metadata 
commands](https://paimon.apache.org/docs/master/pypaimon/cli-query/#table-get). 
    
     Typical questions include: 
    
     - Which files were added or removed by a particular commit or compaction? 
     - Which manifest contains the metadata for a data file being investigated? 
     - Which schema version does a snapshot reference, and which version was 
used to write a particular file? 
     - What file sizes, record counts, sequence-number ranges, and column 
statistics are recorded in the metadata? 
    
     Providing these capabilities through the CLI would reduce the need for 
custom Python scripts or an external query engine when investigating storage 
and metadata issues.
   
   ### Solution
   
    Add three read-only command groups: `manifest`, `snapshot`, and `schema`. 
The syntax below is a proposal and can be adjusted to follow community 
conventions. 
    
     #### Manifest commands 
    
     - List manifests referenced by the latest or a specified snapshot, 
distinguishing base, delta, and optional changelog manifest lists. 
     - Show manifest summaries, including file name, size, schema ID, and 
added/deleted file counts. 
     - Inspect an individual manifest and display its original `ADD`/`DELETE` 
entries, including partition, bucket, data-file path, file schema ID, file 
size, and record count. Expose additional metadata, such as level, 
sequence-number
     ranges, and statistics, when available. 
     - Support filtering entries by partition or bucket and limiting the 
displayed results. 
    
     ```bash 
     # List manifests referenced by the latest snapshot 
     paimon manifest list mydb.orders 
    
     # Inspect the delta manifests for a specific commit 
     paimon manifest list mydb.orders --snapshot-id 42 --type delta 
    
     # Decode a manifest returned by the list command 
     paimon manifest get mydb.orders --file manifest-example --format json 
     ``` 
    
     Manifest inspection should preserve file-level `ADD`/`DELETE` entries 
instead of silently reducing them to the snapshot's live file set. A `DELETE` 
entry removes a file reference; it does not represent a row-level delete or 
prove that
     the physical file has been deleted.
   
     #### Snapshot commands 
    
     - List available snapshots in snapshot-ID order, with ID, schema ID, 
commit time, commit kind, commit user, and record counts. 
     - Retrieve the latest snapshot or a specific snapshot by ID, including its 
manifest-list references and other available snapshot metadata. 
    
     ```bash 
     # Browse available snapshot history 
     paimon snapshot list mydb.orders --limit 20 
    
     # Inspect the latest snapshot or a specific historical snapshot 
     paimon snapshot get mydb.orders 
     paimon snapshot get mydb.orders --snapshot-id 42 --format json 
     ``` 
    
     #### Schema commands 
    
     - List available schema versions. 
     - Retrieve the latest schema, a specific schema ID, or the schema 
referenced by a specified snapshot. 
     - Display schema ID, field IDs, names, types, nullability, partition keys, 
primary keys, table options, and comments. 
    
     ```bash 
     # Browse schema history 
     paimon schema list mydb.orders 
    
     # Inspect the latest schema or a specific version 
     paimon schema get mydb.orders 
     paimon schema get mydb.orders --schema-id 3 --format json 
    
     # Resolve the schema ID recorded in a historical snapshot 
     paimon schema get mydb.orders --snapshot-id 42 --format json 
     ``` 
    
     `--schema-id` and `--snapshot-id` should be mutually exclusive. 
Snapshot-based lookup must use that snapshot's schema ID. Data files can 
reference older schema versions, which should remain visible in manifest 
inspection.
   
     #### Shared behavior 
    
     - Reuse the existing `paimon.yaml` / `-c` catalog configuration and 
PyPaimon metadata readers; keep existing commands compatible. 
     - Support readable table output and structured JSON output, with explicit 
IDs and file references for subsequent inspection. 
     - Read metadata without scanning data-file contents or modifying the 
table. 
     - Resolve the latest snapshot once per command and report the selected 
snapshot ID. 
     - Return clear errors for unavailable snapshot/schema IDs, missing or 
unreadable manifests, and unsupported metadata versions. Never silently 
substitute the latest version for a requested historical version. 
     - Handle empty tables and absent optional metadata explicitly. 
     - Respect ordinary append, primary-key, and Data Evolution table 
semantics. Show mode-specific metadata, such as row-ID ranges and written 
columns, when available. 
    
     The intended debugging workflow is: 
    
     ```text 
     Snapshot 42 ----------------------------> Schema 3 
         | 
         +--> Base / delta manifest lists 
                   | 
                   +--> Manifest files 
                            | 
                            +--> ADD / DELETE entries 
                                     | 
                                     +--> Data-file path and statistics 
                                     +--> File schema ID --> Schema version 
     ```
   
   
     
   
   ### Anything else?
   
     Related capabilities: 
    
     - [PyPaimon 
CLI](https://paimon.apache.org/docs/master/pypaimon/cli/#basic-usage) 
     - [Existing schema and snapshot 
inspection](https://paimon.apache.org/docs/master/pypaimon/cli-query/#table-get)
 
     - [PyPaimon system 
tables](https://paimon.apache.org/docs/master/pypaimon/system-tables/), 
including `$snapshots`, `$schemas`, `$manifests`, and `$files`.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to