PavithranRick opened a new pull request, #17463:
URL: https://github.com/apache/hudi/pull/17463
### Describe the issue this Pull Request addresses
The pull request adds a new procedure, `show_file_group_history`, to provide
comprehensive tracking and debugging capabilities for Hudi file groups across
their entire lifecycle. This procedure allows users to analyze commits, clean
and rollback operations, compactions, clustering, and regular inserts/updates
with detailed metadata and statistics. It supports advanced filtering and is
optimized for large timelines.
### Summary and Changelog
**Overview**
The `show_file_group_history` procedure provides detailed tracking for Hudi
file groups, including operation types, deletion and replacement tracking,
statistics, and performance metrics.
**Features**
- **Complete Operation Tracking**
- Deletion Operations: Clean and Rollback
- Replacement Operations: Compaction and Clustering
- Standard Operations: Inserts, Updates, Commits
- Timeline Coverage: Active and archived timelines
- **Advanced Filtering**
- SQL-like expressions
- Partition-based filtering (Hive-style syntax)
- Operation-specific filtering
- Time-range and state-based filtering
- **Performance Optimizations**
- Limit-aware timeline processing
- Smart partition filtering
- Efficient metadata parsing
- Scalable design for production
**Syntax**
CALL show_file_group_history(
table => 'table_name' | path => '/path/to/table',
file_group_id => 'file_group_id',
[partition => 'partition_filter'],
[show_archived => true|false],
[limit => number],
[filter => 'filter_expression']
)
**Parameters**
Parameter | Type | Required | Default | Description
---------------|---------|----------|---------|-------------
table | String | Yes* | - | Hudi table name (either
table or path required)
path | String | Yes* | - | Hudi table path (either
table or path required)
file_group_id | String | Yes | - | Target file group ID to
analyze
partition | String | No | null | Partition filter (e.g.,
category=electronics)
show_archived | Boolean | No | false | Include archived timeline in
analysis
limit | Integer | No | 100 | Maximum number of results to
return
filter | String | No | null | SQL-like filter expression
for results
**Output Schema**
25-column result set with detailed file group information, including:
- **Core Instant Information:** instant_time, completion_time, action,
timeline_type, state
- **File and Partition Details:** partition_path, file_name, operation_type
- **Statistics and Metrics:** num_writes, num_inserts, num_updates,
num_deletes, file_size_bytes, total_write_bytes, prev_commit
- **Deletion Tracking:** was_deleted, delete_action, delete_instant
- **Replacement Tracking:** was_replaced, replace_action, replace_instant
- **Performance Metrics:** total_write_errors, total_scan_time_ms,
total_upsert_time_ms, total_create_time_ms
- **Additional Information:** prev_base_file, column_stats_available
**Example Usage**
CALL show_file_group_history(
table => 'my_hudi_table',
file_group_id => '00778190-0304-4bff-8cd3-18e85ed482bc-0',
partition => 'category=electronics',
show_archived => true,
limit => 50
);
### Impact
**Public API Changes**
- New Procedure: show_file_group_history
- 25-column output including operation, file, deletion, replacement, and
performance metrics
- Provides powerful debugging tools for Hudi table maintenance
**Performance Impact**
- Limit-aware processing avoids unnecessary timeline loading
- Smart partition filtering reduces scanning overhead
- Handles large archived timelines efficiently
- Shared utility functions reduce maintenance overhead
### Risk Level
low
### Documentation Update
Hudi website documentation needs to include show_file_group_history in the
procedures section with usage examples.
### Contributor's checklist
- [ ] Read through contributor's guide:
https://hudi.apache.org/contribute/how-to-contribute
- [ ] Change Logs and Impact stated clearly
- [ ] Adequate tests added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
