vamshikrishnakyatham opened a new pull request, #13745:
URL: https://github.com/apache/hudi/pull/13745
### Change Logs
# Hudi File Group History Procedure
## Overview
The `show_file_group_history` procedure provides comprehensive tracking and
debugging capabilities for Hudi file groups across their entire lifecycle.
## Features
### Complete Operation Tracking
- **Deletion Operations**: Clean and Rollback operations with detailed
tracking
- **Replacement Operations**: Compaction and Clustering operations with
comprehensive metadata
- **Standard Operations**: Regular commits, updates, and inserts with full
statistics
- **Timeline Coverage**: Both active and archived timeline support
### Advanced Filtering
- SQL-like filter expressions for precise result filtering
- Partition-based filtering with Hive-style syntax
- Operation-specific filtering capabilities
- Time-range and state-based filtering
### Performance Optimizations
- Limit-aware timeline processing to prevent unnecessary data loading
- Smart partition filtering to reduce scanning overhead
- Efficient metadata parsing for large timelines
- Scalable design for production environments
## Syntax
```sql
CALL show_file_group_history(
table => 'table_name' | path => '/path/to/table',
file_group_id => 'file_group_id',
[partition => 'partition_filter'],
[show_archived => true|false],
[limit => number],
[filter => 'filter_expression']
)
```
### Parameters
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `table` | String | Yes* | - | Hudi table name (either `table` or `path`
required) |
| `path` | String | Yes* | - | Hudi table path (either `table` or `path`
required) |
| `file_group_id` | String | Yes | - | Target file group ID to analyze |
| `partition` | String | No | null | Partition filter (e.g.,
`category=electronics`) |
| `show_archived` | Boolean | No | false | Include archived timeline in
analysis |
| `limit` | Integer | No | 100 | Maximum number of results to return |
| `filter` | String | No | null | SQL-like filter expression for results |
## Output Schema
The procedure returns a 25-column result set with comprehensive file group
information:
### Core Instant Information
- `instant_time`: Timestamp of the operation
- `completion_time`: When the operation completed
- `action`: Type of action (commit, clean, rollback, replacecommit, etc.)
- `timeline_type`: Timeline classification (ACTIVE, ARCHIVED)
- `state`: Operation state (COMPLETED, INFLIGHT, REQUESTED)
### File and Partition Details
- `partition_path`: Partition location
- `file_name`: Full file name
- `operation_type`: Specific operation performed (INSERT, UPDATE, DELETE)
### Statistics and Metrics
- `num_writes`: Total number of writes
- `num_inserts`: Number of insert operations
- `num_updates`: Number of update operations
- `num_deletes`: Number of delete operations
- `file_size_bytes`: File size in bytes
- `total_write_bytes`: Total bytes written
- `prev_commit`: Previous commit timestamp
### Deletion Tracking
- `was_deleted`: Boolean indicating if file was deleted
- `delete_action`: Type of deletion operation (`clean`, `rollback`)
- `delete_instant`: Timestamp when file was deleted
### Replacement Tracking
- `was_replaced`: Boolean indicating if file was replaced
- `replace_action`: Type of replacement operation (`compaction`,
`clustering`)
- `replace_instant`: Timestamp when file was replaced
### Performance Metrics
- `total_write_errors`: Number of write errors encountered
- `total_scan_time_ms`: Time spent scanning (milliseconds)
- `total_upsert_time_ms`: Time spent on upsert operations (milliseconds)
- `total_create_time_ms`: Time spent creating files (milliseconds)
### Additional Information
- `prev_base_file`: Previous base file reference
- `column_stats_available`: Whether column statistics are available
### Impact
**Public API Changes:**
- **New Procedures Added**:
- `show_file_group_history(table, file_group_id, [partition],
[show_archived], [limit])`
- **Output Schema**: 27-column result set including instant details, file
statistics, operation metadata, and deletion, replacement tracking
- **User-Facing Features**: Powerful debugging tools for Hudi table
maintenance and troubleshooting
**Performance Impact:**
- **Positive Impact**: Limit-aware processing prevents loading entire
timelines unnecessarily
- **Optimized Scanning**: Smart partition filtering reduces data scanning
overhead
- **Scalable Design**: Handles large archived timelines efficiently through
batched processing
- **Code Efficiency**: Shared utility functions reduce maintenance overhead
and ensure consistency
### Risk level (write none, low medium or high below)
low
**Verification Done:**
- Comprehensive test suites covering basic functionality, partition
filtering, complex updates, various operations on CoW and MoR tables, archived
timeline access, and error handling
- All Scalastyle checks passed with proper Scala idiomatic code (no return
statements)
- No modifications to existing procedures or core Hudi functionality
- Read-only operations with no data modification capabilities
- Extensive validation of edge cases (non-existent file groups/files, empty
timelines, etc.)
- Proper Spark driver port binding configuration for test environments
- File selection logic validated for clean operation testing with
chronologically older files
### Documentation Update
**Required Documentation Updates:**
- Hudi website documentation needs to be updated to include both new
procedures in the procedures section
**Example Usage Documentation:**
```sql
-- File Group History Example
CALL show_file_group_history(
table => 'my_hudi_table',
file_group_id => '00778190-0304-4bff-8cd3-18e85ed482bc-0',
partition => 'category=electronics',
show_archived => true,
limit => 50
);
```
**Output** (just cut up to total_write_bytes here)
<img width="2292" height="163" alt="image"
src="https://github.com/user-attachments/assets/5992c802-2312-4d17-b5f5-b3381a5851bf"
/>
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed modify this to make it compatible for our changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]