suxiaogang223 opened a new pull request, #60556:
URL: https://github.com/apache/doris/pull/60556
### What problem does this PR solve?
## Summary
This PR refactors Paimon system tables (e.g., `table$snapshots`,
`table$binlog`) to use the native table execution path (FileQueryScanNode)
instead of the TVF path (MetadataScanNode). This provides a unified execution
path with regular tables and better integration with query optimization.
## Changes
### 1. New Type Hierarchy for System Tables
Introduced a cleaner type hierarchy for system table types:
```
SysTableType (abstract base)
├── NativeSysTableType (abstract) # Native execution path
│ └── PaimonSysTable # All Paimon system tables
│
└── TvfSysTableType (abstract) # TVF execution path
├── IcebergSysTable # Iceberg metadata tables
└── PartitionsSysTable # Hive partition tables
```
### 2. Native Table Path for Paimon System Tables
- Created `PaimonSysExternalTable` that wraps a `PaimonExternalTable` and
provides access to system table data
- Modified `PaimonScanNode` to handle non-DataSplit splits from system tables
- Added `paimon_sys_table_type` parameter to Thrift and JNI reader for
system table identification
### 3. Removed `paimon_meta` TVF
- Deleted `PaimonMeta.java` and `PaimonTableValuedFunction.java`
- Paimon system tables now use native table path instead of TVF
### 4. O(1) Lookup Optimization
Changed system table type storage from `List` to `Map<String, SysTableType>`
for O(1) lookup:
```java
// Before - O(n) iteration
for (SysTableType type : getSupportedSysTableTypes()) {
if (type.containsMetaTable(tableName)) ...
}
// After - O(1) map lookup
getSupportedSysTableTypes().get(sysTableName)
```
### 5. Removed `SupportedSysTables` Utility Class
Each `SysTableType` subclass now maintains its own static map:
- `PaimonSysTable.SUPPORTED_SYS_TABLES`
- `IcebergSysTable.SUPPORTED_SYS_TABLES`
- `PartitionsSysTable.HIVE_SUPPORTED_SYS_TABLES`
## Files Changed
### New Files
| File | Description |
|------|-------------|
| `NativeSysTableType.java` | Abstract base for native path system tables |
| `TvfSysTableType.java` | Abstract base for TVF path system tables |
| `PaimonSysExternalTable.java` | Paimon system table wrapper |
### Modified Files
| File | Description |
|------|-------------|
| `SysTableType.java` | Simplified base class (renamed from `SysTable.java`)
|
| `PaimonSysTable.java` | Extends `NativeSysTableType`, uses Map |
| `IcebergSysTable.java` | Extends `TvfSysTableType`, uses Map |
| `PartitionsSysTable.java` | Extends `TvfSysTableType`, uses Map |
| `PaimonScanNode.java` | Handles system table splits |
| `PaimonSplit.java` | Unified split handling |
| `BindRelation.java` | Dispatches to native/TVF path |
| `DescribeCommand.java` | Supports describe for native system tables |
| `TableIf.java` | Returns `Map<String, SysTableType>`, O(1) lookup |
| `paimon_jni_reader.cpp` | Reads system table type parameter |
| `PlanNodes.thrift` | Added `paimon_sys_table_type` field |
### Deleted Files
| File | Description |
|------|-------------|
| `PaimonMeta.java` | TVF implementation |
| `PaimonTableValuedFunction.java` | TVF implementation |
| `SupportedSysTables.java` | Centralized list (moved to subclasses) |
## Query Examples
```sql
-- These queries now use native table path (PaimonScanNode)
SELECT * FROM paimon_catalog.db.table$snapshots;
SELECT * FROM paimon_catalog.db.table$binlog;
SELECT * FROM paimon_catalog.db.table$audit_log;
-- Iceberg still uses TVF path (unchanged)
SELECT * FROM iceberg_catalog.db.table$snapshots;
```
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
