link3280 commented on issue #2635:
URL: https://github.com/apache/amoro/issues/2635#issuecomment-2049054887
I propose to align `PartitionBaseInfo` with the iceberg partition metadata
table, which contains the following columns:
```
+-------------------------------+--------------------+--------------------------------------------+--+
| col_name | data_type |
comment |
+-------------------------------+--------------------+--------------------------------------------+--+
| partition | struct<dt:string> |
|
| spec_id | int |
|
| record_count | bigint | Count of records in
data files |
| file_count | int | Count of data files
|
| position_delete_record_count | bigint | Count of records in
position delete files |
| position_delete_file_count | int | Count of position
delete files |
| equality_delete_record_count | bigint | Count of records in
equality delete files |
| equality_delete_file_count | int | Count of equality
delete files |
+-------------------------------+--------------------+--------------------------------------------+--+
```
That would fix the performance issue because we don't have to iterate over
all the entries to count files. However, the downside is that we have to drop
the commit time and the storage size at the partition level which are
calculated based on the entries.
@majin1102 @zhoujinsong @baiyangtx WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
