Reo-LEI commented on a change in pull request #3120:
URL: https://github.com/apache/iceberg/pull/3120#discussion_r719302947
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -123,6 +123,9 @@ private TableProperties() {
public static final String SPLIT_OPEN_FILE_COST =
"read.split.open-file-cost";
public static final long SPLIT_OPEN_FILE_COST_DEFAULT = 4 * 1024 * 1024; //
4MB
+ public static final String READ_DELETE_FILES_WORKER_POOL_SIZE =
"read.deletes.num-threads";
+ public static final int READ_DELETE_FILES_WORKER_POOL_SIZE_DEFAULT = 1; //
read delete files in serial.
Review comment:
I admit that this configuration may not be appropriate as a table
property. But I think this optimization should apply to all engines, not only
flink. Such as I will sync mysql cdc data to iceberg by flink and rewrite and
merge the delete files into data files by spark.
Maybe we chould config this through `SystemProperties` like we config
`iceberg.worker.num-threads`(https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/SystemProperties.java#L35).
And stop propagate the 'tableProperties'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
