deniskuzZ commented on code in PR #4855:
URL: https://github.com/apache/hive/pull/4855#discussion_r1402248724
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -469,12 +474,34 @@ private Long getSnapshotId(Table table, String
branchName) {
/**
* Creates and commits an Iceberg change with the provided data and delete
files.
* If there are no delete files then an Iceberg 'append' is created,
otherwise Iceberg 'overwrite' is created.
- * @param table The table we are changing
- * @param startTime The start time of the commit - used only for logging
- * @param results The object containing the new files we would like to add
to the table
+ *
+ * @param table The table we are changing
+ * @param startTime The start time of the commit - used only for logging
+ * @param results The object containing the new files we would like to
add to the table
+ * @param isCompaction Flag that indicates whether or not the operation is
done as part of compaction
*/
private void commitWrite(Table table, String branchName, Long snapshotId,
long startTime,
- FilesForCommit results, Operation operation) {
+ FilesForCommit results, Operation operation,
boolean isCompaction) {
+
+ if (isCompaction) {
Review Comment:
Could you please extract this logic into the existing `commitOverwrite`
method with extra param `rewritePolicy`
In case of MAJOR compaction set `rewritePolicy`=`Context.ALL_PARTITIONS`
In future, we might map new DDL operation `OPTIMIZE TABLE REWRITE DATA` to
that logic in order to fix issues with partition evolution + rewrite small
files.
````
public class Context {
....
public enum RewriteDataPolicy {DEFAULT, ALL_PARTITIONS}
....
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
