aokolnychyi commented on a change in pull request #2501:
URL: https://github.com/apache/iceberg/pull/2501#discussion_r627034911
##########
File path: api/src/main/java/org/apache/iceberg/actions/RewriteDataFiles.java
##########
@@ -19,92 +19,120 @@
package org.apache.iceberg.actions;
-import org.apache.iceberg.DataFile;
+import java.util.Map;
+import org.apache.iceberg.StructLike;
import org.apache.iceberg.expressions.Expression;
/**
- * An action that rewrites data files.
+ * An action for rewriting datafiles according to a Rewrite Strategy.
Generally used for
+ * optimizing the sizing and layout of datafiles within a table.
*/
-public interface RewriteDataFiles extends SnapshotUpdate<RewriteDataFiles,
RewriteDataFiles.Result> {
+public interface RewriteDataFiles extends Action<RewriteDataFiles,
RewriteDataFiles.Result> {
+
/**
- * Pass a row filter to filter {@link DataFile}s to be rewritten.
- * <p>
- * Note that all files that may contain data matching the filter may be
rewritten.
- * <p>
- * If not set, all files will be rewritten.
- *
- * @param expr a row filter to filter out data files
- * @return this for method chaining
+ * Enable committing groups of files (see max-file-group-size) prior to the
entire compaction completing.
+ * This will produce additional commits but allow for progress even if some
groups fail to commit. This setting
+ * will not change the correctness of the rewrite operation. The default is
false, which produces a single commit
+ * when the entire job has completed.
*/
- RewriteDataFiles filter(Expression expr);
+ String PARTIAL_PROGRESS_ENABLED = "partial-progress.enabled";
+ boolean PARTIAL_PROGRESS_ENABLED_DEFAULT = false;
/**
- * Enables or disables case sensitive expression binding.
- * <p>
- * If not set, defaults to false.
- *
- * @param caseSensitive caseSensitive
- * @return this for method chaining
+ * The maximum amount of Iceberg commits that compaction is allowed to
produce if partial progress is enabled.
*/
- RewriteDataFiles caseSensitive(boolean caseSensitive);
+ String PARTIAL_PROGRESS_MAX_COMMITS = "partial-progress.max-commits";
+ int PARTIAL_PROGRESS_MAX_COMMITS_DEFAULT = 10;
/**
- * Pass a PartitionSpec id to specify which PartitionSpec should be used in
DataFile rewrite
- * <p>
- * If not set, defaults to the table's default spec ID.
- *
- * @param specId PartitionSpec id to rewrite
- * @return this for method chaining
+ * The entire compaction operation is broken down into pieces based on
partitioning and within partitions based
+ * on size into groups. These sub-units of compaction are referred to as
file groups. The largest amount of data that
+ * should be compacted in a single group is controlled by
MAX_FILE_GROUP_SIZE_BYTES. When grouping files, the
+ * underlying compaction strategy will use this value as to limit the files
which will be included in a single file
+ * group. A group will be processed by a single framework "action". For
example, in Spark this means that each group
+ * would be rewritten in its own Spark action. A group will never contain
files for multiple output partitions.
Review comment:
Shall we also mention the files are split into groups for scalability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
