RussellSpitzer commented on code in PR #4759:
URL: https://github.com/apache/iceberg/pull/4759#discussion_r902794468
##########
core/src/main/java/org/apache/iceberg/actions/SortStrategy.java:
##########
@@ -19,27 +19,57 @@
package org.apache.iceberg.actions;
+import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
import org.apache.iceberg.SortOrder;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.FluentIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.util.BinPacking;
+import org.apache.iceberg.util.BinPacking.ListPacker;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SortStrategyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
/**
* A rewrite strategy for data files which aims to reorder data with data
files to optimally lay them out
* in relation to a column. For example, if the Sort strategy is used on a set
of files which is ordered
- * by column x and original has files File A (x: 0 - 50), File B ( x: 10 - 40)
and File C ( x: 30 - 60),
- * this Strategy will attempt to rewrite those files into File A' (x: 0-20),
File B' (x: 21 - 40),
- * File C' (x: 41 - 60).
+ * by column x and original has files File A (x: 0 - 50), File B ( x: 10 -
40), File C ( x: 30 - 60),
+ * File D ( x: 61 - 80), and File E ( x: 81 - 100), this Strategy will attempt
to rewrite files A, B and C
+ * into File A' (x: 0-20), File B' (x: 21 - 40), File C' (x: 41 - 60) and keep
File D and E unchanged.
* <p>
- * Currently the there is no file overlap detection and we will rewrite all
files if {@link SortStrategy#REWRITE_ALL}
- * is true (default: false). If this property is disabled any files that would
be chosen by
- * {@link BinPackStrategy} will be rewrite candidates.
+ * When the {@link BinPackStrategy#REWRITE_ALL} flag is, all files are
selected for rewrite. Otherwise,
+ * only unsorted files are selected. Rewrite will be applied to those selected
files if
+ * 1. There are a certain number of mis-sized data files or
+ * 2. Those data files do not have sortedness score good enough.
Review Comment:
"Those files do not have a good enough sortedness score"
Also there is an issue here as "sortedness score" is not defined in the this
doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
