RussellSpitzer commented on code in PR #15150:
URL: https://github.com/apache/iceberg/pull/15150#discussion_r2957432916
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -1695,10 +1698,54 @@ public void
testSortCustomSortOrderRequiresRepartition() {
shouldHaveMultipleFiles(table);
shouldHaveLastCommitUnsorted(table, "c2");
shouldHaveLastCommitSorted(table, "c3");
+ dataFilesShouldHaveSortOrderIdMatching(table, SortOrder.unsorted());
}
@TestTemplate
- public void testAutoSortShuffleOutput() {
+ public void testSortPastTableSortOrderGetsAppliedToFiles() throws
IOException {
+ int partitions = 4;
+ Table table = createTable();
+ writeRecords(20, SCALE, partitions);
+ shouldHaveLastCommitUnsorted(table, "c3");
+
+ table.updateSpec().addField("c1").commit();
+
+ table.replaceSortOrder().asc("c3").commit();
+ SortOrder c3SortOrder = table.sortOrder();
+
+ table.replaceSortOrder().asc("c2").commit();
+ shouldHaveFiles(table, 20);
+
+ List<Object[]> originalData = currentData();
+ long dataSizeBefore = testDataSize(table);
+
+ RewriteDataFiles.Result result =
+ basicRewrite(table)
+ .sort(SortOrder.builderFor(table.schema()).asc("c3").build())
+ .option(SizeBasedFileRewritePlanner.REWRITE_ALL, "true")
+ .option(
+ RewriteDataFiles.TARGET_FILE_SIZE_BYTES,
Review Comment:
as a minor speedup to this test you don't need to mess with target file
size. We do this in the other tests because we want to make sure there are
multiple files, but in this test we only care about the sort order being set.
So theoretically you could just make it a single file compaction and just check
that it's properly labeled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
