weimingdiit commented on code in PR #2435:
URL: https://github.com/apache/auron/pull/2435#discussion_r3682429775
##########
thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala:
##########
@@ -700,39 +742,45 @@ class AuronIcebergIntegrationSuite
}
}
- test("iceberg changelog scan falls back when delete changes exist") {
- withTable("local.db.t_changelog_delete") {
- withTempView("t_changelog_delete_changes") {
+ test("iceberg native scan supports mixed insert and full-data-file delete
changelog scan") {
Review Comment:
Thanks for pointing this out. I rechecked the behavior of the replaced test
and Iceberg 1.10.1.
The replaced test did not actually exercise a row-level delete. In the CI
run, the insert produced two data files and the DELETE removed one complete
data file, so it covered a DeletedDataFileScanTask with empty
existingDeletes(), which is the case this PR now makes native.
A genuine position-delete changelog case cannot currently reach Auron's
fallback guard. BaseIncrementalChangelogScan rejects any snapshot in the
changelog range that contains delete manifests before Auron receives the
planned tasks. In addition, Iceberg 1.10.1's CreateDataFileChangeTasks
constructs both AddedRowsScanTask and DeletedDataFileScanTask with NO_DELETES,
so a DeletedDataFileScanTask with non-empty existingDeletes() cannot be
produced through the public changelog planner.
The existing "iceberg scan falls back when delete files exist" test still
covers position-delete fallback for regular Iceberg scans. It explicitly
creates a position delete, verifies that the planned FileScanTask contains
deletes, and checks that NativeIcebergTableScan is not used.
Given these limitations, I have not added a changelog integration test that
would either fail inside Iceberg before reaching Auron or not exercise the
intended guard. If useful, I can add a focused synthetic task-level test for
the existingDeletes() guard instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
