merrily01 commented on code in PR #2435:
URL: https://github.com/apache/auron/pull/2435#discussion_r3672907790
##########
thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala:
##########
@@ -700,39 +742,45 @@ class AuronIcebergIntegrationSuite
}
}
- test("iceberg changelog scan falls back when delete changes exist") {
- withTable("local.db.t_changelog_delete") {
- withTempView("t_changelog_delete_changes") {
+ test("iceberg native scan supports mixed insert and full-data-file delete
changelog scan") {
Review Comment:
This replaces the previous `falls back when delete changes exist` test,
which covered row-level deletes. Could we keep a dedicated fallback test (e.g.
non-partitioned table with a position delete, or a DeletedDataFileScanTask with
non-empty existingDeletes())? It would help guard against regressions on the
row-level delete / update paths.
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala:
##########
@@ -559,6 +557,23 @@ object IcebergScanSupport extends Logging {
}
}
+ private def toNativeChangelogDataFileTask(
+ task: ChangelogScanTask): Option[NativeChangelogDataFileTask] = {
+ task match {
+ case added: AddedRowsScanTask
+ if added.operation() == ChangelogOperation.INSERT &&
+ deletesEmpty(added.deletes()) =>
+ Some(NativeChangelogDataFileTask(added.file(), added.start(),
added.length(), added))
+ case deleted: DeletedDataFileScanTask
+ if deleted.operation() == ChangelogOperation.DELETE &&
Review Comment:
Nit: deleted.operation() == ChangelogOperation.DELETE is always true. How
about dropping it or adding a comment for clarity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
