corgy-w opened a new pull request, #10851:
URL: https://github.com/apache/seatunnel/pull/10851
### Purpose of this pull request
This pull request fixes the Iceberg sink `DROP_DATA` save mode
implementation.
Today `DROP_DATA` is mapped to `truncateTable()`, and the Iceberg
`truncateTable()` path uses `deleteFromRowFilter(alwaysTrue())`. That is
effectively a full-table `DELETE WHERE TRUE`, not a lightweight truncate. For
large Iceberg tables, this forces Iceberg to load table metadata, snapshots,
manifest lists, and manifests before the job starts so it can rewrite table
state for the delete commit. In practice this can make SeaTunnel OOM on the
sink side before any source records are processed.
This patch keeps the existing `DROP_DATA` behavior, but changes the
implementation strategy so it rebuilds the table with `dropTableWithoutPurge +
createTable` instead of expressing the operation as a full-table delete commit.
### Does this PR introduce _any_ user-facing change?
Yes.
For Iceberg sink users, `DROP_DATA` still clears the target table before
writing, but it no longer does so through Iceberg's full-table delete commit
path. This changes the internal execution path and avoids the startup-time
metadata scan that can trigger `GC overhead limit exceeded` on large Iceberg
tables.
### How was this patch tested?
- Added targeted unit coverage in `IcebergSaveModeHandlerTest`
- Tried to run:
- `./mvnw -pl seatunnel-connectors-v2/connector-iceberg
-Dtest=IcebergSaveModeHandlerTest test`
- In this local environment the build could not complete because Maven
dependency resolution is configured through
`http://192.168.0.129:8081/repository/maven-public/`, and resolving
`org.apache.seatunnel:connector-common:3.0.0-SNAPSHOT` failed with `502 Bad
Gateway`
- The code change itself was kept narrowly scoped to the Iceberg sink save
mode path and mirrors the same fix already applied and validated on the
internal branches
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
