wgtmac commented on code in PR #1104:
URL: https://github.com/apache/iceberg-cpp/pull/1104#discussion_r4608433768
##########
src/iceberg/transaction.cc:
##########
@@ -99,6 +104,60 @@ std::string FormatCommittedSnapshots(
return detail;
}
+/// Collect files that must survive transaction finalization.
+///
+/// The set includes manifests and manifest lists from all snapshots created by
+/// this transaction. It also includes owned content files still referenced by
+/// retained delete manifests.
+///
+/// \param metadata The metadata after the transaction commit.
+/// \param io FileIO used to read manifest lists and delete manifests.
+/// \param snapshot_ids Snapshot IDs in the committed AddSnapshot changes.
+/// \param content_paths Owned files whose live references have not been found.
+Result<std::unordered_set<std::string>> CommittedTransactionFiles(
+ const TableMetadata& metadata, const std::shared_ptr<FileIO>& io,
+ std::unordered_set<int64_t> snapshot_ids,
+ std::unordered_set<std::string> content_paths) {
+ std::unordered_set<std::string> committed;
+ for (const auto& snapshot : metadata.snapshots) {
Review Comment:
I assume you mean `snapshots` for `metadata`? I believe we are already doing
this in the line below by skipping existing snapshots:
```
if (!snapshot || snapshot_ids.erase(snapshot->snapshot_id) == 0) {
continue;
}
```
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -918,6 +918,8 @@ Result<ExpireSnapshots::SnapshotToRef>
ExpireSnapshots::ComputeRetainedRefs(
}
Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
+ // A no-op apply must not leave an earlier cleanup plan active.
+ apply_result_.reset();
Review Comment:
Agreed. Current behavior is correct but does look ugly. Let me change this.
##########
src/iceberg/update/rewrite_manifests.cc:
##########
@@ -311,11 +326,20 @@ Status RewriteManifests::ValidateDeletedManifests(
int64_t current_snapshot_id) const {
// Directly deleted manifests must still be present in the current snapshot.
for (const auto& manifest : deleted_manifests_) {
- if (!current_manifests.contains(manifest)) {
+ auto current = current_manifests.find(manifest);
+ if (current == current_manifests.end()) {
return ValidationFailed(
"Deleted manifest {} could not be found in the latest snapshot {}",
manifest.manifest_path, current_snapshot_id);
}
+ // A prior update may reuse this path with new inherited values on replay.
+ // The caller's replacement still contains the original values.
+ if (manifest.sequence_number != current->sequence_number ||
+ manifest.min_sequence_number != current->min_sequence_number ||
+ manifest.first_row_id != current->first_row_id) {
Review Comment:
Those changes are exactly what this validation catches: the caller-provided
replacement contains the old inherited values, so replay must reject it. A
metadata-only upgrade leaves existing manifest lists unchanged, but a
subsequent v3 snapshot can assign first_row_id. I added a regression where only
`first_row_id` changes.
##########
src/iceberg/update/snapshot_update.cc:
##########
@@ -498,26 +502,25 @@ void SnapshotUpdate::RegisterStagedFile(const
std::string& path) {
staged_files_.insert(path);
}
-void SnapshotUpdate::UnregisterStagedFile(const std::string& path) {
+void SnapshotUpdate::RegisterStagedContentFile(const std::string& path) {
Review Comment:
Yes. Content files are also staged files. `staged_files_` tracks cleanup
ownership, while `staged_content_files_` identifies the subset whose live
references must be checked before cleanup.
##########
src/iceberg/transaction.cc:
##########
@@ -556,7 +649,6 @@ Result<std::shared_ptr<Table>> Transaction::CommitOnce(bool
is_first_attempt) {
if (metadata_changed || standalone_retry) {
ICEBERG_CHECK(CanRetry(),
"Cannot rebase a transaction containing a non-retryable
update");
- CleanupUpdates();
Review Comment:
Cleanup is deferred until the commit status is known: successful commits
remove unreferenced staged files, and known failures or explicit aborts clean
all owned staging.
Unknown status preserves files from all attempts because the commit may have
succeeded, so this can retain more orphan files than before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]