nevzheng commented on PR #12257:
URL: https://github.com/apache/gravitino/pull/12257#issuecomment-5172527222
@jerryshao
# PR #12257 Review Responses
## Draft response to Jerry — scope and API focus
Jerry, thank you for the review. I heard three related concerns. Please
correct me if I
have misunderstood any of them.
### A. Scope of the deletion model
**What I heard:** The model appears general. Is that intentional, and if so,
should it cover
other Gravitino entity types too?
**Response:** The model is extensible, but V1 is intentionally narrow: it
supports only
`ICEBERG_TABLE` objects through IRC. It defines retention, restore, and
purge for that case;
it does not enable other entity types or catalog backends. I support
expanding later, but each
type needs its own design for source-system semantics, recovery guarantees,
and purge jobs.
I chose this shape because it extends the established
[Iceberg REST asynchronous hard-delete
design](async-iceberg-rest-hard-deletion.md) authored
by Rory. We add a recoverable deletion action ahead of its cleanup path: the
action owns
retention, recovery, and audit; the existing `iceberg_cleanup_job` owns
execution, retries,
and worker progress after expiry. This does not require `table_meta` to be
the source of truth.
### B. IRC-only versus broader catalog scope
**What I heard:** Why not make this IRC-only, like the existing asynchronous
hard-delete work?
Soft deletion for a source catalog without equivalent semantics—for example
Hive—could make
an object invisible through Gravitino while it remains visible in the source
catalog. The
first version should therefore be scoped to the IRC/Iceberg use case,
analogous to the
existing asynchronous hard-deletion feature. It does not define soft-delete
behavior for
Hive, JDBC, or other non-Iceberg catalog types.
### C. API focus and IRC compatibility
**What I heard:** The proposal currently describes the native Gravitino API
more than the IRC
API. Do we intend to add `UNDROP` to IRC, or should this be a Gravitino API
feature?
**Response:** Great question. I re-read Mark's PRD: it asks for the Iceberg
IRC
`/management/...` API (A), so we can meet the immediate requirement through
that route. The
upstream IRC specification does not define `UNDROP`, so it would be a
documented Gravitino
extension.
**B is also a valid option.** I believe soft deletion in the general
Gravitino API is valuable:
it gives users one predictable recovery experience across metadata types and
avoids creating a
parallel Iceberg-only recovery surface that we would later duplicate. This
is primarily a product
and UX decision, not just an implementation choice.
If we choose B, we should roll it out carefully: Iceberg first, then enable
each additional type
only after defining its restore and purge semantics, implementing its
handler, documenting its
limits, and adding targeted tests. Both options should call the same
deletion record and control
plane. I recommend B as the longer-term direction for those reasons.
I do not want to block the Iceberg work on choosing B. We can carry both
options through review
and resolve the public API shape there.
## Next steps
1. **Decide the API shape — @jerryshao @roryqi @lasdf1234 and Mark:** Mark's
PRD advocates an
Iceberg IRC `/management/...` API (A). Nevin is asking whether we should
instead establish a
generalized Gravitino soft-delete/purge API (B), with Iceberg as the
first supported type and
a staged rollout for additional entity types. For now, carry both API
shapes against the same
internal metadata model and control plane; select the initial public API
as this review
resolves. Nevin favors B, provided each type adds its own restore and
purge semantics before
it is enabled. (todo(team))
2. Add an appendix that explains the upstream IRC surface, the two
management API choices, and
their shared deletion record and control plane. (done)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
