ealeonraz commented on issue #12020:
URL: https://github.com/apache/gravitino/issues/12020#issuecomment-5383319785
Status update: implementation is underway on top of current main, accounting
for the new `Coherence` SPI and the two-boundary drop from #12416 (the Redis
drop mirrors both the `.` pass and the hierarchical schema separator pass).
Design summary against the earlier checklist:
**Keyspace and Cluster safety.** Keys are
`ns:{metalake}:D:<identifier>:<TYPE>` for values,
`ns:{metalake}:F:<identifier>` for version fences, and `ns:{metalake}:IDX` for
a lex-ordered index ZSet. The hash tag is the metalake name, so all keys of one
metalake share a slot and every multi-key Lua operation is CROSSSLOT-safe at
any drop depth. The explicit tradeoff: one metalake's cache is bounded by a
single Cluster node. I want to confirm you are fine with that granularity
before I finalize, the alternative is schema-level tags with fence-only
cross-slot drops.
**Drops and stale writes.** A hierarchical drop is a single atomic fence
write (a persistent version marker per container) plus lazy cleanup of the
subtree, so readers never observe a torn drop and a late in-flight load cannot
refill a dropped key: every write goes through a version-checked `putIfNewer`
that compares against ancestor fences. Fences outlive the value TTL and cover
rename by fencing both identifiers. Versions come from Redis itself, not server
clocks.
**Write ordering and failure.** The cache is only touched after the store
commit, and the post-commit step is invalidation (delete plus fence), never
update-in-place, so a Redis failure degrades to a cache miss rather than a
stale hit. A crash between commit and invalidation leaves staleness bounded by
the value TTL; I will document that explicitly as best-effort, consistent with
the agreed shared-cache scope.
Three questions before the PR:
1. The key scheme assumes a name component never contains `.`, which holds
for `DefaultCapability`'s name pattern but is overridable per catalog.
Acceptable assumption, or should the key encode identifiers?
2. New dependencies: Jedis (MIT) as the client and Kryo (BSD-3) for value
serialization, since several Entity implementations are not Java-serializable
in practice. Any preference here, for example Lettuce?
3. Delivery: one PR, or the cache implementation first with the
Cluster-topology integration tests as a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
