vaibhavk1992 commented on PR #933:
URL: https://github.com/apache/incubator-xtable/pull/933#issuecomment-5743299051
@vinishjail97 I read the RFC and here are my thoughts.
1. What happens if the index build fails mid-sync? The RFC says index
updates must
finish before the commit closes, but also says a failed build just
becomes
"unavailable to lookup." Those read a little differently — does a
broken index
build actually fail the sync, or does the sync go through and the index
just
quietly falls behind?
2. What about turning this on for an existing table? If someone enables
record-index
on a table with years of history, that first build sounds like a full
scan.
Would be good to know if that's resumable, or if a hiccup partway
through means
starting over.
3. How do we retract stale secondary-index entries? To update value → keys
when a
row's column value changes, we'd need the old value to remove the old
mapping.
Doesn't look like the RFC says where that old value comes from during
an
incremental sync — probably just needs a sentence.
4. Might secondary-index updates cost more than the record-index ones?
Record-index
feels naturally scoped to the files that changed. Secondary-index on a
low-cardinality column could touch a lot more of the metadata table per
sync.
Worth asking for a rough sense of cost here, since #887 specifically
wanted a
cost/opt-out story.
5. Any race between a sync and a concurrent lookup? The test plan mentions
testing
this, but the RFC doesn't spell out what actually guards against it.
6. Does this change how often Hudi's own metadata table compacts? More
writes into
the MDT every sync could mean more frequent compaction — curious if
that's been
thought about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
