vinothchandar commented on issue #19094:
URL: https://github.com/apache/hudi/issues/19094#issuecomment-5940664885
@chrevanthreddy We organized everything as sub-issues now.. and got the PRs
linked. I filed an issue for Ethan to look at my comment on 19802.
@vinishjail97 will share a review of 19802 and some testing results he has..
and also few fixes he landed on top during tsting.
Ask from you is
- close sub-issues that are not relevant..
- Link all sub-issues fixed by a PR using `resolves #issue1 #issue2` etc in
the PR body, so it gets automatically linked in our view here.
Here's what AI thinks as status of sub-issues. tell us if this is
accurate...
I went through #20107 and #19802 and mapped them against the sub-issues
here, so we know what's actually left. Summary: the two PRs together cover
bootstrap, DDL,
and the whole read path. What's missing is the generation lifecycle and
everything under maintenance.
One note on how I'd describe #20107: it's more than refactoring. It
registers `MetadataPartitionType.VECTOR_INDEX`, changes `HoodieMetadata.avsc`
and `HoodieMetadataPayload`, and adds the `CREATE INDEX ... USING vector`
DDL, so bootstrap and search are reachable once it lands. So its more like the
core storage changes? lets change PR title if you agree as well
| # | Sub-task | Where it lands | Notes |
|---|---|---|---|
| #19095 | RFC | done | merged as `eccfee6dfe7` |
| #19096 | Index options + Spark DDL | #20107 | `VectorIndexOptions`,
`IndexCommands.scala` |
| #19097 | MDT row families + payload schema | #20107, rest #19802 |
`bootstrapBaseline` / `lastContiguousSourceInstant` come in with #19802 |
| #19098 | RaBitQ encoder + multibit scoring | #20107 | `RaBitQEncoder`,
`RaBitQByteLutScorer`, `RaBitQPlaneKernel` |
| #19099 | Spark bootstrap, IVF clustering, postings | #20107 |
`TwoLevelKMeansBootstrap`, `SparkVectorIndexBootstrap` |
| #19100 | Generation visibility + reader consistency | **new work** |
reader side is there (#20107 only serves `ACTIVE`), writer side isn't —
bootstrap hardcodes
generation 1 and writes `ACTIVE` directly, no max+1, no `BUILDING` →
`ACTIVE`, no `T_cut` gate |
| #19101 | Query planning, centroids, probing | #20107 |
`HoodieVectorSearchPlanBuilder`, `VectorIndexPruner` |
| #19102 | Approximate top-K | #20107, **partial** | scan, scoring,
arbitration, tombstones, overlay slack all there; continuation/refill retaining
`maxRerankCandidates` isn't |
| #19103 | Exact rerank | #20107, **partial** | planner, positional reads,
MoR log fetch, RLI batching there; continuation through the budgets isn't |
| #19104 | Drop, rebuild, cleanup | **new work** | no `dropIndex` path in
either PR, `buildClean` is a no-op in both |
| #19105 | Test coverage | both, **partial** | ~30 new test classes, but
codecov has #20107 patch coverage at 34.7% |
| #19498 | Incremental maintenance | #19802 | `SparkVectorIndexUpdater`,
posting deltas, additive `ClusterStats`, markers in the same commit |
| #19499 | Freshness frontier + catch-up replay | #19802, **partial** |
markers, frontier and the planner `FAIL`/`WARN`/`FALLBACK` policy are in;
replay isn't
written yet, and nothing checks continuity across archived instants |
| #19500 | LIRE split/merge | **new work** | three doc comments mention
it, no code |
| #19501 | Tier-1 compaction + locator refresh | **new work** | #19802
lands the `deltaCount` / `tombstoneCount` counters, nothing consumes them |
| #18850–#18857 | the older `[Vector Index]` set | superseded | these
duplicate #19096–#19099 almost exactly; I'll close them unless you want them
kept |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
