shangxinli commented on issue #18135: URL: https://github.com/apache/hudi/issues/18135#issuecomment-6428106081
Thanks for the detailed write-up — the registry-file approach is a real improvement over what I have on the branch in one respect: it keeps the metadata table's FILES schema untouched, where my current implementation (shangxinli/hudi, branch feat/register-only-bootstrap, PR #20333 for the metadata piece) adds an optional sourceBasePath field directly onto HoodieMetadataFileInfo. Yours is coarser (one base path per partition per snapshot, not per file) but that's a fine tradeoff if we only need one external path per partition, and it sidesteps the schema-change sign-off we're waiting on for #20333. Worth deciding which of the two we standardize on before going further, since they're solving the same problem two different ways. A few things that are already built and tested on my branch, so we don't re-litigate them: - Table property: already have hoodie.bootstrap.has.register.only.partitions (boolean), used by both the read-path gate and the write guard. Happy to rename/fold it into whatever we land on for open question 1, but the mechanism exists. - Open question 2 (partition categories): yes, we need three in one table, not two. DateBasedBootstrapModeSelector already assigns FULL_RECORD / METADATA_ONLY / REGISTER_ONLY per partition by age within a single bootstrap, and TestRegisterOnlyBootstrapRead covers SELECT * across all three tiers in one table, including predicate pushdown onto the cold tier. A table that can only hold registered + native partitions won't cover the onboarding case this was filed for. - R12 (insert_overwrite into sealed partitions): this already works on my branch today, not something to design for. RegisterOnlyPartitionGuard explicitly lets INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, and DELETE_PARTITION through untouched — that's the promotion path, by design — and it's covered by a green test. I undersold this in my Oct 23 comment; sorry for the confusion. Let me know if it'd help to get on a call to reconcile the two branches rather than doing this over comments — a good chunk of R1-R11 is already implemented and tested on my side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
