prathik-onehouse commented on PR #20604: URL: https://github.com/apache/hudi/pull/20604#issuecomment-6962974470
Hi Siva, update on the RLI deferral (ENG-52213) The tests for defer.init without any threshold are now in this PR, 24 cases across the datasource, Spark SQL, structured streaming and HoodieStreamer, global + partitioned, v6 + v10, COW + MOR. All four assertions you listed. They pass on unmodified master, and flipping defer.init off fails them at the first assertion. I couldn't reproduce a writer where the deferral is broken. If you had a specific case in mind I'd rather add it than assume it's covered. Flink's the one gap and it rejects defer.init outright anyway. The restructure is built and working (separate branch, not raised yet). Gate moved into initializeIfNeeded, IndexerFactory back to just reporting enabled indexes. Removed the extra params, the "never defer once built" check and the async-index check, all as you said. Cost was 4 call sites reading enabledIndexerMap as "exists" that now read table config. 2 of those were real bugs, the partitioned-RLI file groups and streaming tagging, both the ones you flagged. You were right about compaction holding the build back, it is fixed now. Also checked the MDT init lock per the ticket: it does not cover concurrent init (released before the write, streaming writer outside it entirely), but it does guarantee a writer's requested instant is on the timeline before it decides, which is what makes the pending-instant check sound. The cost is still unsolved. Where it stands is the footer count: a listing plus a footer read per base file, every commit until the index is built. I tried the alternatives and none landed. Estimating from base-file bytes / avg record size was wrong: `AverageRecordSizeEstimator` only measures commits >100MB, so a table under a threshold falls back to the 1024 b/record default and a narrow-record table would've been deferred permanently, with the estimate itself being what skipped the count that would have corrected it. `numInserts - numDeletes` doesn't survive `delete_partition` (writes no stats), clustering (reports rewrites as fresh inserts) or archival. `numWrites` summed over the timeline double-counts on COW, since it's the whole rewritten file rather than the delta. A running counter would be cheapest to read but adds permanent per-commit state, maintained by every engine, to optimise a phase that ends once the index is built. What I think survives is the file system view for liveness plus `numWrites` for the counts, with a footer read only for files whose commit has been archived. Not built. Questions: 1. Is there a writer or configuration where defer.init is broken that the 24 cases don't cover? 2. Is the per-commit footer count acceptable as it is, or do you want the FSV + numWrites version before this goes up? 3. HFile base files can't report a row count, so counting throws. How do you want that handled? 4. Does a streaming write's metadata writer see an RLI built by the locked writer earlier in the same instant, or is its table config fixed at construction? If the latter, that commit's records never reach the index. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
