prathik-onehouse commented on PR #20604:
URL: https://github.com/apache/hudi/pull/20604#issuecomment-6962974470

   Hi Siva, update on the RLI deferral (ENG-52213)
   
   The tests for defer.init without any threshold are now in this PR, 24 cases 
across the datasource, Spark SQL, structured streaming and HoodieStreamer, 
global + partitioned, v6 + v10, COW + MOR. All four assertions you listed. They 
pass on unmodified master, and flipping defer.init off fails them at the first 
assertion.
   
   I couldn't reproduce a writer where the deferral is broken. If you had a 
specific case in mind I'd rather add it than assume it's covered. Flink's the 
one gap and it rejects defer.init outright anyway.
   
   The restructure is built and working (separate branch, not raised yet). Gate 
moved into initializeIfNeeded, IndexerFactory back to just reporting enabled 
indexes. Removed the extra params, the "never defer once built" check and the 
async-index check, all as you said. Cost was 4 call sites reading 
enabledIndexerMap as "exists" that now read table config. 2 of those were real 
bugs, the partitioned-RLI file groups and streaming tagging, both the ones you 
flagged.
   
   You were right about compaction holding the build back, it is fixed now. 
Also checked the MDT init lock per the ticket: it does not cover concurrent 
init (released before the write, streaming writer outside it entirely), but it 
does guarantee a writer's requested instant is on the timeline before it 
decides, which is what makes the pending-instant check sound.
   
   The cost is still unsolved. Where it stands is the footer count: a listing 
plus a footer read per base file, every commit until the index is built. I 
tried the alternatives and none landed. Estimating from base-file bytes / avg 
record size was wrong: `AverageRecordSizeEstimator` only measures commits 
>100MB, so a table under a threshold falls back to the 1024 b/record default 
and a narrow-record table would've been deferred permanently, with the estimate 
itself being what skipped the count that would have corrected it. `numInserts - 
numDeletes` doesn't survive `delete_partition` (writes no stats), clustering 
(reports rewrites as fresh inserts) or archival. `numWrites` summed over the 
timeline double-counts on COW, since it's the whole rewritten file rather than 
the delta. A running counter would be cheapest to read but adds permanent 
per-commit state, maintained by every engine, to optimise a phase that ends 
once the index is built.
   
   What I think survives is the file system view for liveness plus `numWrites` 
for the counts, with a footer read only for files whose commit has been 
archived. Not built.
   
   Questions:
   
   1. Is there a writer or configuration where defer.init is broken that the 24 
cases don't cover?
   2. Is the per-commit footer count acceptable as it is, or do you want the 
FSV + numWrites version before this goes up?
   3. HFile base files can't report a row count, so counting throws. How do you 
want that handled?
   4. Does a streaming write's metadata writer see an RLI built by the locked 
writer earlier in the same instant, or is its table config fixed at 
construction? If the latter, that commit's records never reach the index.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to