rangareddy commented on issue #8189: URL: https://github.com/apache/hudi/issues/8189#issuecomment-4835196519
The slow "Building workload profile" is BLOOM index tagging degrading when an out-of-order Kinesis batch widens the incoming key range and defeats min/max range pruning, turning nearly every file in your 3.5B-record partition into a bloom candidate (the S3 footer-probe storm you saw). Switch to an ordering-invariant index — BUCKET index (deterministic hash→file-group lookup) or Record Level Index on 0.14+ — or, if staying on BLOOM, sort files by record key to keep ranges tight and push bloom/col-stats into the metadata table; also upgrade off 0.11.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
