rangareddy commented on issue #10907: URL: https://github.com/apache/hudi/issues/10907#issuecomment-4852855372
RO tables read only compacted base files, so their freshness is bounded by compaction completing AND the compaction commit being hive-synced — a 6h lag means one of those is behind, not a query bug. With `HIVE_SYNC_SKIP_RO_SUFFIX=true` the base-named table you're querying is the RO view, so recent updates in log files aren't visible until compaction folds them in; for near-real-time, query the _rt snapshot table instead. Two things to check: (1) last_commit_time_sync on the Hive table vs the latest completed compaction .commit in .hoodie/ (sync lag?), and (2) how many `*.compaction.inflight/requested` vs completed commits exist (compaction backlog?). Given 350M records with only 4 compaction tasks and `max_memory=1024`, compaction likely can't keep up and may be emitting the 0-size parquet you saw — raise `COMPACTION_TASKS` substantially, raise COMPACTION_MAX_MEMORY to 2–4GB, and ideally run compaction as a separate offline job rather than inline with the streaming writer. Also double-check the bucket config line — one shows `num.buckets=SIMPLE` and another `=128`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
