abstractdog commented on PR #5558: URL: https://github.com/apache/hive/pull/5558#issuecomment-2500042114
before making this happen, I need to consider one more thing: the aggregated data seems to miss some counters I saw store_io_throttled=96 logged at the end of a query, when the filesystem is closed, which never made its way to the aggregated counters this throttled counter is automatically logged by hadoop on filesystem closure (fs objects are reused and belong to a single query now), which seems to be different then the IO stats collected on the task thread, need to investigate if I can merge them the challenge is that filesystem closure happens after the dag is completed (callback from AM to daemon), so collecting those might mean blocking the query itself, regarding which we need to be extra careful, I'm experimenting with this now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
