[
https://issues.apache.org/jira/browse/HIVE-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-14224:
----------------------------------
Attachment: HIVE-14224.04.patch
Noticed some issues with the previous patch while testing it more.
1. The filename handling was broken with renames.
2. The appender was getting closed outside of the AsyncLogging thread - which
would mean a race in closing it.
This patch changes the approach on informing the logging system that a query is
done by sending a LOG message with a custom marker. This works better in terms
of being invoked on the correct thread - so the Appender.stop() should be
called after relevant log messages for the specific context.
There's still a race caused by queryComplete messages coming from the AM /
wrapping up structures like TaskRunnerCallable locally (we inform the AM of
success before cleaning up everything for a task). This can result in the same
file sitting around with and without a ".done" flag.
Haven't removed the dag-specific logger yet. Will break a subsequent patch.
That can be done in a followup.
[~prasanth_j] - could you take a quick look at the changes again please. We
should probably disable this by default in a subsequent patch (HIVE-14225) due
to the race, and the potential of generating a large number of files - test it
more before enabling by default.
> LLAP rename query specific log files once a query is complete
> -------------------------------------------------------------
>
> Key: HIVE-14224
> URL: https://issues.apache.org/jira/browse/HIVE-14224
> Project: Hive
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: HIVE-14224.02.patch, HIVE-14224.03.patch,
> HIVE-14224.04.patch, HIVE-14224.wip.01.patch
>
>
> Once a query is complete, rename the query specific log file so that YARN can
> aggregate the logs (once it's configured to do so).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
