[
https://issues.apache.org/jira/browse/IMPALA-8572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928178#comment-16928178
]
ASF subversion and git services commented on IMPALA-8572:
---------------------------------------------------------
Commit 32dbf5d78a4ced803a9182bfb0d7c0992497f97e in impala's branch
refs/heads/master from Bharath Vissapragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=32dbf5d ]
IMPALA-8572: Log query events before unregister.
Currently, the query events (audits and lineages) are logged as a
part of query unregistration. This delays the event logging in cases
where the Unregister() is delayed by client for some reason (ex: Hue
does not call Unregister until the browser tab is closed) or the client
goes away without calling Unregister and the query timeout kicks in.
This patch moves this event logging to an earlier stage in the query
lifecycle. Moved the event logging related code into ClientRequestState
for easier code refactoring.
The conditions under which the events are logged are slightly
modified by this patch. Without the patch, events are logged for
unsuccessful queries if atleast a single fetch is perfomed. This patch
relaxes this guarantee to log events for any query that reaches
the FINISHED state (rows are available to fetch by the client) and does
not wait for a fetch to be performed. This simplifies the coordinator
state machine by avoiding unnecessary synchronization.
Added some test coverage for coordinator side code paths for writing
lineages. fe specific lineage tests only verified the correctness of
lineage created but did not test whether it was being flushed correctly
to the disk.
Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e
Reviewed-on: http://gerrit.cloudera.org:8080/14143
Reviewed-by: Bharath Vissapragada <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Move query hook execution to before query unregistration
> --------------------------------------------------------
>
> Key: IMPALA-8572
> URL: https://issues.apache.org/jira/browse/IMPALA-8572
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: radford nguyen
> Assignee: bharath v
> Priority: Critical
>
> The backend currently executes query event hooks during
> {{ImpalaServer::UnregisterQuery}}, which may actually only happen a long time
> after the query has actually executed. We depend on either the client closing
> the query/session, the client's connection dropping, or an idle session
> timing out.
> e.g. the following sequence is possible.
> # User executes query from Hue.
> # User goes home for weekend, leaving Hue tab open in browser
> # If we're lucky, the session timeout expires after some amount of idle time.
> # The query gets unregistered, hooks get executed
> It would generally be desirable to move the lineage logger earlier in the
> query lifecycle, so it occurs as soon as all of the required data is
> available.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
