[
https://issues.apache.org/jira/browse/IMPALA-8572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849579#comment-16849579
]
ASF subversion and git services commented on IMPALA-8572:
---------------------------------------------------------
Commit 31195eb8119ac6a557486a10dc24692bb0202f85 in impala's branch
refs/heads/master from Radford Nguyen
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=31195eb ]
IMPALA-8473: Publish lineage info via hook
This commit introduces a hook mechanism for publishing,
lineage data specifically, but query information more
generally, from Impala.
The legacy behavior of writing the lineage file is
being retained but deprecated.
Hooks can be implemented by downstream consumers (i.e.
runtime dependencies) to hook into supported places during
Impala query execution:
- impalad startup
- query completion
- see IMPALA-8572 for caveat/details
The consumers are to be frontend Java dependencies
intiated at runtime. 2 backend flags configure this
behavior:
- `query_event_hook_classes` specifies a comma-separated
list of hook consumer implementation classes that
are instantiated and registered at impala start up.
- `query_event_hook_nthreads`
specifies the number of threads to use for asynchronous
hook execution. (Relevant if multiple hooks are
registered.)
Lineage information is passed from the backend after
a query completes (but before it returns) and given
to every hook to execute asynchronously. In other words,
a query may complete and return to the user before any
or all hooks have completed executing. An exception
during hook on-query-complete execution will simply be logged
and will not be (directly) fatal to the system.
Tests:
- added unit tests for FE hook execution
- added E2E tests for hook configuration, execution, error
- ran full build, tests
Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1
Reviewed-on: http://gerrit.cloudera.org:8080/13352
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Move query hook execution to before query unregistration
> --------------------------------------------------------
>
> Key: IMPALA-8572
> URL: https://issues.apache.org/jira/browse/IMPALA-8572
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: radford nguyen
> Priority: Major
>
> Placeholder: description coming soon
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
