[
https://issues.apache.org/jira/browse/TEZ-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423894#comment-15423894
]
Hitesh Shah edited comment on TEZ-3404 at 8/17/16 5:10 AM:
-----------------------------------------------------------
bq. how do I ensure that I do not recreate domain on recovery
2 approaches:
Approach 1:
For this, I am not sure if a re-create check is needed. This may impact some of
the above discussions on where the domain creation is triggered from. Consider
this:
- the domain creation should happen before any AM event or DAG event is sent
to history
- if the domain creation fails, the am conf will be updated to disable
history logging. This may impact recovery a bit as we would then need to write
the AM launch event to recovery to track the actual conf used. Today I believe
it is not. On subsequent attempts, the am conf will be read and we will know
whether history is enabled or disabled.
- For dag domains, the domain creation will happen before the dag submitted
event is written so that will contain the correct config value based on
successful creation or not. So not an issue for subsequent attempts where the
dag is recovered.
Approach 2:
- always re-create domains on each attempt for both app and dag - ignore
duplicate creation errors.
To answer your original question, we would need to use the recovery log to
drive whether domain creation failed/succeeded and recover the data to decide
whether to log to history or not.
was (Author: hitesh):
bq. how do I ensure that I do not recreate domain on recovery
2 approaches:
Approach 1:
For this, I am not sure if a re-create check is needed. This may impact some of
the above discussions on where the domain creation is triggered from. Consider
this:
- the domain creation should happen before any AM event or DAG event is sent
to history
- if the domain creation fails, the am conf will be updated to disable
history logging. This may impact recovery a bit as we would then need to write
the AM launch event to recovery to track the actual conf used. Today I believe
it is not. On subsequent attempts, the am conf will be read and we will know
whether history is enabled or disabled.
- For dag domains, the domain creation will happen before the dag submitted
event is written so that will contain the correct config value based on
successful creation or not. So not an issue for subsequent attempts where the
dag is recovered.
Approach 2:
- always re-create domains on each attempt for both app and dag - ignore
duplicate creation errors.
> Remove blocking call for ATS domain creation from client side to AM
> -------------------------------------------------------------------
>
> Key: TEZ-3404
> URL: https://issues.apache.org/jira/browse/TEZ-3404
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Attachments: TEZ-3404-WIP-01.patch, TEZ-3404-WIP-02.patch
>
>
> Today, there is a blocking call for creating the domain on the client side
> which blocks hive query throughput. We need to move this to the AM and still
> retain the same security guarantees.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
