[
https://issues.apache.org/jira/browse/TEZ-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979629#comment-14979629
]
Jeff Zhang edited comment on TEZ-2581 at 10/29/15 1:32 AM:
-----------------------------------------------------------
bq. Didn't quite understand this. What is the reasoning behind this? Also does
"vertexData.getVertexFinishedEvent() != null" imply that full recovery events
have not been seen? Should it be null (based on the code in isVertexFinished())?
It should be "vertexData.getVertexFinishedEvent() == null", will fix it.
bq. Why are we separately looking at vertex group commit members? vertex group
commit is a single operation that commits for all member vertices. Each member
vertex does not have a separate commit operation.
Because it is not known whether this vertex belong to any vertex group when
parsing recovery logs. So here check both vertex level commit and vertex group
level commit.
bq. Log debug or remove?
Will make it as debug
bq. Why is it always _1 ?
This method (RecoveryParser#readRecoveryEvents) is only for testing (verify the
recovery events). Will add dagId to make it more clear.
bq. How is DagRecoveryData.isRecoverable() different from isDAGRecoverable()?
isDAGRecoverable is for checking recoverability based on the summary data.
DagRecoveryData.isRecoverable() is for checking for recoverability based on the
non-summary events. Will reorganize the code the make it easier to understand
was (Author: zjffdu):
>>> Didn't quite understand this. What is the reasoning behind this? Also does
>>> "vertexData.getVertexFinishedEvent() != null" imply that full recovery
>>> events have not been seen? Should it be null (based on the code in
>>> isVertexFinished())?
It should be "vertexData.getVertexFinishedEvent() == null", will fix it.
>>> Why are we separately looking at vertex group commit members? vertex group
>>> commit is a single operation that commits for all member vertices. Each
>>> member vertex does not have a separate commit operation.
Because it is not known whether this vertex belong to any vertex group when
parsing recovery logs. So here check both vertex level commit and vertex group
level commit.
>>> Log debug or remove?
Will make it as debug
>>> Why is it always _1 ?
This method (RecoveryParser#readRecoveryEvents) is only for testing (verify the
recovery events). Will add dagId to make it more clear.
>>> How is DagRecoveryData.isRecoverable() different from isDAGRecoverable()?
isDAGRecoverable is for checking recoverability based on the summary data.
DagRecoveryData.isRecoverable() is for checking for recoverability based on the
non-summary events. Will reorganize the code the make it easier to understand
> Umbrella for Tez Recovery Redesign
> ----------------------------------
>
> Key: TEZ-2581
> URL: https://issues.apache.org/jira/browse/TEZ-2581
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Attachments: TEZ-2581-WIP-1.patch, TEZ-2581-WIP-2.patch,
> TEZ-2581-WIP-3.patch, TEZ-2581-WIP-4.patch, TEZ-2581-WIP-5.patch,
> TEZ-2581-WIP-6.patch, TezRecoveryRedesignProposal.pdf,
> TezRecoveryRedesignV1.1.pdf
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
