[
https://issues.apache.org/jira/browse/TEZ-993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233976#comment-14233976
]
Jeff Zhang edited comment on TEZ-993 at 12/4/14 7:22 AM:
---------------------------------------------------------
Upload new patch. [~hitesh] Please help review it.
* Create a abstract class RecoveryStore to replace RecoveryService &
RecoveryParser. 2 main responsibilities for RecoveryStore
** Store summary event and recovery event
** Load recovery data
* Abstract class RecoveryStore handle the basic application logic like skipping
events of pre_warm dags and events after completed dags, handling recovery
events in a separate thread, while the implementation do the real store events
and load data, don't need care about the application logic.
* Create a FileSystemRecoveryStore of using FileSystem as the RecoveryStore, it
just do the real store events and load recovered data. (Currently it is coupled
with RecoveredDAGData and DAGSummaryData, which is event based (built from
summary and recovery events), because we need the event to restore
DAG/Vertex/Task/TaskAttempt, maybe in the future the recovered data can be not
event based)
* Add unit test in TestRecoveryStore ( move the unit test of RecoveryParser to
this class )
was (Author: zjffdu):
Upload new patch. [~hitesh] Please help review it.
* Create a abstract class RecoveryStore to replace RecoveryService &
RecoveryParser. 2 main responsibilities for RecoveryStore
** Store summary event and recovery event
** Load recovery data
* abstract RecoveryStore handle the basic application logic like skipping
events of pre_warm dags and events after completed dags, handling recovery
events in a separate thread
* Create a FileSystemRecoveryStore of using FileSystem as the RecoveryStore, it
just do the real store events and load recovered data. (Currently it is coupled
with RecoveredDAGData and DAGSummaryData, which is event based (built from
summary and recovery events), because we need the event to restore
DAG/Vertex/Task/TaskAttempt, maybe in the future the recovered data can be not
event based)
* Add unit test in TestRecoveryStore ( move the unit test of RecoveryParser to
this class )
> Remove application logic from RecoveryService
> ---------------------------------------------
>
> Key: TEZ-993
> URL: https://issues.apache.org/jira/browse/TEZ-993
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Jeff Zhang
> Attachments: TEZ-993-3.patch, Tez-993-2.patch, Tez-993.patch
>
>
> Currently RecoveryService storage logic knows a lot about the DAG like which
> dag is pre-warm and does not need to be stored, which events needs special
> treatment etc. This kind of logic couples the DAG and the storage more than
> is probably necessary and can be a source of complications down the road. The
> storage should ideally be simply storing a sequence of arbitrary records
> delimited by a marker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
