[
https://issues.apache.org/jira/browse/APEXCORE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723879#comment-15723879
]
Sanjay M Pujare commented on APEXCORE-586:
------------------------------------------
The following is the implementation plan. We will soon start the implementation
so pls send your feedback as soon as possible.
We will refactor code to move the logic in
com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)
and
com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan()
to the class
com.datatorrent.stram.StreamingContainerManager
So these 2 methods will return JSONObject representation of the logical or
physical plan.
2 new methods will be added to StreamingContainerManager:
writePhysicalPlan()
writeLogicalPlan()
The first method will create an HDFS file with the name
physicalPlan_NNN_MMM.json under the current app's app directory (where NNN is
the current app attempt id and MMM is a running count (maintained in the
StreamingContainerManager object) that is incremented on every call to
the method. The method will call getPhysicalPlan() to get the JSON object and
write it out to the HDFS file. writeLogicalPlan() will be similarly written to
write the logical plan to the file logicalPlan_NNN_MMM.json. There is a
separate count for the MMM for logical plans.
The method
com.datatorrent.stram.plan.physical.PhysicalPlan.onStatusUpdate(PTOperator)
will be modified to accept a parameter for the StreamingContainerManager
object. Inside this method after redoPartition() is called,
writePhysicalPlan() will be called on the StreamingContainerManager object
which will write a new physicalPlan_NNN_MMM.json file.
Similarly in
com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call()
at the end writeLogicalPlan() will be called which will write a new
logicalPlan_NNN_MMM.json file.
Unit tests will be written to cover existing functionality and new
functionality to minimize the possibility of regressions and to cover the new
code.
> Enhancement to persist logical and physical plan snapshots in HDFS
> ------------------------------------------------------------------
>
> Key: APEXCORE-586
> URL: https://issues.apache.org/jira/browse/APEXCORE-586
> Project: Apache Apex Core
> Issue Type: Improvement
> Reporter: Sanjay M Pujare
> Assignee: Sanjay M Pujare
>
> Pls refer to the discussion on dev@apex
> http://apache-apex-developers-list.78491.x6.nabble.com/Proposing-a-new-feature-to-persist-logical-and-physical-plan-snapshots-in-HDFS-td11592.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
