To help Apex developers/users with debugging or troubleshooting “dead”
applications, I am proposing a new feature to persist logical and physical plan
snapshots in HDFS.
Similar to how the Apex engine persists container data per application attempt
in HDFS as containers_NNN.json (where NNN is 1 for first app attempt, 2 for the
second app attempt and so on), we will create 2 more sets of files under the
…/apps/{appId} directory for an application:
logicalPlan_NNN_MMM.json
physicalPlan_NNN_MMM.json
where NNN stands for the app attempt index (similar to NNN above 1, 2, 3 and so
on) and MMM is a running index starting at 1 which stands for a snapshot within
an app attempt. Note that a logical or physical plan may change within an app
attempt for any number of reasons.
The StreamingContainerManager class maintains the current logical/physical
plans in the “plan” member variable. New methods will be added in
StreamingContainerManager to save the logical or physical plan as JSON
representations in the app directory (as described above). The logic is similar
to com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String) and
com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used inside the
Stram Web service. There will be running indexes in StreamingContainerManager
to keep track of MMM for the logical plan and physical plan. The appropriate
save method will be called on the occurrence of any event that updates the
logical or physical plan for example:
inside
com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call()
for logical plan change event
inside
com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping,
String) for physical plan change event (i.e. redoing partitioning)
Once these files are created, any user or a tool (such as the Apex CLI or the
DT Gateway) can look up these files for troubleshooting/researching of “dead”
applications and significant events in their lifetime in terms of logical or
physical plan changes. Pls send me your feedback.
Sanjay
