To help Apex developers/users with debugging or troubleshooting “dead” applications, I am proposing a new feature to persist logical and physical plan snapshots in HDFS.
Similar to how the Apex engine persists container data per application attempt in HDFS as containers_NNN.json (where NNN is 1 for first app attempt, 2 for the second app attempt and so on), we will create 2 more sets of files under the …/apps/{appId} directory for an application: logicalPlan_NNN_MMM.json physicalPlan_NNN_MMM.json where NNN stands for the app attempt index (similar to NNN above 1, 2, 3 and so on) and MMM is a running index starting at 1 which stands for a snapshot within an app attempt. Note that a logical or physical plan may change within an app attempt for any number of reasons. The StreamingContainerManager class maintains the current logical/physical plans in the “plan” member variable. New methods will be added in StreamingContainerManager to save the logical or physical plan as JSON representations in the app directory (as described above). The logic is similar to com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String) and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used inside the Stram Web service. There will be running indexes in StreamingContainerManager to keep track of MMM for the logical plan and physical plan. The appropriate save method will be called on the occurrence of any event that updates the logical or physical plan for example: inside com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call() for logical plan change event inside com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping, String) for physical plan change event (i.e. redoing partitioning) Once these files are created, any user or a tool (such as the Apex CLI or the DT Gateway) can look up these files for troubleshooting/researching of “dead” applications and significant events in their lifetime in terms of logical or physical plan changes. Pls send me your feedback. Sanjay