Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

2016-11-23 Thread Amol Kekre 
Persisted plan on DFS is good. I am +1 for it. This could be both of the
following

1. Attribute : If set, then upon change in plan persist to DFS
2. On demand

Thks
Amol


On Wed, Nov 23, 2016 at 4:15 PM, Sanjay Pujare 
wrote:

> Okay, but this “state” is gone after the app is “dead” isn’t that true?
> Also the reason for this enhancement is debuggability/troubleshooting of
> Apex apps so it is good to have separate explicit user visible files that
> contain the plan information instead of overloading the state for this
> purpose (in my opinion).
>
> In terms of on-demand, it sounds like a good idea - I didn’t think of it.
> But I would like to drill down the use cases. In most cases,
> logical/physical plan changes are spontaneous or rather internal to the app
> so an external entity making a REST call to save the plan on demand might
> not sync up with when the plan changes took place inside the app. So saving
> the plan JSON files on events described previously seems to be the most
> efficient thing to do (as discussed with @Ashwin Putta) but if there are
> use cases I think it is a good idea to do it on demand as well.
>
> On 11/23/16, 3:00 PM, "Amol Kekre"  wrote:
>
> Good idea. Stram does save state, and maybe a script that translates
> may
> work. But explicit plan saving is also a good idea. Could this be "on
> demand"? a rest call that writes out the plan(s) to specifid hdfs
> files?
>
> We could do both (write on any change/set call) and/or on-demand.
>
> Thks
> Amol
>
>
> On Wed, Nov 23, 2016 at 2:40 PM, Sanjay Pujare  >
> wrote:
>
> > To help Apex developers/users with debugging or troubleshooting
> “dead”
> > applications, I am proposing a new feature to persist logical and
> physical
> > plan snapshots in HDFS.
> >
> >
> >
> > Similar to how the Apex engine persists container data per
> application
> > attempt in HDFS as containers_NNN.json (where NNN is 1 for first app
> > attempt, 2 for the second app attempt and so on), we will create 2
> more
> > sets of files under the …/apps/{appId} directory for an application:
> >
> >
> >
> > logicalPlan_NNN_MMM.json
> >
> > physicalPlan_NNN_MMM.json
> >
> >
> >
> > where NNN stands for the app attempt index (similar to NNN above 1,
> 2, 3
> > and so on) and MMM is a running index starting at 1 which stands for
> a
> > snapshot within an app attempt. Note that a logical or physical plan
> may
> > change within an app attempt for any number of reasons.
> >
> >
> >
> > The StreamingContainerManager class maintains the current
> logical/physical
> > plans in the “plan” member variable. New methods will be added in
> > StreamingContainerManager to save the logical or physical plan as
> JSON
> > representations in the app directory (as described above). The logic
> is
> > similar to com.datatorrent.stram.webapp.StramWebServices.
> getLogicalPlan(String)
> > and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan()
> used
> > inside the Stram Web service. There will be running indexes in
> > StreamingContainerManager to keep track of MMM for the logical plan
> and
> > physical plan. The appropriate save method will be called on the
> occurrence
> > of any event that updates the logical or physical plan for example:
> >
> >
> >
> > inside com.datatorrent.stram.StreamingContainerManager.
> > LogicalPlanChangeRunnable.call()  for logical plan change event
> >
> >
> >
> > inside com.datatorrent.stram.plan.physical.PhysicalPlan.
> redoPartitions(PMapping,
> > String) for physical plan change event (i.e. redoing partitioning)
> >
> >
> >
> > Once these files are created, any user or a tool (such as the Apex
> CLI or
> > the DT Gateway) can look up these files for
> troubleshooting/researching of
> > “dead” applications and significant events in their lifetime in
> terms of
> > logical or physical plan changes. Pls send me your feedback.
> >
> >
> >
> > Sanjay
> >
> >
> >
> >
>
>
>
>

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

2016-11-23 Thread Sanjay Pujare 
Okay, but this “state” is gone after the app is “dead” isn’t that true? Also 
the reason for this enhancement is debuggability/troubleshooting of Apex apps 
so it is good to have separate explicit user visible files that contain the 
plan information instead of overloading the state for this purpose (in my 
opinion).

In terms of on-demand, it sounds like a good idea - I didn’t think of it. But I 
would like to drill down the use cases. In most cases, logical/physical plan 
changes are spontaneous or rather internal to the app so an external entity 
making a REST call to save the plan on demand might not sync up with when the 
plan changes took place inside the app. So saving the plan JSON files on events 
described previously seems to be the most efficient thing to do (as discussed 
with @Ashwin Putta) but if there are use cases I think it is a good idea to do 
it on demand as well.

On 11/23/16, 3:00 PM, "Amol Kekre"  wrote:

Good idea. Stram does save state, and maybe a script that translates may
work. But explicit plan saving is also a good idea. Could this be "on
demand"? a rest call that writes out the plan(s) to specifid hdfs files?

We could do both (write on any change/set call) and/or on-demand.

Thks
Amol


On Wed, Nov 23, 2016 at 2:40 PM, Sanjay Pujare 
wrote:

> To help Apex developers/users with debugging or troubleshooting “dead”
> applications, I am proposing a new feature to persist logical and physical
> plan snapshots in HDFS.
>
>
>
> Similar to how the Apex engine persists container data per application
> attempt in HDFS as containers_NNN.json (where NNN is 1 for first app
> attempt, 2 for the second app attempt and so on), we will create 2 more
> sets of files under the …/apps/{appId} directory for an application:
>
>
>
> logicalPlan_NNN_MMM.json
>
> physicalPlan_NNN_MMM.json
>
>
>
> where NNN stands for the app attempt index (similar to NNN above 1, 2, 3
> and so on) and MMM is a running index starting at 1 which stands for a
> snapshot within an app attempt. Note that a logical or physical plan may
> change within an app attempt for any number of reasons.
>
>
>
> The StreamingContainerManager class maintains the current logical/physical
> plans in the “plan” member variable. New methods will be added in
> StreamingContainerManager to save the logical or physical plan as JSON
> representations in the app directory (as described above). The logic is
> similar to 
com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)
> and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used
> inside the Stram Web service. There will be running indexes in
> StreamingContainerManager to keep track of MMM for the logical plan and
> physical plan. The appropriate save method will be called on the 
occurrence
> of any event that updates the logical or physical plan for example:
>
>
>
> inside com.datatorrent.stram.StreamingContainerManager.
> LogicalPlanChangeRunnable.call()  for logical plan change event
>
>
>
> inside 
com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping,
> String) for physical plan change event (i.e. redoing partitioning)
>
>
>
> Once these files are created, any user or a tool (such as the Apex CLI or
> the DT Gateway) can look up these files for troubleshooting/researching of
> “dead” applications and significant events in their lifetime in terms of
> logical or physical plan changes. Pls send me your feedback.
>
>
>
> Sanjay
>
>
>
>

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

2016-11-23 Thread Amol Kekre 
Good idea. Stram does save state, and maybe a script that translates may
work. But explicit plan saving is also a good idea. Could this be "on
demand"? a rest call that writes out the plan(s) to specifid hdfs files?

We could do both (write on any change/set call) and/or on-demand.

Thks
Amol


On Wed, Nov 23, 2016 at 2:40 PM, Sanjay Pujare 
wrote:

> To help Apex developers/users with debugging or troubleshooting “dead”
> applications, I am proposing a new feature to persist logical and physical
> plan snapshots in HDFS.
>
>
>
> Similar to how the Apex engine persists container data per application
> attempt in HDFS as containers_NNN.json (where NNN is 1 for first app
> attempt, 2 for the second app attempt and so on), we will create 2 more
> sets of files under the …/apps/{appId} directory for an application:
>
>
>
> logicalPlan_NNN_MMM.json
>
> physicalPlan_NNN_MMM.json
>
>
>
> where NNN stands for the app attempt index (similar to NNN above 1, 2, 3
> and so on) and MMM is a running index starting at 1 which stands for a
> snapshot within an app attempt. Note that a logical or physical plan may
> change within an app attempt for any number of reasons.
>
>
>
> The StreamingContainerManager class maintains the current logical/physical
> plans in the “plan” member variable. New methods will be added in
> StreamingContainerManager to save the logical or physical plan as JSON
> representations in the app directory (as described above). The logic is
> similar to 
> com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)
> and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used
> inside the Stram Web service. There will be running indexes in
> StreamingContainerManager to keep track of MMM for the logical plan and
> physical plan. The appropriate save method will be called on the occurrence
> of any event that updates the logical or physical plan for example:
>
>
>
> inside com.datatorrent.stram.StreamingContainerManager.
> LogicalPlanChangeRunnable.call()  for logical plan change event
>
>
>
> inside 
> com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping,
> String) for physical plan change event (i.e. redoing partitioning)
>
>
>
> Once these files are created, any user or a tool (such as the Apex CLI or
> the DT Gateway) can look up these files for troubleshooting/researching of
> “dead” applications and significant events in their lifetime in terms of
> logical or physical plan changes. Pls send me your feedback.
>
>
>
> Sanjay
>
>
>
>

Proposing a new feature to persist logical and physical plan snapshots in HDFS

2016-11-23 Thread Sanjay Pujare 
To help Apex developers/users with debugging or troubleshooting “dead” 
applications, I am proposing a new feature to persist logical and physical plan 
snapshots in HDFS.

 

Similar to how the Apex engine persists container data per application attempt 
in HDFS as containers_NNN.json (where NNN is 1 for first app attempt, 2 for the 
second app attempt and so on), we will create 2 more sets of files under the 
…/apps/{appId} directory for an application:

 

logicalPlan_NNN_MMM.json

physicalPlan_NNN_MMM.json

 

where NNN stands for the app attempt index (similar to NNN above 1, 2, 3 and so 
on) and MMM is a running index starting at 1 which stands for a snapshot within 
an app attempt. Note that a logical or physical plan may change within an app 
attempt for any number of reasons.

 

The StreamingContainerManager class maintains the current logical/physical 
plans in the “plan” member variable. New methods will be added in 
StreamingContainerManager to save the logical or physical plan as JSON 
representations in the app directory (as described above). The logic is similar 
to com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String) and 
com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used inside the 
Stram Web service. There will be running indexes in StreamingContainerManager 
to keep track of MMM for the logical plan and physical plan. The appropriate 
save method will be called on the occurrence of any event that updates the 
logical or physical plan for example:

 

inside 
com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call()
  for logical plan change event

 

inside 
com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping, 
String) for physical plan change event (i.e. redoing partitioning)

 

Once these files are created, any user or a tool (such as the Apex CLI or the 
DT Gateway) can look up these files for troubleshooting/researching of “dead” 
applications and significant events in their lifetime in terms of logical or 
physical plan changes. Pls send me your feedback.

 

Sanjay