[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[Tushar Gosavi (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968777#comment-15968777
 ] 

Tushar Gosavi commented on APEXCORE-408:


I have updated the document, please go through it. The approach I am taking is 
summarized below

- extend dag interface with following methods
  - removeOperator/removeStream
- add startTransaction method on LogicalPlan which will return a DAG object, 
which is a instance of TransactionalDAG which is clone of the dag with 
reference to operator and ports from the original dag.
  TransactionDAG maintains the reference to original dag and has a additional 
commit method. On commit  if the new DAG is valid then the same changes are 
applied to the original DAG.
- user can change the cloned dag as per his requirement. Currently I am 
providing the hooks to change the DAG through StatsListener by extending 
StatsListener interface having additional context object. later the hooks could 
also be provided through  DAGExecutionPluginContext which has access to more 
information from stram than statslistener.
- A Commit handler interface is added which commit method is call to notify 
type of changes performed to the DAG. The PhysicalPlan will be changed through 
this handler.


> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960602#comment-15960602
 ] 

ASF GitHub Bot commented on APEXCORE-408:
-

Github user tushargosavi closed the pull request at:

https://github.com/apache/apex-core/pull/410


> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[Tushar Gosavi (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958879#comment-15958879
 ] 

Tushar Gosavi commented on APEXCORE-408:


[~thw] I have opened a pull request 
https://github.com/apache/apex-core/pull/476 for statslistener  changes. I am 
implementing the second approach from the document as it would be simple and 
won't change existing code much. There are few problems I am facing regarding 
the validation done by LogicalPlan while elements are being added to it, like 
extending a stream the port needs to be present in the DAG. When I try to 
extend an existing stream with new operator's input port , as new operator's 
input port is not in the original DAG , it throws as error as port does not 
belong to any operator in the DAG. Trying to solve this problem.

> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899550#comment-15899550
 ] 

Thomas Weise commented on APEXCORE-408:
---

Please remove the word documents and let's use a single drive document that 
will make it easier to collaborate. The final doc can be attached as PDF when 
the work is complete.

> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
> Attachments: dag.docx, dag.docx
>
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574913#comment-15574913
 ] 

ASF GitHub Bot commented on APEXCORE-408:
-

GitHub user tushargosavi opened a pull request:

https://github.com/apache/apex-core/pull/410

APEXCORE-408: Ability to schedule Sub-DAG from running application.

Pull request for dynamic dag modification through stats listener.  It 
provides following
functionality

- StatsListener can access the opearator name for easily detecting which 
opearator stats are being processed.
- StatsListener can create a instance of object through which it can submit 
dag modifications to the engine.
- StatsListener can return dag changes as a response to engine.
- PlanModifier is modified to take a DAG and apply it on the existing 
running DAG and deploy the changes.

The following functionality is not working yet.

- The new opearator does not start from the correct windowId 
(https://issues.apache.org/jira/browse/APEXCORE-532)
- Relanched application failed to start when it was killed after dynamic 
dag modification.
- There is no support for resuming operator from previous state when they 
were removed. This could be achived through
  readig state through external storage on setup.
- persist operator support is not present for newly added streams.
- Not all parts are covered through tests.

The demo application using the feature is available at
https://github.com/tushargosavi/apex-dynamic-scheduling

There are two variations of WordCount application. The first variation 
detects the presence of
new files start a disconnected DAG to process the data.

(https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/WordCountApp.java)

The second application 
(https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/ExtendApp.java),
initially only one reader operator is running in the DAG, and provides 
pendingFiles as auto-metric to stat listener.
On detecting pending files it attaches splitter counter and output operator 
to the read operator. Once files are processed the splitter, counter and
output operators are removed and added back again if new data files are 
added into the directory.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tushargosavi/incubator-apex-core APEXCORE-408

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/410.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #410


commit 5cecd029463de021469afb20650e3afbf75fd088
Author: Tushar R. Gosavi 
Date:   2016-08-08T10:24:41Z

APEXCORE-408: Ability to schedule Sub-DAG from running application.




> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
> Attachments: Dynamic DAG Changes.docx
>
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574910#comment-15574910
 ] 

ASF GitHub Bot commented on APEXCORE-408:
-

Github user tushargosavi closed the pull request at:

https://github.com/apache/apex-core/pull/393


> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
> Attachments: Dynamic DAG Changes.docx
>
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)