[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731307#comment-15731307
 ] 

Thomas Weise commented on APEXCORE-587:
---

Pramod you can take it up, it isn't urgent though.

> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731060#comment-15731060
 ] 

Vlad Rozov commented on APEXCORE-587:
-

JIRA is not assigned :).

> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Pramod Immaneni (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731038#comment-15731038
 ] 

Pramod Immaneni commented on APEXCORE-587:
--

Agree with both of you. But how about adding methods prepareDAG to launcher API 
and launchDAG that takes a prepared DAG and launches it, launchApp will be a 
combination of both. Also yes for making getDAG() part of AppHandler.

Also thomas, the issue is current assigned, do you want to work on it? If so 
please go ahead otherwise I can work on it as well.

> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730893#comment-15730893
 ] 

Vlad Rozov commented on APEXCORE-587:
-

I agree, it should be OK to move getDAG() to the application handle returned by 
the launchApp() API. It may be possible to use prepareDAG() to get newly 
created DAG, but it does need to support preparing more than one 
DAG/Application. I would also consider using a different class to provide such 
functionality in case DAG creation/preparing does not depend on the Launcher 
implementation. Currently, there is only one class that implements DAG, and 
delegating DAG creation to different Launcher implementations is not necessary.

> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730801#comment-15730801
 ] 

Thomas Weise commented on APEXCORE-587:
---

Possible alternatives include providing access to the DAG that was launched 
through the handle and prepareDAG to create the DAG without launching it. 

> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730697#comment-15730697
 ] 

Vlad Rozov commented on APEXCORE-587:
-

IMO, those methods don't belong to the Launcher or EmbeddedLauncher class. It 
should be possible to use the same Launcher instance to start more than one 
application and the current interface of the Launcher class fully support this. 
In contrast, getDAG() assumes that there is one to one relationship between 
DAG/LogicalPlan and Launcher. It will be better to have a different class that 
deals with DAG creation and cloning. prepareDAG() may still be part of the 
Launcher API, but it needs to change to support multiple applications.  

> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-584) User specified blacklist of nodes for resource allocation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730334#comment-15730334
 ] 

ASF GitHub Bot commented on APEXCORE-584:
-

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/430

APEXCORE-584 Exposed the attribute to blacklist the nodes for the res…

…ource allocation purpose

@PramodSSImmaneni please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-584

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #430


commit 4eb5817009cd98fb9f7b20fdc4ffd493f980deaa
Author: Sandesh Hegde 
Date:   2016-12-07T22:51:35Z

APEXCORE-584 Exposed the attribute to blacklist the nodes for the resource 
allocation purpose




> User specified blacklist of nodes for resource allocation
> -
>
> Key: APEXCORE-584
> URL: https://issues.apache.org/jira/browse/APEXCORE-584
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Sandesh
>Assignee: Sandesh
>
> Based on the discussion in the mailing list, there is a need for user 
> specified blacklist of nodes.
> http://mail-archives.apache.org/mod_mbox/apex-dev/201611.mbox/browser



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #430: APEXCORE-584 Exposed the attribute to blacklist...

[sandeshh]

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/430

APEXCORE-584 Exposed the attribute to blacklist the nodes for the res…

…ource allocation purpose

@PramodSSImmaneni please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-584

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #430


commit 4eb5817009cd98fb9f7b20fdc4ffd493f980deaa
Author: Sandesh Hegde 
Date:   2016-12-07T22:51:35Z

APEXCORE-584 Exposed the attribute to blacklist the nodes for the resource 
allocation purpose




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (APEXCORE-585) Latency should be calculated only after the first window has been complete

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-585:

Priority: Minor  (was: Major)

> Latency should be calculated only after the first window has been complete
> --
>
> Key: APEXCORE-585
> URL: https://issues.apache.org/jira/browse/APEXCORE-585
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: David Yan
>Assignee: David Yan
>Priority: Minor
> Fix For: 3.6.0
>
>
> From the latency unit test, it tries to calculate the latency by looking at 
> the difference between current window id and upstream's current window id 
> when the end window stats is not available. But when the current window id is 
> 0 (first window has not been complete), it would return some wacky latency 
> value because the current window id is 0.
> This would not be an issue if the window id was not a simple sequence.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-585) Latency should be calculated only after the first window has been complete

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730305#comment-15730305
 ] 

ASF GitHub Bot commented on APEXCORE-585:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/427


> Latency should be calculated only after the first window has been complete
> --
>
> Key: APEXCORE-585
> URL: https://issues.apache.org/jira/browse/APEXCORE-585
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: David Yan
>Assignee: David Yan
>
> From the latency unit test, it tries to calculate the latency by looking at 
> the difference between current window id and upstream's current window id 
> when the end window stats is not available. But when the current window id is 
> 0 (first window has not been complete), it would return some wacky latency 
> value because the current window id is 0.
> This would not be an issue if the window id was not a simple sequence.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #427: APEXCORE-585 #resolve do not try to calculate l...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/427


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXCORE-588) Remove the usage of deprecated Yarn apis (example: AMCommand)

[Sandesh (JIRA)]

Sandesh created APEXCORE-588:


 Summary: Remove the usage of deprecated Yarn apis (example: 
AMCommand)
 Key: APEXCORE-588
 URL: https://issues.apache.org/jira/browse/APEXCORE-588
 Project: Apache Apex Core
  Issue Type: Improvement
Reporter: Sandesh
Assignee: Sandesh


Remove the usage of deprecated YARN apis, one example is AMCommand.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-site pull request #63: Adding Google Analytics tracking code and privac...

[mx4web]

GitHub user mx4web opened a pull request:

https://github.com/apache/apex-site/pull/63

Adding Google Analytics tracking code and privacy footer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mx4web/apex-site apexcore-553

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-site/pull/63.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #63


commit 20ba4df3d828a6b5f192df632b512df1c0f7f6a6
Author: Michelle Xiao 
Date:   2016-12-07T22:41:20Z

Adding Google Analytics tracking code and privacy footer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Apache Apex Core Release 3.5.0 (RC1)

[Sandesh Hegde]

+1

Followed the steps in mentioned in http://apex.apache.org/verification.html
Verified the launch of an application.

On Tue, Dec 6, 2016 at 10:55 PM Thomas Weise  wrote:

> Dear Community,
>
> Please vote on the following Apache Apex Core 3.5.0 release candidate.
>
> This is a source release with binary artifacts published to Maven.
>
> List of all issues fixed:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12335724=Text=12318823
> User documentation: https://apex.apache.org/docs/apex-3.5/
>
> Staging directory:
> https://dist.apache.org/repos/dist/dev/apex/apache-apex-core-3.5.0-RC1/
> Source zip:
>
> https://dist.apache.org/repos/dist/dev/apex/apache-apex-core-3.5.0-RC1/apache-apex-core-3.5.0-source-release.zip
> Source tar.gz:
>
> https://dist.apache.org/repos/dist/dev/apex/apache-apex-core-3.5.0-RC1/apache-apex-core-3.5.0-source-release.tar.gz
> Maven staging repository:
> https://repository.apache.org/content/repositories/orgapacheapex-1021/
>
> Git source:
>
> https://git-wip-us.apache.org/repos/asf?p=apex-core.git;a=commit;h=refs/tags/v3.5.0-RC1
>  (commit: 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8)
>
> PGP key:
> http://pgp.mit.edu:11371/pks/lookup?op=vindex=t...@apache.org
> KEYS file:
> https://dist.apache.org/repos/dist/release/apex/KEYS
>
> More information at:
> http://apex.apache.org
>
> Please try the release and vote; vote will be open for 72 hours.
>
> [ ] +1 approve (and what verification was done)
> [ ] -1 disapprove (and reason why)
>
> http://www.apache.org/foundation/voting.html
>
> How to verify release candidate:
>
> http://apex.apache.org/verification.html
>
> Thanks,
> Thomas
>

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Munagala V. Ramanath (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729730#comment-15729730
 ] 

Munagala V. Ramanath commented on APEXCORE-558:
---

Yes, fine. [~sanjaypujare] For OSX could you check if using a different font in 
the console makes a difference for bold/italic ?


> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[David Yan (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729693#comment-15729693
 ] 

David Yan commented on APEXCORE-558:


This sounds good to me

> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729682#comment-15729682
 ] 

Sanjay M Pujare commented on APEXCORE-558:
--

I am okay to use bold/italics even if they don't work on MacOS. So the proposal 
is as follows (only for help output):

- command-name itself is highlighted in bold
- arguments will be in italics (current implementation) and optional arguments 
in square brackets

Hopefully this works for everyone?



> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[David Yan (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729663#comment-15729663
 ] 

David Yan commented on APEXCORE-558:


[~sanjaypujare] I can assure you that there are more than 1% of the users who 
use a dark background on a terminal that find the yellow foreground color OK. :)

Since this does pose a problem for users with a white background, I'd vote for 
removing color altogether and just use bold or italics, and if macOS terminal 
does not support bold/italics, it's not the end of the world for those users 
because they can still read the text.

If we must use color, we can use what git-diff uses by default, which is 0;32 
(green) for addition and 0;31 (red) for removal, and I hope we don't end up 
using red.


> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729622#comment-15729622
 ] 

Sanjay M Pujare commented on APEXCORE-558:
--

True, but I can read red and I just could not read yellow over white. For me 
help was practically useless (unless I copy pasted the text in a separate 
window or looked at the source code). There is another issue at least on my 
MacOS bash - none of the bolding/italics curses commands work and only colors 
work.

> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Visitor API for DAG

[Pramod Immaneni]

JIRA it is..

On Wed, Dec 7, 2016 at 10:09 AM, Sanjay Pujare 
wrote:

> Should we continue the discussion in the JIRA?
>
> Making it generic the way you are suggesting – won’t it cause problems of
> consistency etc e.g. different hooks do different things? Also are there a
> use cases for such generic hooks to justify the effort involved?
>
> On 12/7/16, 7:29 AM, "Pramod Immaneni"  wrote:
>
> Tushar,
>
> Why specifically limit it to client side preparation of the DAG before
> the
> application is launched? Why not make it possible to have general hooks
> that can apply even when the application is running in the different
> distributed components of the application such as the containers and
> stram.
> The hooks could be registered to be asynchronous or synchronous. The
> asynchronous ones could be handled via a bus, we already have a bus
> called mbassador that we use today and it could potentially be used for
> this. The entire functionality is akin to something like dtrace
> without the
> dynamic part.
>
> Thanks
>
> On Fri, Nov 25, 2016 at 3:24 AM, Tushar Gosavi  >
> wrote:
>
> > Opened a Jira https://issues.apache.org/jira/browse/APEXCORE-577
> for this.
> >
> > - Tushar.
> >
> >
> > On Mon, Nov 21, 2016 at 9:59 PM, Amol Kekre 
> wrote:
> > > Ananth,
> > > The current API allows changing properties of a running app. The
> new
> > > proposed API is not needed to do so.
> > >
> > > Thks
> > > Amol
> > >
> > >
> > > On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi <
> tus...@datatorrent.com>
> > > wrote:
> > >
> > >> Hi Ananth,
> > >>
> > >> We can not change runtime properties through this API. The current
> > >> flow of apex application execution is
> > >>
> > >> 1) StramClient prepares application, inject properties from
> > >> properties.xml / user provided xml files and validates dag
> > >> 2) StramClient copies required jars and serialized plan to HDFS
> and
> > >> launch master container.
> > >> 3) Application master reads serialised plan from HDFS and starts
> > >> deploying StramClient as per deployment plan.
> > >>
> > >> The visitor will examine the DAG in stage 1, hence visitor can
> only
> > >> change the initial state. The execution of the application is not
> > >> affected.
> > >>
> > >> - Tushar.
> > >>
> > >>
> > >> On Sat, Nov 19, 2016 at 3:40 AM, ananth 
> wrote:
> > >> > How does this work for the stateful operators ? Can we use this
> to
> > >> override
> > >> > properties that are deserialized ?
> > >> >
> > >> > Regards,
> > >> >
> > >> > Ananth
> > >> >
> > >> >
> > >> >
> > >> > On 18/11/16 05:53, Tushar Gosavi wrote:
> > >> >>
> > >> >> The code will execute before application master is launched,
> it is
> > >> >> just one time activity during application startup. Few use
> cases I
> > >> >> could think are
> > >> >>
> > >> >> - Operator validation/configuration validator
> > >> >>jdbc operator could check if database is accessible with
> given
> > >> >> credentials.
> > >> >>file output operator could if directory exists and
> filesystem is
> > >> >> writable.
> > >> >>
> > >> >> - Injection of properties in operators from external sources.
> > >> >>
> > >> >> - If two operator wants to exchange some information based on
> > >> >> configuration, they could do it through visitor. for example
> > >> >> TUPLE_SCHEMA can be set on downstream operator port based on
> > operators
> > >> >> input TUPLE_SCHEMA and its configuration (for example
> projection
> > >> >> operator which drops few columns, could create a new class
> with fewer
> > >> >> fields and set it as tuple schema on downstream operator port).
> > >> >>
> > >> >> - For pojo enabled operator (port where TUPLE_SCHEMA is
> defined), a
> > >> >> efficient stream codec could be written using asm library for
> > >> >> serialisation and use that as stream codec instead of default
> one.
> > >> >>
> > >> >> -Tushar.
> > >> >>
> > >> >>
> > >> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <
> > san...@datatorrent.com
> > >> >
> > >> >> wrote:
> > >> >>>
> > >> >>> There is a risk if the user written code blocks the thread or
> > crashes
> > >> the
> > >> >>> process. What are the real life examples of this use case?
> > >> >>>
> > >> >>>
> > >> >>> On 11/17/16, 9:21 AM, "amol kekre" 
> wrote:
> > >> >>>
> > >> >>>  +1. Opening up the API for users to put in their own
> code is
> > good.
> > >> >>> In
> > >> >>>  general we should enable users to 

[jira] [Commented] (APEXCORE-577) Visitor API for DAG

[Pramod Immaneni (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729596#comment-15729596
 ] 

Pramod Immaneni commented on APEXCORE-577:
--

Why specifically limit it to client side preparation of the DAG before the 
application is launched? Why not make it possible to have general hooks that 
can apply even when the application is running in the different distributed 
components of the application such as the containers and stram. The hooks could 
be registered to be asynchronous or synchronous. The asynchronous ones could be 
handled via a bus, we already have a bus called mbassador that we use today and 
it could potentially be used for this. The entire functionality is akin to 
something like dtrace without the dynamic part.

> Visitor API for DAG
> ---
>
> Key: APEXCORE-577
> URL: https://issues.apache.org/jira/browse/APEXCORE-577
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Tushar Gosavi
>
> Support an Visitor like api for DAG, which could be used to write plugins for 
> Apex.
> change DAG interface to add a visitor
> {code}
> registerVisitor(DagVisitor visitor)
> {code}
> The DagVisitor interface
> {code}
> interface DagVisitor {
>   boolean startDAG(DAG* dag);
>   boolean visitOperator(OperatorMeta *ometa)
>   boolean visitStream(StreamMeta *smeta);
>   boolean endDAG()
> }
> {code}
> dev thread discussions
> https://lists.apache.org/thread.html/1c37e0954cee029c9de537cc35ecb35beebdf49aba17de89bd2ce9ed@%3Cdev.apex.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Pramod Immaneni (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729585#comment-15729585
 ] 

Pramod Immaneni commented on APEXCORE-558:
--

Red typically means something is wrong, wouldn't you want something more 
re-assuring for help.

> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729554#comment-15729554
 ] 

Sanjay M Pujare commented on APEXCORE-558:
--

I'll revisit this after we have a consensus on what needs to be done. Otherwise 
changing yellow to red was a simple change that would have eliminated a lot of 
inconvenience for 99% of users (myself included).

> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729504#comment-15729504
 ] 

Thomas Weise commented on APEXCORE-587:
---

https://github.com/apache/incubator-beam/pull/1517#pullrequestreview-11859394


> Allow access to DAG through the Launcher API
> 
>
> Key: APEXCORE-587
> URL: https://issues.apache.org/jira/browse/APEXCORE-587
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Thomas Weise
>Priority: Minor
>
> It is useful for clients to access the DAG without launching it. The existing 
> methods from EmbeddedLauncher (getDAG, prepareDAG) should be available 
> through the parent class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXCORE-587) Allow access to DAG through the Launcher API

[Thomas Weise (JIRA)]

Thomas Weise created APEXCORE-587:
-

 Summary: Allow access to DAG through the Launcher API
 Key: APEXCORE-587
 URL: https://issues.apache.org/jira/browse/APEXCORE-587
 Project: Apache Apex Core
  Issue Type: Improvement
Reporter: Thomas Weise
Priority: Minor


It is useful for clients to access the DAG without launching it. The existing 
methods from EmbeddedLauncher (getDAG, prepareDAG) should be available through 
the parent class.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Visitor API for DAG

[Sanjay Pujare]

Should we continue the discussion in the JIRA?

Making it generic the way you are suggesting – won’t it cause problems of 
consistency etc e.g. different hooks do different things? Also are there a use 
cases for such generic hooks to justify the effort involved?

On 12/7/16, 7:29 AM, "Pramod Immaneni"  wrote:

Tushar,

Why specifically limit it to client side preparation of the DAG before the
application is launched? Why not make it possible to have general hooks
that can apply even when the application is running in the different
distributed components of the application such as the containers and stram.
The hooks could be registered to be asynchronous or synchronous. The
asynchronous ones could be handled via a bus, we already have a bus
called mbassador that we use today and it could potentially be used for
this. The entire functionality is akin to something like dtrace without the
dynamic part.

Thanks

On Fri, Nov 25, 2016 at 3:24 AM, Tushar Gosavi 
wrote:

> Opened a Jira https://issues.apache.org/jira/browse/APEXCORE-577 for this.
>
> - Tushar.
>
>
> On Mon, Nov 21, 2016 at 9:59 PM, Amol Kekre  wrote:
> > Ananth,
> > The current API allows changing properties of a running app. The new
> > proposed API is not needed to do so.
> >
> > Thks
> > Amol
> >
> >
> > On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi 
> > wrote:
> >
> >> Hi Ananth,
> >>
> >> We can not change runtime properties through this API. The current
> >> flow of apex application execution is
> >>
> >> 1) StramClient prepares application, inject properties from
> >> properties.xml / user provided xml files and validates dag
> >> 2) StramClient copies required jars and serialized plan to HDFS and
> >> launch master container.
> >> 3) Application master reads serialised plan from HDFS and starts
> >> deploying StramClient as per deployment plan.
> >>
> >> The visitor will examine the DAG in stage 1, hence visitor can only
> >> change the initial state. The execution of the application is not
> >> affected.
> >>
> >> - Tushar.
> >>
> >>
> >> On Sat, Nov 19, 2016 at 3:40 AM, ananth  wrote:
> >> > How does this work for the stateful operators ? Can we use this to
> >> override
> >> > properties that are deserialized ?
> >> >
> >> > Regards,
> >> >
> >> > Ananth
> >> >
> >> >
> >> >
> >> > On 18/11/16 05:53, Tushar Gosavi wrote:
> >> >>
> >> >> The code will execute before application master is launched, it is
> >> >> just one time activity during application startup. Few use cases I
> >> >> could think are
> >> >>
> >> >> - Operator validation/configuration validator
> >> >>jdbc operator could check if database is accessible with given
> >> >> credentials.
> >> >>file output operator could if directory exists and filesystem is
> >> >> writable.
> >> >>
> >> >> - Injection of properties in operators from external sources.
> >> >>
> >> >> - If two operator wants to exchange some information based on
> >> >> configuration, they could do it through visitor. for example
> >> >> TUPLE_SCHEMA can be set on downstream operator port based on
> operators
> >> >> input TUPLE_SCHEMA and its configuration (for example projection
> >> >> operator which drops few columns, could create a new class with 
fewer
> >> >> fields and set it as tuple schema on downstream operator port).
> >> >>
> >> >> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
> >> >> efficient stream codec could be written using asm library for
> >> >> serialisation and use that as stream codec instead of default one.
> >> >>
> >> >> -Tushar.
> >> >>
> >> >>
> >> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <
> san...@datatorrent.com
> >> >
> >> >> wrote:
> >> >>>
> >> >>> There is a risk if the user written code blocks the thread or
> crashes
> >> the
> >> >>> process. What are the real life examples of this use case?
> >> >>>
> >> >>>
> >> >>> On 11/17/16, 9:21 AM, "amol kekre"  wrote:
> >> >>>
> >> >>>  +1. Opening up the API for users to put in their own code is
> good.
> >> >>> In
> >> >>>  general we should enable users to register their code in a lot
> of
> >> >>> scenerios.
> >> >>>
> >> >>>  Thks
> >> >>>  Amol
> >> >>>
> >> >>>  On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi
> >> >>> 
> >> >>>  wrote:
> >> >>>
> >> >>>  > Yes, It could happen after current DAG validation and before
  

Re: Visitor API for DAG

[Pramod Immaneni]

Tushar,

Why specifically limit it to client side preparation of the DAG before the
application is launched? Why not make it possible to have general hooks
that can apply even when the application is running in the different
distributed components of the application such as the containers and stram.
The hooks could be registered to be asynchronous or synchronous. The
asynchronous ones could be handled via a bus, we already have a bus
called mbassador that we use today and it could potentially be used for
this. The entire functionality is akin to something like dtrace without the
dynamic part.

Thanks

On Fri, Nov 25, 2016 at 3:24 AM, Tushar Gosavi 
wrote:

> Opened a Jira https://issues.apache.org/jira/browse/APEXCORE-577 for this.
>
> - Tushar.
>
>
> On Mon, Nov 21, 2016 at 9:59 PM, Amol Kekre  wrote:
> > Ananth,
> > The current API allows changing properties of a running app. The new
> > proposed API is not needed to do so.
> >
> > Thks
> > Amol
> >
> >
> > On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi 
> > wrote:
> >
> >> Hi Ananth,
> >>
> >> We can not change runtime properties through this API. The current
> >> flow of apex application execution is
> >>
> >> 1) StramClient prepares application, inject properties from
> >> properties.xml / user provided xml files and validates dag
> >> 2) StramClient copies required jars and serialized plan to HDFS and
> >> launch master container.
> >> 3) Application master reads serialised plan from HDFS and starts
> >> deploying StramClient as per deployment plan.
> >>
> >> The visitor will examine the DAG in stage 1, hence visitor can only
> >> change the initial state. The execution of the application is not
> >> affected.
> >>
> >> - Tushar.
> >>
> >>
> >> On Sat, Nov 19, 2016 at 3:40 AM, ananth  wrote:
> >> > How does this work for the stateful operators ? Can we use this to
> >> override
> >> > properties that are deserialized ?
> >> >
> >> > Regards,
> >> >
> >> > Ananth
> >> >
> >> >
> >> >
> >> > On 18/11/16 05:53, Tushar Gosavi wrote:
> >> >>
> >> >> The code will execute before application master is launched, it is
> >> >> just one time activity during application startup. Few use cases I
> >> >> could think are
> >> >>
> >> >> - Operator validation/configuration validator
> >> >>jdbc operator could check if database is accessible with given
> >> >> credentials.
> >> >>file output operator could if directory exists and filesystem is
> >> >> writable.
> >> >>
> >> >> - Injection of properties in operators from external sources.
> >> >>
> >> >> - If two operator wants to exchange some information based on
> >> >> configuration, they could do it through visitor. for example
> >> >> TUPLE_SCHEMA can be set on downstream operator port based on
> operators
> >> >> input TUPLE_SCHEMA and its configuration (for example projection
> >> >> operator which drops few columns, could create a new class with fewer
> >> >> fields and set it as tuple schema on downstream operator port).
> >> >>
> >> >> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
> >> >> efficient stream codec could be written using asm library for
> >> >> serialisation and use that as stream codec instead of default one.
> >> >>
> >> >> -Tushar.
> >> >>
> >> >>
> >> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <
> san...@datatorrent.com
> >> >
> >> >> wrote:
> >> >>>
> >> >>> There is a risk if the user written code blocks the thread or
> crashes
> >> the
> >> >>> process. What are the real life examples of this use case?
> >> >>>
> >> >>>
> >> >>> On 11/17/16, 9:21 AM, "amol kekre"  wrote:
> >> >>>
> >> >>>  +1. Opening up the API for users to put in their own code is
> good.
> >> >>> In
> >> >>>  general we should enable users to register their code in a lot
> of
> >> >>> scenerios.
> >> >>>
> >> >>>  Thks
> >> >>>  Amol
> >> >>>
> >> >>>  On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi
> >> >>> 
> >> >>>  wrote:
> >> >>>
> >> >>>  > Yes, It could happen after current DAG validation and before
> the
> >> >>>  > application master is launched.
> >> >>>  >
> >> >>>  > - Tushar.
> >> >>>  >
> >> >>>  >
> >> >>>  > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath
> >> >>> 
> >> >>>  > wrote:
> >> >>>  > > When would the visits happen ? Just before normal
> validation ?
> >> >>>  > >
> >> >>>  > > Ram
> >> >>>  > >
> >> >>>  > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi
> >> >>> 
> >> >>>  > wrote:
> >> >>>  > >
> >> >>>  > >> Hi All,
> >> >>>  > >>
> >> >>>  > >> How about adding visitor like API for DAG in Apex, and an
> api
> >> >>> to
> >> >>>  > >> register visitor for the DAG.
> >> >>>  > >> Possible use cases are
> >> >>>  > >> -  Validator visitor which could validate the