Re: [VOTE] Apache Apex Malhar Release 3.6.0 (RC1)

+1 (binding) Verified checksums Verified LICENSE, NOTICE and README.md. Build with: mvn clean apache-rat:check verify -Dlicense.skip=false -Pall-modules install -DskipTests Thank you, Vlad On 11/30/16 22:56, Bhupesh Chawda wrote: +1 - Verified signatures - Build and test successful

Re: "ExcludeNodes" for an Apex application

Yes, Ram explained to me that in practice this would be a useful feature for Apex devops who typically have no control over Hadoop/Yarn cluster. On 11/30/16, 9:22 PM, "Mohit Jotwani" wrote: This is a practical scenario where developers would be required to exclude

Re: [VOTE] Apache Apex Malhar Release 3.6.0 (RC1)

+1 (binding) Verified existance of LICENSE, NOTICE, README.md and CHANGELOG.md files Built with this command: mvn clean apache-rat:check verify -Dlicense.skip=false -Pall-modules install with no errors. Verified pi demo On Wed, Nov 30, 2016 at 11:37 AM, Siyuan Hua

[jira] [Resolved] (APEXMALHAR-2022) S3 Output Module for file copy

[ https://issues.apache.org/jira/browse/APEXMALHAR-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXMALHAR-2022. Resolution: Fixed Fix Version/s: 3.7.0 > S3 Output Module for file copy

[jira] [Commented] (APEXMALHAR-2022) S3 Output Module for file copy

[ https://issues.apache.org/jira/browse/APEXMALHAR-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710953#comment-15710953 ] ASF GitHub Bot commented on APEXMALHAR-2022: Github user asfgit closed the pull request

[GitHub] apex-malhar pull request #483: APEXMALHAR-2022 Developed S3 Output Module

Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/483 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

Re: [DISCUSSION] Custom Control Tuples

I would like to work on https://issues.apache.org/jira/browse/APEXCORE-580. ~ Bhupesh On Thu, Dec 1, 2016 at 5:42 AM, Sandesh Hegde wrote: > I am interested in working on the following subtask > > https://issues.apache.org/jira/browse/APEXCORE-581 > > Thanks > > > On

[jira] [Commented] (APEXMALHAR-2361) Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance

[ https://issues.apache.org/jira/browse/APEXMALHAR-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710326#comment-15710326 ] David Yan commented on APEXMALHAR-2361: --- I think removing it is better since you don't want to

[jira] [Commented] (APEXMALHAR-2361) Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance

[ https://issues.apache.org/jira/browse/APEXMALHAR-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710303#comment-15710303 ] bright chen commented on APEXMALHAR-2361: - Probably we can handle DISCARDING by update value

[jira] [Updated] (APEXCORE-581) Delivery of Custom Control Tuples

[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan updated APEXCORE-581: --- Description: The behavior should be as follow: - The control tuples should only be sent to

Re: [DISCUSSION] Custom Control Tuples

I am interested in working on the following subtask https://issues.apache.org/jira/browse/APEXCORE-581 Thanks On Wed, Nov 30, 2016 at 2:07 PM David Yan wrote: > I have created an umbrella ticket for control tuple support: > >

[GitHub] apex-malhar pull request #519: APEXMALHAR-2362 #resolve clearing the removed...

GitHub user davidyan74 opened a pull request: https://github.com/apache/apex-malhar/pull/519 APEXMALHAR-2362 #resolve clearing the removedSets at endWindow in SpillableSetMultimapImpl @brightchen please review and merge You can merge this pull request into a Git repository by

[jira] [Moved] (APEXMALHAR-2362) SpillableSetMulitmapImpl.removedSets keeps growing

[ https://issues.apache.org/jira/browse/APEXMALHAR-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan moved APEXCORE-582 to APEXMALHAR-2362: Workflow: Default workflow, editable Closed status (was:

[jira] [Created] (APEXCORE-582) SpillableSetMulitmapImpl.removedSets keeps growing

David Yan created APEXCORE-582: -- Summary: SpillableSetMulitmapImpl.removedSets keeps growing Key: APEXCORE-582 URL: https://issues.apache.org/jira/browse/APEXCORE-582 Project: Apache Apex Core

[jira] [Commented] (APEXMALHAR-2361) Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance

[ https://issues.apache.org/jira/browse/APEXMALHAR-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710094#comment-15710094 ] David Yan commented on APEXMALHAR-2361: --- It makes sense. This potentially skips a lot of

[jira] [Commented] (APEXMALHAR-2339) Windowed Operator benchmarking

[ https://issues.apache.org/jira/browse/APEXMALHAR-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710031#comment-15710031 ] bright chen commented on APEXMALHAR-2339: - SpillableWindowedKeyedStorage.remove(Window) will

[jira] [Created] (APEXMALHAR-2361) Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance

bright chen created APEXMALHAR-2361: --- Summary: Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance Key: APEXMALHAR-2361 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2361

[jira] [Commented] (APEXMALHAR-2359) Optimise fire trigger to avoid go through all data

[ https://issues.apache.org/jira/browse/APEXMALHAR-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709958#comment-15709958 ] ASF GitHub Bot commented on APEXMALHAR-2359: GitHub user brightchen opened a pull

[GitHub] apex-malhar pull request #518: APEXMALHAR-2359 #resolve #comment Optimise fi...

GitHub user brightchen opened a pull request: https://github.com/apache/apex-malhar/pull/518 APEXMALHAR-2359 #resolve #comment Optimise fire trigger to avoid go t… …hrough all data You can merge this pull request into a Git repository by running: $ git pull

Re: [DISCUSSION] Custom Control Tuples

I have created an umbrella ticket for control tuple support: https://issues.apache.org/jira/browse/APEXCORE-579 Currently it has two subtasks. Please have a look at them and see whether I'm missing anything or if you have anything to add. You are welcome to add more subtasks or comment on the

[jira] [Updated] (APEXCORE-581) Delivery of Custom Control Tuples

[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan updated APEXCORE-581: --- Description: The behavior should be as follow: - The control tuples should only be sent to

[jira] [Updated] (APEXCORE-580) Interface for processing and emitting control tuples

[ https://issues.apache.org/jira/browse/APEXCORE-580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan updated APEXCORE-580: --- Description: DefaultOutputPort needs to have a emitControl method so that operator code can call

[jira] [Updated] (APEXCORE-580) Interface for processing and emitting control tuples

[ https://issues.apache.org/jira/browse/APEXCORE-580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan updated APEXCORE-580: --- Summary: Interface for processing and emitting control tuples (was: Add methods for processing and

[jira] [Updated] (APEXCORE-580) Add methods for processing and emitting control tuples

[ https://issues.apache.org/jira/browse/APEXCORE-580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan updated APEXCORE-580: --- Description: DefaultOutputPort needs to have a emitControl method so that operator code can call

[jira] [Created] (APEXCORE-581) Delivery of Custom Control Tuples

David Yan created APEXCORE-581: -- Summary: Delivery of Custom Control Tuples Key: APEXCORE-581 URL: https://issues.apache.org/jira/browse/APEXCORE-581 Project: Apache Apex Core Issue Type:

[jira] [Created] (APEXCORE-580) Add methods for processing and emitting control tuples

David Yan created APEXCORE-580: -- Summary: Add methods for processing and emitting control tuples Key: APEXCORE-580 URL: https://issues.apache.org/jira/browse/APEXCORE-580 Project: Apache Apex Core

Re: "ExcludeNodes" for an Apex application

Apex has automatic blacklisting of the troublesome nodes, please take a look at the following attributes, MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Context.DAGContext.html#MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST

Re: "ExcludeNodes" for an Apex application

Not sure if this is what Milind had in mind but we often run into situations where the dev group working with Apex has no control over cluster configuration -- to make any changes to the cluster they need to go through an elaborate process that can take many days. Meanwhile, if they notice that a

Re: [VOTE] Apache Apex Malhar Release 3.6.0 (RC1)

+1 Verified checksums Verified compilation Verified build and test Verified pi demo On Wed, Nov 30, 2016 at 9:50 AM, Tushar Gosavi wrote: > +1 > > Verified checksums > Verified compilation > > - Tushar. > > > On Wed, Nov 30, 2016 at 7:43 PM, Thomas Weise

Re: "ExcludeNodes" for an Apex application

I agree, Randomly rebooting node is Yarn issue. Even anti-affinity between apps should be Yarn in long run. We could contribute the above jira. Thks Amol On Wed, Nov 30, 2016 at 10:58 AM, Sanjay Pujare wrote: > To me both use cases appear to be generic resource

Re: "ExcludeNodes" for an Apex application

To me both use cases appear to be generic resource management use cases. For example, a randomly rebooting node is not good for any purpose esp. long running apps so it is a bit of a stretch to imagine that these nodes will be acceptable for some batch jobs in Yarn. So such a node should be

Re: "ExcludeNodes" for an Apex application

But then, what's the solution to the 2 problem scenarios that Milind describes ? Ram On Wed, Nov 30, 2016 at 10:34 AM, Sanjay Pujare wrote: > I think “exclude nodes” and such is really the job of the resource manager > i.e. Yarn. So I am not sure taking over some of

Re: "ExcludeNodes" for an Apex application

I think “exclude nodes” and such is really the job of the resource manager i.e. Yarn. So I am not sure taking over some of these tasks in Apex would be very useful. I agree with Amol that apps should be node neutral. Resource management in Yarn together with fault tolerance in Apex should

Re: [VOTE] Apache Apex Malhar Release 3.6.0 (RC1)

Can folks please verify the release. Thanks -- sent from mobile On Nov 26, 2016 6:32 PM, "Thomas Weise" wrote: > Dear Community, > > Please vote on the following Apache Apex Malhar 3.6.0 release candidate. > > This is a source release with binary artifacts published to Maven.

[jira] [Commented] (APEXMALHAR-2022) S3 Output Module for file copy

[ https://issues.apache.org/jira/browse/APEXMALHAR-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708479#comment-15708479 ] ASF GitHub Bot commented on APEXMALHAR-2022: Github user chaithu14 closed the pull

[jira] [Commented] (APEXMALHAR-2022) S3 Output Module for file copy

[ https://issues.apache.org/jira/browse/APEXMALHAR-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708481#comment-15708481 ] ASF GitHub Bot commented on APEXMALHAR-2022: GitHub user chaithu14 reopened a pull

[GitHub] apex-malhar pull request #483: APEXMALHAR-2022 Developed S3 Output Module

GitHub user chaithu14 reopened a pull request: https://github.com/apache/apex-malhar/pull/483 APEXMALHAR-2022 Developed S3 Output Module You can merge this pull request into a Git repository by running: $ git pull https://github.com/chaithu14/incubator-apex-malhar

[GitHub] apex-malhar pull request #483: APEXMALHAR-2022 Developed S3 Output Module

Github user chaithu14 closed the pull request at: https://github.com/apache/apex-malhar/pull/483 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[jira] [Commented] (APEXMALHAR-2022) S3 Output Module for file copy

[ https://issues.apache.org/jira/browse/APEXMALHAR-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708392#comment-15708392 ] ASF GitHub Bot commented on APEXMALHAR-2022: Github user chaithu14 closed the pull

[GitHub] apex-malhar pull request #483: APEXMALHAR-2022 Developed S3 Output Module

GitHub user chaithu14 reopened a pull request: https://github.com/apache/apex-malhar/pull/483 APEXMALHAR-2022 Developed S3 Output Module You can merge this pull request into a Git repository by running: $ git pull https://github.com/chaithu14/incubator-apex-malhar

[jira] [Commented] (APEXMALHAR-2022) S3 Output Module for file copy

[ https://issues.apache.org/jira/browse/APEXMALHAR-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708393#comment-15708393 ] ASF GitHub Bot commented on APEXMALHAR-2022: GitHub user chaithu14 reopened a pull

[GitHub] apex-malhar pull request #483: APEXMALHAR-2022 Developed S3 Output Module

Github user chaithu14 closed the pull request at: https://github.com/apache/apex-malhar/pull/483 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the