[jira] [Commented] (APEXCORE-501) Sub Partitioning the parallel partitions

[ https://issues.apache.org/jira/browse/APEXCORE-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408917#comment-15408917 ] Yogi Devendra commented on APEXCORE-501: Link for the discussion:

[jira] [Created] (APEXCORE-501) Sub Partitioning the parallel partitions

Yogi Devendra created APEXCORE-501: -- Summary: Sub Partitioning the parallel partitions Key: APEXCORE-501 URL: https://issues.apache.org/jira/browse/APEXCORE-501 Project: Apache Apex Core

[GitHub] apex-malhar pull request #335: APEXMALHAR-1701 Deduper with Managed state

Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/335 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

Re: empty operator/stream/module names

When we need to change plan dynamically through dtcli, we need name to delete or attach to existing operator/port. I am fine with using system generated name when user do not provide name while adding operator/module/stream. -Tushar. On Thu, Aug 4, 2016 at 10:33 PM, Sanjay Pujare

[jira] [Updated] (APEXMALHAR-1701) Deduper : create a deduper backed by Managed State

[ https://issues.apache.org/jira/browse/APEXMALHAR-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kolhatkar updated APEXMALHAR-1701: -- Assignee: Bhupesh Chawda > Deduper : create a deduper backed by Managed

Re: empty operator/stream/module names

Names are already validated for uniqueness. I believe we still allow names that are not fully compatible with the REST API, but otherwise all unique names are considered to be valid. Vlad On 8/4/16 10:44, Sanjay Pujare wrote: That’s a good point. System generated names can still be made to

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408560#comment-15408560 ] ASF GitHub Bot commented on APEXCORE-495: - Github user sandeshh commented on a diff in the pull

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73609127 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -180,4 +188,40 @@ public void

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408380#comment-15408380 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73589696 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -180,4 +188,40 @@ public void

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408361#comment-15408361 ] ASF GitHub Bot commented on APEXCORE-495: - Github user sandeshh commented on a diff in the pull

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user sandeshh commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73584686 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -180,4 +188,40 @@ public void

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408347#comment-15408347 ] ASF GitHub Bot commented on APEXCORE-495: - Github user sandeshh commented on a diff in the pull

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73583092 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -53,16 +56,16 @@ private static File configFile;

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408332#comment-15408332 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408330#comment-15408330 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73582758 --- Diff: engine/src/test/resources/testAppPackage/mydtapp/src/main/resources/com/example/mydtapp/Application/ExtRes.html --- @@ -0,0 +1,21 @@ +

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73582496 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -126,10 +130,14 @@ public void

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408318#comment-15408318 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73582122 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -103,8 +107,8 @@ public static void finished() public void

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408314#comment-15408314 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

Re: [Proposal] Named Checkpoints

hmm! actually it may be a good debugging tool too. Keep the named checkpoints around. The feature is to keep checkpoints around, which can be done by giving a feature to not delete checkpoints, but then naming them makes it more operational. Send a command from cli->get checkpoint -> know it is

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73581887 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -180,4 +188,40 @@ public void

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73581539 --- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java --- @@ -180,4 +188,40 @@ public void

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408307#comment-15408307 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/apex-core/pull/360#discussion_r73581176 --- Diff: engine/src/main/java/com/datatorrent/stram/client/AppPackage.java --- @@ -49,6 +49,8 @@ import net.lingala.zip4j.exception.ZipException;

[jira] [Commented] (APEXCORE-495) Enhancing the configuration package to store apps

[ https://issues.apache.org/jira/browse/APEXCORE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408301#comment-15408301 ] ASF GitHub Bot commented on APEXCORE-495: - Github user davidyan74 commented on a diff in the

Re: empty operator/stream/module names

That’s a good point. System generated names can still be made to work for this use-case but I see the reason for having a name. But then another set of questions come up: we need to validate the name for uniqueness within an app, valid syntax etc. May be it’s already being done. On 8/4/16,

Re: empty operator/stream/module names

A couple of minor issues with empty names: It will not be possible to configure such operators from an XML file other than through wildcards. Also, the AM log messages will not be as informative with an empty name, e.g. 2016-08-02 10:04:24,294 INFO com.datatorrent.stram.ResourceRequestHandler:

Re: empty operator/stream/module names

It will not be possible to configure such operators from an XML file other than through wildcards -- but maybe that's OK. Ram On Thu, Aug 4, 2016 at 10:03 AM, Sanjay Pujare wrote: > I differ. For the UI to render a DAG the names are useful, but if the name > is not

Re: empty operator/stream/module names

I agree with Sanjay. Error should be what the engine cannot do without. The lesser work an user can get away with (a choice for a forced option) the better. So if users want they can name these objects, but if they don't it still works. Thks Amol On Thu, Aug 4, 2016 at 10:03 AM, Sanjay Pujare

Re: empty operator/stream/module names

I differ. For the UI to render a DAG the names are useful, but if the name is not required by the engine i.e. the engine is able to execute your application fine with empty or null strings as names, is there any reason to make them mandatory? On the other hand, we can come up with a scheme

Re: empty operator/stream/module names

I don't see any reason to allow either. Ram On Thu, Aug 4, 2016 at 8:51 AM, Vlad Rozov wrote: > Currently addOperator/addStream/addModule allows both null and empty > string in the operator/stream/module names. Is there any reason to allow > empty string? Should empty

empty operator/stream/module names

Currently addOperator/addStream/addModule allows both null and empty string in the operator/stream/module names. Is there any reason to allow empty string? Should empty string and null be disallowed in those APIs? Vlad

Re: [Proposal] Named Checkpoints

Sandesh, Agreed. There should be external APIs to acces the feature, if we want to integrate it to 3rd party tools. Thanks, A _ Sent with difficulty, I mean handheld ;) On 4 Aug 2016 8:47 pm, "Sandesh Hegde" wrote: > @Chinmay > We

[jira] [Commented] (APEXCORE-448) Make operator name available in OperatorContext

[ https://issues.apache.org/jira/browse/APEXCORE-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407955#comment-15407955 ] ASF GitHub Bot commented on APEXCORE-448: - Github user PramodSSImmaneni commented on the issue:

[jira] [Commented] (APEXCORE-448) Make operator name available in OperatorContext

[ https://issues.apache.org/jira/browse/APEXCORE-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407941#comment-15407941 ] ASF GitHub Bot commented on APEXCORE-448: - Github user vrozov commented on the issue:

[GitHub] apex-core issue #364: APEXCORE-448 Made operator name available in operator ...

Github user vrozov commented on the issue: https://github.com/apache/apex-core/pull/364 As far as I can see not all changes are correct (please see my comments regarding UnifierDeployInfo and access from two different threads to the operatorName). It will be good to have them fixed

[GitHub] apex-core issue #364: APEXCORE-448 Made operator name available in operator ...

Github user PramodSSImmaneni commented on the issue: https://github.com/apache/apex-core/pull/364 I second it, please reopen this PR and I will merge it. I think we have spent enough time on the tests and everyone's inputs have been valuable. There will be other opportunities in

Re: [Proposal] Named Checkpoints

@Chinmay We can enhance the existing checkpoint tuple but that one is more frequently used than this feature, so why burden Checkpoint tuple with an extra field? @Aniruddha It is better to leave the scheduling to the users, they can use any tool that they are already familiar with. On Thu, Aug

Re: [Proposal] Named Checkpoints

+1 On the idea, it would be awesome to have. Question: Can we further develop this brilliant idea into:- Scheduled checkpoints ( To save as dynamically named checkpoint)? This would be on the lines of logrotate / general backup strategies. Thanks, A _ Sent

Re: [Proposal] Named Checkpoints

+1 Ram On Thu, Aug 4, 2016 at 12:10 AM, Sandesh Hegde wrote: > Hello Team, > > This thread is to discuss the Named Checkpoint feature for Apex. ( > https://issues.apache.org/jira/browse/APEXCORE-498) > > Named checkpoints allow following workflow, > > 1. Users can

[jira] [Commented] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds

[ https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407850#comment-15407850 ] ASF GitHub Bot commented on APEXMALHAR-2176: GitHub user yogidevendra opened a pull

[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...

GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/361 APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method You can merge this pull request into a Git repository by running: $ git pull

[jira] [Comment Edited] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds

[ https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407815#comment-15407815 ] Yogi Devendra edited comment on APEXMALHAR-2176 at 8/4/16 2:10 PM: ---

[jira] [Commented] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds

[ https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407815#comment-15407815 ] Yogi Devendra commented on APEXMALHAR-2176: --- additional setter method for listItem with

[jira] [Created] (APEXMALHAR-2175) POJOUtils is considering java reserved words as fields in the expression

Yogi Devendra created APEXMALHAR-2175: - Summary: POJOUtils is considering java reserved words as fields in the expression Key: APEXMALHAR-2175 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2175

[jira] [Commented] (APEXCORE-448) Make operator name available in OperatorContext

[ https://issues.apache.org/jira/browse/APEXCORE-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407454#comment-15407454 ] ASF GitHub Bot commented on APEXCORE-448: - Github user tweise commented on the issue:

[GitHub] apex-core issue #364: APEXCORE-448 Made operator name available in operator ...

Github user tweise commented on the issue: https://github.com/apache/apex-core/pull/364 This pull request attracts a lot of attention... Not to spoil the party, but this looks like a rather minor change and there are many other pull requests that I would consider of much higher

Re: [Proposal] Named Checkpoints

Nice feature. +1 for it. One question, instead of creating a new tuple type, can this be done by modifying current checkpoint tuple? On Thu, Aug 4, 2016 at 2:02 PM, Yogi Devendra wrote: > This will be awesome feature. I can see the usecases for production > scenario

Re: [Proposal] Named Checkpoints

This will be awesome feature. I can see the usecases for production scenario as well as developement/troubleshooting environments. Excellent value add. ~ Yogi On 4 August 2016 at 12:40, Sandesh Hegde wrote: > Hello Team, > > This thread is to discuss the Named

Enricher - Support to load data from other file formats in FSLoader

Hello Community, I would like to take up this JIRA issue https://issues.apache.org/jira/browse/APEXMALHAR-2151 In the current implementation, FSLoader loads the data from a file but the data needs to be in JSON format. Support for reading from different formats would be a good addition. To be

[GitHub] apex-malhar pull request #360: APEXMALHAR-2174-S3-ReaderIssue Fixed the S3 r...

Github user yogidevendra commented on a diff in the pull request: https://github.com/apache/apex-malhar/pull/360#discussion_r73466833 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/S3BlockReader.java --- @@ -47,63 +56,144 @@ public S3BlockReader() public void