[jira] [Created] (APEXMALHAR-2258) JavaExpressionParser does not cast type correctly when expression is binary
Chinmay Kolhatkar created APEXMALHAR-2258: - Summary: JavaExpressionParser does not cast type correctly when expression is binary Key: APEXMALHAR-2258 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2258 Project: Apache Apex Malhar Issue Type: Bug Reporter: Chinmay Kolhatkar Assignee: Chinmay Kolhatkar JavaExpressionParser does not cast type correctly when expression is binary. It should cast total of resolved expression to return type instead of just first part of expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2256) POJOInnerJoinOperator should use getDeclaredField of java reflection
[ https://issues.apache.org/jira/browse/APEXMALHAR-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509033#comment-15509033 ] ASF GitHub Bot commented on APEXMALHAR-2256: Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/421 > POJOInnerJoinOperator should use getDeclaredField of java reflection > > > Key: APEXMALHAR-2256 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2256 > Project: Apache Apex Malhar > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > > POJOInnerJoinOperator should use getDeclaredField of java reflection. > Currently its using getField because of which unless the POJO fields are > public, they're not see by reflection. In reality the fields in POJO will be > private and getters/setters will be public. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2258) JavaExpressionParser does not cast type correctly when expression is binary
[ https://issues.apache.org/jira/browse/APEXMALHAR-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509042#comment-15509042 ] ASF GitHub Bot commented on APEXMALHAR-2258: GitHub user chinmaykolhatkar opened a pull request: https://github.com/apache/apex-malhar/pull/422 APEXMALHAR-2258 JavaExpressionParser should handle casting of binary expression correctly @bhupeshchawda Please merge. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chinmaykolhatkar/apex-malhar APEXMALHAR-2258_JavaExpressionParser Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #422 commit d713e521e62849c8116e7c45bfce40ead3c60a73 Author: Chinmay Kolhatkar Date: 2016-09-21T07:16:54Z APEXMALHAR-2258 JavaExpressionParser should handle casting of binary expression correctly. > JavaExpressionParser does not cast type correctly when expression is binary > --- > > Key: APEXMALHAR-2258 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2258 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > > JavaExpressionParser does not cast type correctly when expression is binary. > It should cast total of resolved expression to return type instead of just > first part of expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXMALHAR-2256) POJOInnerJoinOperator should use getDeclaredField of java reflection
[ https://issues.apache.org/jira/browse/APEXMALHAR-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXMALHAR-2256. Resolution: Fixed Merged > POJOInnerJoinOperator should use getDeclaredField of java reflection > > > Key: APEXMALHAR-2256 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2256 > Project: Apache Apex Malhar > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > Fix For: 3.6.0 > > > POJOInnerJoinOperator should use getDeclaredField of java reflection. > Currently its using getField because of which unless the POJO fields are > public, they're not see by reflection. In reality the fields in POJO will be > private and getters/setters will be public. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXMALHAR-2256) POJOInnerJoinOperator should use getDeclaredField of java reflection
[ https://issues.apache.org/jira/browse/APEXMALHAR-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXMALHAR-2256: --- Fix Version/s: 3.6.0 > POJOInnerJoinOperator should use getDeclaredField of java reflection > > > Key: APEXMALHAR-2256 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2256 > Project: Apache Apex Malhar > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > Fix For: 3.6.0 > > > POJOInnerJoinOperator should use getDeclaredField of java reflection. > Currently its using getField because of which unless the POJO fields are > public, they're not see by reflection. In reality the fields in POJO will be > private and getters/setters will be public. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] apex-malhar pull request #421: APEXMALHAR-2256 Replacing getFields with getD...
Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #422: APEXMALHAR-2258 JavaExpressionParser should h...
GitHub user chinmaykolhatkar opened a pull request: https://github.com/apache/apex-malhar/pull/422 APEXMALHAR-2258 JavaExpressionParser should handle casting of binary expression correctly @bhupeshchawda Please merge. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chinmaykolhatkar/apex-malhar APEXMALHAR-2258_JavaExpressionParser Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #422 commit d713e521e62849c8116e7c45bfce40ead3c60a73 Author: Chinmay Kolhatkar Date: 2016-09-21T07:16:54Z APEXMALHAR-2258 JavaExpressionParser should handle casting of binary expression correctly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #422: APEXMALHAR-2258 JavaExpressionParser should h...
Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/422 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXMALHAR-2258) JavaExpressionParser does not cast type correctly when expression is binary
[ https://issues.apache.org/jira/browse/APEXMALHAR-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509192#comment-15509192 ] ASF GitHub Bot commented on APEXMALHAR-2258: Github user asfgit closed the pull request at: https://github.com/apache/apex-malhar/pull/422 > JavaExpressionParser does not cast type correctly when expression is binary > --- > > Key: APEXMALHAR-2258 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2258 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > Fix For: 3.6.0 > > > JavaExpressionParser does not cast type correctly when expression is binary. > It should cast total of resolved expression to return type instead of just > first part of expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXMALHAR-2258) JavaExpressionParser does not cast type correctly when expression is binary
[ https://issues.apache.org/jira/browse/APEXMALHAR-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXMALHAR-2258. Resolution: Fixed Fix Version/s: 3.6.0 Merged > JavaExpressionParser does not cast type correctly when expression is binary > --- > > Key: APEXMALHAR-2258 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2258 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > Fix For: 3.6.0 > > > JavaExpressionParser does not cast type correctly when expression is binary. > It should cast total of resolved expression to return type instead of just > first part of expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXMALHAR-2259) Create Fixed Length Parser Operator
Chinmay Kolhatkar created APEXMALHAR-2259: - Summary: Create Fixed Length Parser Operator Key: APEXMALHAR-2259 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2259 Project: Apache Apex Malhar Issue Type: Task Affects Versions: 3.5.0 Reporter: Chinmay Kolhatkar Assignee: Hitesh Kapoor Create Fixed Length Parser TODO: The description needs to be updated for design and requirement of Fixed length parser. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Improving Apex relaunch time.
Relaunching from the same location can be one of the options. On Tue, Sep 20, 2016, 10:17 PM Tushar Gosavi wrote: > In case of application failure, we will like to have ability to > quickly restart the application while keeping the old state for > failure > analysis. Also the problem remains the same when we want to start from > savepoint, where we will need to copy state from > savepoint to application. > > -Tushar. > > > > On Tue, Sep 20, 2016 at 8:34 PM, Sandesh Hegde > wrote: > > How about re-launching the app from the same location? > > > > If at all they want to store the state we can provide savepoint feature. > > > > On Tue, Sep 20, 2016 at 4:39 AM Tushar Gosavi > > wrote: > > > >> We have observed that application relaunch takes long time. > >> The one major reason for delay in application startup during relaunch > >> is time taken to copy state of exisitng application to new application. > >> This state could grow in GBs and copy is performed in single thread > before > >> new application is submitted to Yarn. > >> > >> The state of previous application constists > >> - jars > >> - stram checkpoint/recovery file. > >> - events > >> - container file > >> - stats recording if enabled. > >> - operator checkpoints > >> - operator data. > >> > >> We could avoid copying debugging data like stat recording which could > >> run in TB for long > >> running application and is not required for functioning of new > application. > >> > >> Similarly operator checkpoints could be read in parallel when they are > >> launched for first time, > >> This will also help in copying only required checkpoints and will be > >> done in parallel > >> by multiple containers/threads. > >> > >> For operator data stored in application directory, we could copy it > >> completely for now, but > >> in future we could provide an callback which will allow operator > >> partition to read only > >> required state from previous location. > >> > >> let me know your though on this. > >> > >> Regards, > >> - Tushar. > >> >
[jira] [Updated] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXMALHAR-2251: --- Description: Dependency on com.datatorrent.netlet.util.Slice should be replaced with either java.nio.ByteBuffer or internal malhar class that provides similar functionality. (was: The ) Issue Type: Improvement (was: Bug) Summary: Replace dependency on com.datatorrent.netlet.util.Slice (was: Slice hashCode should not depend on offset) > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510579#comment-15510579 ] Thomas Weise commented on APEXMALHAR-2251: -- We discussed to extend the netlet Slice class, override hashCode and then use this derived class everywhere in Malhar. This will remove the dependency everywhere else. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
excluding hadoop dependencies from application package
Is subject already documented? Thank you, Vlad
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510860#comment-15510860 ] Vlad Rozov commented on APEXMALHAR-2251: Extending netlet Slice class will not remove dependency as well as using delegation pattern. There is not much functionality in the Slice class that justifies its usage in Malhar. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2255) Use latest java sdk for couchbase
[ https://issues.apache.org/jira/browse/APEXMALHAR-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510868#comment-15510868 ] Vlad Rozov commented on APEXMALHAR-2255: Is the request to upgrade JDK or upgrade couchbase client library? It seems that it is later. Please update title accordingly and specify upgrade version. Is it backward compatible in a sense that it can be used to connect to older version of couchbase server? > Use latest java sdk for couchbase > - > > Key: APEXMALHAR-2255 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2255 > Project: Apache Apex Malhar > Issue Type: Test >Reporter: Priyanka Gugale > > Right now the Couchbase connector in Malhar uses couchbase-client library > which is outdated. We should instead use java-client library from couchbase > to connect to couchbase server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: excluding hadoop dependencies from application package
Some info here: http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts Ram On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov wrote: > Is subject already documented? > > Thank you, > > Vlad >
[jira] [Closed] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandesh closed APEXCORE-451. Resolution: Fixed Fix Version/s: 3.5.0 > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511027#comment-15511027 ] David Yan commented on APEXMALHAR-2251: --- Extending the netlet Slice class in Malhar is just a temporary measure to fix the hashCode issue and it paves the path of getting rid of netlet entirely when we migrate to Netty. It's a one time fix for everywhere that uses Slice. When we get rid of netlet, we can discuss what to do with the Slice class (whether we want it to extend from ByteBuffer or just copy the code from netlet minus the buggy hashCode function), without changing code that uses Slice. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: excluding hadoop dependencies from application package
It may be useful to have this enforced in the archetype by default. There may be situations where users want to include these dependencies, but those are exception cases. On Wed, Sep 21, 2016 at 12:24 PM, Munagala Ramanath wrote: > Some info here: > http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts > > Ram > > > On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov > wrote: > > > Is subject already documented? > > > > Thank you, > > > > Vlad > > >
Re: excluding hadoop dependencies from application package
Candidate to be added here? https://apex.apache.org/docs/apex/development_best_practices/ On Wed, Sep 21, 2016 at 12:24 PM, Munagala Ramanath wrote: > Some info here: > http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts > > Ram > > > On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov > wrote: > > > Is subject already documented? > > > > Thank you, > > > > Vlad > > >
Re: excluding hadoop dependencies from application package
copy-dependencies can be modified to skip hadoop and apex-engine dependencies: http://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html On Wed, Sep 21, 2016 at 1:13 PM, Thomas Weise wrote: > It may be useful to have this enforced in the archetype by default. > > There may be situations where users want to include these dependencies, > but those are exception cases. > > > On Wed, Sep 21, 2016 at 12:24 PM, Munagala Ramanath > wrote: > >> Some info here: >> http://docs.datatorrent.com/troubleshooting/#hadoop-dependen >> cies-conflicts >> >> Ram >> >> >> On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov >> wrote: >> >> > Is subject already documented? >> > >> > Thank you, >> > >> > Vlad >> > >> > >
[jira] [Commented] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511106#comment-15511106 ] Thomas Weise commented on APEXCORE-451: --- [~sandesh] JIRAs for the next release are to be marked resolved (not closed) by the committer that merges the change. > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandesh reopened APEXCORE-451: -- > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511127#comment-15511127 ] Sandesh commented on APEXCORE-451: -- [~thomasWeise] Sure, will open it now. [~shubhamp] can you please mark this jira appropriately. > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511146#comment-15511146 ] Thomas Weise commented on APEXCORE-451: --- See http://apex.apache.org/contributing.html#merging-a-pull-request-committers- Anyone merging PRs should have read it. And in case it can be made clearer, please submit a change to this page. > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: excluding hadoop dependencies from application package
+1 to both suggestions - changing copy dependencies in the apex application archetype and documenting in https://apex.apache.org/docs/apex/development_best_practices/. Thank you, Vlad On 9/21/16 13:20, Pramod Immaneni wrote: Candidate to be added here? https://apex.apache.org/docs/apex/development_best_practices/ On Wed, Sep 21, 2016 at 12:24 PM, Munagala Ramanath wrote: Some info here: http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts Ram On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov wrote: Is subject already documented? Thank you, Vlad
[jira] [Resolved] (APEXMALHAR-2217) Remove some redundant code in WindowedStorage and WindowedKeyedStorage
[ https://issues.apache.org/jira/browse/APEXMALHAR-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan resolved APEXMALHAR-2217. --- Resolution: Fixed Fix Version/s: 3.6.0 Fixed by https://github.com/apache/apex-malhar/pull/407 > Remove some redundant code in WindowedStorage and WindowedKeyedStorage > -- > > Key: APEXMALHAR-2217 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2217 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Siyuan Hua >Assignee: David Yan > Fix For: 3.6.0 > > > WindowedStorage has an entrySet() method which return the windowstorage > itself which is redundant. WindowedKeyedStorage has 2 methods remove and > migrateWindow can both inherit from parent interface -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511280#comment-15511280 ] shubham pathak commented on APEXCORE-451: - [~davidyan] Could you mark the jira appropriately ? > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-451) get-app-package-operators in ApexCLI to contain "type" property to indicate operator or module
[ https://issues.apache.org/jira/browse/APEXCORE-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan resolved APEXCORE-451. Resolution: Fixed > get-app-package-operators in ApexCLI to contain "type" property to indicate > operator or module > --- > > Key: APEXCORE-451 > URL: https://issues.apache.org/jira/browse/APEXCORE-451 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: shubham pathak >Assignee: shubham pathak > Fix For: 3.5.0 > > > Apex apps can contain both operators and modules. We would need an > additional property in the response of get-app-package-operators to > distinguish between the two. > We can have a "type" property that will be enum of values > e.g. "type" : "operator" to indicate operator >"type" : "composite" to indicate a module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] apex-core pull request #391: APEXCORE-533 layering of the properties in case...
Github user asfgit closed the pull request at: https://github.com/apache/apex-core/pull/391 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXCORE-533) Layering of AppPackage properties & configPackage properties in case of ConfigApps is not working.
[ https://issues.apache.org/jira/browse/APEXCORE-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511320#comment-15511320 ] ASF GitHub Bot commented on APEXCORE-533: - Github user asfgit closed the pull request at: https://github.com/apache/apex-core/pull/391 > Layering of AppPackage properties & configPackage properties in case of > ConfigApps is not working. > -- > > Key: APEXCORE-533 > URL: https://issues.apache.org/jira/browse/APEXCORE-533 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sandesh >Assignee: Sandesh > > Config Apps are not inheriting the properties set in the AppPackage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] apex-malhar pull request #404: APEXMALHAR-2190 #resolve #comment Use reusabl...
Github user brightchen closed the pull request at: https://github.com/apache/apex-malhar/pull/404 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #404: APEXMALHAR-2190 #resolve #comment Use reusabl...
GitHub user brightchen reopened a pull request: https://github.com/apache/apex-malhar/pull/404 APEXMALHAR-2190 #resolve #comment Use reusable buffer to serial spill⦠â¦able data structure You can merge this pull request into a Git repository by running: $ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2190-PR Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/404.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #404 commit 3541026395f2d13bc128dd7524eef7753fdd5dc6 Author: brightchen Date: 2016-08-16T00:46:27Z APEXMALHAR-2190 #resolve #comment Use reusable buffer to serial spillable data structure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXMALHAR-2190) Use reusable buffer to serial spillable data structure
[ https://issues.apache.org/jira/browse/APEXMALHAR-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511463#comment-15511463 ] ASF GitHub Bot commented on APEXMALHAR-2190: Github user brightchen closed the pull request at: https://github.com/apache/apex-malhar/pull/404 > Use reusable buffer to serial spillable data structure > -- > > Key: APEXMALHAR-2190 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2190 > Project: Apache Apex Malhar > Issue Type: Task >Reporter: bright chen >Assignee: bright chen > Original Estimate: 240h > Remaining Estimate: 240h > > Spillable Data Structure created lots of temporary memory to serial data lot > of of memory copy( see SliceUtils.concatenate(byte[], byte[]). Which used up > memory very quickly. See APEXMALHAR-2182. > Use a shared memory to avoid allocate temporary memory and memory copy > some basic ideas > - SerToLVBuffer interface provides a method serTo(T object, LengthValueBuffer > buffer): instead of create a memory and then return the serialized data, this > method let the caller pass in the buffer. So different objects or object with > embed objects can share the same LengthValueBuffer > - LengthValueBuffer: It is a buffer which manage the memory as length and > value(which is the generic format of serialized data). which provide length > placeholder mechanism to avoid temporary memory and data copy when the length > can be know after data serialized > - memory management classes: includes interface ByteStream and it's > implementations: Block, FixedBlock, BlocksStream. Which provides a mechanism > to dynamic allocate and manage memory. Which basically provides following > function. I tried other some other stream mechamism such as > ByteArrayInputStream, but it can meet 3rd criteria, and don't have good > performance(50% loss) > - dynamic allocate memory > - reset memory for reuse > - BlocksStream make sure the output slices will not be changed when need > extra memory; Block can change the reference of output slices buffer is data > was moved due to reallocate of memory(BlocksStream is better solution). > - WindowableBlocksStream extends from BlocksStream and provides function to > reset memory window by window instead of reset all memory. It provides > certain amount of cache( as bytes ) in memory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2190) Use reusable buffer to serial spillable data structure
[ https://issues.apache.org/jira/browse/APEXMALHAR-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511464#comment-15511464 ] ASF GitHub Bot commented on APEXMALHAR-2190: GitHub user brightchen reopened a pull request: https://github.com/apache/apex-malhar/pull/404 APEXMALHAR-2190 #resolve #comment Use reusable buffer to serial spill… …able data structure You can merge this pull request into a Git repository by running: $ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2190-PR Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/404.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #404 commit 3541026395f2d13bc128dd7524eef7753fdd5dc6 Author: brightchen Date: 2016-08-16T00:46:27Z APEXMALHAR-2190 #resolve #comment Use reusable buffer to serial spillable data structure > Use reusable buffer to serial spillable data structure > -- > > Key: APEXMALHAR-2190 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2190 > Project: Apache Apex Malhar > Issue Type: Task >Reporter: bright chen >Assignee: bright chen > Original Estimate: 240h > Remaining Estimate: 240h > > Spillable Data Structure created lots of temporary memory to serial data lot > of of memory copy( see SliceUtils.concatenate(byte[], byte[]). Which used up > memory very quickly. See APEXMALHAR-2182. > Use a shared memory to avoid allocate temporary memory and memory copy > some basic ideas > - SerToLVBuffer interface provides a method serTo(T object, LengthValueBuffer > buffer): instead of create a memory and then return the serialized data, this > method let the caller pass in the buffer. So different objects or object with > embed objects can share the same LengthValueBuffer > - LengthValueBuffer: It is a buffer which manage the memory as length and > value(which is the generic format of serialized data). which provide length > placeholder mechanism to avoid temporary memory and data copy when the length > can be know after data serialized > - memory management classes: includes interface ByteStream and it's > implementations: Block, FixedBlock, BlocksStream. Which provides a mechanism > to dynamic allocate and manage memory. Which basically provides following > function. I tried other some other stream mechamism such as > ByteArrayInputStream, but it can meet 3rd criteria, and don't have good > performance(50% loss) > - dynamic allocate memory > - reset memory for reuse > - BlocksStream make sure the output slices will not be changed when need > extra memory; Block can change the reference of output slices buffer is data > was moved due to reallocate of memory(BlocksStream is better solution). > - WindowableBlocksStream extends from BlocksStream and provides function to > reset memory window by window instead of reset all memory. It provides > certain amount of cache( as bytes ) in memory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-533) Layering of AppPackage properties & configPackage properties in case of ConfigApps is not working.
[ https://issues.apache.org/jira/browse/APEXCORE-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Yan resolved APEXCORE-533. Resolution: Fixed Fix Version/s: 3.5.0 > Layering of AppPackage properties & configPackage properties in case of > ConfigApps is not working. > -- > > Key: APEXCORE-533 > URL: https://issues.apache.org/jira/browse/APEXCORE-533 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sandesh >Assignee: Sandesh > Fix For: 3.5.0 > > > Config Apps are not inheriting the properties set in the AppPackage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511551#comment-15511551 ] Thomas Weise commented on APEXMALHAR-2251: -- +1 for using ByteBuffer instead of Slice > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511564#comment-15511564 ] David Yan commented on APEXMALHAR-2251: --- +1 on ByteBuffer as well. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511607#comment-15511607 ] Munagala V. Ramanath commented on APEXMALHAR-2251: -- Direct or indirect ? Also, ByteBuffers need some care to use correctly -- since people often forget to flip(). +1 for indirect > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512077#comment-15512077 ] Vlad Rozov commented on APEXMALHAR-2251: Do you mean Direct or Heap? Majority of the code should work with ByteBuffer and not worry whether it is Direct or Heap allocated. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512167#comment-15512167 ] Thomas Weise commented on APEXMALHAR-2251: -- There shouldn't be a situation where the distinction matters for replacement of Slice. Instead of constructing Slice, it would be ByteBuffer.wrap(...). Also did a few tests with equals, any two buffers with same byte sequence are considered equal. Compared to Slice a ByteBuffer adds a couple of fields, since it also deals with the position for manipulation etc. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: excluding hadoop dependencies from application package
If we modify copy-dependency-mojo we will need to keep it updated after every release of mojo. I would prefer having it documented as well as a comment in our archetype pom file would help. -Priyanka On Thu, Sep 22, 2016 at 2:55 AM, Vlad Rozov wrote: > +1 to both suggestions - changing copy dependencies in the apex > application archetype and documenting in https://apex.apache.org/docs/a > pex/development_best_practices/. > > Thank you, > > Vlad > > > On 9/21/16 13:20, Pramod Immaneni wrote: > >> Candidate to be added here? >> >> https://apex.apache.org/docs/apex/development_best_practices/ >> >> On Wed, Sep 21, 2016 at 12:24 PM, Munagala Ramanath >> wrote: >> >> Some info here: >>> http://docs.datatorrent.com/troubleshooting/#hadoop-dependen >>> cies-conflicts >>> >>> Ram >>> >>> >>> On Wed, Sep 21, 2016 at 12:00 PM, Vlad Rozov >>> wrote: >>> >>> Is subject already documented? Thank you, Vlad >
PrintWriter for LogicalPlan
Hi All, For testing of calcite generated DAG, I want to verify whether the correct DAG is created with right operators and streams and whether properties are set properly. Is there any PrintWriter for the LogicalPlan object printing components of DAG? If not, I am planning to write one for Calcite testing. I was wondering if that would be useful addition to apex-core for testing purpose. Please share your opinion. Thanks, Chinmay.
Re: Python support
I would like to help in contributing to this feature. On Wed, Sep 21, 2016 at 12:26 AM, Sasha Parfenov wrote: > +1 on both executing Python code in an operator and high level API for > constructing Pipelines in Python. > > There is a large user base of engineers and data scientists which use > Python on regular basis for crunching through big data. Providing them > with a powerful new platform for big data processing, wrapped in a familiar > language, will open Apex to a much broader user base and help grow the > project. > > Given the potentially new user base of Python developers, it may make sense > to prioritize the high level API for pipeline construction. This will > allow users to build simple applications with existing library operators, > and we can get feedback on what areas they would like to see improved next > - custom Python operator support or more built-in library operators. > > Thanks, > Sasha > > On Thu, Sep 15, 2016 at 2:06 PM, Thomas Weise wrote: > > > Hi, > > > > Python (not Jython) seems to be a popular language and frequently used > for > > data analysis, especially where flexibility matters. It has a > comprehensive > > library and it is generally considered low barrier to entry. I have also > > seen Python used in critical back-end components, although that's > probably > > not very common? > > > > I think Python support could potentially expand the user base for Apex. > > There are 2 main areas that can be considered: > > > > 1) Support to execute Python code through an operator > > 2) A client API that lets users construct pipelines in Python > > > > The former can exist without the latter. And it would enable users to > > leverage existing code that otherwise would have to be rewritten in a JVM > > language. The engine could ship scripts/packages so they are > automatically > > distributed on the cluster. > > > > A useful client API probably requires back-end support for lambda > functions > > and more complex UDFs. > > > > Would be great to get some feedback, especially from those that have > > experience with Python, on how an integration could potentially open up > new > > use cases for Apex. > > > > Thanks, > > Thomas > > >
Re: PrintWriter for LogicalPlan
There is a LogicalPlanSerializer in apex engine, which will generate json equivalent of DAG, which can be printed easily. But for verification check if right components are added in DAG by examining dag structure than printing it. - Tushar. On Thu, Sep 22, 2016 at 11:39 AM, Chinmay Kolhatkar wrote: > Hi All, > > For testing of calcite generated DAG, I want to verify whether the correct > DAG is created with right operators and streams and whether properties are > set properly. > > Is there any PrintWriter for the LogicalPlan object printing components of > DAG? > > If not, I am planning to write one for Calcite testing. > I was wondering if that would be useful addition to apex-core for testing > purpose. > > Please share your opinion. > > Thanks, > Chinmay.
Re: PrintWriter for LogicalPlan
Hi Tushar, Thanks for the information. Can you please provide me pointers on how to use it? I can always examine the LogicalPlan structure but for a test case, I think its an overkill, instead if I have a utility which can print LogicalPlan (similar to LogicalPlanSerializer), then its just the matter for string comparison which one has to do in test for verification. -Chinmay. On Thu, Sep 22, 2016 at 11:59 AM, Tushar Gosavi wrote: > There is a LogicalPlanSerializer in apex engine, which will generate > json equivalent of DAG, which can be printed easily. > But for verification check if right components are added in DAG by > examining dag structure than printing it. > > - Tushar. > > > On Thu, Sep 22, 2016 at 11:39 AM, Chinmay Kolhatkar > wrote: > > Hi All, > > > > For testing of calcite generated DAG, I want to verify whether the > correct > > DAG is created with right operators and streams and whether properties > are > > set properly. > > > > Is there any PrintWriter for the LogicalPlan object printing components > of > > DAG? > > > > If not, I am planning to write one for Calcite testing. > > I was wondering if that would be useful addition to apex-core for testing > > purpose. > > > > Please share your opinion. > > > > Thanks, > > Chinmay. >
[jira] [Updated] (APEXCORE-310) apex cli - support to kill the app by appname
[ https://issues.apache.org/jira/browse/APEXCORE-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-310: --- Summary: apex cli - support to kill the app by appname (was: dtcli - support to kill the app by appname) > apex cli - support to kill the app by appname > - > > Key: APEXCORE-310 > URL: https://issues.apache.org/jira/browse/APEXCORE-310 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Priority: Minor > > dtcli should support the ability to kill the app by appname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2255) Use latest java sdk for couchbase
[ https://issues.apache.org/jira/browse/APEXMALHAR-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512385#comment-15512385 ] Priyanka Gugale commented on APEXMALHAR-2255: - The request is to upgrade *couchbase java sdk* i.e. couchbase client library. I will update the title. Here is the compatibility matrix: http://developer.couchbase.com/documentation/server/current/sdk/java/compatibility-versions-features.html I would suggest we should pick the latest at time from: https://mvnrepository.com/artifact/com.couchbase.client/java-client (by checking compatibility from above link) > Use latest java sdk for couchbase > - > > Key: APEXMALHAR-2255 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2255 > Project: Apache Apex Malhar > Issue Type: Test >Reporter: Priyanka Gugale > > Right now the Couchbase connector in Malhar uses couchbase-client library > which is outdated. We should instead use java-client library from couchbase > to connect to couchbase server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXMALHAR-2255) Use latest couchbase client library
[ https://issues.apache.org/jira/browse/APEXMALHAR-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Gugale updated APEXMALHAR-2255: Summary: Use latest couchbase client library (was: Use latest java sdk for couchbase) > Use latest couchbase client library > --- > > Key: APEXMALHAR-2255 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2255 > Project: Apache Apex Malhar > Issue Type: Test >Reporter: Priyanka Gugale > > Right now the Couchbase connector in Malhar uses couchbase-client library > which is outdated. We should instead use java-client library from couchbase > to connect to couchbase server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: PrintWriter for LogicalPlan
Hi Chinmay, take a look at following gist for example. https://gist.github.com/anonymous/8df8e05ffc16e620bbc030e1034da3c4 - Tushar. On Thu, Sep 22, 2016 at 12:05 PM, Chinmay Kolhatkar wrote: > Hi Tushar, > > Thanks for the information. Can you please provide me pointers on how to > use it? > > I can always examine the LogicalPlan structure but for a test case, I think > its an overkill, instead if I have a utility which can print LogicalPlan > (similar to LogicalPlanSerializer), then its just the matter for string > comparison which one has to do in test for verification. > > -Chinmay. > > > > On Thu, Sep 22, 2016 at 11:59 AM, Tushar Gosavi > wrote: > >> There is a LogicalPlanSerializer in apex engine, which will generate >> json equivalent of DAG, which can be printed easily. >> But for verification check if right components are added in DAG by >> examining dag structure than printing it. >> >> - Tushar. >> >> >> On Thu, Sep 22, 2016 at 11:39 AM, Chinmay Kolhatkar >> wrote: >> > Hi All, >> > >> > For testing of calcite generated DAG, I want to verify whether the >> correct >> > DAG is created with right operators and streams and whether properties >> are >> > set properly. >> > >> > Is there any PrintWriter for the LogicalPlan object printing components >> of >> > DAG? >> > >> > If not, I am planning to write one for Calcite testing. >> > I was wondering if that would be useful addition to apex-core for testing >> > purpose. >> > >> > Please share your opinion. >> > >> > Thanks, >> > Chinmay. >>
