[jira] [Created] (APEXMALHAR-2564) Extend the UI Console to show physical operator info

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2564:
-

 Summary: Extend the UI Console to show physical operator info
 Key: APEXMALHAR-2564
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2564
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXMALHAR-2562) Update backend to serve console UI from stram

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2562:
-

 Summary: Update backend to serve console UI from stram
 Key: APEXMALHAR-2562
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2562
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXMALHAR-2563) Extend the UI Console to show logical operator info

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2563:
-

 Summary: Extend the UI Console to show logical operator info
 Key: APEXMALHAR-2563
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2563
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXMALHAR-2561) Update build steps for release build

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2561:
-

 Summary: Update build steps for release build
 Key: APEXMALHAR-2561
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2561
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXMALHAR-2560) Create react project which shows application level information

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2560:
-

 Summary: Create react project which shows application level 
information
 Key: APEXMALHAR-2560
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2560
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXMALHAR-2559) Add operational elements to UI Console

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2559:
-

 Summary: Add operational elements to UI Console 
 Key: APEXMALHAR-2559
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2559
 Project: Apache Apex Malhar
  Issue Type: New Feature
  Components: ui console
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


This includes control related operations allowed from the UI, as follows:
   a. Stop/Kill application
   b. Stop/Kill containers
   c. Stack trace dump of containers
   d. Logs related?
   d. etc...??



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXMALHAR-2559) Add operational elements to UI Console

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2559:
-

Assignee: (was: Chinmay Kolhatkar)

> Add operational elements to UI Console 
> ---
>
> Key: APEXMALHAR-2559
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2559
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: ui console
>Reporter: Chinmay Kolhatkar
>Priority: Critical
>
> This includes control related operations allowed from the UI, as follows:
>    a. Stop/Kill application
>    b. Stop/Kill containers
>    c. Stack trace dump of containers
>    d. Logs related?
>    d. etc...??



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXMALHAR-2558) Create UI Console for showing realtime apex app information

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2558:
-

Assignee: Chinmay Kolhatkar

> Create UI Console for showing realtime apex app information
> ---
>
> Key: APEXMALHAR-2558
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2558
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: ui console
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>Priority: Critical
>
> This includes following sub-features:
> 1. The webpage will be hosted on the same port as that of stram webservice 
> but at a different path. Say, http://:/ui
>  
> 2. The webpage will be a static page and will be a compiled from reactJS 
> application.
>  
> 3. There will be a categories of readonly information that will be shown on 
> the UI:
>    a. Application level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Start time/ Running span
>        iv. Stram events
>        v. Total memory used/available
>        vi. Number of container allocated/deployed
>        v. Other information which is available in stram
>    b. Logical Operator level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Important events related to logical operator
>        iv. Container list in which the operator is deployed
>        v. Any other information available in stram
>        vi. Logical DAG View
>    c. Container level information
>        i. Status
>        ii. Number of tuples processed/emitted
>        iii. Important events related to logical operator
>        iv. Any other information available in stram
>        v. Physical DAG View



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXMALHAR-2558) Create UI Console for showing realtime apex app information

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2558:
-

 Summary: Create UI Console for showing realtime apex app 
information
 Key: APEXMALHAR-2558
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2558
 Project: Apache Apex Malhar
  Issue Type: New Feature
  Components: ui console
Reporter: Chinmay Kolhatkar


This includes following sub-features:
1. The webpage will be hosted on the same port as that of stram webservice but 
at a different path. Say, http://:/ui
 
2. The webpage will be a static page and will be a compiled from reactJS 
application.
 
3. There will be a categories of readonly information that will be shown on the 
UI:
   a. Application level information
       i. Status
       ii. Number of tuples processed/emitted
       iii. Start time/ Running span
       iv. Stram events
       v. Total memory used/available
       vi. Number of container allocated/deployed
       v. Other information which is available in stram
   b. Logical Operator level information
       i. Status
       ii. Number of tuples processed/emitted
       iii. Important events related to logical operator
       iv. Container list in which the operator is deployed
       v. Any other information available in stram
       vi. Logical DAG View
   c. Container level information
       i. Status
       ii. Number of tuples processed/emitted
       iii. Important events related to logical operator
       iv. Any other information available in stram
       v. Physical DAG View



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-289) Encrypted Streams in Apex DAG

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-289:
--

Assignee: (was: Chinmay Kolhatkar)

> Encrypted Streams in Apex DAG
> -
>
> Key: APEXCORE-289
> URL: https://issues.apache.org/jira/browse/APEXCORE-289
> Project: Apache Apex Core
>  Issue Type: New Feature
>Reporter: Chinmay Kolhatkar
>Priority: Major
>  Labels: roadmap
>
> We should support encrypted streams in a DAG for Apex.
> Basically there will be 2 ways user can configure the streams for encryption:
> 1) App wide attributes- Using which all the stream in the DAG will have 
> encrypted channel.
> 2) Stream based attribute - Using this user can set a certain stream to flow 
> over encrypted channel.
> Encrypted for the streams should done at Network/Buffer Server levels.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXMALHAR-1979) Ability to generate compiled class from given json schema

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-1979:
-

Assignee: (was: Chinmay Kolhatkar)

> Ability to generate compiled class from given json schema
> -
>
> Key: APEXMALHAR-1979
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1979
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Chinmay Kolhatkar
>Priority: Major
>
> Ability to generate compiled class from given json schema.
> Will add the details on json schema soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXMALHAR-2053) Add utility for making schema dynamically configuration to DAG via populateDAG

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2053:
-

Assignee: (was: Chinmay Kolhatkar)

> Add utility for making schema dynamically configuration to DAG via populateDAG
> --
>
> Key: APEXMALHAR-2053
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2053
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: utilities
>Affects Versions: 3.3.0
>Reporter: Chinmay Kolhatkar
>Priority: Major
>
> Added utility using which schema can be made as a configurable property.
> As a part of this also add SchemaRegistry interface which can be used as 
> guideline for retrieving schemas from different sources. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-623) Schema propagation support in Apex

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-623:
--

Assignee: (was: Chinmay Kolhatkar)

> Schema propagation support in Apex
> --
>
> Key: APEXCORE-623
> URL: https://issues.apache.org/jira/browse/APEXCORE-623
> Project: Apache Apex Core
>  Issue Type: New Feature
>Reporter: Chinmay Kolhatkar
>Priority: Major
>
> In this feature, following will be done:
> 1. Add an interface which can be implemented by operators to help in 
> communicating the output schema of its port.
> 2. Update Logical Plan to provide propagation of schema across the pipeline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (APEXCORE-813) Update docker build steps to use apex bin dist package

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466285#comment-16466285
 ] 

Chinmay Kolhatkar commented on APEXCORE-813:


[~thw] and others (I wish, there was a way to tag all community):

There are 2 ways to deal with this jira:

1. When we release the apex-core, we also publish binary dist package on 
dist.apache.org and then let the docker hub automated build (basically during 
apex image building) pick it from dist.apache.org and use it.

2. Irrespective of whether the binary package is present on dist.apache.org, 
during docker image build process, we use the given branch (from which 
Dockerfile is used) to rebuild the binary package and use it.

 

First one is clearer but then its not possible to create images on SNAPSHOT 
build unless we parameterize the docker build process in some way.

Second way will ensure always building a new binary dist from the branch but 
not necessary mean its using the apache released dist package.

 

Which approach seems better?

 

> Update docker build steps to use apex bin dist package
> --
>
> Key: APEXCORE-813
> URL: https://issues.apache.org/jira/browse/APEXCORE-813
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>Priority: Major
>
> Currently docker build steps picks up apex deb from bigtop CI for installing 
> in apex docker image.
> Because of this docker image generation becomes dependent on update to bigtop 
> and then CI build to be generated from that.
> Instead the the binary package of apex can be used in docker image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-812) Update Docker Hub build account with build src from apache repository

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-812:
--

Assignee: Chinmay Kolhatkar

> Update Docker Hub build account with build src from apache repository
> -
>
> Key: APEXCORE-812
> URL: https://issues.apache.org/jira/browse/APEXCORE-812
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>Priority: Major
>
> Currently the docker hub is pulling code from 
> *_github:chinmaykolhatkar/docker-pool_* repository to build the docker image. 
> Instead of that it should pick the build src from git:apache/apex-core from 
> docker/ folder instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-813) Update docker build steps to use apex bin dist package

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-813:
--

Assignee: Chinmay Kolhatkar

> Update docker build steps to use apex bin dist package
> --
>
> Key: APEXCORE-813
> URL: https://issues.apache.org/jira/browse/APEXCORE-813
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>Priority: Major
>
> Currently docker build steps picks up apex deb from bigtop CI for installing 
> in apex docker image.
> Because of this docker image generation becomes dependent on update to bigtop 
> and then CI build to be generated from that.
> Instead the the binary package of apex can be used in docker image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXCORE-814) Rename envvar DT_CLIENT_OPTS to APEX_CLIENT_OPTS

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-814:
--

 Summary: Rename envvar DT_CLIENT_OPTS to APEX_CLIENT_OPTS
 Key: APEXCORE-814
 URL: https://issues.apache.org/jira/browse/APEXCORE-814
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Chinmay Kolhatkar


Currently DT_CLIENT_OPTS is the environment variables from which apexcli reads 
the client specific options. This should be renamed to APEX_CLIENT_OPTS.

Also any documentation related work towards this should also be addressed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXCORE-813) Update docker build steps to generate apex deb from dist binary package from apex-core repo

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-813:
--

 Summary: Update docker build steps to generate apex deb from dist 
binary package from apex-core repo
 Key: APEXCORE-813
 URL: https://issues.apache.org/jira/browse/APEXCORE-813
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Chinmay Kolhatkar


Currently docker build steps picks up apex deb from bigtop CI for installing in 
apex docker image.

Because of this docker image generation becomes dependent on update to bigtop 
and then CI build to be generated from that.

Instead the the binary package of apex can be used in docker image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXCORE-812) Update Docker Hub build account with build src from apache repository

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-812:
--

 Summary: Update Docker Hub build account with build src from 
apache repository
 Key: APEXCORE-812
 URL: https://issues.apache.org/jira/browse/APEXCORE-812
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Chinmay Kolhatkar


Currently the docker hub is pulling code from 
*_github:chinmaykolhatkar/docker-pool_* repository to build the docker image. 
Instead of that it should pick the build src from git:apache/apex-core from 
docker/ folder instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (APEXCORE-811) Move docker build sourcecode from private repository to apex-core repo

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-811:
--

 Summary: Move docker build sourcecode from private repository to 
apex-core repo
 Key: APEXCORE-811
 URL: https://issues.apache.org/jira/browse/APEXCORE-811
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Currently the docker source code for building apex docker is present in a 
personal repository:

[https://github.com/chinmaykolhatkar/docker-pool]

The code should be moved to apex-core repository and the docker build should be 
triggered from there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (APEXCORE-729) Include an example in the apex core release binary

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXCORE-729:
---
Summary: Include an example in the apex core release binary  (was: Include 
an example in the release)

> Include an example in the apex core release binary
> --
>
> Key: APEXCORE-729
> URL: https://issues.apache.org/jira/browse/APEXCORE-729
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> Include a self-contained sample example that the user can run easily and see 
> how the platform works, in the binary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (APEXCORE-729) Include an example in the release

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXCORE-729:
---
Issue Type: Task  (was: Sub-task)
Parent: (was: APEXCORE-727)

> Include an example in the release
> -
>
> Key: APEXCORE-729
> URL: https://issues.apache.org/jira/browse/APEXCORE-729
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> Include a self-contained sample example that the user can run easily and see 
> how the platform works, in the binary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (APEXCORE-729) Include an example in the release

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463263#comment-16463263
 ] 

Chinmay Kolhatkar commented on APEXCORE-729:


I suggest we make this as a top level task as this would required work in both 
apex-malhar and apex-core... The other 2 subtasks in parent task can 
independently get done. What do you guys think? 

> Include an example in the release
> -
>
> Key: APEXCORE-729
> URL: https://issues.apache.org/jira/browse/APEXCORE-729
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> Include a self-contained sample example that the user can run easily and see 
> how the platform works, in the binary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-728) Include licenses for dependencies

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-728:
--

Assignee: Chinmay Kolhatkar

> Include licenses for dependencies
> -
>
> Key: APEXCORE-728
> URL: https://issues.apache.org/jira/browse/APEXCORE-728
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> The correct LICENSE and NOTICE files should be included based on the 
> dependencies that are being packaged in the binary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-731) Modifications to the build to generate binary package

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-731:
--

Assignee: Chinmay Kolhatkar  (was: Thomas Weise)

> Modifications to the build to generate binary package
> -
>
> Key: APEXCORE-731
> URL: https://issues.apache.org/jira/browse/APEXCORE-731
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> The maven build would need to be updated to also optionally generate package 
> containing the binaries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-729) Include an example in the release

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-729:
--

Assignee: Chinmay Kolhatkar

> Include an example in the release
> -
>
> Key: APEXCORE-729
> URL: https://issues.apache.org/jira/browse/APEXCORE-729
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> Include a self-contained sample example that the user can run easily and see 
> how the platform works, in the binary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (APEXCORE-727) Create and publish binary releases along with the source releases

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXCORE-727:
--

Assignee: Chinmay Kolhatkar  (was: Pramod Immaneni)

> Create and publish binary releases along with the source releases
> -
>
> Key: APEXCORE-727
> URL: https://issues.apache.org/jira/browse/APEXCORE-727
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Pramod Immaneni
>Assignee: Chinmay Kolhatkar
>Priority: Minor
>
> Currently, we only do source releases. It would be easier for the users if 
> there were binaries readily available. Create and publish binary release on 
> the website along with sources when a release is made.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (APEXCORE-740) Setup plugins does not identify operator classes because they're loaded through different classloaders

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036569#comment-16036569
 ] 

Chinmay Kolhatkar commented on APEXCORE-740:


RCA: 
StramAppLauncher.loadDependencies is getting called multiple times. This method 
loads the new jars in apa and adds to the thread context class loader. The 
issue is that it just overrides the parent class loader. It should rather build 
over the parent/existing. Hence each time loadDependencies is called, same 
classes gets loaded and becomes part of seperate class loader.
Now the order of execution happens this way: loadDependencies(), load setup 
plugin, loadDependencies(), load operator classes. These operator classes are 
different classes as compared to one which setup plugin understands.

> Setup plugins does not identify operator classes because they're loaded 
> through different classloaders
> --
>
> Key: APEXCORE-740
> URL: https://issues.apache.org/jira/browse/APEXCORE-740
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Apexcli loads setup plugin and operator classes with different thread context 
> class loader. Because of this, the setup plugin does not understand the 
> operator classes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXCORE-740) Setup plugins does not identify operator classes because they're loaded through different plugin classloaders

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-740:
--

 Summary: Setup plugins does not identify operator classes because 
they're loaded through different plugin classloaders
 Key: APEXCORE-740
 URL: https://issues.apache.org/jira/browse/APEXCORE-740
 Project: Apache Apex Core
  Issue Type: Bug
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Apexcli loads setup plugin and operator classes with different thread context 
class loader. Because of this, the setup plugin does not understand the 
operator classes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (APEXCORE-740) Setup plugins does not identify operator classes because they're loaded through different classloaders

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXCORE-740:
---
Summary: Setup plugins does not identify operator classes because they're 
loaded through different classloaders  (was: Setup plugins does not identify 
operator classes because they're loaded through different plugin classloaders)

> Setup plugins does not identify operator classes because they're loaded 
> through different classloaders
> --
>
> Key: APEXCORE-740
> URL: https://issues.apache.org/jira/browse/APEXCORE-740
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Apexcli loads setup plugin and operator classes with different thread context 
> class loader. Because of this, the setup plugin does not understand the 
> operator classes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-720) In DAGExecution plugin, Context.getDAG returns null

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007939#comment-16007939
 ] 

Chinmay Kolhatkar commented on APEXCORE-720:


RCA:

Plugin event of type: ApexPluginDispatcher.DAG_CHANGE is fired before any 
execution plugin is discovered. This event is internal event and is supposed to 
creates a clone of LogicalPlan and set in DAGExecutionPlugin.Context for 
retrieval in Context.getDAG call.

But the dispatch method was checking whether any plugin exists. Hence the 
clonedDAG variable was not getting set in DAGExecutionPlugin.Context.

Because of this, the call to Context.getDAG was returning null and only after 
any further change in LogicalPlan it was taking effect.

> In DAGExecution plugin, Context.getDAG returns null
> ---
>
> Key: APEXCORE-720
> URL: https://issues.apache.org/jira/browse/APEXCORE-720
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> When plugin calls context.getDAG method, it returns null.
> The method should return a LogicalPlan object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXCORE-720) In DAGExecution plugin, Context.getDAG returns null

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-720:
--

 Summary: In DAGExecution plugin, Context.getDAG returns null
 Key: APEXCORE-720
 URL: https://issues.apache.org/jira/browse/APEXCORE-720
 Project: Apache Apex Core
  Issue Type: Bug
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


When plugin calls context.getDAG method, it returns null.
The method should return a LogicalPlan object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2296) Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2296:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)
> --
>
> Key: APEXMALHAR-2296
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2296
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Add support for SQL GROUP BY (Aggregate RelNode)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2295) Apex SQL: Replace POJOInnerJoinOperator with WindowedJoinOperator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2295:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Replace POJOInnerJoinOperator with WindowedJoinOperator
> -
>
> Key: APEXMALHAR-2295
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2295
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Replace POJOInnerJoinOperator with WindowedJoinOperator



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2294) Apex SQL: Make operator names user friendly for user to configure via properties.xml

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2294:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Make operator names user friendly for user to configure via 
> properties.xml
> 
>
> Key: APEXMALHAR-2294
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2294
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Make operator names user friendly for user to configure via properties.xml



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2293) Apex SQL: Add sql line cli to sql support of apex

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2293:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Add sql line cli to sql support of apex
> -
>
> Key: APEXMALHAR-2293
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2293
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Add cli like support to malhar-sql.
> The support can be done similar to: 
> https://github.com/apache/storm/blob/master/external/sql/README.md
> It should support following type of statements:
> 1. CREATE EXTERNAL TABLE with kafka, file, jdbc specifications
> 2. DROP EXTERNAL TABLE
> 3. CREATE FUNCTION
> 4. DELETE FUNCTION
> 5. INSERT and SELECT statements



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2292) Apex SQL: Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2292:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Add generic endpoint to malhar-sql
> 
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: sql
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (APEXMALHAR-2453) Add sort Accumulation for Windowed operator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2453.
---
   Resolution: Done
Fix Version/s: 3.8.0

> Add sort Accumulation for Windowed operator
> ---
>
> Key: APEXMALHAR-2453
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2453
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Ajay Gupta
>Assignee: Ajay Gupta
> Fix For: 3.8.0
>
>
> Currently there is no support for sorting data received as a part of a event 
> window. To add support for sort, we need to add an accumulation. The 
> accumulation will compare the input stream of tuples and output a sorted list 
> at end-window watermark/trigger. The accumulation will support both ascending 
> and descending (reverse) sort.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (APEXMALHAR-2453) Add sort Accumulation for Windowed operator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2453:
--
Summary: Add sort Accumulation for Windowed operator  (was: Sort 
Accumulation for Windowed operator)

> Add sort Accumulation for Windowed operator
> ---
>
> Key: APEXMALHAR-2453
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2453
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Ajay Gupta
>Assignee: Ajay Gupta
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (APEXMALHAR-2453) Sort Accumulation for Windowed operator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2453:
--
Issue Type: New Feature  (was: Task)

> Sort Accumulation for Windowed operator
> ---
>
> Key: APEXMALHAR-2453
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2453
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Ajay Gupta
>Assignee: Ajay Gupta
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (APEXMALHAR-2430) Optimize Join accumulation by changing the data structure in accumulation method

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2430.
---
   Resolution: Done
Fix Version/s: 3.7.0

> Optimize Join accumulation by changing the data structure in accumulation 
> method
> 
>
> Key: APEXMALHAR-2430
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2430
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
> Fix For: 3.7.0
>
>
> Instead of having a list of map we can have List> 
> The key to multimap is a unique object generated for set of keys configured  
> and value of multimap can contain more than one object for the same key.
> This way, while making the comparison, it becomes straightforward how to get 
> the matching pairs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (APEXMALHAR-2429) Ambiguity in passing "key" parameter to Join accumulation

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2429.
---
   Resolution: Done
Fix Version/s: 3.7.0

> Ambiguity in passing "key" parameter to Join accumulation
> -
>
> Key: APEXMALHAR-2429
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2429
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
> Fix For: 3.7.0
>
>
> Currently we are passing a single String array as parameter and assuming that 
> first entry is key from first stream and second entry from second stream and 
> so and so forth. There is a scope to improve this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (APEXMALHAR-2437) Malhar build - ratcheck failing

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar closed APEXMALHAR-2437.
-
Resolution: Invalid

> Malhar build - ratcheck failing
> ---
>
> Key: APEXMALHAR-2437
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2437
> Project: Apache Apex Malhar
>  Issue Type: Bug
>  Components: build
>Reporter: Chinmay Kolhatkar
>Priority: Critical
> Attachments: rat.txt
>
>
> Ratcheck is failing in malhar master because of the changes in examples -> 
> demos folder.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2437) Malhar build - ratcheck failing

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2437:
-

 Summary: Malhar build - ratcheck failing
 Key: APEXMALHAR-2437
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2437
 Project: Apache Apex Malhar
  Issue Type: Bug
  Components: build
Reporter: Chinmay Kolhatkar
Priority: Critical
 Attachments: rat.txt

Ratcheck is failing in malhar master because of the changes in examples -> 
demos folder.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (APEXMALHAR-2413) Improve PojoInnerJoin Accumulation

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2413.
---
   Resolution: Done
Fix Version/s: 3.7.0

> Improve PojoInnerJoin Accumulation
> --
>
> Key: APEXMALHAR-2413
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2413
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: algorithms
>Affects Versions: 3.6.0
>Reporter: Chinmay Kolhatkar
>Assignee: Hitesh Kapoor
> Fix For: 3.7.0
>
>
> This contains a series of subtasks:
> 1. Enable PojoJoinInner accumulation to emit POJO instead of Map
> 2. Improve performance of PojoInnerJoin accumulation by using PojoUtils 
> instead of java reflection
> 3. Enable PojoInnerJoin accumulation to accept group of keys for join instead 
> of a single key.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2413) Improve PojoInnerJoin Accumulation

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2413:
-

Assignee: Hitesh Kapoor  (was: Chinmay Kolhatkar)

> Improve PojoInnerJoin Accumulation
> --
>
> Key: APEXMALHAR-2413
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2413
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: algorithms
>Affects Versions: 3.6.0
>Reporter: Chinmay Kolhatkar
>Assignee: Hitesh Kapoor
>
> This contains a series of subtasks:
> 1. Enable PojoJoinInner accumulation to emit POJO instead of Map
> 2. Improve performance of PojoInnerJoin accumulation by using PojoUtils 
> instead of java reflection
> 3. Enable PojoInnerJoin accumulation to accept group of keys for join instead 
> of a single key.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2414) Improve performance of PojoInnerJoin accum by using PojoUtils

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2414:
-

Assignee: (was: Chinmay Kolhatkar)

> Improve performance of PojoInnerJoin accum by using PojoUtils
> -
>
> Key: APEXMALHAR-2414
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2414
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: algorithms
>Reporter: Chinmay Kolhatkar
>
> Currently PojoInnerJoin accumulation uses java reflection to deal with 
> objects. Use PojoUtils instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2414) Improve performance of PojoInnerJoin accum by using PojoUtils

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2414:
-

Assignee: Chinmay Kolhatkar

> Improve performance of PojoInnerJoin accum by using PojoUtils
> -
>
> Key: APEXMALHAR-2414
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2414
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: algorithms
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Currently PojoInnerJoin accumulation uses java reflection to deal with 
> objects. Use PojoUtils instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2414) Improve performance of PojoInnerJoin accum by using PojoUtils

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2414:
-

 Summary: Improve performance of PojoInnerJoin accum by using 
PojoUtils
 Key: APEXMALHAR-2414
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2414
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar


Currently PojoInnerJoin accumulation uses java reflection to deal with objects. 
Use PojoUtils instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2415) Enable PojoInnerJoin accum to allow multiple keys for join purpose

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2415:
-

 Summary: Enable PojoInnerJoin accum to allow multiple keys for 
join purpose
 Key: APEXMALHAR-2415
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2415
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Affects Versions: 3.6.0
Reporter: Chinmay Kolhatkar


Enable PojoInnerJoin accum to allow multiple keys for join purpose



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (APEXMALHAR-2409) Improve PojoInnerJoin accumulation to emit a POJO instead of Map

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2409:
--
Issue Type: Sub-task  (was: Improvement)
Parent: APEXMALHAR-2413

> Improve PojoInnerJoin accumulation to emit a POJO instead of Map
> 
>
> Key: APEXMALHAR-2409
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2409
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 3.6.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Currently POJO Inner Join accumulation emits a Map. It should emit a POJO.
> The parameter to the accumulation should accept POJO list mapping from input 
> to output variables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2413) Improve PojoInnerJoin Accumulation

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2413:
-

 Summary: Improve PojoInnerJoin Accumulation
 Key: APEXMALHAR-2413
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2413
 Project: Apache Apex Malhar
  Issue Type: Improvement
  Components: algorithms
Affects Versions: 3.6.0
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


This contains a series of subtasks:
1. Enable PojoJoinInner accumulation to emit POJO instead of Map
2. Improve performance of PojoInnerJoin accumulation by using PojoUtils instead 
of java reflection
3. Enable PojoInnerJoin accumulation to accept group of keys for join instead 
of a single key.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2409) Improve PojoInnerJoin accumulation to emit a POJO instead of Map

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2409:
-

 Summary: Improve PojoInnerJoin accumulation to emit a POJO instead 
of Map
 Key: APEXMALHAR-2409
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2409
 Project: Apache Apex Malhar
  Issue Type: Improvement
  Components: streams
Affects Versions: 3.6.0
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Currently POJO Inner Join accumulation emits a Map. It should emit a POJO.

The parameter to the accumulation should accept POJO list mapping from input to 
output variables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (APEXMALHAR-2399) In PojoInnerJoin accumulation default constructor is directly throwing an exception which messes up in default serialization.

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2399.
---
   Resolution: Fixed
Fix Version/s: 3.7.0

> In PojoInnerJoin accumulation default constructor is directly throwing an 
> exception which messes up in default serialization.
> -
>
> Key: APEXMALHAR-2399
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2399
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
> Fix For: 3.7.0
>
>
> In PojoInnerJoin accumulation default constructor is directly throwing an 
> exception which messes up in default serialization. The exception is thrown 
> saying to specify the number of streams. Currently we only merge 2 streams so 
> there is no need to throw this exception also it messes up the serialization 
> and the applicaion can't be loaded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-623) Schema propagation support in Apex

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838061#comment-15838061
 ] 

Chinmay Kolhatkar commented on APEXCORE-623:


Design doc: 
https://docs.google.com/document/d/1ibLQ1KYCLTeufG7dLoHyN_tRQXEM3LR-7o_S0z_porQ/edit?usp=sharing

> Schema propagation support in Apex
> --
>
> Key: APEXCORE-623
> URL: https://issues.apache.org/jira/browse/APEXCORE-623
> Project: Apache Apex Core
>  Issue Type: New Feature
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> In this feature, following will be done:
> 1. Add an interface which can be implemented by operators to help in 
> communicating the output schema of its port.
> 2. Update Logical Plan to provide propagation of schema across the pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-623) Schema propagation support in Apex

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838050#comment-15838050
 ] 

Chinmay Kolhatkar commented on APEXCORE-623:


Discussion on mailing list: 
https://lists.apache.org/thread.html/7e806086da8876d9e256a9d9ddb449946961afa1e357b365c4d8f3bf@%3Cdev.apex.apache.org%3E

> Schema propagation support in Apex
> --
>
> Key: APEXCORE-623
> URL: https://issues.apache.org/jira/browse/APEXCORE-623
> Project: Apache Apex Core
>  Issue Type: New Feature
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> In this feature, following will be done:
> 1. Add an interface which can be implemented by operators to help in 
> communicating the output schema of its port.
> 2. Update Logical Plan to provide propagation of schema across the pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXCORE-623) Schema propagation support in Apex

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXCORE-623:
--

 Summary: Schema propagation support in Apex
 Key: APEXCORE-623
 URL: https://issues.apache.org/jira/browse/APEXCORE-623
 Project: Apache Apex Core
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


In this feature, following will be done:
1. Add an interface which can be implemented by operators to help in 
communicating the output schema of its port.
2. Update Logical Plan to provide propagation of schema across the pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2259) Create Fixed Length Parser Operator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2259.
---
   Resolution: Done
Fix Version/s: 3.7.0

> Create Fixed Length Parser Operator
> ---
>
> Key: APEXMALHAR-2259
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2259
> Project: Apache Apex Malhar
>  Issue Type: Task
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Hitesh Kapoor
> Fix For: 3.7.0
>
> Attachments: Mail_thread.pdf
>
>
> An operator for parsing fixed width records has to be implemented.
> This operator shall be used to parse fixed width byte array/tuples based on a 
> JSON Schema and emit the parsed bytearray on one port; converted POJO object 
> on another port and the failed bytearray/tuples on an error port.
> User will provide a JSON schema definition



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (APEXCORE-576) Apex/Malhar Extensions

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704240#comment-15704240
 ] 

Chinmay Kolhatkar edited comment on APEXCORE-576 at 11/29/16 5:22 AM:
--

Here is the thought process behind why I proposed this idea.
I think we do have a good level of contributions to Apex and Malhar as of now. 
But IMO this is growing or going to grow in future. In which case, for 
accepting the code into apex or malhar with right level of maturity might be 
tough. At the same time, external contributors including individuals or 
enterprise may or may not find enough time to contribute to mainstream 
following all the processes but might have a valuable code which would enhance 
Apex.

Having a registry like mechanism where one could register their code (in their 
own repositry) to be working with Apache Apex gives a low barrier to entry at 
the same time providing larger feature set and contribution to Apache Apex 
overall. We'll need to have a process defined for what is "low barrier" to get 
into this registry and what is "standard barrier" to get into mainstream.
With this registry we could have an indexing of such extensions for easy search 
and usage, making life easier for users of apex too.

Spark packages were released at the beginning of 2015, and there are already 
290+ packages there, which is a good number in 2 years IMO. Their processes 
might not suit us at all, but the idea is worth investigating I think.

I believe this is a win-win situation for Apex community, apex user and apex 
contributors. With our growing community and usage, we would quickly need a way 
to manage the right quality and large contributions and I think this can solve 
that problem.



was (Author: chinmay):
Hi All,

Here is the thought process behind why I proposed this idea.
I think we do have a good level of contributions to Apex and Malhar as of now. 
But IMO this is growing or going to grow in future. In which case, for 
accepting the code into apex or malhar with right level of maturity might be 
tough. At the same time, external contributors including individuals or 
enterprise may or may not find enough time to contribute to mainstream 
following all the processes and they might have a valuable code which would 
enhance Apex.

Having a registry like mechanism where one could register their code to be 
working with Apache Apex gives a low barrier to entry at the same time 
providing larger feature set and contribution to Apache Apex overall. We'll 
need to have a process defined for what is "low barrier" to get into this 
registry and what is "standard barrier" to get into mainstream.
With this registry we could have an indexing of such extensions for each search 
and usage making life easier for users of apex too.

Spark packages were released at the beginning of 2015, and there are already 
290+ packages there which is a good number in 2 years IMO.

We need not follow the process they have at all, but the idea is worth 
investigating I think.

I believe this is a win-win situation for Apex community, apex user and apex 
contributors. With our growing community and usage, we would quickly need a way 
to manage the quality and contribution and I think this can solve that problem.

-Chinmay.


> Apex/Malhar Extensions
> --
>
> Key: APEXCORE-576
> URL: https://issues.apache.org/jira/browse/APEXCORE-576
> Project: Apache Apex Core
>  Issue Type: New Feature
>  Components: Website
>Reporter: Chinmay Kolhatkar
>
> The purpose of this task is to provide a way to external contributors to make 
> better contributions to Apache Apex project.
> The idea looks something like this:
> 1. One could have extension to apex core/malhar in their own repository and 
> just register itself with Apache Apex. 
> 2. As it matures and find more and more use we can consume that in mainstream 
> releases.
> 3. Some possibilities of of Apex extensions are as follows:
> a. Operators - DataSources, DataDestinations, Parsers, Formatters, 
> Processing etc.
> b. Apex Engine Plugables
> c. External Integrations
> d. Integration with other platform like Machine learning, Graph engines 
> etc.
> e. Application which are ready to use.
> d. Apex related tools which can ease the development and usage of apex.
> The initial discussion about this has happened here:
> https://lists.apache.org/thread.html/3d6ca2b46c53df77f37f54d64e18607a623c5a54f439e1afebfbef35@%3Cdev.apex.apache.org%3E
> More concrete discussion/implementation proposal required for this task to 
> progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-576) Apex/Malhar Extensions

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704240#comment-15704240
 ] 

Chinmay Kolhatkar commented on APEXCORE-576:


Hi All,

Here is the thought process behind why I proposed this idea.
I think we do have a good level of contributions to Apex and Malhar as of now. 
But IMO this is growing or going to grow in future. In which case, for 
accepting the code into apex or malhar with right level of maturity might be 
tough. At the same time, external contributors including individuals or 
enterprise may or may not find enough time to contribute to mainstream 
following all the processes and they might have a valuable code which would 
enhance Apex.

Having a registry like mechanism where one could register their code to be 
working with Apache Apex gives a low barrier to entry at the same time 
providing larger feature set and contribution to Apache Apex overall. We'll 
need to have a process defined for what is "low barrier" to get into this 
registry and what is "standard barrier" to get into mainstream.
With this registry we could have an indexing of such extensions for each search 
and usage making life easier for users of apex too.

Spark packages were released at the beginning of 2015, and there are already 
290+ packages there which is a good number in 2 years IMO.

We need not follow the process they have at all, but the idea is worth 
investigating I think.

I believe this is a win-win situation for Apex community, apex user and apex 
contributors. With our growing community and usage, we would quickly need a way 
to manage the quality and contribution and I think this can solve that problem.

-Chinmay.


> Apex/Malhar Extensions
> --
>
> Key: APEXCORE-576
> URL: https://issues.apache.org/jira/browse/APEXCORE-576
> Project: Apache Apex Core
>  Issue Type: New Feature
>  Components: Website
>Reporter: Chinmay Kolhatkar
>
> The purpose of this task is to provide a way to external contributors to make 
> better contributions to Apache Apex project.
> The idea looks something like this:
> 1. One could have extension to apex core/malhar in their own repository and 
> just register itself with Apache Apex. 
> 2. As it matures and find more and more use we can consume that in mainstream 
> releases.
> 3. Some possibilities of of Apex extensions are as follows:
> a. Operators - DataSources, DataDestinations, Parsers, Formatters, 
> Processing etc.
> b. Apex Engine Plugables
> c. External Integrations
> d. Integration with other platform like Machine learning, Graph engines 
> etc.
> e. Application which are ready to use.
> d. Apex related tools which can ease the development and usage of apex.
> The initial discussion about this has happened here:
> https://lists.apache.org/thread.html/3d6ca2b46c53df77f37f54d64e18607a623c5a54f439e1afebfbef35@%3Cdev.apex.apache.org%3E
> More concrete discussion/implementation proposal required for this task to 
> progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-576) Apex/Malhar Extensions

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704211#comment-15704211
 ] 

Chinmay Kolhatkar commented on APEXCORE-576:


One can have their extensions in their own github repository and just affiliate 
their extensions with Apache Apex via this registry. Because their code resides 
in their own open repository, the process of affiliation to Apex is lesser. 
This is not a direct contribution to Apache Apex but rather an indirect 
contribution to have features with Apache Apex. As the extensions matures, they 
can be pulled into mainstream with right standards followed.

> Apex/Malhar Extensions
> --
>
> Key: APEXCORE-576
> URL: https://issues.apache.org/jira/browse/APEXCORE-576
> Project: Apache Apex Core
>  Issue Type: New Feature
>  Components: Website
>Reporter: Chinmay Kolhatkar
>
> The purpose of this task is to provide a way to external contributors to make 
> better contributions to Apache Apex project.
> The idea looks something like this:
> 1. One could have extension to apex core/malhar in their own repository and 
> just register itself with Apache Apex. 
> 2. As it matures and find more and more use we can consume that in mainstream 
> releases.
> 3. Some possibilities of of Apex extensions are as follows:
> a. Operators - DataSources, DataDestinations, Parsers, Formatters, 
> Processing etc.
> b. Apex Engine Plugables
> c. External Integrations
> d. Integration with other platform like Machine learning, Graph engines 
> etc.
> e. Application which are ready to use.
> d. Apex related tools which can ease the development and usage of apex.
> The initial discussion about this has happened here:
> https://lists.apache.org/thread.html/3d6ca2b46c53df77f37f54d64e18607a623c5a54f439e1afebfbef35@%3Cdev.apex.apache.org%3E
> More concrete discussion/implementation proposal required for this task to 
> progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-576) Apex/Malhar Extensions

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704203#comment-15704203
 ] 

Chinmay Kolhatkar commented on APEXCORE-576:


The malhar is rich ecosystem right now, but I doubt this would dilute the 
existing ecosystem. The idea is to provide contributor an way to affiliate 
their code with Apache Apex (I believe affiliate is the right word). Once their 
components matures, they can chose to raise the pull request to malhar (i.e. 
mainstream) with proper standard. For involving to mainstream, the standards as 
they are currently remains intact.

> Apex/Malhar Extensions
> --
>
> Key: APEXCORE-576
> URL: https://issues.apache.org/jira/browse/APEXCORE-576
> Project: Apache Apex Core
>  Issue Type: New Feature
>  Components: Website
>Reporter: Chinmay Kolhatkar
>
> The purpose of this task is to provide a way to external contributors to make 
> better contributions to Apache Apex project.
> The idea looks something like this:
> 1. One could have extension to apex core/malhar in their own repository and 
> just register itself with Apache Apex. 
> 2. As it matures and find more and more use we can consume that in mainstream 
> releases.
> 3. Some possibilities of of Apex extensions are as follows:
> a. Operators - DataSources, DataDestinations, Parsers, Formatters, 
> Processing etc.
> b. Apex Engine Plugables
> c. External Integrations
> d. Integration with other platform like Machine learning, Graph engines 
> etc.
> e. Application which are ready to use.
> d. Apex related tools which can ease the development and usage of apex.
> The initial discussion about this has happened here:
> https://lists.apache.org/thread.html/3d6ca2b46c53df77f37f54d64e18607a623c5a54f439e1afebfbef35@%3Cdev.apex.apache.org%3E
> More concrete discussion/implementation proposal required for this task to 
> progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-576) Apex/Malhar Extensions

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704198#comment-15704198
 ] 

Chinmay Kolhatkar commented on APEXCORE-576:


Yes.. These extensions are outside of Apache Apex.. I agree with your idea of 
this being a proxy ticket. Once things are fleshed out we can create actual 
Jira tasks for this.

> Apex/Malhar Extensions
> --
>
> Key: APEXCORE-576
> URL: https://issues.apache.org/jira/browse/APEXCORE-576
> Project: Apache Apex Core
>  Issue Type: New Feature
>  Components: Website
>Reporter: Chinmay Kolhatkar
>
> The purpose of this task is to provide a way to external contributors to make 
> better contributions to Apache Apex project.
> The idea looks something like this:
> 1. One could have extension to apex core/malhar in their own repository and 
> just register itself with Apache Apex. 
> 2. As it matures and find more and more use we can consume that in mainstream 
> releases.
> 3. Some possibilities of of Apex extensions are as follows:
> a. Operators - DataSources, DataDestinations, Parsers, Formatters, 
> Processing etc.
> b. Apex Engine Plugables
> c. External Integrations
> d. Integration with other platform like Machine learning, Graph engines 
> etc.
> e. Application which are ready to use.
> d. Apex related tools which can ease the development and usage of apex.
> The initial discussion about this has happened here:
> https://lists.apache.org/thread.html/3d6ca2b46c53df77f37f54d64e18607a623c5a54f439e1afebfbef35@%3Cdev.apex.apache.org%3E
> More concrete discussion/implementation proposal required for this task to 
> progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-576) Apex/Malhar Extensions

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704195#comment-15704195
 ] 

Chinmay Kolhatkar commented on APEXCORE-576:


Yes all these needs to be sorted out. This Jira is a placeholder an idea and 
further discussion. We need to come up with an doc having a proposal for this 
before we approach implementing it.

> Apex/Malhar Extensions
> --
>
> Key: APEXCORE-576
> URL: https://issues.apache.org/jira/browse/APEXCORE-576
> Project: Apache Apex Core
>  Issue Type: New Feature
>  Components: Website
>Reporter: Chinmay Kolhatkar
>
> The purpose of this task is to provide a way to external contributors to make 
> better contributions to Apache Apex project.
> The idea looks something like this:
> 1. One could have extension to apex core/malhar in their own repository and 
> just register itself with Apache Apex. 
> 2. As it matures and find more and more use we can consume that in mainstream 
> releases.
> 3. Some possibilities of of Apex extensions are as follows:
> a. Operators - DataSources, DataDestinations, Parsers, Formatters, 
> Processing etc.
> b. Apex Engine Plugables
> c. External Integrations
> d. Integration with other platform like Machine learning, Graph engines 
> etc.
> e. Application which are ready to use.
> d. Apex related tools which can ease the development and usage of apex.
> The initial discussion about this has happened here:
> https://lists.apache.org/thread.html/3d6ca2b46c53df77f37f54d64e18607a623c5a54f439e1afebfbef35@%3Cdev.apex.apache.org%3E
> More concrete discussion/implementation proposal required for this task to 
> progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2257) Add documentation for Transform operator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2257.
---
   Resolution: Done
Fix Version/s: 3.6.0

> Add documentation for Transform operator
> 
>
> Key: APEXMALHAR-2257
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2257
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: Chaitanya
>Assignee: Chaitanya
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2354) Add support for heuristic watermarks in WindowedOperator

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692958#comment-15692958
 ] 

Chinmay Kolhatkar commented on APEXMALHAR-2354:
---

Discussion is happening here:
https://lists.apache.org/thread.html/a34fb3419c43a1ed09a47d303155d8f0dc347af191e860695a48c71e@%3Cdev.apex.apache.org%3E


> Add support for heuristic watermarks in WindowedOperator
> 
>
> Key: APEXMALHAR-2354
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2354
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> The purpose of this improvement is to add support to plugin heuristic 
> algorithms for determining watermarks in WindowedOperator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2354) Add support for heuristic watermarks in WindowedOperator

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2354:
-

 Summary: Add support for heuristic watermarks in WindowedOperator
 Key: APEXMALHAR-2354
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2354
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


The purpose of this improvement is to add support to plugin heuristic 
algorithms for determining watermarks in WindowedOperator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2353) timeExpression should not be null for time based Dedup

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2353.
---
   Resolution: Fixed
Fix Version/s: 3.6.0

> timeExpression should not be null for time based Dedup  
> 
>
> Key: APEXMALHAR-2353
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2353
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Bhupesh Chawda
>Assignee: Bhupesh Chawda
>Priority: Minor
> Fix For: 3.6.0
>
>
> Time Based Dedup has timeExpression as optional. It has to be supplied by the 
> user.
> In the current setting, if the user does not specify a timeExpression, then a 
> different time (System time) will be passed for each tuple, irrespective of 
> whether the tuple is a duplicate or a unique. This is a bug since even the 
> duplicate tuples may fall in different buckets and will be concluded as 
> unique. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2293) Apex SQL: Add sql line cli to sql support of apex

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657010#comment-15657010
 ] 

Chinmay Kolhatkar commented on APEXMALHAR-2293:
---

[~thw], [~davidyan] Here is my understanding about this jira. This jira has 
following asks:
1. Make the endpoints configurable from outside... Something like a JSon 
configuration which can be passed to Endpoint interface and individual 
endpoints can read and configure it.
2. Have a registry of some kind which is accessible externally for registering 
different types of endpoints.
3. Use above concept in sql cli.

I propose following changes:
1. Add a method called "getSchema" which will return the URI for given endpoint.
1. Create a EndpointRegistry class which has a map of URI -> Endoint. URI can 
be like kafka, file, jdbc etc...
2. In the static block of this class there will be used ServiceLoader of java 
to find all the classes in classpath that implements Endpoint interface. Once 
found these classes can be added to the Map provided.
3. This way if anyone wants to add an Endpoint externally, all one has to do is 
define the class and add to classpath.
4. Alternativey EndpointRegistry can have method like reload which can reload 
the schema.
5. A new method should be added to Endpoint which takes a String a property for 
that class. This string can then be interpreted by Endpoint to configure the 
property.
6. Similar can be done for MessageFormats as well.
7. SQL cli can then use above APIs to dynamically define the table. (Example of 
sql cli is available with storm-sql)

Advantage of this:
1. Endpoints could get automatically registered with malhar-sql.
2. One of create custom endpoints in own project and just use it with 
malhar-sql.
3. Endpoints can completely configured externally.

Please share your opinion.

Thanks,
Chinmay.

> Apex SQL: Add sql line cli to sql support of apex
> -
>
> Key: APEXMALHAR-2293
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2293
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Add cli like support to malhar-sql.
> The support can be done similar to: 
> https://github.com/apache/storm/blob/master/external/sql/README.md
> It should support following type of statements:
> 1. CREATE EXTERNAL TABLE with kafka, file, jdbc specifications
> 2. DROP EXTERNAL TABLE
> 3. CREATE FUNCTION
> 4. DELETE FUNCTION
> 5. INSERT and SELECT statements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2293) Apex SQL: Add sql line cli to sql support of apex

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656620#comment-15656620
 ] 

Chinmay Kolhatkar commented on APEXMALHAR-2293:
---

I've moved couple of jiras as sub-task to this one a they're related and need 
to be captured in a single context.

> Apex SQL: Add sql line cli to sql support of apex
> -
>
> Key: APEXMALHAR-2293
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2293
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Add cli like support to malhar-sql.
> The support can be done similar to: 
> https://github.com/apache/storm/blob/master/external/sql/README.md
> It should support following type of statements:
> 1. CREATE EXTERNAL TABLE with kafka, file, jdbc specifications
> 2. DROP EXTERNAL TABLE
> 3. CREATE FUNCTION
> 4. DELETE FUNCTION
> 5. INSERT and SELECT statements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) Apex SQL: Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Issue Type: Sub-task  (was: New Feature)
Parent: APEXMALHAR-2293

> Apex SQL: Add generic endpoint to malhar-sql
> 
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: sql
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2294) Apex SQL: Make operator names user friendly for user to configure via properties.xml

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2294:
--
Issue Type: Sub-task  (was: Improvement)
Parent: APEXMALHAR-2293

> Apex SQL: Make operator names user friendly for user to configure via 
> properties.xml
> 
>
> Key: APEXMALHAR-2294
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2294
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: sql
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Make operator names user friendly for user to configure via properties.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2167) Add user documentation for Json Formatter

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2167.
---
   Resolution: Done
Fix Version/s: 3.6.0

> Add user documentation for Json Formatter
> -
>
> Key: APEXMALHAR-2167
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2167
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: shubham pathak
>Assignee: shubham pathak
> Fix For: 3.6.0
>
>
> Add user documentation for Json Formatter operator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2167) Add user documentation for Json Formatter

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2167:
--
Summary: Add user documentation for Json Formatter  (was: Add user 
documentation for Json Formatter on apex docs)

> Add user documentation for Json Formatter
> -
>
> Key: APEXMALHAR-2167
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2167
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: shubham pathak
>Assignee: shubham pathak
>
> Add user documentation for Json Formatter operator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2166) Add user documentation for Json Parser

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2166.
---
   Resolution: Done
Fix Version/s: 3.6.0

> Add user documentation for Json Parser
> --
>
> Key: APEXMALHAR-2166
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2166
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: shubham pathak
>Assignee: shubham pathak
> Fix For: 3.6.0
>
>
> Add user documentation for JSON Parser operator. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2166) Add user documentation for Json Parser

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2166:
--
Summary: Add user documentation for Json Parser  (was: Add user 
documentation for Json Parser on apex docs)

> Add user documentation for Json Parser
> --
>
> Key: APEXMALHAR-2166
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2166
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: shubham pathak
>Assignee: shubham pathak
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2290) JDBCPOJOInsertOutput Operator - Optimization required to populate metadata for columns from database

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar resolved APEXMALHAR-2290.
---
Resolution: Fixed

> JDBCPOJOInsertOutput Operator - Optimization required to populate metadata 
> for columns from database
> 
>
> Key: APEXMALHAR-2290
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2290
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
> Fix For: 3.6.0
>
>
> In some cases the JDBC output operator does not deploy properly. The operator 
> gets killed and does not gets deployed. 
> The setup method uses "select * from tablename" to just get types of fields 
> in the table. If the tables already has lot of data then select queries runs 
> for more than 30 sec and the operator is killed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2290) JDBCPOJOInsertOutput Operator - Optimization required to populate metadata for columns from database

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2290:
--
Summary: JDBCPOJOInsertOutput Operator - Optimization required to populate 
metadata for columns from database  (was: JDBC operator does not deploy in 
certain cases)

> JDBCPOJOInsertOutput Operator - Optimization required to populate metadata 
> for columns from database
> 
>
> Key: APEXMALHAR-2290
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2290
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Hitesh Kapoor
>Assignee: Hitesh Kapoor
> Fix For: 3.6.0
>
>
> In some cases the JDBC output operator does not deploy properly. The operator 
> gets killed and does not gets deployed. 
> The setup method uses "select * from tablename" to just get types of fields 
> in the table. If the tables already has lot of data then select queries runs 
> for more than 30 sec and the operator is killed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2315) Ignore Join Test for because of issues in POJOInnerJoinOperator

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2315:
-

 Summary: Ignore Join Test for because of issues in 
POJOInnerJoinOperator
 Key: APEXMALHAR-2315
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2315
 Project: Apache Apex Malhar
  Issue Type: Bug
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar
Priority: Minor


Ignore Join Test for because of issues in POJOInnerJoinOperator

sql integration uses POJOInnerJoinOperator because of which issues occuring 
because of managed state and inner join operator interaction is causing this 
test to fail.

There is a Jira (APEXMALHAR-2295) present to replace POJOInnerJoinOperator with 
WIndowed variant of it. As a part of that, these tests should be resumed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-1818) Integrate Calcite to support SQL

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15597039#comment-15597039
 ] 

Chinmay Kolhatkar commented on APEXMALHAR-1818:
---

[~julianhyde] Created a PR for powered by page of calcite: 
https://github.com/apache/calcite/pull/315

[~thw] Please comment if any change of text is required.


> Integrate Calcite to support SQL
> 
>
> Key: APEXMALHAR-1818
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1818
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: query operators, sql
>Reporter: Amol
>Assignee: Chinmay Kolhatkar
>  Labels: roadmap
> Fix For: 3.6.0
>
>
> Once we have ability to generate a subdag, we should take a look at 
> integrating Calcite into Apex. The operator that enables populate DAG, should 
> use Calcite to generate the DAG, given a SQL query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-1818) Integrate Calcite to support SQL

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595795#comment-15595795
 ] 

Chinmay Kolhatkar commented on APEXMALHAR-1818:
---

Thanks to everyone for making this happen.

As a part of this PR, we had a number of improvements that came up. I've 
created jiras for them:
Here is the list: 
https://issues.apache.org/jira/browse/APEXMALHAR-2311?jql=project%20%3D%20APEXMALHAR%20AND%20component%20%3D%20sql

Those will be the next item to keep improving.

Feel free to add any jiras as required.


> Integrate Calcite to support SQL
> 
>
> Key: APEXMALHAR-1818
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1818
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: query operators, sql
>Reporter: Amol
>Assignee: Chinmay Kolhatkar
>  Labels: roadmap
> Fix For: 3.6.0
>
>
> Once we have ability to generate a subdag, we should take a look at 
> integrating Calcite into Apex. The operator that enables populate DAG, should 
> use Calcite to generate the DAG, given a SQL query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-1818) Integrate Calcite to support SQL

[Chinmay Kolhatkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594477#comment-15594477
 ] 

Chinmay Kolhatkar commented on APEXMALHAR-1818:
---

Sure will create a PR for powered by page once we have this PR merged.

> Integrate Calcite to support SQL
> 
>
> Key: APEXMALHAR-1818
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1818
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: query operators, sql
>Reporter: Amol
>Assignee: Chinmay Kolhatkar
>  Labels: roadmap
>
> Once we have ability to generate a subdag, we should take a look at 
> integrating Calcite into Apex. The operator that enables populate DAG, should 
> use Calcite to generate the DAG, given a SQL query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-1818) Integrate Calcite to support SQL

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-1818:
--
Component/s: sql

> Integrate Calcite to support SQL
> 
>
> Key: APEXMALHAR-1818
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1818
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: query operators, sql
>Reporter: Amol
>Assignee: Chinmay Kolhatkar
>  Labels: roadmap
>
> Once we have ability to generate a subdag, we should take a look at 
> integrating Calcite into Apex. The operator that enables populate DAG, should 
> use Calcite to generate the DAG, given a SQL query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2304) Apex SQL: Add examples for SQL in Apex in demos folder

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2304:
--
Component/s: sql

> Apex SQL: Add examples for SQL in Apex in demos folder
> --
>
> Key: APEXMALHAR-2304
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2304
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2311) Apex SQL: Allow operators to be configured from outside

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2311:
--
Component/s: sql

> Apex SQL: Allow operators to be configured from outside
> ---
>
> Key: APEXMALHAR-2311
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2311
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Currently there is no way to configure the operators added against given 
> Relational Tree from external file e.g. properties.xml etc..
> There should be a way to configure the added operators externally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2310) Apex SQL: Reuse the schema generated automatically in SQL DAG

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2310:
--
Component/s: sql

> Apex SQL: Reuse the schema generated automatically in SQL DAG
> -
>
> Key: APEXMALHAR-2310
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2310
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Currently, for each stream a new class (POJO) is generated and assigned to 
> port TUPLE_CLASS. In there is repetition of the same schema, we should reuse 
> the class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2296) Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2296:
--
Component/s: sql

> Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)
> --
>
> Key: APEXMALHAR-2296
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2296
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Add support for SQL GROUP BY (Aggregate RelNode)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2311) Apex SQL: Allow operators to be configured from outside

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2311:
-

 Summary: Apex SQL: Allow operators to be configured from outside
 Key: APEXMALHAR-2311
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2311
 Project: Apache Apex Malhar
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar


Currently there is no way to configure the operators added against given 
Relational Tree from external file e.g. properties.xml etc..

There should be a way to configure the added operators externally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2310) Apex SQL: Reuse the schema generated automatically in SQL DAG

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2310:
-

 Summary: Apex SQL: Reuse the schema generated automatically in SQL 
DAG
 Key: APEXMALHAR-2310
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2310
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Chinmay Kolhatkar


Currently, for each stream a new class (POJO) is generated and assigned to port 
TUPLE_CLASS. In there is repetition of the same schema, we should reuse the 
class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2117) JDBCLoader for Enricher should generate default queryStmt even when value is empty string

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2117:
--
Assignee: (was: Chinmay Kolhatkar)

> JDBCLoader for Enricher should generate default queryStmt even when value is 
> empty string 
> --
>
> Key: APEXMALHAR-2117
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2117
> Project: Apache Apex Malhar
>  Issue Type: Bug
>  Components: utilities
>Affects Versions: 3.4.0
>Reporter: Chinmay Kolhatkar
>
> Currently JDBCLoader creates a default queryStmt based on properties set only 
> when the queryStmt is set to null.
> It should do that same when its set to empty string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2038) Whilelist tests in contrib instead of skipTests=true

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2038:
--
Labels: newbie  (was: )

> Whilelist tests in contrib instead of skipTests=true
> 
>
> Key: APEXMALHAR-2038
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2038
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: build
>Reporter: Chinmay Kolhatkar
>  Labels: newbie
>
> Currently malhar contrib has skipTests=true which makes none of the tests to 
> run in travis build. But there are tests which do not need external setup to 
> run the test.
> Hence,instead of skipTests=true, we could use includes/excludes/test 
> parameters and whilelist the tests which are not dependent on external system 
> and make them run.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2038) Whilelist tests in contrib instead of skipTests=true

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2038:
--
Assignee: (was: Chinmay Kolhatkar)

> Whilelist tests in contrib instead of skipTests=true
> 
>
> Key: APEXMALHAR-2038
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2038
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: build
>Reporter: Chinmay Kolhatkar
>
> Currently malhar contrib has skipTests=true which makes none of the tests to 
> run in travis build. But there are tests which do not need external setup to 
> run the test.
> Hence,instead of skipTests=true, we could use includes/excludes/test 
> parameters and whilelist the tests which are not dependent on external system 
> and make them run.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2304) Apex SQL: Add examples for SQL in Apex in demos folder

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2304:
-

 Summary: Apex SQL: Add examples for SQL in Apex in demos folder
 Key: APEXMALHAR-2304
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2304
 Project: Apache Apex Malhar
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2296) Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2296:
-

 Summary: Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)
 Key: APEXMALHAR-2296
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2296
 Project: Apache Apex Malhar
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Add support for SQL GROUP BY (Aggregate RelNode)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2295) Apex SQL: Replace POJOInnerJoinOperator with WindowedJoinOperator

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2295:
-

 Summary: Apex SQL: Replace POJOInnerJoinOperator with 
WindowedJoinOperator
 Key: APEXMALHAR-2295
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2295
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Replace POJOInnerJoinOperator with WindowedJoinOperator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2294) Apex SQL: Make operator names user friendly for user to configure via properties.xml

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2294:
-

 Summary: Apex SQL: Make operator names user friendly for user to 
configure via properties.xml
 Key: APEXMALHAR-2294
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2294
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Make operator names user friendly for user to configure via properties.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) Apex SQL: Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Summary: Apex SQL: Add generic endpoint to malhar-sql  (was: Add generic 
endpoint to malhar-sql)

> Apex SQL: Add generic endpoint to malhar-sql
> 
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) malhar-sql: Add generic endpoint to sql support of Apex

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Summary: malhar-sql: Add generic endpoint to sql support of Apex  (was: Add 
generic endpoint to sql support of Apex)

> malhar-sql: Add generic endpoint to sql support of Apex
> ---
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Summary: Add generic endpoint to malhar-sql  (was: malhar-sql: Add generic 
endpoint to sql support of Apex)

> Add generic endpoint to malhar-sql
> --
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2293) Add sql line cli to sql support of apex

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2293:
-

 Summary: Add sql line cli to sql support of apex
 Key: APEXMALHAR-2293
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2293
 Project: Apache Apex Malhar
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Add cli like support to malhar-sql.
The support can be done similar to: 
https://github.com/apache/storm/blob/master/external/sql/README.md

It should support following type of statements:
1. CREATE EXTERNAL TABLE with kafka, file, jdbc specifications
2. DROP EXTERNAL TABLE
3. CREATE FUNCTION
4. DELETE FUNCTION
5. INSERT and SELECT statements




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)