EmbeddedKafka Server for testing kafka related code

[Chinmay Kolhatkar]

Dear Community,

While testing kafka related endpoint code for Apex-Calcite integration, I
found an interesting embedded kafka implementation over the internet and I
used it in my code.

The implementation is based on ready to use EmbeddedZookeer and KafkaServer
from mvn dependency kafka_2.11:0.9.0.0:test.

I found this local kafka server very stable for testing purpose. Would it
be a good addition to malhar-kafka module in test folder?

The EmbeddedKafka server is here:
https://github.com/chinmaykolhatkar/apex-malhar/blob/calcite/sql/src/test/java/org/apache/apex/malhar/sql/EmbeddedKafka.java

In above class, I've also added method for publishing and consume from
topic.

Please share your opinion.

Thanks,
Chinmay.

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chandni Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572169#comment-15572169
 ] 

Chandni Singh commented on APEXMALHAR-2284:
---

1.  put needs to done only after you retrieve the value from getAsync(k1).  You 
may have to wait for the future to finish. 

2. it is an async call back and tries to retrieve the latest value.  Whatever 
is put in the bucket is the latest. So if you are doing a put, without 
completing get then it is a problem.


> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: (APEXMALHAR-2290) JDBC operator does not deploy after failure in certain cases

[Chinmay Kolhatkar]

+1

On Thu, Oct 13, 2016 at 7:48 PM, Hitesh Kapoor 
wrote:

> Hi All,
>
> Thank you for your suggestions.
> Following the approach as suggested by Chinmay in option 1.
> Will make the necessary code changes, test it and create a PR.
>
> --Hitesh
>
> On Thu, Oct 13, 2016 at 11:20 AM, Pradeep A. Dalvi 
> wrote:
>
> > +1 for Option 1 to use conn.getMetaData()
> >
> > --prad
> >
> > On Wed, Oct 12, 2016 at 11:39 PM, Chinmay Kolhatkar 
> > wrote:
> >
> > > Hi Hitesh,
> > >
> > > Instead of limiting the row count please use one of the following 2
> > > approaches:
> > >
> > > 1. ResultSet rsColumns = null;
> > > DatabaseMetaData meta = conn.getMetaData();
> > > rsColumns = meta.getColumns(null, null, "tablename", null);
> > > while (rsColumns.next()) {
> > >   System.out.println(rsColumns.getString("TYPE_NAME"));
> > >   System.out.println(rsColumns.getString("COLUMN_NAME"));
> > > }
> > >
> > > Example given here:
> > > http://www.java2s.com/Code/Java/Database-SQL-JDBC/
> > > GetColumnNameAndTypeForATable.htm
> > >
> > > 2. Execute the select statement putting a always false where clause
> like
> > > "select * from table where 1 = 2"
> > >
> > > I would prefer option 1 over option 2.
> > >
> > > -Chinmay.
> > >
> > >
> > > On Wed, Oct 12, 2016 at 8:38 PM, Hitesh Kapoor  >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > This issue occurs when we try to insert records in a table which has
> a
> > > lot
> > > > of data.
> > > > The setup method of JdbcPOJOInsertOutputOperator generates the
> metadata
> > > of
> > > > the columns in the table. To do so it fires a query of the form
> > "Select *
> > > > from tablename" and the extracts the required meta data like column
> > > > name,data type and if it allowed to be NULL.
> > > > When the table has a lot of data this "Select" query takes up a lot
> of
> > > time
> > > > (more than 30 sec) and the operator gets killed.
> > > > The fix is straight forward and simple as suggested by Sandeep is to
> > > limit
> > > > the maximum rows returned by the select query to 1. I am using the
> JDBC
> > > > function setMaxRows() to achieve this.
> > > > Will be opening a PR for the same. This fix won't have corresponding
> > unit
> > > > test cases and I will test the changes externally via an app.
> > > >
> > > > Regards,
> > > > Hitesh Kapoor
> > > >
> > >
> >
>

Re: (APEXMALHAR-2290) JDBC operator does not deploy after failure in certain cases

[Hitesh Kapoor]

Hi All,

Thank you for your suggestions.
Following the approach as suggested by Chinmay in option 1.
Will make the necessary code changes, test it and create a PR.

--Hitesh

On Thu, Oct 13, 2016 at 11:20 AM, Pradeep A. Dalvi  wrote:

> +1 for Option 1 to use conn.getMetaData()
>
> --prad
>
> On Wed, Oct 12, 2016 at 11:39 PM, Chinmay Kolhatkar 
> wrote:
>
> > Hi Hitesh,
> >
> > Instead of limiting the row count please use one of the following 2
> > approaches:
> >
> > 1. ResultSet rsColumns = null;
> > DatabaseMetaData meta = conn.getMetaData();
> > rsColumns = meta.getColumns(null, null, "tablename", null);
> > while (rsColumns.next()) {
> >   System.out.println(rsColumns.getString("TYPE_NAME"));
> >   System.out.println(rsColumns.getString("COLUMN_NAME"));
> > }
> >
> > Example given here:
> > http://www.java2s.com/Code/Java/Database-SQL-JDBC/
> > GetColumnNameAndTypeForATable.htm
> >
> > 2. Execute the select statement putting a always false where clause like
> > "select * from table where 1 = 2"
> >
> > I would prefer option 1 over option 2.
> >
> > -Chinmay.
> >
> >
> > On Wed, Oct 12, 2016 at 8:38 PM, Hitesh Kapoor 
> > wrote:
> >
> > > Hi All,
> > >
> > > This issue occurs when we try to insert records in a table which has a
> > lot
> > > of data.
> > > The setup method of JdbcPOJOInsertOutputOperator generates the metadata
> > of
> > > the columns in the table. To do so it fires a query of the form
> "Select *
> > > from tablename" and the extracts the required meta data like column
> > > name,data type and if it allowed to be NULL.
> > > When the table has a lot of data this "Select" query takes up a lot of
> > time
> > > (more than 30 sec) and the operator gets killed.
> > > The fix is straight forward and simple as suggested by Sandeep is to
> > limit
> > > the maximum rows returned by the select query to 1. I am using the JDBC
> > > function setMaxRows() to achieve this.
> > > Will be opening a PR for the same. This fix won't have corresponding
> unit
> > > test cases and I will test the changes externally via an app.
> > >
> > > Regards,
> > > Hitesh Kapoor
> > >
> >
>

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chandni Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572183#comment-15572183
 ] 

Chandni Singh commented on APEXMALHAR-2284:
---

I think the proper fix for this may take time. IMO we should disable 
POJOInnerJoinOperatorTest but keep the JIRA open.


> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #420: APEXMALHAR-2242 Added user documentation for ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/420


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: EmbeddedKafka Server for testing kafka related code

[Sanjay Pujare]

Strong +1 for using embedded kafka. Found embedded servers very useful in 
ActiveMQ and SQS (ElasticMQ) unit tests. My only question/concern is how well 
can you simulate different versions of Kafka.

On 10/13/16, 7:25 AM, "Chinmay Kolhatkar"  wrote:

Dear Community,

While testing kafka related endpoint code for Apex-Calcite integration, I
found an interesting embedded kafka implementation over the internet and I
used it in my code.

The implementation is based on ready to use EmbeddedZookeer and KafkaServer
from mvn dependency kafka_2.11:0.9.0.0:test.

I found this local kafka server very stable for testing purpose. Would it
be a good addition to malhar-kafka module in test folder?

The EmbeddedKafka server is here:

https://github.com/chinmaykolhatkar/apex-malhar/blob/calcite/sql/src/test/java/org/apache/apex/malhar/sql/EmbeddedKafka.java

In above class, I've also added method for publishing and consume from
topic.

Please share your opinion.

Thanks,
Chinmay.

[jira] [Commented] (APEXCORE-559) Support quit which should behave the same as "exit"

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572630#comment-15572630
 ] 

Sanjay M Pujare commented on APEXCORE-559:
--

bash is a Linux/Unix command line shell and emulates the original Unix sh . It 
probably wants to limit the reserved words as shell commands so I can 
understand why they don't support quit. But most application level CLIs support 
quit and I was surprised to notice Apex CLI doesn't. BTW, "quit" and "exit" can 
have different semantics and I am okay to keep that difference if it applies 
here. For example, "quit" means abort whatever is going on and just quit 
whereas "exit" is allowed only if the system is quiescent.

> Support quit which should behave the same as "exit"
> ---
>
> Key: APEXCORE-559
> URL: https://issues.apache.org/jira/browse/APEXCORE-559
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Sanjay M Pujare
>Priority: Trivial
>
> Most CLIs accept both quit and exit and we should enhance the Apex CLI to 
> accept "quit" which should have the same semantics as "exit".
> apex> quit
> Invalid command 'quit'. Type "help" for list of commands



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2220) Move the FunctionOperator to Malhar library

[Dongming Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572382#comment-15572382
 ] 

Dongming Liang commented on APEXMALHAR-2220:


[~siyuan] Did you have a chance to look into my question above? Thanks.

> Move the FunctionOperator to Malhar library
> ---
>
> Key: APEXMALHAR-2220
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2220
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Siyuan Hua
>Assignee: Dongming Liang
>
> FunctionOperator initially is just designed for high-level API and we think 
> it can also useful if people want to build stateless transformation and work 
> with other operator directly. FunctionOperator can be reused. Thus we should 
> move FO to malhar library



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXCORE-557) Upgrade netlet dependency to 1.3.0

[Vlad Rozov (JIRA)]

Vlad Rozov created APEXCORE-557:
---

 Summary: Upgrade netlet dependency to 1.3.0
 Key: APEXCORE-557
 URL: https://issues.apache.org/jira/browse/APEXCORE-557
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2242) Add documentation for 0.9 version of Kafka Input Operator.

[Munagala V. Ramanath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munagala V. Ramanath resolved APEXMALHAR-2242.
--
   Resolution: Fixed
Fix Version/s: 3.6.0

Merged.

> Add documentation for 0.9 version of Kafka Input Operator.
> --
>
> Key: APEXMALHAR-2242
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2242
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: Chaitanya
>Assignee: Chaitanya
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Sanjay M Pujare (JIRA)]

Sanjay M Pujare created APEXCORE-558:


 Summary: Do not use yellow color to display command strings in 
help output
 Key: APEXCORE-558
 URL: https://issues.apache.org/jira/browse/APEXCORE-558
 Project: Apache Apex Core
  Issue Type: Bug
 Environment: MacOS, Linux
Reporter: Sanjay M Pujare
Priority: Minor


Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
strings in yellow which is extremely hard to read and reduces usability of the 
CLI. Should be changed to red or purple or just be bolded in black.

In the following output "alias" and "begin-macro" are in yellow and hard to 
read.

apex> help
GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:

alias alias-name command
Create a command alias
begin-macro name




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2242) Add documentation for 0.9 version of Kafka Input Operator.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572645#comment-15572645
 ] 

ASF GitHub Bot commented on APEXMALHAR-2242:


Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/420


> Add documentation for 0.9 version of Kafka Input Operator.
> --
>
> Key: APEXMALHAR-2242
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2242
> Project: Apache Apex Malhar
>  Issue Type: Documentation
>Reporter: Chaitanya
>Assignee: Chaitanya
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: EmbeddedKafka Server for testing kafka related code

[Thomas Weise]

+1, how hard is it to change the existing tests in the Kafka module to use
it?

Different versions are IMO not an issue for the existing tests, they
already run against a fixed version (the broker is launched as part of the
test).

On Thu, Oct 13, 2016 at 9:59 AM, Sanjay Pujare 
wrote:

> Strong +1 for using embedded kafka. Found embedded servers very useful in
> ActiveMQ and SQS (ElasticMQ) unit tests. My only question/concern is how
> well can you simulate different versions of Kafka.
>
> On 10/13/16, 7:25 AM, "Chinmay Kolhatkar"  wrote:
>
> Dear Community,
>
> While testing kafka related endpoint code for Apex-Calcite
> integration, I
> found an interesting embedded kafka implementation over the internet
> and I
> used it in my code.
>
> The implementation is based on ready to use EmbeddedZookeer and
> KafkaServer
> from mvn dependency kafka_2.11:0.9.0.0:test.
>
> I found this local kafka server very stable for testing purpose. Would
> it
> be a good addition to malhar-kafka module in test folder?
>
> The EmbeddedKafka server is here:
> https://github.com/chinmaykolhatkar/apex-malhar/
> blob/calcite/sql/src/test/java/org/apache/apex/malhar/
> sql/EmbeddedKafka.java
>
> In above class, I've also added method for publishing and consume from
> topic.
>
> Please share your opinion.
>
> Thanks,
> Chinmay.
>
>
>
>

[jira] [Created] (APEXCORE-559) Support quit which should behave the same as "exit"

[Sanjay M Pujare (JIRA)]

Sanjay M Pujare created APEXCORE-559:


 Summary: Support quit which should behave the same as "exit"
 Key: APEXCORE-559
 URL: https://issues.apache.org/jira/browse/APEXCORE-559
 Project: Apache Apex Core
  Issue Type: Improvement
Reporter: Sanjay M Pujare
Priority: Trivial


Most CLIs accept both quit and exit and we should enhance the Apex CLI to 
accept "quit" which should have the same semantics as "exit".

apex> quit
Invalid command 'quit'. Type "help" for list of commands




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output

[Munagala V. Ramanath (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572586#comment-15572586
 ] 

Munagala V. Ramanath commented on APEXCORE-558:
---

Yes, agree, the yellow has inadequate contrast when used with light color 
schemes.


> Do not use yellow color to display command strings in help output
> -
>
> Key: APEXCORE-558
> URL: https://issues.apache.org/jira/browse/APEXCORE-558
> Project: Apache Apex Core
>  Issue Type: Bug
> Environment: MacOS, Linux
>Reporter: Sanjay M Pujare
>Priority: Minor
>
> Apex CLI Help output (at least on MacOS and Linux terminals) shows command 
> strings in yellow which is extremely hard to read and reduces usability of 
> the CLI. Should be changed to red or purple or just be bolded in black.
> In the following output "alias" and "begin-macro" are in yellow and hard to 
> read.
> apex> help
> GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
> alias alias-name command
>   Create a command alias
> begin-macro name
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-559) Support quit which should behave the same as "exit"

[Munagala V. Ramanath (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572610#comment-15572610
 ] 

Munagala V. Ramanath commented on APEXCORE-559:
---

I agree that this improves usability (both Python and Ruby accept both) though 
there are counter examples, e.g. bash
which does not like 'quit'.


> Support quit which should behave the same as "exit"
> ---
>
> Key: APEXCORE-559
> URL: https://issues.apache.org/jira/browse/APEXCORE-559
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Sanjay M Pujare
>Priority: Trivial
>
> Most CLIs accept both quit and exit and we should enhance the Apex CLI to 
> accept "quit" which should have the same semantics as "exit".
> apex> quit
> Invalid command 'quit'. Type "help" for list of commands



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2297) Readme files for Demos

[Sandesh (JIRA)]

Sandesh created APEXMALHAR-2297:
---

 Summary: Readme files for Demos
 Key: APEXMALHAR-2297
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2297
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Sandesh


We have many demos under, 
https://github.com/apache/apex-malhar/tree/master/demos, none of them have 
Readme files.

It is useful to have Readme file along with the demos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Updated] (APEXCORE-553) Add google analytics tracking to apex website

[Michelle Xiao]

I suggest to install Google Tag Manager on Apex website, and add Google
Analytics there.

GA is the analytics tool that provides reports about activity on the site.
GTM is a tag manager that can output tags based on defined rules. The main
benefit of GTM is that we can use it for multiple tags, especially some
customized event tracking, e.g. Download Button click, file download.

Current DataTorrent website uses GTM, we have 10 customized tags so far,
the corresponding event reports can be found in Google Analytics.

On Wed, Oct 12, 2016 at 6:49 PM, Ashwin Chandra Putta (JIRA) <
j...@apache.org> wrote:

>
>  [ https://issues.apache.org/jira/browse/APEXCORE-553?page=
> com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Ashwin Chandra Putta updated APEXCORE-553:
> --
> Description:
> Add Google Analytics tracking to the apex.apache.org website so that we
> can learn about website traffic and what users are looking for.
>
> {code}
> 
>   (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]
> =i[r]||function(){
>   (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
> Date();a=s.createElement(o),
>   m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.
> parentNode.insertBefore(a,m)
>   })(window,document,'script','https://www.google-analytics.
> com/analytics.js','ga');
>
>   ga('create', 'UA-85540278-1', 'auto');
>   ga('send', 'pageview');
>
> 
> {code}
>
> Add privacy footer to indicate collection of data. Reference:
> http://www.apache.org/foundation/policies/privacy.html
>
> Discussion: https://lists.apache.org/thread.html/
> f1af3f19ecd2532eb2f5cea2a81fda744912112df5259e2d55c05b88@%
> 3Cdev.apex.apache.org%3E
>
>   was:
> Add Google Analytics tracking to the apex.apache.org website so that we
> can learn about website traffic and what users are looking for.
>
> {code}
> 
>   (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]
> =i[r]||function(){
>   (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
> Date();a=s.createElement(o),
>   m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.
> parentNode.insertBefore(a,m)
>   })(window,document,'script','https://www.google-analytics.
> com/analytics.js','ga');
>
>   ga('create', 'UA-85540278-1', 'auto');
>   ga('send', 'pageview');
>
> 
> {code}
>
> Discussion: https://lists.apache.org/thread.html/
> f1af3f19ecd2532eb2f5cea2a81fda744912112df5259e2d55c05b88@%
> 3Cdev.apex.apache.org%3E
>
>
> > Add google analytics tracking to apex website
> > -
> >
> > Key: APEXCORE-553
> > URL: https://issues.apache.org/jira/browse/APEXCORE-553
> > Project: Apache Apex Core
> >  Issue Type: Improvement
> >  Components: Website
> >Reporter: Ashwin Chandra Putta
> >Assignee: Michelle Xiao
> >
> > Add Google Analytics tracking to the apex.apache.org website so that we
> can learn about website traffic and what users are looking for.
> > {code}
> > 
> >   (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]
> =i[r]||function(){
> >   (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
> Date();a=s.createElement(o),
> >   m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.
> parentNode.insertBefore(a,m)
> >   })(window,document,'script','https://www.google-analytics.
> com/analytics.js','ga');
> >   ga('create', 'UA-85540278-1', 'auto');
> >   ga('send', 'pageview');
> > 
> > {code}
> > Add privacy footer to indicate collection of data. Reference:
> http://www.apache.org/foundation/policies/privacy.html
> > Discussion: https://lists.apache.org/thread.html/
> f1af3f19ecd2532eb2f5cea2a81fda744912112df5259e2d55c05b88@%
> 3Cdev.apex.apache.org%3E
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

[jira] [Commented] (APEXMALHAR-2220) Move the FunctionOperator to Malhar library

[Siyuan Hua (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573091#comment-15573091
 ] 

Siyuan Hua commented on APEXMALHAR-2220:


[~d9liang] The malhar-library should be depends on malhar-stream. The classes 
related to this tickets can be moved from malhar-stream to malhar-library 
including the function interface and function operator. They should not depend 
on anything from malhar-stream

> Move the FunctionOperator to Malhar library
> ---
>
> Key: APEXMALHAR-2220
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2220
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Siyuan Hua
>Assignee: Dongming Liang
>
> FunctionOperator initially is just designed for high-level API and we think 
> it can also useful if people want to build stateless transformation and work 
> with other operator directly. FunctionOperator can be reused. Thus we should 
> move FO to malhar library



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574244#comment-15574244
 ] 

Thomas Weise commented on APEXMALHAR-2284:
--

That's fine. Can you summarize how the issue will be addressed though. The 
operator is broken right now or not?

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chaitanya (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574274#comment-15574274
 ] 

Chaitanya commented on APEXMALHAR-2284:
---

Right now the operator is broken.

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chaitanya (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571020#comment-15571020
 ] 

Chaitanya commented on APEXMALHAR-2284:
---

This was failing due to the below merged PR:
https://github.com/apache/apex-malhar/pull/444

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chaitanya (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571020#comment-15571020
 ] 

Chaitanya edited comment on APEXMALHAR-2284 at 10/13/16 6:46 AM:
-

[~thw] This was failing due to the below merged PR:
https://github.com/apache/apex-malhar/pull/444.
Please suggest the right approach further.


was (Author: chaithu):
This was failing due to the below merged PR:
https://github.com/apache/apex-malhar/pull/444

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chandni Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571074#comment-15571074
 ] 

Chandni Singh edited comment on APEXMALHAR-2284 at 10/13/16 6:57 AM:
-

Are you sure about that?

The tests were failing before merging the PR on a different PR as well, if I 
remember correctly.

Also why do they fail intermittently?


was (Author: csingh):
Are you sure about that?

The tests were failing before merging the PR on a different PR as well, if I 
remember correctly?

Also why do they fail intermittently?

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chaitanya (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571291#comment-15571291
 ] 

Chaitanya commented on APEXMALHAR-2284:
---

[~csingh] If you feel that adding the variable "asyncReadSource" is not 
feasible, Please suggest how to achieve the above scenario can be valid using 
ManagedState.

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chandni Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571119#comment-15571119
 ] 

Chandni Singh commented on APEXMALHAR-2284:
---

Chaitanya,

Please feel free to assign the ticket to me if my PR broke these tests. Having 
a variable {code}asyncReadSource{code} didn't really make much sense to me. 
Changing how reader thread fetch value from a bucket doesn't need to be 
altered.  
Anyways if this was done for tests, then there are better ways of doing it. 



> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: can operators emit on a different from the operator itself thread?

[Amol Kekre]

Vlad,
I agree that the check should be ON by default. Ability to turn it off for
entire app is fine, per port is not needed.

Thks
Amol

On Wed, Oct 12, 2016 at 10:34 PM, Tushar Gosavi 
wrote:

> +1 for on by default and ability to turn it off for entire application.
>
> - Tushar.
>
>
> On Thu, Oct 13, 2016 at 11:00 AM, Pradeep A. Dalvi 
> wrote:
> > +1 for ON by default
> > +1 for disabling it for all output ports
> >
> > With the kind of issues we have observed being faced by developers in the
> > past, I strongly believe this check should be ON by default.
> > However at the same time I feel, it shall be one-time check, mostly in
> > Development phase and before going into Production. Having said that, if
> > disabling it at application level i.e. for all operators and their
> > respective output ports would it make implementation simpler, then that
> can
> > be targeted first. Thoughts?
> >
> > --prad
> >
> > On Thu, Oct 13, 2016 at 7:32 AM, Vlad Rozov 
> wrote:
> >
> >> I run jmh test and check takes 1ns on my MacBook Pro and on the lab
> >> machine. This corresponds to 3% degradation at 30 million
> events/second. I
> >> think we can move forward with the check ON by default. Do we need an
> >> ability to turn OFF check for a specific operator and/or port? My
> thought
> >> is that such ability is not necessary and it should be OK to disable
> check
> >> for all output ports in an application.
> >>
> >> Vlad
> >>
> >>
> >> On 10/12/16 11:56, Amol Kekre wrote:
> >>
> >>> In case there turns out to be a penalty, we can introduce a "check for
> >>> thread affinity" mode that triggers this check. My initial thought is
> to
> >>> make this check ON by default. We should wait till benchmarks are
> >>> available
> >>> before discussing adding this check.
> >>>
> >>> Thks
> >>> Amol
> >>>
> >>>
> >>> On Wed, Oct 12, 2016 at 11:07 AM, Sanjay Pujare <
> san...@datatorrent.com>
> >>> wrote:
> >>>
> >>> A JIRA has been created for adding this thread affinity check
>  https://issues.apache.org/jira/browse/APEXCORE-510 . I have made this
>  enhancement in a branch
>  https://github.com/sanjaypujare/apex-core/tree/malhar-510.
>  thread_affinity
>  and I have been benchmarking the performance with this change. I will
> be
>  publishing the results in the above JIRA where we can discuss them and
>  hopefully agree on merging this change.
> 
>  On Thu, Aug 11, 2016 at 1:41 PM, Sanjay Pujare <
> san...@datatorrent.com>
>  wrote:
> 
>  You are right, I was subconsciously thinking about the THREAD_LOCAL
> case
> > with a single container and a simple DAG and in that case Vlad’s
> >
>  assumption
> 
> > might not be valid but may be it is.
> >
> > On 8/11/16, 11:47 AM, "Munagala Ramanath" 
> wrote:
> >
> >  If I understand Vlad correctly, what he is saying is that each
> >
>  operator
> 
> >  saves currentThread in
> >  its own setup() and checks it in its own output methods. The
> > threads
> >
>  in
> 
> >  different operators are
> >  running potentially on different nodes and/or processes and
> there
> >
>  will
> 
> > be
> >  no connection between them.
> >
> >  Ram
> >
> >  On Thu, Aug 11, 2016 at 11:41 AM, Sanjay Pujare <
> > san...@datatorrent.com>
> >  wrote:
> >
> >  > Name check is expensive, agreed, but there isn’t anything else
> > currently.
> >  > Ideally the stram engine (considering that it is an engine
> >
>  providing
> 
> >  > resources like threads etc) should use a ThreadFactory or a
> > ThreadGroup to
> >  > create operator threads so identification and adding
> > functionality
> >
>  is
> 
> >  > easier.
> >  >
> >  > The idea of checking for the same thread between setup() and
> > emit()
> > won’t
> >  > work because the emit() check will have to be in the Sink
> > hierarchy
> > and
> >  > AFAIK a Sink object doesn’t have access to the corresponding
> > operator,
> >  > right? Another more fundamental problem probably is that these
> > threads
> >  > don’t have to match. The emit() for any operator (or rather a
> > Sink
> > related
> >  > to an operator) is ultimately triggered by an emitTuple() on
> the
> > topmost
> >  > input operator in that path which happens in that input
> > operator’s
> > thread
> >  > which doesn’t have to match the thread calling setup() in the
> > downstream
> >  > operators, right?
> >  >
> >  >
> >  > On 8/11/16, 10:59 AM, "Vlad Rozov" 
> >
>  wrote:
> 
> >  >
> >  > Name verification is too 

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chandni Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571074#comment-15571074
 ] 

Chandni Singh commented on APEXMALHAR-2284:
---

Are you sure about that?

The tests were failing before merging the PR on a different PR as well, if I 
remember correctly?

Also why do they fail intermittently?

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chandni Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571119#comment-15571119
 ] 

Chandni Singh edited comment on APEXMALHAR-2284 at 10/13/16 7:27 AM:
-

Chaitanya,

Please feel free to assign the ticket to me if my PR broke these tests. Having 
a variable {code}asyncReadSource{code} didn't really make much sense to me. 
Changing how reader thread fetch value from a bucket doesn't need to be exposed 
and configured.  
Anyways if this was done for tests, then there are better ways of doing it. 




was (Author: csingh):
Chaitanya,

Please feel free to assign the ticket to me if my PR broke these tests. Having 
a variable {code}asyncReadSource{code} didn't really make much sense to me. 
Changing how reader thread fetch value from a bucket doesn't need to be 
altered.  
Anyways if this was done for tests, then there are better ways of doing it. 



> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2284) POJOInnerJoinOperatorTest fails in Travis CI

[Chaitanya (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571170#comment-15571170
 ] 

Chaitanya commented on APEXMALHAR-2284:
---

getAsync() method has two parts. First it looks into the bucket memory, if it 
founds then it returns value with immediateFuture. If not then it returns the 
Future task with ValueFetchTask.
Task of ValueFetchTask is first checks into bucket memory again and if not 
found in memory then returns the value from disk.

In case of join operator, key is not present in first memory lookup, but it may 
present in the second lookup. 

> POJOInnerJoinOperatorTest fails in Travis CI
> 
>
> Key: APEXMALHAR-2284
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2284
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Chaitanya
>Priority: Blocker
>
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/166322754/log.txt
> {code}
> Failed tests: 
>   POJOInnerJoinOperatorTest.testEmitMultipleTuplesFromStream2:337 Number of 
> tuple emitted  expected:<2> but was:<4>
>   POJOInnerJoinOperatorTest.testInnerJoinOperator:184 Number of tuple emitted 
>  expected:<1> but was:<2>
>   POJOInnerJoinOperatorTest.testMultipleValues:236 Number of tuple emitted  
> expected:<2> but was:<3>
>   POJOInnerJoinOperatorTest.testUpdateStream1Values:292 Number of tuple 
> emitted  expected:<1> but was:<2>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) Add generic endpoint to sql support of Apex

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Issue Type: New Feature  (was: Improvement)

> Add generic endpoint to sql support of Apex
> ---
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2293) Add sql line cli to sql support of apex

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2293:
-

 Summary: Add sql line cli to sql support of apex
 Key: APEXMALHAR-2293
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2293
 Project: Apache Apex Malhar
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Add cli like support to malhar-sql.
The support can be done similar to: 
https://github.com/apache/storm/blob/master/external/sql/README.md

It should support following type of statements:
1. CREATE EXTERNAL TABLE with kafka, file, jdbc specifications
2. DROP EXTERNAL TABLE
3. CREATE FUNCTION
4. DELETE FUNCTION
5. INSERT and SELECT statements




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Summary: Add generic endpoint to malhar-sql  (was: malhar-sql: Add generic 
endpoint to sql support of Apex)

> Add generic endpoint to malhar-sql
> --
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) malhar-sql: Add generic endpoint to sql support of Apex

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Summary: malhar-sql: Add generic endpoint to sql support of Apex  (was: Add 
generic endpoint to sql support of Apex)

> malhar-sql: Add generic endpoint to sql support of Apex
> ---
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2292) Apex SQL: Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2292:
--
Summary: Apex SQL: Add generic endpoint to malhar-sql  (was: Add generic 
endpoint to malhar-sql)

> Apex SQL: Add generic endpoint to malhar-sql
> 
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2294) Apex SQL: Make operator names user friendly for user to configure via properties.xml

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2294:
-

 Summary: Apex SQL: Make operator names user friendly for user to 
configure via properties.xml
 Key: APEXMALHAR-2294
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2294
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Make operator names user friendly for user to configure via properties.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2295) Apex SQL: Replace POJOInnerJoinOperator with WindowedJoinOperator

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2295:
-

 Summary: Apex SQL: Replace POJOInnerJoinOperator with 
WindowedJoinOperator
 Key: APEXMALHAR-2295
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2295
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Replace POJOInnerJoinOperator with WindowedJoinOperator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXMALHAR-2296) Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2296:
-

 Summary: Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)
 Key: APEXMALHAR-2296
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2296
 Project: Apache Apex Malhar
  Issue Type: New Feature
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Add support for SQL GROUP BY (Aggregate RelNode)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)