[jira] [Commented] (APEXMALHAR-2548) Pubsub operators do not use the correct URL to connect to web socket server in SSL mode

2017-11-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238640#comment-16238640
 ] 

ASF GitHub Bot commented on APEXMALHAR-2548:


PramodSSImmaneni closed pull request #676: APEXMALHAR-2548 Using the correct 
websocket scheme when connecting to a SSL cluster
URL: https://github.com/apache/apex-malhar/pull/676
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java 
b/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java
index 98dfebd7b8..82c9214fa9 100644
--- 
a/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java
+++ 
b/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java
@@ -18,19 +18,24 @@
  */
 package com.datatorrent.apps.logstream;
 
-import java.net.URI;
-import java.util.*;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Set;
 
 import org.apache.apex.malhar.contrib.misc.streamquery.SelectOperator;
 import 
org.apache.apex.malhar.contrib.misc.streamquery.condition.EqualValueCondition;
-
-import org.apache.commons.lang.StringUtils;
+import org.apache.apex.malhar.lib.utils.PubSubHelper;
 import org.apache.hadoop.conf.Configuration;
 
+import com.datatorrent.api.DAG;
+import com.datatorrent.api.Operator.InputPort;
+import com.datatorrent.api.StreamingApplication;
+import com.datatorrent.contrib.redis.RedisKeyValPairOutputOperator;
+import com.datatorrent.contrib.redis.RedisMapOutputOperator;
+import com.datatorrent.contrib.redis.RedisNumberSummationMapOutputOperator;
 import com.datatorrent.lib.algo.TopN;
 import com.datatorrent.lib.io.ConsoleOutputOperator;
 import com.datatorrent.lib.io.PubSubWebSocketOutputOperator;
@@ -43,13 +48,6 @@
 import com.datatorrent.lib.util.AbstractDimensionTimeBucketOperator;
 import com.datatorrent.lib.util.DimensionTimeBucketSumOperator;
 
-import com.datatorrent.api.DAG;
-import com.datatorrent.api.Operator.InputPort;
-import com.datatorrent.api.StreamingApplication;
-import com.datatorrent.contrib.redis.RedisKeyValPairOutputOperator;
-import com.datatorrent.contrib.redis.RedisMapOutputOperator;
-import com.datatorrent.contrib.redis.RedisNumberSummationMapOutputOperator;
-
 /**
  * Log stream processing application based on Apex platform.
  * This application consumes log data generated by running systems and services
@@ -156,14 +154,12 @@ private SYSTEM_KEYS(String value)
 
   private InputPort wsOutput(DAG dag, String operatorName)
   {
-String daemonAddress = dag.getValue(DAG.GATEWAY_CONNECT_ADDRESS);
-if (!StringUtils.isEmpty(daemonAddress)) {
-  URI uri = URI.create("ws://" + daemonAddress + "/pubsub");
+if (PubSubHelper.isGatewayConfigured(dag)) {
   String appId = "appid";
   //appId = dag.attrValue(DAG.APPLICATION_ID, null); // will be used once 
UI is able to pick applications from list and listen to corresponding 
application
   String topic = "apps.logstream." + appId + "." + operatorName;
   PubSubWebSocketOutputOperator wsOut = 
dag.addOperator(operatorName, new PubSubWebSocketOutputOperator());
-  wsOut.setUri(uri);
+  wsOut.setUri(PubSubHelper.getURI(dag));
   wsOut.setTopic(topic);
   return wsOut.input;
 }
diff --git 
a/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
 
b/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
index 73c38ef6cb..eed9a683e7 100644
--- 
a/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
+++ 
b/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
@@ -20,11 +20,13 @@
 
 import java.io.Serializable;
 import java.net.URI;
+
 import org.apache.apex.examples.frauddetect.operator.HdfsStringOutputOperator;
 import org.apache.apex.examples.frauddetect.operator.MongoDBOutputOperator;
+import org.apache.apex.malhar.lib.utils.PubSubHelper;
 import org.apache.hadoop.conf.Configuration;
+
 import com.datatorrent.api.Context;
-import com.datatorrent.api.Context.DAGContext;
 import com.datatorrent.api.DAG;
 import com.datatorrent.api.StreamingApplication;
 import com.datatorrent.api.annotation.ApplicationAnnotation;
@@ -88,11 +90,7 @@ public void populateDAG(DAG dag, Configuration conf)
   {
 
 try {
-  String gatewayAddress = dag.getValue(DAGContext.GATEWAY_CONNECT_ADDRESS);
-  if (gatewayAddress == null) {
-gatewayAddress = "localhost:9090";
-  }
-  URI duri = URI.create("ws://" + gatewayAddress + "/pubsub");
+

[jira] [Commented] (APEXMALHAR-2548) Pubsub operators do not use the correct URL to connect to web socket server in SSL mode

2017-11-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238641#comment-16238641
 ] 

ASF GitHub Bot commented on APEXMALHAR-2548:


PramodSSImmaneni opened a new pull request #676: APEXMALHAR-2548 Using the 
correct websocket scheme when connecting to a SSL cluster
URL: https://github.com/apache/apex-malhar/pull/676
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Pubsub operators do not use the correct URL to connect to web socket server 
> in SSL mode
> ---
>
> Key: APEXMALHAR-2548
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2548
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Pramod Immaneni
>Assignee: Pramod Immaneni
>Priority: Major
>
> In SSL mode, pub sub operators need to use wss protocol instead of ws. Today 
> they use ws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (APEXCORE-791) Gateway security settings need to be available in the DAG

2017-11-03 Thread Vlad Rozov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/APEXCORE-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov resolved APEXCORE-791.
-
   Resolution: Fixed
Fix Version/s: 3.7.0

> Gateway security settings need to be available in the DAG
> -
>
> Key: APEXCORE-791
> URL: https://issues.apache.org/jira/browse/APEXCORE-791
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Pramod Immaneni
>Assignee: Pramod Immaneni
>Priority: Major
> Fix For: 3.7.0
>
>
> Gateway connect address attribute is available while constructing the DAG but 
> other gateway security-related attributes such as GATEWAY_USE_SSL are not. 
> These need to be made available as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (APEXCORE-791) Gateway security settings need to be available in the DAG

2017-11-03 Thread Vlad Rozov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/APEXCORE-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-791:

Priority: Minor  (was: Major)

> Gateway security settings need to be available in the DAG
> -
>
> Key: APEXCORE-791
> URL: https://issues.apache.org/jira/browse/APEXCORE-791
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Pramod Immaneni
>Assignee: Pramod Immaneni
>Priority: Minor
> Fix For: 3.7.0
>
>
> Gateway connect address attribute is available while constructing the DAG but 
> other gateway security-related attributes such as GATEWAY_USE_SSL are not. 
> These need to be made available as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (APEXCORE-791) Gateway security settings need to be available in the DAG

2017-11-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXCORE-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238283#comment-16238283
 ] 

ASF GitHub Bot commented on APEXCORE-791:
-

vrozov closed pull request #586: APEXCORE-791 Making gateway security related 
settings available during construction of the DAG
URL: https://github.com/apache/apex-core/pull/586
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java 
b/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java
index 07641d23e2..65be99a6d0 100644
--- a/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java
+++ b/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java
@@ -71,6 +71,7 @@
 import org.apache.apex.engine.plugin.ApexPluginDispatcher;
 import org.apache.apex.engine.plugin.NoOpApexPluginDispatcher;
 import org.apache.apex.engine.util.CascadeStorageAgent;
+import org.apache.apex.engine.util.PubSubWebSocketClientBuilder;
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.builder.ToStringBuilder;
@@ -197,7 +198,6 @@
 public class StreamingContainerManager implements PlanContext
 {
   private static final Logger LOG = 
LoggerFactory.getLogger(StreamingContainerManager.class);
-  public static final String GATEWAY_LOGIN_URL_PATH = "/ws/v2/login";
   public static final String BUILTIN_APPDATA_URL = "builtin";
   public static final String CONTAINERS_INFO_FILENAME_FORMAT = 
"containers_%d.json";
   public static final String OPERATORS_INFO_FILENAME_FORMAT = 
"operators_%d.json";
@@ -556,23 +556,12 @@ private void setupStringCodecs()
 
   private void setupWsClient()
   {
-String gatewayAddress = 
plan.getLogicalPlan().getValue(LogicalPlan.GATEWAY_CONNECT_ADDRESS);
-boolean gatewayUseSsl = 
plan.getLogicalPlan().getValue(LogicalPlan.GATEWAY_USE_SSL);
-String gatewayUserName = 
plan.getLogicalPlan().getValue(LogicalPlan.GATEWAY_USER_NAME);
-String gatewayPassword = 
plan.getLogicalPlan().getValue(LogicalPlan.GATEWAY_PASSWORD);
-int timeout = 
plan.getLogicalPlan().getValue(LogicalPlan.PUBSUB_CONNECT_TIMEOUT_MILLIS);
-
-if (gatewayAddress != null) {
+wsClient = new 
PubSubWebSocketClientBuilder().setContext(plan.getLogicalPlan()).build();
+if (wsClient != null) {
   try {
-wsClient = new SharedPubSubWebSocketClient((gatewayUseSsl ? "wss://" : 
"ws://") + gatewayAddress + "/pubsub", timeout);
-if (gatewayUserName != null && gatewayPassword != null) {
-  wsClient.setLoginUrl((gatewayUseSsl ? "https://; : "http://;) + 
gatewayAddress + GATEWAY_LOGIN_URL_PATH);
-  wsClient.setUserName(gatewayUserName);
-  wsClient.setPassword(gatewayPassword);
-}
-wsClient.setup();
-  } catch (Exception ex) {
-LOG.warn("Cannot establish websocket connection to {}", 
gatewayAddress, ex);
+wsClient.openConnection();
+  } catch (Exception e) {
+LOG.warn("Cannot establish websocket connection to uri {}", 
wsClient.getUri(), e);
   }
 }
   }
diff --git 
a/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java 
b/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java
index e41b348a0c..49998cfa00 100644
--- 
a/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java
+++ 
b/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java
@@ -26,6 +26,8 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import org.apache.apex.engine.util.PubSubWebSocketClientBuilder;
+
 import com.datatorrent.api.Component;
 import com.datatorrent.api.Context;
 import com.datatorrent.api.Context.OperatorContext;
@@ -39,7 +41,6 @@
 import com.datatorrent.api.StatsListener;
 import com.datatorrent.api.StatsListener.OperatorRequest;
 import com.datatorrent.api.StringCodec;
-import com.datatorrent.stram.StreamingContainerManager;
 import com.datatorrent.stram.api.ContainerContext;
 import com.datatorrent.stram.api.ContainerEvent.ContainerStatsEvent;
 import com.datatorrent.stram.api.ContainerEvent.NodeActivationEvent;
@@ -66,10 +67,6 @@
 public class TupleRecorderCollection extends HashMap implements Component
 {
   private int tupleRecordingPartFileSize;
-  private String gatewayAddress;
-  private boolean gatewayUseSsl = false;
-  private String gatewayUserName;
-  private String gatewayPassword;
   private long tupleRecordingPartFileTimeMillis;
   private String appPath;
   private String appId;
@@ -89,13 +86,11 @@ public void setup(Context ctx)

[jira] [Created] (APEXCORE-793) Revisit PubSubWebSocketClient and SharedPubSubWebSocketClient

2017-11-03 Thread Vlad Rozov (JIRA) 
Vlad Rozov created APEXCORE-793:
---

 Summary: Revisit PubSubWebSocketClient and 
SharedPubSubWebSocketClient
 Key: APEXCORE-793
 URL: https://issues.apache.org/jira/browse/APEXCORE-793
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Vlad Rozov
Assignee: Vlad Rozov






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Malhar release 3.8.0

2017-11-03 Thread Vlad Rozov 
Sorry, I don't fully understand your point. Is the suggestion not to add 
maven plugin and only rely on manual sanity check or do extra manual 
sanity check on top of the plugin? Any reason not to go with the second 
approach?


Thank you,

Vlad

On 11/3/17 08:36, Justin Mclean wrote:

Hi,


The JIRA mentioned is to unblock the release. IMO the license plugin should
be taken up as separate activity.

JFYI - In my experience the license plugin is not going to catch everything and 
tit may produce false positives. The best way is to manually look at the 
release artefact and it’s dependancies.

Thanks,
Justin

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Ambarish Pande 
Congratulations Tushar!

On Fri, 3 Nov 2017 at 11:48 PM, Sanjay Pujare 
wrote:

> Congratulations, Tushar!
>
> Sanjay
>
>
> On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni 
> wrote:
>
> > A bit delayed but nevertheless important announcement, Apache Apex PMC is
> > pleased to announce Tushar Gosavi as a new PMC member.
> >
> > Tushar has been contributing to Apex from the beginning of the project
> and
> > has been working on the codebase for over 3 years. He is among the few
> who
> > have a wide breadth of contribution, including both core and malhar, from
> > internal changes to user facing api, from input/output operators to
> > components that support operators, and has a good overall understanding
> of
> > the codebase and how it works.
> >
> > His salient contributions over the years are
> >
> >- Module support in Apex
> >- Operator additions and improvements such as S3, File input and
> output,
> >partitionable unique count and dynamic partitioning improvements
> >- Initial WAL implementation from which subsequent implementations
> were
> >derived for different use cases
> >- Plugin support for Apex
> >- Various bug fixes and improvements in both malhar and core that you
> >can find in the JIRA
> >- Participated in long-term project maintenance tasks such as
> >refactoring operators and demos
> >- Participated in important feature discussions
> >- Reviewed and committed pull requests from contributors
> >- Participated in conducting and teaching in an Apex workshop at a
> >university and speaking at Apex conference organized by DataTorrent
> >
> > Conference talks & Presentations
> >
> >1. Presentations at VIIT and PICT Pune
> >2. http://www.apexbigdata.com/pune-platform-talk-9.html
> >3. http://www.apexbigdata.com/pune-integration-talk-4.html
> >4. Webinar on Smart Partitioning with Apex. (
> >https://www.youtube.com/watch?v=HCATB1zlLE4
> >)
> >5. Presented about customer use case at Pune Meetup in 2016
> >
> > Pramod for the Apache Apex PMC.
> >
>
-- 

___

Ambarish Pande

Associate Software Engineer

E: ambar...@datatorrent.com | M: +91-9028293982

www.datatorrent.com  |  apex.apache.org

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Sanjay Pujare 
Congratulations, Ananth

Sanjay


On Fri, Nov 3, 2017 at 1:50 AM, Thomas Weise  wrote:

> The Project Management Committee (PMC) for Apache Apex is pleased to
> announce Ananth Gundabattula as new committer.
>
> Ananth has been contributing to the project for about a year. Highlights:
>
> * Cassandra and Kudu operators with in-depth analysis/design work
> * Good collaboration, adherence to contributor guidelines and ownership of
> work
> * Work beyond feature focus such as fixing pre-existing test issues that
> impact CI
> * Presented at YOW Data and Dataworks Summit Australia
> * Enthusiast, contributes on his own time
>
> Welcome, Ananth, and congratulations!
> Thomas, for the Apache Apex PMC.
>

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Sanjay Pujare 
Congratulations, Tushar!

Sanjay


On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni 
wrote:

> A bit delayed but nevertheless important announcement, Apache Apex PMC is
> pleased to announce Tushar Gosavi as a new PMC member.
>
> Tushar has been contributing to Apex from the beginning of the project and
> has been working on the codebase for over 3 years. He is among the few who
> have a wide breadth of contribution, including both core and malhar, from
> internal changes to user facing api, from input/output operators to
> components that support operators, and has a good overall understanding of
> the codebase and how it works.
>
> His salient contributions over the years are
>
>- Module support in Apex
>- Operator additions and improvements such as S3, File input and output,
>partitionable unique count and dynamic partitioning improvements
>- Initial WAL implementation from which subsequent implementations were
>derived for different use cases
>- Plugin support for Apex
>- Various bug fixes and improvements in both malhar and core that you
>can find in the JIRA
>- Participated in long-term project maintenance tasks such as
>refactoring operators and demos
>- Participated in important feature discussions
>- Reviewed and committed pull requests from contributors
>- Participated in conducting and teaching in an Apex workshop at a
>university and speaking at Apex conference organized by DataTorrent
>
> Conference talks & Presentations
>
>1. Presentations at VIIT and PICT Pune
>2. http://www.apexbigdata.com/pune-platform-talk-9.html
>3. http://www.apexbigdata.com/pune-integration-talk-4.html
>4. Webinar on Smart Partitioning with Apex. (
>https://www.youtube.com/watch?v=HCATB1zlLE4
>)
>5. Presented about customer use case at Pune Meetup in 2016
>
> Pramod for the Apache Apex PMC.
>

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Chinmay Kolhatkar 
Congrats Tushar!!


On Fri, Nov 3, 2017 at 10:21 PM, Chaitanya Chebolu <
chaita...@datatorrent.com> wrote:

> Congratulations Tushar!!!
>
> On Fri, Nov 3, 2017 at 10:06 PM, Venkatesh Kottapalli <
> venkat...@datatorrent.com> wrote:
>
> > Congratulations Tushar!
> >
> > -Venky
> >
> > On Fri, Nov 3, 2017 at 9:24 AM Hitesh Kapoor 
> > wrote:
> >
> > > Congratulations Tushar.
> > >
> > > Regards,
> > > Hitesh Kapoor
> > >
> > > On 03-Nov-2017 9:53 PM, "Ashwin Chandra Putta" <
> ashwinchand...@gmail.com
> > >
> > > wrote:
> > >
> > > > Congratulations Tushar!!
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > > > On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni <
> > pra...@datatorrent.com>
> > > > wrote:
> > > >
> > > > > A bit delayed but nevertheless important announcement, Apache Apex
> > PMC
> > > is
> > > > > pleased to announce Tushar Gosavi as a new PMC member.
> > > > >
> > > > > Tushar has been contributing to Apex from the beginning of the
> > project
> > > > and
> > > > > has been working on the codebase for over 3 years. He is among the
> > few
> > > > who
> > > > > have a wide breadth of contribution, including both core and
> malhar,
> > > from
> > > > > internal changes to user facing api, from input/output operators to
> > > > > components that support operators, and has a good overall
> > understanding
> > > > of
> > > > > the codebase and how it works.
> > > > >
> > > > > His salient contributions over the years are
> > > > >
> > > > >- Module support in Apex
> > > > >- Operator additions and improvements such as S3, File input and
> > > > output,
> > > > >partitionable unique count and dynamic partitioning improvements
> > > > >- Initial WAL implementation from which subsequent
> implementations
> > > > were
> > > > >derived for different use cases
> > > > >- Plugin support for Apex
> > > > >- Various bug fixes and improvements in both malhar and core
> that
> > > you
> > > > >can find in the JIRA
> > > > >- Participated in long-term project maintenance tasks such as
> > > > >refactoring operators and demos
> > > > >- Participated in important feature discussions
> > > > >- Reviewed and committed pull requests from contributors
> > > > >- Participated in conducting and teaching in an Apex workshop
> at a
> > > > >university and speaking at Apex conference organized by
> > DataTorrent
> > > > >
> > > > > Conference talks & Presentations
> > > > >
> > > > >1. Presentations at VIIT and PICT Pune
> > > > >2. http://www.apexbigdata.com/pune-platform-talk-9.html
> > > > >3. http://www.apexbigdata.com/pune-integration-talk-4.html
> > > > >4. Webinar on Smart Partitioning with Apex. (
> > > > >https://www.youtube.com/watch?v=HCATB1zlLE4
> > > > >)
> > > > >5. Presented about customer use case at Pune Meetup in 2016
> > > > >
> > > > > Pramod for the Apache Apex PMC.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > >
> >
>
>
>
> --
>
> *Chaitanya*
>
> Software Engineer
>
> E: chaita...@datatorrent.com | Twitter: @chaithu1403
>
> www.datatorrent.com  |  apex.apache.org
>

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Chaitanya Chebolu 
Congratulations Tushar!!!

On Fri, Nov 3, 2017 at 10:06 PM, Venkatesh Kottapalli <
venkat...@datatorrent.com> wrote:

> Congratulations Tushar!
>
> -Venky
>
> On Fri, Nov 3, 2017 at 9:24 AM Hitesh Kapoor 
> wrote:
>
> > Congratulations Tushar.
> >
> > Regards,
> > Hitesh Kapoor
> >
> > On 03-Nov-2017 9:53 PM, "Ashwin Chandra Putta"  >
> > wrote:
> >
> > > Congratulations Tushar!!
> > >
> > > Regards,
> > > Ashwin.
> > >
> > > On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni <
> pra...@datatorrent.com>
> > > wrote:
> > >
> > > > A bit delayed but nevertheless important announcement, Apache Apex
> PMC
> > is
> > > > pleased to announce Tushar Gosavi as a new PMC member.
> > > >
> > > > Tushar has been contributing to Apex from the beginning of the
> project
> > > and
> > > > has been working on the codebase for over 3 years. He is among the
> few
> > > who
> > > > have a wide breadth of contribution, including both core and malhar,
> > from
> > > > internal changes to user facing api, from input/output operators to
> > > > components that support operators, and has a good overall
> understanding
> > > of
> > > > the codebase and how it works.
> > > >
> > > > His salient contributions over the years are
> > > >
> > > >- Module support in Apex
> > > >- Operator additions and improvements such as S3, File input and
> > > output,
> > > >partitionable unique count and dynamic partitioning improvements
> > > >- Initial WAL implementation from which subsequent implementations
> > > were
> > > >derived for different use cases
> > > >- Plugin support for Apex
> > > >- Various bug fixes and improvements in both malhar and core that
> > you
> > > >can find in the JIRA
> > > >- Participated in long-term project maintenance tasks such as
> > > >refactoring operators and demos
> > > >- Participated in important feature discussions
> > > >- Reviewed and committed pull requests from contributors
> > > >- Participated in conducting and teaching in an Apex workshop at a
> > > >university and speaking at Apex conference organized by
> DataTorrent
> > > >
> > > > Conference talks & Presentations
> > > >
> > > >1. Presentations at VIIT and PICT Pune
> > > >2. http://www.apexbigdata.com/pune-platform-talk-9.html
> > > >3. http://www.apexbigdata.com/pune-integration-talk-4.html
> > > >4. Webinar on Smart Partitioning with Apex. (
> > > >https://www.youtube.com/watch?v=HCATB1zlLE4
> > > >)
> > > >5. Presented about customer use case at Pune Meetup in 2016
> > > >
> > > > Pramod for the Apache Apex PMC.
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashwin.
> > >
> >
>



-- 

*Chaitanya*

Software Engineer

E: chaita...@datatorrent.com | Twitter: @chaithu1403

www.datatorrent.com  |  apex.apache.org

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Chaitanya Chebolu 
Congrats Ananth!!

On Fri, Nov 3, 2017 at 9:49 PM, Bhupesh Chawda 
wrote:

> Congratulations Ananth!
>
> ~ Bhupesh
>
>
> ___
>
> Bhupesh Chawda
>
> E: bhup...@datatorrent.com | Twitter: @bhupeshsc
>
> www.datatorrent.com  |  apex.apache.org
>
>
>
> On Fri, Nov 3, 2017 at 9:37 PM, Tushar Gosavi 
> wrote:
>
> > Congratulations Ananth.
> >
> > - Tushar.
> >
> >
> > On Fri, Nov 3, 2017 at 9:28 PM, Ashwin Chandra Putta <
> > ashwinchand...@gmail.com> wrote:
> >
> > > Congratulations Ananth. Well deserved!
> > >
> > > On Fri, Nov 3, 2017 at 2:08 AM, Chinmay Kolhatkar <
> > chin...@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > Congrats Ananth!!
> > > >
> > > > On Fri, Nov 3, 2017 at 2:37 PM, Priyanka Gugale 
> > > wrote:
> > > >
> > > > > Congrats Ananth!!
> > > > >
> > > > > -Priyanka
> > > > >
> > > > > On Fri, Nov 3, 2017 at 2:20 PM, Thomas Weise 
> wrote:
> > > > >
> > > > > > The Project Management Committee (PMC) for Apache Apex is pleased
> > to
> > > > > > announce Ananth Gundabattula as new committer.
> > > > > >
> > > > > > Ananth has been contributing to the project for about a year.
> > > > Highlights:
> > > > > >
> > > > > > * Cassandra and Kudu operators with in-depth analysis/design work
> > > > > > * Good collaboration, adherence to contributor guidelines and
> > > ownership
> > > > > of
> > > > > > work
> > > > > > * Work beyond feature focus such as fixing pre-existing test
> issues
> > > > that
> > > > > > impact CI
> > > > > > * Presented at YOW Data and Dataworks Summit Australia
> > > > > > * Enthusiast, contributes on his own time
> > > > > >
> > > > > > Welcome, Ananth, and congratulations!
> > > > > > Thomas, for the Apache Apex PMC.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashwin.
> > >
> >
>



-- 

*Chaitanya*

Software Engineer

E: chaita...@datatorrent.com | Twitter: @chaithu1403

www.datatorrent.com  |  apex.apache.org

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Venkatesh Kottapalli 
Congratulations Tushar!

-Venky

On Fri, Nov 3, 2017 at 9:24 AM Hitesh Kapoor  wrote:

> Congratulations Tushar.
>
> Regards,
> Hitesh Kapoor
>
> On 03-Nov-2017 9:53 PM, "Ashwin Chandra Putta" 
> wrote:
>
> > Congratulations Tushar!!
> >
> > Regards,
> > Ashwin.
> >
> > On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni 
> > wrote:
> >
> > > A bit delayed but nevertheless important announcement, Apache Apex PMC
> is
> > > pleased to announce Tushar Gosavi as a new PMC member.
> > >
> > > Tushar has been contributing to Apex from the beginning of the project
> > and
> > > has been working on the codebase for over 3 years. He is among the few
> > who
> > > have a wide breadth of contribution, including both core and malhar,
> from
> > > internal changes to user facing api, from input/output operators to
> > > components that support operators, and has a good overall understanding
> > of
> > > the codebase and how it works.
> > >
> > > His salient contributions over the years are
> > >
> > >- Module support in Apex
> > >- Operator additions and improvements such as S3, File input and
> > output,
> > >partitionable unique count and dynamic partitioning improvements
> > >- Initial WAL implementation from which subsequent implementations
> > were
> > >derived for different use cases
> > >- Plugin support for Apex
> > >- Various bug fixes and improvements in both malhar and core that
> you
> > >can find in the JIRA
> > >- Participated in long-term project maintenance tasks such as
> > >refactoring operators and demos
> > >- Participated in important feature discussions
> > >- Reviewed and committed pull requests from contributors
> > >- Participated in conducting and teaching in an Apex workshop at a
> > >university and speaking at Apex conference organized by DataTorrent
> > >
> > > Conference talks & Presentations
> > >
> > >1. Presentations at VIIT and PICT Pune
> > >2. http://www.apexbigdata.com/pune-platform-talk-9.html
> > >3. http://www.apexbigdata.com/pune-integration-talk-4.html
> > >4. Webinar on Smart Partitioning with Apex. (
> > >https://www.youtube.com/watch?v=HCATB1zlLE4
> > >)
> > >5. Presented about customer use case at Pune Meetup in 2016
> > >
> > > Pramod for the Apache Apex PMC.
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
>

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Bhupesh Chawda 
Congratulations Tushar!

~ Bhupesh


___

Bhupesh Chawda

E: bhup...@datatorrent.com | Twitter: @bhupeshsc

www.datatorrent.com  |  apex.apache.org



On Fri, Nov 3, 2017 at 9:54 PM, Hitesh Kapoor 
wrote:

> Congratulations Tushar.
>
> Regards,
> Hitesh Kapoor
>
> On 03-Nov-2017 9:53 PM, "Ashwin Chandra Putta" 
> wrote:
>
> > Congratulations Tushar!!
> >
> > Regards,
> > Ashwin.
> >
> > On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni 
> > wrote:
> >
> > > A bit delayed but nevertheless important announcement, Apache Apex PMC
> is
> > > pleased to announce Tushar Gosavi as a new PMC member.
> > >
> > > Tushar has been contributing to Apex from the beginning of the project
> > and
> > > has been working on the codebase for over 3 years. He is among the few
> > who
> > > have a wide breadth of contribution, including both core and malhar,
> from
> > > internal changes to user facing api, from input/output operators to
> > > components that support operators, and has a good overall understanding
> > of
> > > the codebase and how it works.
> > >
> > > His salient contributions over the years are
> > >
> > >- Module support in Apex
> > >- Operator additions and improvements such as S3, File input and
> > output,
> > >partitionable unique count and dynamic partitioning improvements
> > >- Initial WAL implementation from which subsequent implementations
> > were
> > >derived for different use cases
> > >- Plugin support for Apex
> > >- Various bug fixes and improvements in both malhar and core that
> you
> > >can find in the JIRA
> > >- Participated in long-term project maintenance tasks such as
> > >refactoring operators and demos
> > >- Participated in important feature discussions
> > >- Reviewed and committed pull requests from contributors
> > >- Participated in conducting and teaching in an Apex workshop at a
> > >university and speaking at Apex conference organized by DataTorrent
> > >
> > > Conference talks & Presentations
> > >
> > >1. Presentations at VIIT and PICT Pune
> > >2. http://www.apexbigdata.com/pune-platform-talk-9.html
> > >3. http://www.apexbigdata.com/pune-integration-talk-4.html
> > >4. Webinar on Smart Partitioning with Apex. (
> > >https://www.youtube.com/watch?v=HCATB1zlLE4
> > >)
> > >5. Presented about customer use case at Pune Meetup in 2016
> > >
> > > Pramod for the Apache Apex PMC.
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
>

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Ashwin Chandra Putta 
Congratulations Tushar!!

Regards,
Ashwin.

On Fri, Nov 3, 2017 at 9:17 AM, Pramod Immaneni 
wrote:

> A bit delayed but nevertheless important announcement, Apache Apex PMC is
> pleased to announce Tushar Gosavi as a new PMC member.
>
> Tushar has been contributing to Apex from the beginning of the project and
> has been working on the codebase for over 3 years. He is among the few who
> have a wide breadth of contribution, including both core and malhar, from
> internal changes to user facing api, from input/output operators to
> components that support operators, and has a good overall understanding of
> the codebase and how it works.
>
> His salient contributions over the years are
>
>- Module support in Apex
>- Operator additions and improvements such as S3, File input and output,
>partitionable unique count and dynamic partitioning improvements
>- Initial WAL implementation from which subsequent implementations were
>derived for different use cases
>- Plugin support for Apex
>- Various bug fixes and improvements in both malhar and core that you
>can find in the JIRA
>- Participated in long-term project maintenance tasks such as
>refactoring operators and demos
>- Participated in important feature discussions
>- Reviewed and committed pull requests from contributors
>- Participated in conducting and teaching in an Apex workshop at a
>university and speaking at Apex conference organized by DataTorrent
>
> Conference talks & Presentations
>
>1. Presentations at VIIT and PICT Pune
>2. http://www.apexbigdata.com/pune-platform-talk-9.html
>3. http://www.apexbigdata.com/pune-integration-talk-4.html
>4. Webinar on Smart Partitioning with Apex. (
>https://www.youtube.com/watch?v=HCATB1zlLE4
>)
>5. Presented about customer use case at Pune Meetup in 2016
>
> Pramod for the Apache Apex PMC.
>



-- 

Regards,
Ashwin.

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Bhupesh Chawda 
Congratulations Ananth!

~ Bhupesh


___

Bhupesh Chawda

E: bhup...@datatorrent.com | Twitter: @bhupeshsc

www.datatorrent.com  |  apex.apache.org



On Fri, Nov 3, 2017 at 9:37 PM, Tushar Gosavi 
wrote:

> Congratulations Ananth.
>
> - Tushar.
>
>
> On Fri, Nov 3, 2017 at 9:28 PM, Ashwin Chandra Putta <
> ashwinchand...@gmail.com> wrote:
>
> > Congratulations Ananth. Well deserved!
> >
> > On Fri, Nov 3, 2017 at 2:08 AM, Chinmay Kolhatkar <
> chin...@datatorrent.com
> > >
> > wrote:
> >
> > > Congrats Ananth!!
> > >
> > > On Fri, Nov 3, 2017 at 2:37 PM, Priyanka Gugale 
> > wrote:
> > >
> > > > Congrats Ananth!!
> > > >
> > > > -Priyanka
> > > >
> > > > On Fri, Nov 3, 2017 at 2:20 PM, Thomas Weise  wrote:
> > > >
> > > > > The Project Management Committee (PMC) for Apache Apex is pleased
> to
> > > > > announce Ananth Gundabattula as new committer.
> > > > >
> > > > > Ananth has been contributing to the project for about a year.
> > > Highlights:
> > > > >
> > > > > * Cassandra and Kudu operators with in-depth analysis/design work
> > > > > * Good collaboration, adherence to contributor guidelines and
> > ownership
> > > > of
> > > > > work
> > > > > * Work beyond feature focus such as fixing pre-existing test issues
> > > that
> > > > > impact CI
> > > > > * Presented at YOW Data and Dataworks Summit Australia
> > > > > * Enthusiast, contributes on his own time
> > > > >
> > > > > Welcome, Ananth, and congratulations!
> > > > > Thomas, for the Apache Apex PMC.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
>

[ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

2017-11-03 Thread Pramod Immaneni 
A bit delayed but nevertheless important announcement, Apache Apex PMC is
pleased to announce Tushar Gosavi as a new PMC member.

Tushar has been contributing to Apex from the beginning of the project and
has been working on the codebase for over 3 years. He is among the few who
have a wide breadth of contribution, including both core and malhar, from
internal changes to user facing api, from input/output operators to
components that support operators, and has a good overall understanding of
the codebase and how it works.

His salient contributions over the years are

   - Module support in Apex
   - Operator additions and improvements such as S3, File input and output,
   partitionable unique count and dynamic partitioning improvements
   - Initial WAL implementation from which subsequent implementations were
   derived for different use cases
   - Plugin support for Apex
   - Various bug fixes and improvements in both malhar and core that you
   can find in the JIRA
   - Participated in long-term project maintenance tasks such as
   refactoring operators and demos
   - Participated in important feature discussions
   - Reviewed and committed pull requests from contributors
   - Participated in conducting and teaching in an Apex workshop at a
   university and speaking at Apex conference organized by DataTorrent

Conference talks & Presentations

   1. Presentations at VIIT and PICT Pune
   2. http://www.apexbigdata.com/pune-platform-talk-9.html
   3. http://www.apexbigdata.com/pune-integration-talk-4.html
   4. Webinar on Smart Partitioning with Apex. (
   https://www.youtube.com/watch?v=HCATB1zlLE4
   )
   5. Presented about customer use case at Pune Meetup in 2016

Pramod for the Apache Apex PMC.

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Tushar Gosavi 
Congratulations Ananth.

- Tushar.


On Fri, Nov 3, 2017 at 9:28 PM, Ashwin Chandra Putta <
ashwinchand...@gmail.com> wrote:

> Congratulations Ananth. Well deserved!
>
> On Fri, Nov 3, 2017 at 2:08 AM, Chinmay Kolhatkar  >
> wrote:
>
> > Congrats Ananth!!
> >
> > On Fri, Nov 3, 2017 at 2:37 PM, Priyanka Gugale 
> wrote:
> >
> > > Congrats Ananth!!
> > >
> > > -Priyanka
> > >
> > > On Fri, Nov 3, 2017 at 2:20 PM, Thomas Weise  wrote:
> > >
> > > > The Project Management Committee (PMC) for Apache Apex is pleased to
> > > > announce Ananth Gundabattula as new committer.
> > > >
> > > > Ananth has been contributing to the project for about a year.
> > Highlights:
> > > >
> > > > * Cassandra and Kudu operators with in-depth analysis/design work
> > > > * Good collaboration, adherence to contributor guidelines and
> ownership
> > > of
> > > > work
> > > > * Work beyond feature focus such as fixing pre-existing test issues
> > that
> > > > impact CI
> > > > * Presented at YOW Data and Dataworks Summit Australia
> > > > * Enthusiast, contributes on his own time
> > > >
> > > > Welcome, Ananth, and congratulations!
> > > > Thomas, for the Apache Apex PMC.
> > > >
> > >
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Ashwin Chandra Putta 
Congratulations Ananth. Well deserved!

On Fri, Nov 3, 2017 at 2:08 AM, Chinmay Kolhatkar 
wrote:

> Congrats Ananth!!
>
> On Fri, Nov 3, 2017 at 2:37 PM, Priyanka Gugale  wrote:
>
> > Congrats Ananth!!
> >
> > -Priyanka
> >
> > On Fri, Nov 3, 2017 at 2:20 PM, Thomas Weise  wrote:
> >
> > > The Project Management Committee (PMC) for Apache Apex is pleased to
> > > announce Ananth Gundabattula as new committer.
> > >
> > > Ananth has been contributing to the project for about a year.
> Highlights:
> > >
> > > * Cassandra and Kudu operators with in-depth analysis/design work
> > > * Good collaboration, adherence to contributor guidelines and ownership
> > of
> > > work
> > > * Work beyond feature focus such as fixing pre-existing test issues
> that
> > > impact CI
> > > * Presented at YOW Data and Dataworks Summit Australia
> > > * Enthusiast, contributes on his own time
> > >
> > > Welcome, Ananth, and congratulations!
> > > Thomas, for the Apache Apex PMC.
> > >
> >
>



-- 

Regards,
Ashwin.

Re: Malhar release 3.8.0

2017-11-03 Thread Justin Mclean 
Hi,

> The JIRA mentioned is to unblock the release. IMO the license plugin should
> be taken up as separate activity.

JFYI - In my experience the license plugin is not going to catch everything and 
tit may produce false positives. The best way is to manually look at the 
release artefact and it’s dependancies.

Thanks,
Justin

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Chinmay Kolhatkar 
Congrats Ananth!!

On Fri, Nov 3, 2017 at 2:37 PM, Priyanka Gugale  wrote:

> Congrats Ananth!!
>
> -Priyanka
>
> On Fri, Nov 3, 2017 at 2:20 PM, Thomas Weise  wrote:
>
> > The Project Management Committee (PMC) for Apache Apex is pleased to
> > announce Ananth Gundabattula as new committer.
> >
> > Ananth has been contributing to the project for about a year. Highlights:
> >
> > * Cassandra and Kudu operators with in-depth analysis/design work
> > * Good collaboration, adherence to contributor guidelines and ownership
> of
> > work
> > * Work beyond feature focus such as fixing pre-existing test issues that
> > impact CI
> > * Presented at YOW Data and Dataworks Summit Australia
> > * Enthusiast, contributes on his own time
> >
> > Welcome, Ananth, and congratulations!
> > Thomas, for the Apache Apex PMC.
> >
>

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Priyanka Gugale 
Congrats Ananth!!

-Priyanka

On Fri, Nov 3, 2017 at 2:20 PM, Thomas Weise  wrote:

> The Project Management Committee (PMC) for Apache Apex is pleased to
> announce Ananth Gundabattula as new committer.
>
> Ananth has been contributing to the project for about a year. Highlights:
>
> * Cassandra and Kudu operators with in-depth analysis/design work
> * Good collaboration, adherence to contributor guidelines and ownership of
> work
> * Work beyond feature focus such as fixing pre-existing test issues that
> impact CI
> * Presented at YOW Data and Dataworks Summit Australia
> * Enthusiast, contributes on his own time
>
> Welcome, Ananth, and congratulations!
> Thomas, for the Apache Apex PMC.
>

[ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

2017-11-03 Thread Thomas Weise 
The Project Management Committee (PMC) for Apache Apex is pleased to
announce Ananth Gundabattula as new committer.

Ananth has been contributing to the project for about a year. Highlights:

* Cassandra and Kudu operators with in-depth analysis/design work
* Good collaboration, adherence to contributor guidelines and ownership of
work
* Work beyond feature focus such as fixing pre-existing test issues that
impact CI
* Presented at YOW Data and Dataworks Summit Australia
* Enthusiast, contributes on his own time

Welcome, Ananth, and congratulations!
Thomas, for the Apache Apex PMC.

[jira] [Commented] (APEXMALHAR-2549) JdbcIOAppTest.testApplication is flaky

2017-11-03 Thread Thomas Weise (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237217#comment-16237217
 ] 

Thomas Weise commented on APEXMALHAR-2549:
--

More frequent flakes:

com.datatorrent.lib.db.jdbc.JdbcIOAppTest.testApplication
com.datatorrent.lib.db.jdbc.JdbcInputOperatorApplicationTest.testJdbcPOJOInputOperatorApplication
com.datatorrent.lib.db.jdbc.JdbcInputOperatorApplicationTest.testJdbcPOJOPollInputOperatorApplication
com.datatorrent.lib.db.jdbc.JdbcPojoOperatorApplicationTest.testApplication
com.datatorrent.lib.parser.XmlParserApplicationTest.testApplication
com.datatorrent.lib.transform.TransformOperatorAppTest.testExpressionApplication


> JdbcIOAppTest.testApplication is flaky
> --
>
> Key: APEXMALHAR-2549
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2549
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Priority: Major
>
> One of the tests that from time to time shows up as CI failure:
> java.lang.AssertionError: Events in store expected:<10> but was:<0>
>   at 
> com.datatorrent.lib.db.jdbc.JdbcIOAppTest.testApplication(JdbcIOAppTest.java:127)
> https://builds.apache.org/job/Apex_Malhar_PR/org.apache.apex$malhar-library/1726/testReport/junit/com.datatorrent.lib.db.jdbc/JdbcIOAppTest/testApplication/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (APEXMALHAR-2549) JdbcIOAppTest.testApplication is flaky

2017-11-03 Thread Thomas Weise (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237203#comment-16237203
 ] 

Thomas Weise commented on APEXMALHAR-2549:
--

It is no surprise that this test fails randomly:

{code}
try {
  LocalMode lma = LocalMode.newInstance();
  Configuration conf = new Configuration(false);
  lma.prepareDAG(new JdbcIOApp(), conf);
  LocalMode.Controller lc = lma.getController();
  lc.runAsync();
  // wait for records to be added to table
  Thread.sleep(3000);
  lc.shutdown();
  Assert.assertEquals("Events in store", 10, getNumOfEventsInStore());
} catch (ConstraintViolationException e) {
  Assert.fail("constraint violations: " + e.getConstraintViolations());
}
{code}

> JdbcIOAppTest.testApplication is flaky
> --
>
> Key: APEXMALHAR-2549
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2549
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Thomas Weise
>Priority: Major
>
> One of the tests that from time to time shows up as CI failure:
> java.lang.AssertionError: Events in store expected:<10> but was:<0>
>   at 
> com.datatorrent.lib.db.jdbc.JdbcIOAppTest.testApplication(JdbcIOAppTest.java:127)
> https://builds.apache.org/job/Apex_Malhar_PR/org.apache.apex$malhar-library/1726/testReport/junit/com.datatorrent.lib.db.jdbc/JdbcIOAppTest/testApplication/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (APEXMALHAR-2548) Pubsub operators do not use the correct URL to connect to web socket server in SSL mode

2017-11-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237171#comment-16237171
 ] 

ASF GitHub Bot commented on APEXMALHAR-2548:


tweise closed pull request #676: APEXMALHAR-2548 Using the correct websocket 
scheme when connecting to a SSL cluster
URL: https://github.com/apache/apex-malhar/pull/676
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java 
b/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java
index 98dfebd7b8..82c9214fa9 100644
--- 
a/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java
+++ 
b/apps/logstream/src/main/java/com/datatorrent/apps/logstream/Application.java
@@ -18,19 +18,24 @@
  */
 package com.datatorrent.apps.logstream;
 
-import java.net.URI;
-import java.util.*;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Set;
 
 import org.apache.apex.malhar.contrib.misc.streamquery.SelectOperator;
 import 
org.apache.apex.malhar.contrib.misc.streamquery.condition.EqualValueCondition;
-
-import org.apache.commons.lang.StringUtils;
+import org.apache.apex.malhar.lib.utils.PubSubHelper;
 import org.apache.hadoop.conf.Configuration;
 
+import com.datatorrent.api.DAG;
+import com.datatorrent.api.Operator.InputPort;
+import com.datatorrent.api.StreamingApplication;
+import com.datatorrent.contrib.redis.RedisKeyValPairOutputOperator;
+import com.datatorrent.contrib.redis.RedisMapOutputOperator;
+import com.datatorrent.contrib.redis.RedisNumberSummationMapOutputOperator;
 import com.datatorrent.lib.algo.TopN;
 import com.datatorrent.lib.io.ConsoleOutputOperator;
 import com.datatorrent.lib.io.PubSubWebSocketOutputOperator;
@@ -43,13 +48,6 @@
 import com.datatorrent.lib.util.AbstractDimensionTimeBucketOperator;
 import com.datatorrent.lib.util.DimensionTimeBucketSumOperator;
 
-import com.datatorrent.api.DAG;
-import com.datatorrent.api.Operator.InputPort;
-import com.datatorrent.api.StreamingApplication;
-import com.datatorrent.contrib.redis.RedisKeyValPairOutputOperator;
-import com.datatorrent.contrib.redis.RedisMapOutputOperator;
-import com.datatorrent.contrib.redis.RedisNumberSummationMapOutputOperator;
-
 /**
  * Log stream processing application based on Apex platform.
  * This application consumes log data generated by running systems and services
@@ -156,14 +154,12 @@ private SYSTEM_KEYS(String value)
 
   private InputPort wsOutput(DAG dag, String operatorName)
   {
-String daemonAddress = dag.getValue(DAG.GATEWAY_CONNECT_ADDRESS);
-if (!StringUtils.isEmpty(daemonAddress)) {
-  URI uri = URI.create("ws://" + daemonAddress + "/pubsub");
+if (PubSubHelper.isGatewayConfigured(dag)) {
   String appId = "appid";
   //appId = dag.attrValue(DAG.APPLICATION_ID, null); // will be used once 
UI is able to pick applications from list and listen to corresponding 
application
   String topic = "apps.logstream." + appId + "." + operatorName;
   PubSubWebSocketOutputOperator wsOut = 
dag.addOperator(operatorName, new PubSubWebSocketOutputOperator());
-  wsOut.setUri(uri);
+  wsOut.setUri(PubSubHelper.getURI(dag));
   wsOut.setTopic(topic);
   return wsOut.input;
 }
diff --git 
a/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
 
b/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
index 73c38ef6cb..eed9a683e7 100644
--- 
a/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
+++ 
b/examples/frauddetect/src/main/java/org/apache/apex/examples/frauddetect/Application.java
@@ -20,11 +20,13 @@
 
 import java.io.Serializable;
 import java.net.URI;
+
 import org.apache.apex.examples.frauddetect.operator.HdfsStringOutputOperator;
 import org.apache.apex.examples.frauddetect.operator.MongoDBOutputOperator;
+import org.apache.apex.malhar.lib.utils.PubSubHelper;
 import org.apache.hadoop.conf.Configuration;
+
 import com.datatorrent.api.Context;
-import com.datatorrent.api.Context.DAGContext;
 import com.datatorrent.api.DAG;
 import com.datatorrent.api.StreamingApplication;
 import com.datatorrent.api.annotation.ApplicationAnnotation;
@@ -88,11 +90,7 @@ public void populateDAG(DAG dag, Configuration conf)
   {
 
 try {
-  String gatewayAddress = dag.getValue(DAGContext.GATEWAY_CONNECT_ADDRESS);
-  if (gatewayAddress == null) {
-gatewayAddress = "localhost:9090";
-  }
-  URI duri = URI.create("ws://" + gatewayAddress + "/pubsub");
+  URI duri

[jira] [Commented] (APEXMALHAR-2548) Pubsub operators do not use the correct URL to connect to web socket server in SSL mode

2017-11-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237172#comment-16237172
 ] 

ASF GitHub Bot commented on APEXMALHAR-2548:


PramodSSImmaneni opened a new pull request #676: APEXMALHAR-2548 Using the 
correct websocket scheme when connecting to a SSL cluster
URL: https://github.com/apache/apex-malhar/pull/676
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Pubsub operators do not use the correct URL to connect to web socket server 
> in SSL mode
> ---
>
> Key: APEXMALHAR-2548
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2548
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Pramod Immaneni
>Assignee: Pramod Immaneni
>Priority: Major
>
> In SSL mode, pub sub operators need to use wss protocol instead of ws. Today 
> they use ws.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: checking dependencies for known vulnerabilities

2017-11-03 Thread Thomas Weise 
After the PR CI build fails it should be straightforward to see if the
dependency was part of the PR or not.


On Thu, Nov 2, 2017 at 11:50 PM, Pramod Immaneni 
wrote:

> Check is fine as long as we can identify whether the CVE was introduced by
> a PR. It might still be useful to fix a CVE even if there is no release,
> downstream projects might be able to use it.
>
> On Thu, Nov 2, 2017 at 4:42 PM, Thomas Weise  wrote:
>
> > Check as part of PR build is needed to ensure no new issues are
> introduced.
> > I don't think it is necessary to have another build to notice
> pre-existing
> > issues since releases won't occur without prior PRs.
> >
> > Whitelisting suggestion is for pre-existing problems and not for those
> > introduced by a PR.
> >
> >
> > On Thu, Nov 2, 2017 at 11:35 PM, Pramod Immaneni  >
> > wrote:
> >
> > > If the PR introduces a CVE by all means lets fail it initally and later
> > > look at whitelisting it if needed. Why fail all PRs that haven't caused
> > the
> > > problem. What happens when there are no PRs in a given period, wouldn't
> > it
> > > be better to know above crtiical CVEs (that this proposal is trying to
> > > address) sooner than wait till some random PR is raised. What if there
> > are
> > > no PRs for a month. I think it makes sense to have a separate build
> that
> > > will catch these errors.
> > >
> > > On Thu, Nov 2, 2017 at 3:17 PM, Ananth G 
> wrote:
> > >
> > > > Only the first PR should be broken if cve with a higher score than
> > agreed
> > > > is introduced .Once you whitelist subsequent builds will not fail?
> > >
> > >
> > > > - The dependency check plugin is enabled to fail/break the build if a
> > > > violation is detected
> > > > - A PR introducing a risk will either “acknowledge” the cve by
> creating
> > > > the whitelist entry as part of the process or will break the build
> > > > - The reviewer/s would notice either the changes to the whitelist
> file
> > or
> > > > the build break
> > > > - The review process would agree upon the approach and ensure JIRA
> > ticket
> > > > creation or question why an alternative cannot be used
> > > > - whitelist would be maintained in a suppressions file example given
> > here
> > > > https://jeremylong.github.io/DependencyCheck/general/
> suppression.html
> > > > - subsequent PR update should no longer break the build
> > > > - there is a list of JIRA items for cve which needs to be addressed
> in
> > > the
> > > > subsequent releases as well as a set of artefacts in the project
> > modules
> > > > that summarise the community’s awareness of the issue.
> > > >
> > > >
> > > > Regards
> > > > Ananth
> > > >
> > > > > On 3 Nov 2017, at 8:38 am, Pramod Immaneni  >
> > > > wrote:
> > > > >
> > > > > The question is which build? We can definitely make it a mandatory
> > > > release
> > > > > step and also have nightly builds that are meant for just these,
> > > > detection
> > > > > of CVEs and even possible infra changes that might break tests.
> Those
> > > > will
> > > > > work even if there are no PRs.
> > > > >
> > > > >> On Thu, Nov 2, 2017 at 1:21 PM, Ananth G 
> > > > wrote:
> > > > >>
> > > > >> My vote would be to break the build. This can then force a
> > > > “whitelisting”
> > > > >> configuration in the maven plugin to be created as part of the
> > review
> > > > >> process ( along with a new JIRA ticket ).
> > > > >>
> > > > >> The concern would then be to ensure that the community works
> > towards a
> > > > >> resolution of the JIRA. Not breaking the build for me is tech debt
> > > > slipping
> > > > >> without anyones notice.  Fixing the JIRA is a hygiene process
> which
> > I
> > > > >> believe cannot take a back burner as compared contributor welfare
> > and
> > > > would
> > > > >> need commitments from the contributor ( and/or others).
> > > > >>
> > > > >> On a related note, looking at apache spark, there seems to be CVE
> > > > listings
> > > > >> which the spark community has taken care of as the releases
> > > progressed.
> > > > >> http://www.cvedetails.com/vulnerability-list/vendor_id-
> > > > >> 45/product_id-38954/Apache-Spark.html  > > > >> vulnerability-list/vendor_id-45/product_id-38954/Apache-
> Spark.html>
> > .
> > > > >>
> > > > >> Regards,
> > > > >> Ananth
> > > > >>
> > > > >>
> > > > >>> On 3 Nov 2017, at 4:48 am, Sanjay Pujare  >
> > > > wrote:
> > > > >>>
> > > > >>> I like this suggestion. Blocking the PR is too drastic. I also
> > second
> > > > >>> Pramod's point (made elsewhere) that we should try to encourage
> > > > >>> contribution instead of discouraging it by resorting to drastic
> > > > measures.
> > > > >>> If you institute drastic measures to achieve a desired effect
> (e.g.
> > > > >> getting
> > > > >>> contributors to look into CVEs and build infrastructure issues)
> it
> >