[GitHub] storm issue #2689: STORM-3061: rocket, jms, and mqtt updates

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2689
  
Hi @revans2 Now I'm on vacation. I will add a usage document for RocketMQ 
example after I get back next week.


---

[GitHub] storm issue #2518: STORM-2902: Some improvements for storm-rocketmq module

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2518
  
Also rebased PR.


---

[GitHub] storm issue #2518: STORM-2902: Some improvements for storm-rocketmq module

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2518
  
@hustfxj Added logs and fixed warnings for RocketMqSpout/RocketMqBolt.


---

[GitHub] storm pull request #2518: STORM-2902: Some improvements for storm-rocketmq m...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2518#discussion_r180281409
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMqSpout.java
 ---
@@ -60,14 +58,14 @@
 public class RocketMqSpout implements IRichSpout {
 // TODO add metrics
--- End diff --

Thanks. Will update.


---

[GitHub] storm pull request #2518: STORM-2902: Some improvements for storm-rocketmq m...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2518#discussion_r178704108
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/RocketMqConfig.java
 ---
@@ -23,28 +23,20 @@
 import java.util.Properties;
 import java.util.UUID;
 
+import org.apache.commons.lang.StringUtils;
 import org.apache.commons.lang.Validate;
 import org.apache.rocketmq.client.ClientConfig;
 import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
 import org.apache.rocketmq.client.exception.MQClientException;
 import org.apache.rocketmq.client.producer.DefaultMQProducer;
 import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
-import org.apache.rocketmq.remoting.common.RemotingUtil;
 
 /**
  * RocketMqConfig for Consumer/Producer.
  */
--- End diff --

@vongosling This is limited by the storm checkstyle rule 
`AbbreviationAsWordInName`. Refer to 
https://github.com/apache/storm/blob/master/storm-checkstyle/src/main/resources/storm/storm_checkstyle.xml#L213


---

[GitHub] storm issue #2518: STORM-2902: Some improvements for storm-rocketmq module

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2518
  
@vongosling Added unit tests and rebased PR.


---

[GitHub] storm issue #2518: STORM-2902: Some improvements for storm-rocketmq module

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2518
  
@vongosling Will add more unit tests later.


---

[GitHub] storm issue #2518: STORM-2902: Some improvements for storm-rocketmq module

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2518
  
Any comments are welcome..


---

[GitHub] storm issue #2518: STORM-2902: Some improvements for storm-rocketmq module

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2518
  
@harshach @vongosling @hustfxj Do yo have time to take a look?


---

[GitHub] storm pull request #2518: STORM-2902: Some improvements for storm-rocketmq m...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2518

STORM-2902: Some improvements for storm-rocketmq module

## Brief changelog
* Upgraded RocketMQ version to 4.2.0 which brings improvements and new 
features like batch sending
* Imporved retry policy for RocketMQ consumer push mode to avoid data loss 
in some scenes
* Batch sending supported for bolt and trident state
* Allow running several consumer instances in one process, that is to say, 
different topics in one worker is possible

## Verifying this change
local tests check passed ✅
apache-rat check passed ✅
checkstyle check passed ✅


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm rocketmq-improvements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2518


commit d3232136655647f1968b2b8b0ba22f84f7d1f96c
Author: Xin Wang <xinwang@...>
Date:   2017-12-24T13:27:40Z

STORM-2902: Some improvements for storm-rocketmq module




---

[GitHub] storm issue #2446: STORM-2843: [Flux] properties file not found when loading...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2446
  
@HeartSaVioR Created new PR for this.


---

[GitHub] storm issue #2445: STORM-2843: [Flux] properties file not found when loading...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2445
  
@HeartSaVioR Created new PR for this.


---

[GitHub] storm pull request #2479: STORM-2843: [Flux] [1.x-branch]properties file not...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2479

STORM-2843: [Flux] [1.x-branch]properties file not found when loading 
resources f…

…rom classpath

STORM-2843 for 1.x-branch

https://issues.apache.org/jira/browse/STORM-2843

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2843-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2479


commit febb31452163c361e3cc8e8bf63403662d74012a
Author: Xin Wang <xinwang@...>
Date:   2017-12-24T16:08:06Z

STORM-2843: [Flux] properties file not found when loading resources from 
classpath




---

[GitHub] storm pull request #2446: STORM-2843: [Flux] properties file not found when ...

[vesense]

Github user vesense closed the pull request at:

https://github.com/apache/storm/pull/2446


---

[GitHub] storm pull request #2478: STORM-2843: [Flux] properties file not found when ...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2478

STORM-2843: [Flux] properties file not found when loading resources f…

…rom classpath

STORM-2843 for master branch

https://issues.apache.org/jira/browse/STORM-2843

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2843

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2478.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2478


commit 6482b2a140e8cc19ea2357822543324a1c570d38
Author: Xin Wang <xinwang@...>
Date:   2017-12-24T15:06:15Z

STORM-2843: [Flux] properties file not found when loading resources from 
classpath




---

[GitHub] storm pull request #2445: STORM-2843: [Flux] properties file not found when ...

[vesense]

Github user vesense closed the pull request at:

https://github.com/apache/storm/pull/2445


---

[GitHub] storm issue #2445: STORM-2843: [Flux] properties file not found when loading...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2445
  
@HeartSaVioR PR updated. Could you take a look again?


---

[GitHub] storm issue #2469: STORM-2861: Explicit reference kafka-schema-registry-clie...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2469
  
@ptgoetz @satishd 
Yes, I know that kafka-avro-serializer depends on 
kafka-schema-registry-client and avro.
There were io.confluent.kafka.schemaregistry.client.XXX not found errors 
when compiling the storm-hdfs code, and this PR fixed the issue. (To tell the 
truth, it's a bit strange.)
Another benefit is that users can know clearly about which avro version 
they are using. By default the avro module is from hadoop-common dependencies.


---

[GitHub] storm pull request #2469: STORM-2861: Explicit reference kafka-schema-regist...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2469

STORM-2861: Explicit reference kafka-schema-registry-client

https://issues.apache.org/jira/browse/STORM-2861

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2861

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2469.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2469


commit f0bd79585c74fa9e63d427ecd90830605e18c46b
Author: Xin Wang <best.wang...@163.com>
Date:   2017-12-19T12:00:32Z

STORM-2861: Explicit reference kafka-schema-registry-client

https://issues.apache.org/jira/browse/STORM-2861




---

[GitHub] storm pull request #2445: STORM-2843: [Flux] properties file not found when ...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2445#discussion_r157729717
  
--- Diff: 
flux/flux-core/src/main/java/org/apache/storm/flux/parser/FluxParser.java ---
@@ -39,51 +39,57 @@
 
 private FluxParser(){}
 
-// TODO refactor input stream processing (see parseResource() method).
 public static TopologyDef parseFile(String inputFile, boolean 
dumpYaml, boolean processIncludes,
String propertiesFile, boolean envSub) throws IOException {
-   
+
 FileInputStream in = new FileInputStream(inputFile);
-TopologyDef topology = parseInputStream(in, dumpYaml, 
processIncludes, propertiesFile, envSub);
+InputStream propertiesIn = null;
+if (propertiesFile != null) {
+propertiesIn = new FileInputStream(propertiesFile);
+}
+TopologyDef topology = parseInputStream(in, dumpYaml, 
processIncludes, propertiesIn, propertiesFile, envSub);
 in.close();
-
+
 return topology;
 }
 
 public static TopologyDef parseResource(String resource, boolean 
dumpYaml, boolean processIncludes,
String propertiesFile, boolean envSub) throws IOException {
 
 InputStream in = FluxParser.class.getResourceAsStream(resource);
-TopologyDef topology = parseInputStream(in, dumpYaml, 
processIncludes, propertiesFile, envSub);
+InputStream propertiesIn = null;
+if (propertiesFile != null) {
+propertiesIn = 
FluxParser.class.getResourceAsStream(propertiesFile);
+}
+TopologyDef topology = parseInputStream(in, dumpYaml, 
processIncludes, propertiesIn, propertiesFile, envSub);
--- End diff --

Make sense. Yaml file and included yaml file should share the same 
properties instance. PR for master and 1.x are updated. Please take a look 
again.


---

[GitHub] storm issue #2448: Quick fix: correcting markdown format

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2448
  
Thanks @Ethanlm Merged into master.


---

[GitHub] storm pull request #2446: STORM-2843: [Flux] properties file not found when ...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2446

STORM-2843: [Flux] properties file not found when loading resources...

…from classpath

STORM-2843 for 1.x-branch

https://issues.apache.org/jira/browse/STORM-2843

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2843-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2446.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2446


commit e2deed3d448ef956ba94ca7698349b0cf88d8d60
Author: Xin Wang <xinw...@apache.org>
Date:   2017-12-05T10:03:38Z

STORM-2843: properties file not found when loading resources from classpath




---

[GitHub] storm pull request #2445: STORM-2843: [Flux] properties file not found when ...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2445

STORM-2843: [Flux] properties file not found when loading resources...

…from classpath

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2843

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2445.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2445


commit c144f73e853db28dba7f6bdee7a9490100f019bd
Author: Xin Wang <xinw...@apache.org>
Date:   2017-12-05T09:45:36Z

STORM-2843: properties file not found when loading resources from classpath




---

[GitHub] storm pull request #2444: STORM-2842: Fixed links for YARN Integr...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2444

STORM-2842: Fixed links for YARN Integration

This is very minor change. Fixed links for YARN Integration.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2842

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2444


commit 6e2b2dc0f29a1db41d81820395c7382631d29494
Author: Xin Wang <best.wang...@163.com>
Date:   2017-12-05T08:01:15Z

STORM-2842: Fixed links for YARN Integration




---

[GitHub] storm issue #2377: [STORM-2780] MetricsConsumer record unnecessary timestamp

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2377
  
I prefer to Ieave it as it is, since the log time and the 
taskInfo.timestamp is not same. Perhaps someone uses it, I'm not sure.


---

[GitHub] storm issue #2234: STORM-2652: fix error in open method of JmsSpout

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2234
  
@omerhadari I assigned the Jira to you and added you to the Jira 
contributors.
@srdo I added you to the Jira committers/PMCs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2227: [STORM-2643] add confirmation before profiling/debugging ...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2227
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2220: STORM-2638: Add in maximum object size for Pacemaker Thri...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2220
  
Sorry for the delay. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2220: STORM-2638: Add in maximum object size for Pacemaker Thri...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2220
  
Looks good to me. It would be better if we can apply new code style to the 
changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2209: [STORM-2622] Add owner resource summary on storm UI

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2209
  
Looks great. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2214: STORM-2634: Apply new code style to storm-sql-runt...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2214

STORM-2634: Apply new code style to storm-sql-runtime

Reduce the max allowed violation count to 0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2634

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2214.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2214


commit 2a36ec7bbe88d61685886b9de291b7fcc027b899
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-15T08:10:39Z

STORM-2634: Apply new code style to storm-sql-runtime




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2213: STORM-2633: Apply new code style to storm-sql-redi...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2213

STORM-2633: Apply new code style to storm-sql-redis

Reduce the max allowed violation count to 0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2633

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2213.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2213


commit 2dbe80a508c9b17ecf60fdc9256412841f4c0819
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-15T03:59:55Z

STORM-2633: Apply new code style to storm-sql-redis




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2212: STORM-2632: Apply new code style to storm-sql-kafk...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2212

STORM-2632: Apply new code style to storm-sql-kafka

Reduce the max allowed violation count to 0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2632

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2212


commit c2d2bd2fc997bc80b03c61ab8ced7cac8110990f
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-15T03:52:10Z

STORM-2632: Apply new code style to storm-sql-kafka




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2211: STORM-2631: Apply new code style to storm-sql-mong...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2211

STORM-2631: Apply new code style to storm-sql-mongodb

Reduce the max allowed violation count to 0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2631

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2211.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2211


commit 38c91267278ecf40e14c5f2da9ddc53624c08c69
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-15T03:43:22Z

STORM-2631: Apply new code style to storm-sql-mongodb




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2210: STORM-2630: Apply new code style to storm-sql-hdfs

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2210

STORM-2630: Apply new code style to storm-sql-hdfs

Reduce the max allowed violation count to 0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2630

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2210.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2210


commit 9a7cb048556db04eab4d75629c0bded04e62d8e8
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-15T03:37:15Z

STORM-2630: Apply new code style to storm-sql-hdfs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2196: STORM-2619: Correct the JDK version in Setting-up-...

[vesense]

Github user vesense closed the pull request at:

https://github.com/apache/storm/pull/2196


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2196: STORM-2619: Correct the JDK version in Setting-up-a-Storm...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2196
  
OK. No problem. I will close this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2196: STORM-2619: Correct the JDK version in Setting-up-...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2196

STORM-2619: Correct the JDK version in Setting-up-a-Storm-cluster.md

The Storm 2.0 needs JDK 1.8+

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2619

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2196


commit c931e79644aeedc6260388a87cd5c521b2d51c2c
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-10T13:47:17Z

STORM-2619: Correct the JDK version in Setting-up-a-Storm-cluster.md




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2194: STORM-2594: Apply new code style to storm-rocketmq

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2194

STORM-2594: Apply new code style to storm-rocketmq

Reduce the max allowed violation count to 0.

Changed the class names including MQ to Mq. 
>Since storm-rocketmq is only merged into master branch and storm 2.0 
hasn't been released, so the changes should hurt nothing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2594

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2194.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2194


commit 1d7be760198189711165abac48acfd3bc332083a
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-09T13:02:30Z

STORM-2594: Apply new code style to storm-rocketmq




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2193: STORM-2589: Apply new code style to storm-mongodb

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2193

STORM-2589: Apply new code style to storm-mongodb

Reduce the max allowed violation count to 0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2589

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2193


commit 3d4c59e5d3f20cae205a82336b2cd59ff51fac84
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-09T10:06:22Z

STORM-2589: Apply new code style to storm-mongodb




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2192: STORM-2618: Add TridentKafkaStateUpdater for storm-kafka-...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2192
  
This can be cherry-picked to 1.x branch as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2192: STORM-2618: Add TridentKafkaStateUpdater for storm...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2192

STORM-2618: Add TridentKafkaStateUpdater for storm-kafka-client

* Added TridentKafkaStateUpdater, otherwise users have to create a class 
themselves.

usage scenario:
```
stream.partitionPersist(stateFactory, fields, new 
TridentKafkaStateUpdater(), new Fields());
```

* Fixed documents

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2618

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2192


commit 24dc055226d377f04e1b3116ceb5e5261133bd16
Author: Xin Wang <best.wang...@163.com>
Date:   2017-07-09T05:40:57Z

STORM-2618: Add TridentKafkaStateUpdater for storm-kafka-client




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2189: [STORM-2615] Add topology readonly user configuration

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2189
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2189: [STORM-2615] Add topology readonly user configurat...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2189#discussion_r126295391
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/security/auth/authorizer/SimpleACLAuthorizer.java
 ---
@@ -157,6 +162,25 @@ public boolean permit(ReqContext context, String 
operation, Map<String, Object>
 }
 
 if (checkUserGroupAllowed(userGroups, topoGroups)) return true;
+
+if (_topoReadOnlyCommands.contains(operation)){
+Set topoReadOnlyUsers = new HashSet();
--- End diff --

nit: -> `Set topoReadOnlyUsers = new HashSet<>();`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2191: Fix typo in storm.py file

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2191
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2183: STORM-2609: Simple command line DRPC Client

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2183
  
+1
We'd better add the new command to document. (Command-line-client.html)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2092: STORM-2493: update documents to reflect the changes

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2092
  
@harshach / @revans2 Can you take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2092: STORM-2493: update documents to reflect the change...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2092#discussion_r113874786
  
--- Diff: docs/storm-pmml.md ---
@@ -0,0 +1,37 @@
+#Storm PMML Bolt
--- End diff --

Nice catch. Will fix.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2092: STORM-2493: update documents to reflect the change...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2092

STORM-2493: update documents to reflect the changes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2493-documents

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2092.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2092


commit e80f9e20db208555753b93f48c4af175dbe47c6a
Author: vesense <best.wang...@163.com>
Date:   2017-04-27T04:40:57Z

STORM-2493: update documents to reflect the changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2091: fix typo in storm-client pom file: kyro -> kryo

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2091
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2082: expose Tuple for node.js

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2082
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2087: Fixed DemoTest.java link

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2087
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113431656
  
--- Diff: 
examples/storm-starter/src/jvm/org/apache/storm/starter/LambdaTopology.java ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.starter;
+
+import org.apache.storm.Config;
+import org.apache.storm.topology.ConfigurableTopology;
+import org.apache.storm.topology.TopologyBuilder;
+import org.apache.storm.tuple.Values;
+
+import java.io.Serializable;
+import java.util.UUID;
+
+public class LambdaTopology extends ConfigurableTopology {
+public static void main(String[] args) {
+ConfigurableTopology.start(new LambdaTopology(), args);
+}
+
+@Override
+protected int run(String[] args) throws Exception {
+TopologyBuilder builder = new TopologyBuilder();
+
+// example. spout1: generate random strings
+// bolt1: get the first part of a string
+// bolt2: output the tuple
+
+// NOTE: Variable used in lambda expression should be final or 
effectively final
+// (or it will cause compilation error),
+// and variable type should implement the Serializable interface 
if it isn't primitive type
+// (or it will cause not serializable exception).
+Prefix prefix = new Prefix("Hello lambda:");
--- End diff --

This is just an example to prove:
>variable type should implement the Serializable interface if it isn't 
primitive type (or it will cause not serializable exception).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2089
  
@arunmahadevan Addressed your comments. Please take a look again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113403773
  
--- Diff: 
examples/storm-starter/src/jvm/org/apache/storm/starter/LambdaTopology.java ---
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.starter;
+
+import org.apache.storm.Config;
+import org.apache.storm.topology.ConfigurableTopology;
+import org.apache.storm.topology.TopologyBuilder;
+import org.apache.storm.tuple.Values;
+
+import java.io.Serializable;
+import java.util.UUID;
+
+public class LambdaTopology extends ConfigurableTopology {
+public static void main(String[] args) {
+ConfigurableTopology.start(new LambdaTopology(), args);
+}
+
+@Override
+protected int run(String[] args) throws Exception {
+TopologyBuilder builder = new TopologyBuilder();
+
+// example. spout1: generate random strings
+// bolt1: get the first part of a string
+// bolt2: output the tuple
+
+// NOTE: Variable used in lambda expression should be final or 
effectively final
+// (or it will cause compilation error),
+// and variable type should implement the Serializable interface 
if it isn't primitive type
+// (or it will cause not serializable exception).
+Prefix prefix = new Prefix("Hello lambda:");
+String suffix = ":so cool!";
+
+builder.setSpout("spout1", () -> UUID.randomUUID().toString());
+builder.setBolt("bolt1", (tuple, collector) -> {
+String[] parts = tuple.getStringByField("lambda").split("\\-");
+collector.emit(new Values(prefix + parts[0] + suffix));
--- End diff --

I think we should support multiple fields. I added a parameter `fields` to 
support user defined output fields, and updated the example.
```
setBolt(String id, SerializableBiConsumer<Tuple,BasicOutputCollector> 
biConsumer, String... fields)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113403731
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java ---
@@ -353,6 +421,31 @@ public void setStateSpout(String id, IRichStateSpout 
stateSpout, Number parallel
 }
 
 /**
+ * Define a new spout in this topology.
+ *
+ * @param id the id of this component. This id is referenced by other 
components that want to consume this spout's outputs.
+ * @param supplier lambda expression that implements tuple generating 
for this spout
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not 
positive
+ */
+public SpoutDeclarer setSpout(String id, SerializableSupplier 
supplier) throws IllegalArgumentException {
+return setSpout(id, supplier, null);
+}
+
+/**
+ * Define a new spout in this topology with the specified parallelism. 
If the spout declares
+ * itself as non-distributed, the parallelism_hint will be ignored and 
only one task
+ * will be allocated to this component.
+ *
+ * @param id the id of this component. This id is referenced by other 
components that want to consume this spout's outputs.
+ * @param parallelism_hint the number of tasks that should be assigned 
to execute this spout. Each task will run on a thread in a process somewhere 
around the cluster.
+ * @param supplier lambda expression that implements tuple generating 
for this spout
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not 
positive
+ */
+public SpoutDeclarer setSpout(String id, SerializableSupplier 
supplier, Number parallelism_hint) throws IllegalArgumentException {
--- End diff --

Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113403709
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java ---
@@ -353,6 +421,31 @@ public void setStateSpout(String id, IRichStateSpout 
stateSpout, Number parallel
 }
 
 /**
+ * Define a new spout in this topology.
+ *
+ * @param id the id of this component. This id is referenced by other 
components that want to consume this spout's outputs.
+ * @param supplier lambda expression that implements tuple generating 
for this spout
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not 
positive
+ */
+public SpoutDeclarer setSpout(String id, SerializableSupplier 
supplier) throws IllegalArgumentException {
--- End diff --

Nice catch. Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2089
  
@srdo Addressed your comments. Please take a look again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113364466
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/lambda/LambdaBiConsumerBolt.java ---
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.lambda;
+
+import org.apache.storm.topology.BasicOutputCollector;
+import org.apache.storm.tuple.Tuple;
+
+public class LambdaBiConsumerBolt extends AbstractLambdaBolt {
+
+private SerializableBiConsumer<Tuple,BasicOutputCollector> biConsumer;
+
+public 
LambdaBiConsumerBolt(SerializableBiConsumer<Tuple,BasicOutputCollector> 
biConsumer) {
--- End diff --

I prefer to keep the current names. Both Consumer and BiConsumer are 
functional interfaces accepting parameters and returning nothing, these two 
bolts are corresponding processor. And users don't access them directly when 
writing spout/bolt by lambda.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113364472
  
--- Diff: 
examples/storm-starter/src/jvm/org/apache/storm/starter/LambdaTopology.java ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.starter;
+
+import org.apache.storm.Config;
+import org.apache.storm.topology.ConfigurableTopology;
+import org.apache.storm.topology.TopologyBuilder;
+import org.apache.storm.tuple.Values;
+
+import java.util.UUID;
+
+public class LambdaTopology extends ConfigurableTopology {
+public static void main(String[] args) {
+ConfigurableTopology.start(new LambdaTopology(), args);
+}
+
+@Override
+protected int run(String[] args) throws Exception {
+TopologyBuilder builder = new TopologyBuilder();
+
+// example. spout1: generate random strings
+// bolt1: get the first part of a string
+// bolt2: output the tuple
+builder.setSpout("spout1", () -> UUID.randomUUID().toString());
--- End diff --

Variable used in lambda expression should be final or effectively final (or 
it will cause compilation error), and variable type should implement the 
Serializable interface if it isn't primitive type (or it will cause not 
serializable exception).

@srdo Thanks for your reminding. I will add some NOTE to remind users.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2089#discussion_r113364469
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java ---
@@ -316,6 +322,68 @@ public BoltDeclarer setBolt(String id, IWindowedBolt 
bolt, Number parallelism_hi
 }
 
 /**
+ * Define a new bolt in this topology. This defines a lambda basic 
bolt, which is a
+ * simpler to use but more restricted kind of bolt. Basic bolts are 
intended
+ * for non-aggregation processing and automate the anchoring/acking 
process to
+ * achieve proper reliability in the topology.
+ *
+ * @param id the id of this component. This id is referenced by other 
components that want to consume this bolt's outputs.
+ * @param biConsumer lambda expression which is the instance of 
functional interface BiConsumer
--- End diff --

Good catch. Will update.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2089: STORM-2490: Lambda support

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2089

STORM-2490: Lambda support

https://issues.apache.org/jira/browse/STORM-2490

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2490-lambda

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2089


commit 7b19d6b8a32726fb45c6095111a27e13411efd66
Author: vesense <best.wang...@163.com>
Date:   2017-04-25T04:55:24Z

STORM-2490: Lambda support




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2085: Adding Cassandra Object Mapper statement builder.

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2085
  
@tandrup Yes, you need to apply apache license to new files.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2077: [STORM-2480] principle should be modified to principal

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2077
  
Merged, and I added you to the contributors list.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2077: [STORM-2480] principle should be modified to principal

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2077
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111882508
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,177 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
Integer.parseInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING).toString()));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer, context);
+
+if (ordered) {
+consumer.registerMessageListener(new MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+   
ConsumeOrderlyContext context) {
+if (process(msgs)) {
+return ConsumeOrderlyStatus.SUCCESS;
+} else {
+return 
ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
+}
+}

[GitHub] storm issue #2024: STORM-2349: Add one RocketMQ plugin for the Apache Storm

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2024
  
@vongosling @hustfxj @harshach Updated. Can you take a look again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111867114
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.ObjectReader;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private static MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
ObjectReader.getInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING)));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+// Since RocketMQ Consumer is thread-safe, RocketMQSpout uses a 
single
+// consumer instance across threads to improve the performance.
+synchronized (RocketMQSpout.class) {
+if (consumer == null) {
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer);
+
+if (ordered) {
+consumer.registerMessageListener(new 
MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+ 

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111551137
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.ObjectReader;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private static MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
ObjectReader.getInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING)));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+// Since RocketMQ Consumer is thread-safe, RocketMQSpout uses a 
single
+// consumer instance across threads to improve the performance.
+synchronized (RocketMQSpout.class) {
+if (consumer == null) {
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer);
+
+if (ordered) {
+consumer.registerMessageListener(new 
MessageListenerOrderly() {
--- End diff --

In fact, the RocketMQ "push" mode is still pulling data from broker. 
PushConsumer is a high level consumer API, wrapping the pulling details

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111551156
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.ObjectReader;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private static MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
ObjectReader.getInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING)));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+// Since RocketMQ Consumer is thread-safe, RocketMQSpout uses a 
single
+// consumer instance across threads to improve the performance.
+synchronized (RocketMQSpout.class) {
+if (consumer == null) {
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer);
+
+if (ordered) {
+consumer.registerMessageListener(new 
MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+ 

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111551133
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.ObjectReader;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private static MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
ObjectReader.getInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING)));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+// Since RocketMQ Consumer is thread-safe, RocketMQSpout uses a 
single
+// consumer instance across threads to improve the performance.
+synchronized (RocketMQSpout.class) {
--- End diff --

Maybe my code comment is not so clear. thread-safe is just precondition, 
the important is that this is related to the RocketMQ internal 
implementation(sharing queue, threads, etc.), "Consumer concurrency / Only one 
consumer instance per process" is the way official recommend.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feat

[GitHub] storm pull request #2071: STORM-1858: KafkaBolt: sharing a single producer i...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2071

STORM-1858: KafkaBolt: sharing a single producer instance across threads

https://issues.apache.org/jira/browse/STORM-1858

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-1858

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2071.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2071


commit 40eefa8cbaed4a5a6cfa145bf76d7b81d871719e
Author: vesense <best.wang...@163.com>
Date:   2017-04-14T03:18:31Z

STORM-1858: KafkaBolt: sharing a single producer instance across threads




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2069: STORM-2475: Fix parsing of host:port to deal with IPv6 ad...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2069
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2024: STORM-2349: Add one RocketMQ plugin for the Apache Storm

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2024
  
@vongosling @hustfxj Updated. Can you take a look again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2067: STORM-2474: fix external README format

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2067

STORM-2474: fix external README format

This is a really minor change to fix the file format.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2067.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2067


commit ab4204e757ae373426fc675035405c9e88c90efb
Author: Xin Wang <best.wang...@163.com>
Date:   2017-04-13T09:53:06Z

STORM-2474: fix external README format




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111346264
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,177 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
Integer.parseInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING).toString()));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer, context);
+
+if (ordered) {
+consumer.registerMessageListener(new MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+   
ConsumeOrderlyContext context) {
+if (process(msgs)) {
+return ConsumeOrderlyStatus.SUCCESS;
+} else {
+return 
ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
+}
+}

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111345560
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,177 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+boolean ordered = getBoolean(properties, 
RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
Integer.parseInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING).toString()));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer, context);
+
+if (ordered) {
+consumer.registerMessageListener(new MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+   
ConsumeOrderlyContext context) {
+if (process(msgs)) {
+return ConsumeOrderlyStatus.SUCCESS;
+} else {
+return 
ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
+}
+}

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r111344417
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.Utils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private boolean ordered;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+ordered = 
getBoolean(properties,RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
Utils.getInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING)));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer, context);
+
+if (ordered) {
+consumer.registerMessageListener(new MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+   
ConsumeOrderlyContext context) {
+if (process(msgs)) {
+return ConsumeOrderlyStatus.SUCCESS;
+} else {
+return 
ConsumeOrderlyS

[GitHub] storm issue #2064: STORM-1114: Handle race condition in Storm/Trident transa...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2064
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2061: STORM-2469: fix integration-test dependencies issu...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2061

STORM-2469: fix integration-test dependencies issue

This is the hot fix for integration-test dependencies issue

* add storm-server. since __the ExclamationTopology depends on 
LocalCluster__
* remove unused dependencies: selenium-firefox-driver, selenium-support, 
storm-solr, storm-starter, storm-hdfs

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2469

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2061.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2061


commit a9dbe7ca48d123c8d60394da89e8bae7524bf7e8
Author: Xin Wang <best.wang...@163.com>
Date:   2017-04-12T06:12:12Z

STORM-2469: fix integration-test dependencies issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2058: [STORM-2466] The example of jaas.conf in jaas_kerberos.co...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2058
  
I guess the change is a copy from 
http://storm.apache.org/releases/1.1.0/SECURITY.html "Storm Kerberos 
Configuration" section.
Overall looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2056: [STORM-2464] update storm-mongodb.md

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2056
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2057: [STORM-2465] modify storm-redis's READEME.md and update s...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2057
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2059: STORM-2463: fix DRPCTest.testDequeueAfterTimeout test fai...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2059
  
@revans2 100ms dosen't work, but 1s works well. 500ms sometimes still get 
failure.
 I changed the value to 1s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2059: STORM-2463: fix DRPCTest.testDequeueAfterTimeout t...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/2059

STORM-2463: fix DRPCTest.testDequeueAfterTimeout test failure



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-2463

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2059


commit b44f473bf72b0fb2178def0532a75cc1d1f60b4d
Author: Xin Wang <best.wang...@163.com>
Date:   2017-04-11T13:31:29Z

STORM-2463: fix DRPCTest.testDequeueAfterTimeout




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2048: [STORM-2461] modify storm-hdfs's READEME.md and update st...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2048
  
@liu-zhaokun Will merge it later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2046: STORM-2458: Kafka Spout should allow hooking up custom lo...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2046
  
@souravmitra Yes, I will merge it later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2046: STORM-2458: Kafka Spout should allow hooking up custom lo...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2046
  
Looks good to me. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2048: [STORM-2461] modify storm-hdfs's READEME.md and update st...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2048
  
Thanks @liu-zhaokun Keep the good work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2048: [STORM-2461] modify storm-hdfs's READEME.md and update st...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2048
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2024: STORM-2349: Add one RocketMQ plugin for the Apache Storm

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2024
  
POM files updated & Rebased.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2024: STORM-2349: Add one RocketMQ plugin for the Apache Storm

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2024
  
Hi @hustfxj I guess you know Apache RocketMQ a lot, please take a look if 
you have time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2024: STORM-2349: Add one RocketMQ plugin for the Apache Storm

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2024
  
@vongosling Thanks for your comments. And I will rebase the code on master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r110599823
  
--- Diff: 
external/storm-rocketmq/src/main/java/org/apache/storm/rocketmq/spout/RocketMQSpout.java
 ---
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.rocketmq.spout;
+
+import org.apache.commons.lang.Validate;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.MQPushConsumer;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
+import 
org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import 
org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.storm.Config;
+import org.apache.storm.rocketmq.DefaultMessageRetryManager;
+import org.apache.storm.rocketmq.MessageRetryManager;
+import org.apache.storm.rocketmq.MessageSet;
+import org.apache.storm.rocketmq.RocketMQConfig;
+import org.apache.storm.rocketmq.SpoutConfig;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.IRichSpout;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+import org.apache.storm.utils.Utils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import static org.apache.storm.rocketmq.RocketMQUtils.getBoolean;
+import static org.apache.storm.rocketmq.RocketMQUtils.getInteger;
+
+/**
+ * RocketMQSpout uses MQPushConsumer as the default implementation.
+ * PushConsumer is a high level consumer API, wrapping the pulling details
+ * Looks like broker push messages to consumer
+ */
+public class RocketMQSpout implements IRichSpout {
+// TODO add metrics
+
+private MQPushConsumer consumer;
+private SpoutOutputCollector collector;
+private boolean ordered;
+private BlockingQueue queue;
+
+private Properties properties;
+private MessageRetryManager messageRetryManager;
+
+public RocketMQSpout(Properties properties) {
+this.properties = properties;
+}
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) {
+Validate.notEmpty(properties, "Consumer properties can not be 
empty");
+ordered = 
getBoolean(properties,RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, false);
+
+int queueSize = getInteger(properties, SpoutConfig.QUEUE_SIZE, 
Utils.getInt(conf.get(Config.TOPOLOGY_MAX_SPOUT_PENDING)));
+queue = new LinkedBlockingQueue<>(queueSize);
+
+consumer = new DefaultMQPushConsumer();
+RocketMQConfig.buildConsumerConfigs(properties, 
(DefaultMQPushConsumer)consumer, context);
+
+if (ordered) {
+consumer.registerMessageListener(new MessageListenerOrderly() {
+@Override
+public ConsumeOrderlyStatus 
consumeMessage(List msgs,
+   
ConsumeOrderlyContext context) {
+if (process(msgs)) {
+return ConsumeOrderlyStatus.SUCCESS;
+} else {
+return 
ConsumeOrderlyS

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r110599801
  
--- Diff: external/storm-rocketmq/pom.xml ---
@@ -0,0 +1,89 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+storm
+org.apache.storm
+2.0.0-SNAPSHOT
+../../pom.xml
+
+
+storm-rocketmq
+storm-rocketmq
+
+jar
+
+
+
+    vesense
+Xin Wang
+xinw...@apache.org
+
+
+
+
+
+
+org.apache.storm
+storm-core
+${project.version}
+provided
+
+
+org.apache.rocketmq
+rocketmq-client
+4.0.0-incubating
+
+
+commons-lang
+commons-lang
+
+
+
+org.mockito
+mockito-core
+test
+${mockito.version}
+
+
+org.slf4j
+log4j-over-slf4j
+test
--- End diff --

This is not used. Will remove.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2024: STORM-2349: Add one RocketMQ plugin for the Apache...

[vesense]

Github user vesense commented on a diff in the pull request:

https://github.com/apache/storm/pull/2024#discussion_r110599786
  
--- Diff: external/storm-rocketmq/pom.xml ---
@@ -0,0 +1,89 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+storm
+org.apache.storm
+2.0.0-SNAPSHOT
+../../pom.xml
+
+
+storm-rocketmq
+storm-rocketmq
+
+jar
+
+
+
+    vesense
+Xin Wang
+xinw...@apache.org
+
+
+
+
+
+
+org.apache.storm
+storm-core
+${project.version}
+provided
+
+
+org.apache.rocketmq
+rocketmq-client
+4.0.0-incubating
+
+
+commons-lang
+commons-lang
--- End diff --

This is used for validating parameters. It will cause `ClassNotFound` 
exception without the dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2045: [STORM-2457] update and modify storm/hbase/README.md and ...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2045
  
@liu-zhaokun Yes, I will merge it later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2044: [STORM-2454] the default returned value of this method wh...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2044
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2045: [STORM-2457] update and modify storm/hbase/README.md and ...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2045
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2042: STORM-2453 Move non-connectors into the top directory

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2042
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---