[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

[revans2]

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2241
  
@roshannaik I just created a new pull request to master for an updated 
version of ThroughputVsLatency and I added in the test tools that I created for 
being able to capture and simulate topologies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2289: STORM-2702: storm-loadgen

[revans2]

GitHub user revans2 opened a pull request:

https://github.com/apache/storm/pull/2289

STORM-2702: storm-loadgen

Some tools to be able to generate load on a cluster for testing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/revans2/incubator-storm STORM-2702

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2289


commit 0d10b8afe7e282d04b67f1a0c1c90db801842b14
Author: Robert (Bobby) Evans 
Date:   2017-08-21T18:29:59Z

STORM-2702: Part 1.  Move files as needed

commit 6c2dcbedabb88970697f42b6f66bf64177e2ac9c
Author: Robert (Bobby) Evans 
Date:   2017-08-21T19:36:10Z

STORM-2702: storm-loadgen




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2288: [STORM-2700] Should not check Blobstore ACL if the...

[Ethanlm]

GitHub user Ethanlm opened a pull request:

https://github.com/apache/storm/pull/2288

[STORM-2700] Should not check Blobstore ACL if the validation is disabled

See: https://issues.apache.org/jira/browse/STORM-2700

`canUserReadBlob ` function should take 
`STORM_BLOBSTORE_ACL_VALIDATION_ENABLED` config into consideration. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ethanlm/storm STORM-2700

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2288


commit d1b73321399658693ba31fa533079615cb2c6cd9
Author: Ethan Li 
Date:   2017-08-21T19:08:29Z

[STORM-2700] Should not check Blobstore ACL if the validation is disabled




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2282: STORM-2694: Add KafkaTupleListener to storm-kafka-client

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2282
  
Thanks for making the changes. +1 once the license issue is fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2282: STORM-2694: Add KafkaTupleListener to storm-kafka-...

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2282#discussion_r134280097
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/EmptyKafkaTupleListener.java
 ---
@@ -0,0 +1,33 @@
+package org.apache.storm.kafka.spout;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+
+public class EmptyKafkaTupleListener implements KafkaTupleListener {
+
+@Override
+public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collector) { }
+
+@Override
+public void onEmit(List tuple, KafkaSpoutMessageId msgId) { }
+
+@Override
+public void onAck(KafkaSpoutMessageId msgId) { }
+
+@Override
+public void onPartitionsReassigned(KafkaSpoutMessageId msgId) { }
+
+@Override
+public void onRetry(KafkaSpoutMessageId msgId) { }
+
+@Override
+public void onMaxRetryReached(KafkaSpoutMessageId msgId) { }
+
+@Override
+public String toString() {
+return "EmptyKafkaTupleListener";
--- End diff --

Makes sense


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2282: STORM-2694: Add KafkaTupleListener to storm-kafka-...

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2282#discussion_r134280004
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/EmptyKafkaTupleListener.java
 ---
@@ -0,0 +1,35 @@
+package org.apache.storm.kafka.spout;
--- End diff --

You need to add the Apache license to the top of the new files. You can 
just copy it from one of the other ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2270: [STORM-2686] Add Locality Aware Shuffle Grouping

[Ethanlm]

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2270
  
@roshannaik  Thanks for the review. I did some performance testing with 
ACKing disabled on single Worker case:

 Performance testing on ConstSpoutNullBoltTopo with ACKing disabled.

1. Config: 
`topology.message.timeout: 300; 
  topology.max.spout.pending: 5000;
  topology.acker.executors: 0`
2. 1 VM 
3. Launched 1 workers, 1 spout task and 1 bolt task. ACKing disabled.
4. All experiments ran 300s.
5. For clarity, only show the outputs at 240s.
6. Numbers fluctuate slightly during the experiments.

Grouping | transsferred (messages) | transfer rate (message/s) | 
spout_transferred | spout_acks | spout_throughput (acks/s)
-- | -- | -- | -- | -- | --
LocalityASG | 105218480 | 1753641 | 105218480 | 105218500 | 1753641
LocalOrShuffle(loadaware disabled) | 112664640 | 1877744 | 112664640 | 
112664620 | 1877744
Shuffle (loadaware disabled) | 108752120 | 1812535 | 108752120 | 108752100 
| 1812535
LocalOrShuffle | 79656420 | 1327607 | 79656420 | 79656420 | 1327607
Shuffle | 79809260 | 1330154 | 79809260 | 79809280 | 1330154

The results show that `ConcurrentMap` in `LocalityAwareShuffleGrouping` 
introduces a very small overhead. We can also observe that `LoadAware` actually 
introduces much overhead in this single Worker case. It's probably because of 
`AtomicReference>` data structure.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2287: [STORM-2699] Put all the version information of third par...

[BigOneLiu]

Github user BigOneLiu commented on the issue:

https://github.com/apache/storm/pull/2287
  
@HeartSaVioR 
Could you please review my pr? thx


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2287: [STORM-2699] Put all the version information of th...

[BigOneLiu]

GitHub user BigOneLiu opened a pull request:

https://github.com/apache/storm/pull/2287

[STORM-2699] Put all the version information of third party component into 
the main pom 


[https://issues.apache.org/jira/browse/STORM-2699](https://issues.apache.org/jira/browse/STORM-2699)
I think it's better to put all the version information of third party 
components into the main pom for more efficient version control


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/BigOneLiu/storm master0821

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2287






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2270: [STORM-2686] Add Locality Aware Shuffle Grouping

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2270
  
I will try some runs on STORM-2306 in single worker mode and ACK-disabled.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2270: [STORM-2686] Add Locality Aware Shuffle Grouping

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2270
  
So the ACKer bolt is usually the bigger bottleneck ...compared to the 
groupings. So with ACKing enabled it will hide any bottlenecks in the 
groupings. The only thing slower than ACKer bolt is the inter-worker message 
transfer code.  

 Latency is unlikely to take a noticeable hit...unless you have lots of 
hops  Spot->Bolt->Bolt-Bolt->Bolt->Bolt. Regardless, good you have validated 
that there is latency is.   

The real observable hit if any will be in the throughput. You should 
examine the ConstSpoutNull Bolt with ACKing disabled.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2270: [STORM-2686] Add Locality Aware Shuffle Grouping

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2270
  
taking a look... 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---