[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151906#comment-16151906
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-gossip/pull/68


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147264#comment-16147264
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

GitHub user pxsalehi reopened a pull request:

https://github.com/apache/incubator-gossip/pull/68

GOSSIP-41 Transfer gossip data in bulk

This is the first version I had in mind. The simplest way is to create a 
bulk message which carries a list of messages. The handler simply unpacks the 
list of messages and the rest will stay the same. I have failing tests to go 
through yet. But maybe someone can let me know if this is the way to go or 
there is a better alternative.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pxsalehi/incubator-gossip GOSSIP_41_bulk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-gossip/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68


commit 7c457eb3f497ba2368d74e4f8afa36e7910c3e16
Author: pxsalehi 
Date:   2017-08-28T13:59:15Z

GOSSIP-41 Transfer gossip data in bulk




> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147260#comment-16147260
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user pxsalehi closed the pull request at:

https://github.com/apache/incubator-gossip/pull/68


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145741#comment-16145741
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user pxsalehi commented on the issue:

https://github.com/apache/incubator-gossip/pull/68
  
Done!


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144717#comment-16144717
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r135690087
  
--- Diff: 
gossip-itest/src/test/java/org/apache/gossip/PerNodeDataEventTest.java ---
@@ -23,45 +23,57 @@
 import org.apache.gossip.model.PerNodeDataMessage;
 import org.junit.Assert;
 import org.junit.Test;
-import org.junit.platform.runner.JUnitPlatform;
 import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.UnknownHostException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.UUID;
+import java.util.*;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 
-@RunWith(JUnitPlatform.class)
+@RunWith(Parameterized.class)
 public class PerNodeDataEventTest extends AbstractIntegrationBase {
   
   private String receivedKey = "";
   private String receivingNodeId = "";
   private Object receivingNodeDataNewValue = "";
   private Object receivingNodeDataOldValue = "";
   private Semaphore lock = new Semaphore(0);
-  
-  
+  private int base;
+  private boolean bulkTransfer;
+
+  public PerNodeDataEventTest(int base, boolean bulkTransfer) {
+this.base = base;
+this.bulkTransfer = bulkTransfer;
+  }
+
--- End diff --

Lets handle that in a separate ticket.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143797#comment-16143797
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user pxsalehi commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r135529679
  
--- Diff: 
gossip-itest/src/test/java/org/apache/gossip/PerNodeDataEventTest.java ---
@@ -23,45 +23,57 @@
 import org.apache.gossip.model.PerNodeDataMessage;
 import org.junit.Assert;
 import org.junit.Test;
-import org.junit.platform.runner.JUnitPlatform;
 import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.UnknownHostException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.UUID;
+import java.util.*;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 
-@RunWith(JUnitPlatform.class)
+@RunWith(Parameterized.class)
 public class PerNodeDataEventTest extends AbstractIntegrationBase {
   
   private String receivedKey = "";
   private String receivingNodeId = "";
   private Object receivingNodeDataNewValue = "";
   private Object receivingNodeDataOldValue = "";
   private Semaphore lock = new Semaphore(0);
-  
-  
+  private int base;
+  private boolean bulkTransfer;
+
+  public PerNodeDataEventTest(int base, boolean bulkTransfer) {
+this.base = base;
+this.bulkTransfer = bulkTransfer;
+  }
+
--- End diff --

While going through the tests, I noticed the same can be done in 
`TenNodeThreeSeedTest` by parameterizing base.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143745#comment-16143745
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r135516447
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/manager/AbstractActiveGossiper.java 
---
@@ -92,22 +102,59 @@ public final void sendSharedData(LocalMember me, 
LocalMember member){
   UdpSharedDataMessage message = new UdpSharedDataMessage();
   message.setUuid(UUID.randomUUID().toString());
   message.setUriFrom(me.getId());
-  message.setExpireAt(innerEntry.getValue().getExpireAt());
-  message.setKey(innerEntry.getValue().getKey());
-  message.setNodeId(innerEntry.getValue().getNodeId());
-  message.setTimestamp(innerEntry.getValue().getTimestamp());
-  message.setPayload(innerEntry.getValue().getPayload());
-  message.setReplicable(innerEntry.getValue().getReplicable());
+  copySharedDataMessage(innerEntry.getValue(), message);
   gossipCore.sendOneWay(message, member.getUri());
 }
-sharedDataHistogram.update(System.currentTimeMillis() - startTime);
   }
-  
+
+  /** Send shared data by batching together several entries. */
+  private void sendSharedDataInBulkInternal(LocalMember me, LocalMember 
member) {
+UdpSharedDataBulkMessage udpMessage = new UdpSharedDataBulkMessage();
+udpMessage.setUuid(UUID.randomUUID().toString());
+udpMessage.setUriFrom(me.getId());
+for (Entry innerEntry : 
gossipCore.getSharedData().entrySet()) {
+  if (innerEntry.getValue().getReplicable() != null && 
!innerEntry.getValue().getReplicable()
+  .shouldReplicate(me, member, innerEntry.getValue())) {
+continue;
+  }
+  SharedDataMessage message = new SharedDataMessage();
+  copySharedDataMessage(innerEntry.getValue(), message);
+  udpMessage.addMessage(message);
+  if (udpMessage.getMessages().size() == 
gossipSettings.getBulkTransferSize()) {
+gossipCore.sendOneWay(udpMessage, member.getUri());
+udpMessage = new UdpSharedDataBulkMessage();
+udpMessage.setUuid(UUID.randomUUID().toString());
+udpMessage.setUriFrom(me.getId());
+  }
+}
+if (udpMessage.getMessages().size() > 0) {
+  gossipCore.sendOneWay(udpMessage, member.getUri());
+}
+  }
+
+  private void copySharedDataMessage(SharedDataMessage original, 
SharedDataMessage copy) {
+copy.setExpireAt(original.getExpireAt());
+copy.setKey(original.getKey());
+copy.setNodeId(original.getNodeId());
+copy.setTimestamp(original.getTimestamp());
+copy.setPayload(original.getPayload());
+copy.setReplicable(original.getReplicable());
+  }
+
   public final void sendPerNodeData(LocalMember me, LocalMember member){
 if (member == null){
   return;
 }
 long startTime = System.currentTimeMillis();
+if (gossipSettings.isBulkTransfer()) {
+  sendPerNodeDataInBulkInternal(me, member);
+} else
--- End diff --

same deal use { } even for one line if


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143744#comment-16143744
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r135516918
  
--- Diff: 
gossip-itest/src/test/java/org/apache/gossip/PerNodeDataEventTest.java ---
@@ -23,45 +23,57 @@
 import org.apache.gossip.model.PerNodeDataMessage;
 import org.junit.Assert;
 import org.junit.Test;
-import org.junit.platform.runner.JUnitPlatform;
 import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.UnknownHostException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.UUID;
+import java.util.*;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 
-@RunWith(JUnitPlatform.class)
+@RunWith(Parameterized.class)
 public class PerNodeDataEventTest extends AbstractIntegrationBase {
   
   private String receivedKey = "";
   private String receivingNodeId = "";
   private Object receivingNodeDataNewValue = "";
   private Object receivingNodeDataOldValue = "";
   private Semaphore lock = new Semaphore(0);
-  
-  
+  private int base;
+  private boolean bulkTransfer;
+
+  public PerNodeDataEventTest(int base, boolean bulkTransfer) {
+this.base = base;
+this.bulkTransfer = bulkTransfer;
+  }
+
--- End diff --

Slick. I like this.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143746#comment-16143746
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r135516238
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/manager/AbstractActiveGossiper.java 
---
@@ -78,12 +79,21 @@ public final void sendShutdownMessage(LocalMember me, 
LocalMember target){
 m.setShutdownAtNanos(gossipManager.getClock().nanoTime());
 gossipCore.sendOneWay(m, target.getUri());
   }
-  
-  public final void sendSharedData(LocalMember me, LocalMember member){
-if (member == null){
+
+  public final void sendSharedData(LocalMember me, LocalMember member) {
+if (member == null) {
   return;
 }
 long startTime = System.currentTimeMillis();
+if (gossipSettings.isBulkTransfer()) {
+  sendSharedDataInBulkInternal(me, member);
+} else
--- End diff --

use { } even for one line if so your intention is clear.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143187#comment-16143187
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user pxsalehi commented on the issue:

https://github.com/apache/incubator-gossip/pull/68
  
Instead of new tests I have used parameterized tests to run existing tests 
with bulk transfer activated.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137782#comment-16137782
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r134644887
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/manager/AbstractActiveGossiper.java 
---
@@ -109,22 +120,31 @@ public final void sendPerNodeData(LocalMember me, 
LocalMember member){
 }
 long startTime = System.currentTimeMillis();
 for (Entry> 
entry : gossipCore.getPerNodeData().entrySet()){
+  UdpPerNodeDataMessage udpMessage = new UdpPerNodeDataMessage();
+  udpMessage.setUuid(UUID.randomUUID().toString());
+  udpMessage.setUriFrom(me.getId());
   for (Entry innerEntry : 
entry.getValue().entrySet()){
 if (innerEntry.getValue().getReplicable() != null && 
!innerEntry.getValue().getReplicable()
 .shouldReplicate(me, member, innerEntry.getValue())) {
   continue;
 }
-UdpPerNodeDataMessage message = new UdpPerNodeDataMessage();
-message.setUuid(UUID.randomUUID().toString());
-message.setUriFrom(me.getId());
+PerNodeDataMessage message = new PerNodeDataMessage();
 message.setExpireAt(innerEntry.getValue().getExpireAt());
 message.setKey(innerEntry.getValue().getKey());
 message.setNodeId(innerEntry.getValue().getNodeId());
 message.setTimestamp(innerEntry.getValue().getTimestamp());
 message.setPayload(innerEntry.getValue().getPayload());
 message.setReplicable(innerEntry.getValue().getReplicable());
-gossipCore.sendOneWay(message, member.getUri());   
+udpMessage.addMessage(message);
+if (udpMessage.getMessages().size() == 100) {
--- End diff --

This needs to be a variable that can be controlled through settings


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137783#comment-16137783
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r134644941
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/manager/AbstractActiveGossiper.java 
---
@@ -84,22 +84,33 @@ public final void sendSharedData(LocalMember me, 
LocalMember member){
   return;
 }
 long startTime = System.currentTimeMillis();
+
+UdpSharedDataMessage udpMessage = new UdpSharedDataMessage();
+udpMessage.setUuid(UUID.randomUUID().toString());
+udpMessage.setUriFrom(me.getId());
+
 for (Entry innerEntry : 
gossipCore.getSharedData().entrySet()){
   if (innerEntry.getValue().getReplicable() != null && 
!innerEntry.getValue().getReplicable()
   .shouldReplicate(me, member, innerEntry.getValue())) {
 continue;
   }
-  UdpSharedDataMessage message = new UdpSharedDataMessage();
-  message.setUuid(UUID.randomUUID().toString());
-  message.setUriFrom(me.getId());
+  SharedDataMessage message = new SharedDataMessage();
   message.setExpireAt(innerEntry.getValue().getExpireAt());
   message.setKey(innerEntry.getValue().getKey());
   message.setNodeId(innerEntry.getValue().getNodeId());
   message.setTimestamp(innerEntry.getValue().getTimestamp());
   message.setPayload(innerEntry.getValue().getPayload());
   message.setReplicable(innerEntry.getValue().getReplicable());
-  gossipCore.sendOneWay(message, member.getUri());
+  udpMessage.addMessage(message);
+  if (udpMessage.getMessages().size() == 100) {
--- End diff --

Use the same variable to control this.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137781#comment-16137781
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r134644585
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/model/PerNodeDataBulkMessage.java 
---
@@ -0,0 +1,27 @@
+package org.apache.gossip.model;
--- End diff --

All files Requires apache licence header


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137784#comment-16137784
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r134644532
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/model/PerNodeDataBulkMessage.java 
---
@@ -0,0 +1,27 @@
+package org.apache.gossip.model;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
--- End diff --

No author comments. 


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137786#comment-16137786
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

Github user edwardcapriolo commented on a diff in the pull request:

https://github.com/apache/incubator-gossip/pull/68#discussion_r134644652
  
--- Diff: 
gossip-base/src/main/java/org/apache/gossip/model/PerNodeDataBulkMessage.java 
---
@@ -0,0 +1,27 @@
+package org.apache.gossip.model;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Created by pxsalehi on 22.08.17.
+ */
+public class PerNodeDataBulkMessage extends Base {
+private List messages = new ArrayList<>();
+
+public void addMessage(PerNodeDataMessage msg) {
+messages.add(msg);
+}
+
+public List getMessages() {
--- End diff --

Indent is two spaces. Use our style sheet.


> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (GOSSIP-41) Transfer gossip data in bulk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GOSSIP-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136790#comment-16136790
 ] 

ASF GitHub Bot commented on GOSSIP-41:
--

GitHub user pxsalehi opened a pull request:

https://github.com/apache/incubator-gossip/pull/68

[WIP] GOSSIP-41 Transfer gossip data in bulk

This is the first version I had in mind. The simplest way is to create a 
bulk message which carries a list of messages. The handler simply unpacks the 
list of messages and the rest will stay the same. I have failing tests to go 
through yet. But maybe someone can let me know if this is the way to go or 
there is a better alternative.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pxsalehi/incubator-gossip GOSSIP_41_bulk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-gossip/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68






> Transfer gossip data in bulk
> 
>
> Key: GOSSIP-41
> URL: https://issues.apache.org/jira/browse/GOSSIP-41
> Project: Gossip
>  Issue Type: Improvement
>Reporter: Edward Capriolo
>Assignee: Pooya Salehi
>
> Currently the process that transfers shared data and per node data iterates a 
> map transmitting a single datum at a time using one way communication.  We 
> could transmit more key values at once by simply saying if keys < 100 
> transfer all at once, or by collecting keys and sending in groups of 100. 
> This would greatly lower chatter communicating data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)