Repository: geode-examples Updated Branches: refs/heads/feature/GEODE-2231 da58df518 -> 65443bb77
GEODE-2231 Derive BadEmployeeKey from EmployeeKey - Modify the Producer and Consumer to deal with both regions - Revise to the README (not completed yet) Project: http://git-wip-us.apache.org/repos/asf/geode-examples/repo Commit: http://git-wip-us.apache.org/repos/asf/geode-examples/commit/65443bb7 Tree: http://git-wip-us.apache.org/repos/asf/geode-examples/tree/65443bb7 Diff: http://git-wip-us.apache.org/repos/asf/geode-examples/diff/65443bb7 Branch: refs/heads/feature/GEODE-2231 Commit: 65443bb77dfe8fe90c764d5122ffffc69ce16287 Parents: da58df5 Author: Karen Miller <kmil...@pivotal.io> Authored: Mon Feb 6 17:35:34 2017 -0800 Committer: Karen Miller <kmil...@pivotal.io> Committed: Mon Feb 6 17:35:34 2017 -0800 ---------------------------------------------------------------------- partitioned/README.md | 82 +++++++++++++------- partitioned/scripts/startAll.sh | 5 +- .../examples/partitioned/BadEmployeeKey.java | 38 +-------- .../geode/examples/partitioned/BaseClient.java | 33 +++++--- .../geode/examples/partitioned/Consumer.java | 34 +++++--- .../geode/examples/partitioned/Producer.java | 60 +++++++++++++- .../partitioned/BadEmployeeKeyTest.java | 22 ------ .../examples/partitioned/ConsumerTest.java | 16 +--- .../examples/partitioned/ProducerTest.java | 4 +- 9 files changed, 169 insertions(+), 125 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/README.md ---------------------------------------------------------------------- diff --git a/partitioned/README.md b/partitioned/README.md index d82e398..1c51058 100644 --- a/partitioned/README.md +++ b/partitioned/README.md @@ -17,59 +17,76 @@ limitations under the License. # Geode partitioned region example -This basic example demonstrates the properties of a partitioned region. -Two servers host a partitioned region, which has no redundancy. -The producer puts 50 entries into the partitioned region. -The consumer prints the number of entries in the region. +This basic example demonstrates the basic properties of partitioned regions: + +- Data entries are distributed across all servers that host a region. +The distribution is like database sharding, except that the distribution +occurs automatically. It is also similar to data striping on disks, +except that the distribution is not based on hardware. + +- Hashing distributes entries among buckets that reside on servers. +A good hash code is important in order to spread the entries among buckets. + +In this example, +two servers host two partitioned regions. +There is no redundancy, so that the basic properties of partitioning +may be observed. +The Producer code puts the same 10 entries into each of the two +partitioned regions. +The Consumer gets and prints the entries from each of the two regions. Due to partitioning, the entries are distributed among the two servers hosting the region. Since there is no redundancy of the data within the region, when one of the servers goes away, the entries hosted within that server are also gone. +The two regions are the same, except for the hash code implementation. +The ```EmployeeRegion``` has a good hashing function, +and the ```BadEmployeeRegion``` has a pointedly poor hash code implementation. +The hash code is so bad that all entries in the +```BadEmployeeRegion``` end up in the same bucket. + This example is a simple demonstration of some basic Geode APIs, -as well how to write tests using mocks for Geode applications. +as well as providing ```gfsh``` command examples. ## Steps 1. From the ```geode-examples/partitioned``` directory, -build the jar (with the EmployeeKey and EmployeeData classes), -which is put onto the classpath when starting the servers: +build the jar (with the ```EmployeeKey```, ```BadEmployeeKey```, +and ```EmployeeData``` classes): $ ../gradlew build 1. From the ```geode-examples/partitioned``` directory, -run a script that starts a locator and two servers: +run a script that starts a locator and two servers. +The JAR built in the first step is placed +onto the classpath when starting the servers: $ scripts/startAll.sh - Each of the servers hosts the partitioned region called ```EmployeeRegion```. - -2. Run the producer to put 50 entries into ```EmployeeRegion```: + Each of the servers hosts both partitioned regions. +1. Run the producer to put the same 10 entries into both the +```EmployeeRegion``` and the ```BadEmployeeRegion```: $ ../gradlew run -Pmain=Producer ... ... - INFO: Done. Inserted 50 entries. - + INFO: Inserted 10 entries in EmployeeRegion. + INFO: Inserted 10 entries in BadEmployeeRegion. To see contents of the region keys, use a ```gfsh``` query: $ $GEODE_HOME/bin/gfsh ... gfsh>connect gfsh>query --query="select e.key from /EmployeeRegion.entries e" - - or, to see contents of the region values, use a ```gfsh``` query: + Or, to see contents of the region values, use a ```gfsh``` query: gfsh>query --query="select * from /EmployeeRegion" - -3. Run the consumer to observe that there are 50 entries in ```EmployeeRegion```: +1. Run the consumer to get and log all 10 entries in each region, +the```EmployeeRegion``` and the ```BadEmployeeRegion```: $ ../gradlew run -Pmain=Consumer - ... - ... - INFO: Done. 50 entries available on the server(s). - Note that this observation may also be made with ```gfsh```: + Note that the quantity of entries may also be observed with ```gfsh```: $ $GEODE_HOME/bin/gfsh ... @@ -85,12 +102,25 @@ run a script that starts a locator and two servers: Type | Name | Value ------ | ----------- | --------- - Region | size | 50 + Region | size | 10 | data-policy | PARTITION gfsh>quit -4. Kill one of the servers: + As an alternative, ```gfsh``` maybe used to identify how many entries + there are for each region on each server by looking at statistics. + + gfsh>show metrics --categories=partition --region=/BadEmployeeRegion --member=server1 + + Within the output, the result for ```totalBucketSize``` identifies + the number of entries hosted on the specified server. + Vary the command to see both ```server1``` and ```server2```, as well as + ```EmployeeRegion``` and ```BadEmployeeRegion```. + + Note that for the ```BadEmployeeRegion```, one of the servers will host + all the entries, while the other server will not have any of the entries. + This is due to the bad hash code generated for those keys. +1. Kill one of the servers: $ $GEODE_HOME/bin/gfsh ... @@ -99,12 +129,12 @@ run a script that starts a locator and two servers: gfsh>quit 5. Run the consumer a second time, and notice that only approximately half of -the entries are still available: +the entries of the ```EmployeeRegion``` are still available: $ ../gradlew run -Pmain=Consumer ... ... - INFO: Done. 25 entries available on the server(s). + INFO: Done. 6 entries available on the server(s). Again, this observation may also be made with ```gfsh```: @@ -121,7 +151,7 @@ the entries are still available: Type | Name | Value ------ | ----------- | --------- - Region | size | 25 + Region | size | 4 | data-policy | PARTITION gfsh>quit http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/scripts/startAll.sh ---------------------------------------------------------------------- diff --git a/partitioned/scripts/startAll.sh b/partitioned/scripts/startAll.sh index e3e6ab9..165fd44 100755 --- a/partitioned/scripts/startAll.sh +++ b/partitioned/scripts/startAll.sh @@ -36,9 +36,12 @@ do gfsh start server --locators=localhost[${GEODE_LOCATOR_PORT}] --name=server$N --server-port=0 --mcast-port=0 --classpath=${PWD}/build/libs/partitioned-0.1.0-SNAPSHOT.jar done -# create a region using GFSH +# create 2 regions with the same data using GFSH, one that works well, +# good keys, and the other that implements a bad hashCode on the key gfsh -e "connect --locator=localhost[${GEODE_LOCATOR_PORT}]" -e "create region --name=EmployeeRegion --type=PARTITION" +gfsh -e "connect --locator=localhost[${GEODE_LOCATOR_PORT}]" -e "create region --name=BadEmployeeRegion --type=PARTITION" + gfsh -e "connect --locator=localhost[${GEODE_LOCATOR_PORT}]" -e "list members" exit 0 http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/main/java/org/apache/geode/examples/partitioned/BadEmployeeKey.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/BadEmployeeKey.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/BadEmployeeKey.java index fef04d4..cf3f98a 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/BadEmployeeKey.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/BadEmployeeKey.java @@ -19,44 +19,14 @@ import java.util.logging.Logger; import java.io.Serializable; import org.apache.geode.cache.client.ClientCache; -public class BadEmployeeKey implements Serializable { +public class BadEmployeeKey extends EmployeeKey { private static final long serialVersionUID = 1L; - static final Logger logger = Logger.getAnonymousLogger(); - private String name; - private int emplNumber; - public BadEmployeeKey() {} public BadEmployeeKey(String n, int en) { - this.name = n; - this.emplNumber = en; - } - - public String getName() { - return (name); - } - - public int getEmplNumber() { - return (emplNumber); - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (o == null || getClass() != o.getClass()) { - return false; - } - - BadEmployeeKey that = (BadEmployeeKey) o; - - if (emplNumber != that.emplNumber) { - return false; - } - return name.equals(that.name); + super(n, en); } /* @@ -71,8 +41,4 @@ public class BadEmployeeKey implements Serializable { return 1; } - public String toString() { - return ("Name: " + this.name + " Employee Number: " + this.emplNumber); - } - } http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java index 3608518..0b66cb6 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java @@ -27,27 +27,42 @@ public abstract class BaseClient { static final Logger logger = Logger.getAnonymousLogger(); protected ClientCache clientCache; - protected void setRegion(Region region) { - this.region = region; + protected void setRegion1(Region region) { + this.region1 = region; } - private Region region; + protected void setRegion2(Region region) { + this.region2 = region; + } + + private Region region1; + private Region region2; private final String locatorHost = System.getProperty("GEODE_LOCATOR_HOST", "localhost"); private final int locatorPort = Integer.getInteger("GEODE_LOCATOR_PORT", 10334); - protected static final String REGION_NAME = "EmployeeRegion"; + protected static final String REGION1_NAME = "EmployeeRegion"; + protected static final String REGION2_NAME = "BadEmployeeRegion"; static final int NUM_ENTRIES = 10; public BaseClient() { this.clientCache = getClientCache(); } - protected Region getRegion() { - if (region == null) { - region = getClientCache() + protected Region getRegion1() { + if (region1 == null) { + region1 = getClientCache() + .<EmployeeKey, EmployeeData>createClientRegionFactory(ClientRegionShortcut.PROXY) + .create(REGION1_NAME); + } + return (region1); + } + + protected Region getRegion2() { + if (region2 == null) { + region2 = getClientCache() .<EmployeeKey, EmployeeData>createClientRegionFactory(ClientRegionShortcut.PROXY) - .create(REGION_NAME); + .create(REGION2_NAME); } - return region; + return (region2); } protected ClientCache getClientCache() { http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java index ba1e3a4..9fd47a5 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java @@ -22,8 +22,6 @@ public class Consumer extends BaseClient { public static void main(String[] args) { Consumer c = new Consumer(); - int numEntries = c.countEntriesOnServer(); - logger.info(String.format("%d entries in the region on the server(s).", numEntries)); c.printRegionContents(); } @@ -33,20 +31,32 @@ public class Consumer extends BaseClient { this.clientCache = clientCache; } - public int countEntriesOnServer() { - int size = this.getRegion().keySetOnServer().size(); - return size; - } public void printRegionContents() { - Region myRegion = this.getRegion(); - Set<EmployeeKey> setOfKeys = myRegion.keySetOnServer(); /* - * for each key in setOfKeys print the entry + * Print EmployeeRegion size and contents + */ + Region r1 = this.getRegion1(); + Set<EmployeeKey> setOfKeys1 = r1.keySetOnServer(); + logger.info(setOfKeys1.size() + " entries in EmployeeRegion on the server(s)."); + if (setOfKeys1.size() > 0) { + logger.info("Contents of EmployeeRegion:"); + for (EmployeeKey key : setOfKeys1) { + logger.info(r1.get(key).toString()); + } + } + + /* + * Print BadEmployeeRegion size and contents */ - logger.info("Region contents:"); - for (EmployeeKey key : setOfKeys) { - logger.info(myRegion.get(key).toString()); + Region r2 = this.getRegion2(); + Set<EmployeeKey> setOfKeys2 = r2.keySetOnServer(); + logger.info(setOfKeys2.size() + " entries in BadEmployeeRegion on the server(s)."); + if (setOfKeys2.size() > 0) { + logger.info("Contents of BadEmployeeRegion:"); + for (EmployeeKey key : setOfKeys2) { + logger.info(r2.get(key).toString()); + } } } http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java index a9e8f83..4869e01 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java @@ -21,7 +21,10 @@ import org.apache.geode.cache.client.ServerOperationException; public class Producer extends BaseClient { public static void main(String[] args) { - new Producer().populateRegion(); + + Producer p = new Producer(); + p.populateRegion(); + p.populateBadRegion(); } public Producer() {} @@ -32,7 +35,7 @@ public class Producer extends BaseClient { public void populateRegion() { - Region r = getRegion(); + Region r = getRegion1(); EmployeeKey k1 = new EmployeeKey("Alex Able", 160); EmployeeData d1 = new EmployeeData(k1, 70000, 40); try { @@ -78,6 +81,57 @@ public class Producer extends BaseClient { EmployeeData d10 = new EmployeeData(k10, 70000, 40); r.put(k10, d10); - logger.info("Inserted 10 entries."); + logger.info("Inserted 10 entries in EmployeeRegion."); + } + + public void populateBadRegion() { + + Region r = getRegion2(); + BadEmployeeKey k1 = new BadEmployeeKey("Alex Able", 160); + EmployeeData d1 = new EmployeeData(k1, 70000, 40); + try { + r.put(k1, d1); + } catch (ServerOperationException e) { + logger.info("ServerOperationException " + e.getMessage()); + logger.info("ServerOperationException " + e.getCause()); + } + + BadEmployeeKey k2 = new BadEmployeeKey("Bertie Bell", 170); + EmployeeData d2 = new EmployeeData(k2, 72000, 40); + r.put(k2, d2); + + BadEmployeeKey k3 = new BadEmployeeKey("Kris Call", 180); + EmployeeData d3 = new EmployeeData(k3, 68000, 40); + r.put(k3, d3); + + BadEmployeeKey k4 = new BadEmployeeKey("Dale Driver", 190); + EmployeeData d4 = new EmployeeData(k4, 81500, 36); + r.put(k4, d4); + + BadEmployeeKey k5 = new BadEmployeeKey("Frankie Forth", 201); + EmployeeData d5 = new EmployeeData(k5, 45000, 40); + r.put(k5, d5); + + BadEmployeeKey k6 = new BadEmployeeKey("Jamie Jive", 220); + EmployeeData d6 = new EmployeeData(k6, 56500, 40); + r.put(k6, d6); + + BadEmployeeKey k7 = new BadEmployeeKey("Morgan Minnow", 230); + EmployeeData d7 = new EmployeeData(k7, 65000, 36); + r.put(k7, d7); + + BadEmployeeKey k8 = new BadEmployeeKey("Pat Puts", 240); + EmployeeData d8 = new EmployeeData(k8, 67000, 40); + r.put(k8, d8); + + BadEmployeeKey k9 = new BadEmployeeKey("Ricky Reliable", 2500); + EmployeeData d9 = new EmployeeData(k9, 71000, 40); + r.put(k9, d9); + + BadEmployeeKey k10 = new BadEmployeeKey("Taylor Tack", 260); + EmployeeData d10 = new EmployeeData(k10, 70000, 40); + r.put(k10, d10); + + logger.info("Inserted 10 entries in BadEmployeeRegion."); } } http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/test/java/org/apache/geode/examples/partitioned/BadEmployeeKeyTest.java ---------------------------------------------------------------------- diff --git a/partitioned/src/test/java/org/apache/geode/examples/partitioned/BadEmployeeKeyTest.java b/partitioned/src/test/java/org/apache/geode/examples/partitioned/BadEmployeeKeyTest.java index 1c16de8..dacd669 100644 --- a/partitioned/src/test/java/org/apache/geode/examples/partitioned/BadEmployeeKeyTest.java +++ b/partitioned/src/test/java/org/apache/geode/examples/partitioned/BadEmployeeKeyTest.java @@ -34,30 +34,8 @@ public class BadEmployeeKeyTest { } @Test - public void testGetName() { - assertEquals("First Last", k.getName()); - } - - @Test - public void testGetEmplNumber() { - assertEquals(3001, k.getEmplNumber()); - } - - @Test - public void testEquals() { - BadEmployeeKey otherKey = new BadEmployeeKey("First Last", 3001); - assertTrue(k.equals(otherKey)); - BadEmployeeKey nonMatchingKey = new BadEmployeeKey("Othername", 1); - assertFalse(k.equals(nonMatchingKey)); - } - - @Test public void testHashCode() { assertEquals(1, k.hashCode()); } - @Test - public void testToString() { - assertEquals("Name: First Last Employee Number: 3001", k.toString()); - } } http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java ---------------------------------------------------------------------- diff --git a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java index 37a7842..11dac48 100644 --- a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java +++ b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java @@ -41,17 +41,12 @@ public class ConsumerTest { @Before public void setup() { - when(region.getName()).thenReturn(Consumer.REGION_NAME); + when(region.getName()).thenReturn(Consumer.REGION1_NAME); when(keys.size()).thenReturn(Consumer.NUM_ENTRIES); when(region.keySetOnServer()).thenReturn(keys); when(clientCache.getRegion(any())).thenReturn(region); consumer = new Consumer(clientCache); - consumer.setRegion(region); - } - - @Test - public void numberOfEntriesOnServerShouldMatchConsumerEntries() throws Exception { - assertEquals(consumer.NUM_ENTRIES, consumer.countEntriesOnServer()); + consumer.setRegion1(region); } @Test @@ -59,12 +54,5 @@ public class ConsumerTest { assertTrue(consumer.NUM_ENTRIES > 0); } - @Test - public void countingEntriesWithoutConnectionShouldThrowNoAvailableLocatorsException() - throws Exception { - consumer = new Consumer(); - expectedException.expect(NoAvailableLocatorsException.class); - assertEquals(consumer.NUM_ENTRIES, consumer.countEntriesOnServer()); - } } http://git-wip-us.apache.org/repos/asf/geode-examples/blob/65443bb7/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java ---------------------------------------------------------------------- diff --git a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java index 439f9c1..d794820 100644 --- a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java +++ b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java @@ -40,7 +40,7 @@ public class ProducerTest { @Before public void setup() throws Exception { - when(region.getName()).thenReturn(Producer.REGION_NAME); + when(region.getName()).thenReturn(Producer.REGION1_NAME); when(region.keySetOnServer()).thenReturn(keys); when(clientCache.getRegion(any())).thenReturn(region); } @@ -48,7 +48,7 @@ public class ProducerTest { @Test public void populateRegionShouldReturnCorrectNumberOfEntries() throws Exception { producer = new Producer(clientCache); - producer.setRegion(region); + producer.setRegion1(region); producer.populateRegion(); verify(region, times(producer.NUM_ENTRIES)).put(any(), any());
