Repository: geode-examples Updated Branches: refs/heads/feature/GEODE-2231 17fd0ab82 -> 912a37234
GEODE-2231 Rework README, code, and tests to present partitioning example in two parts Project: http://git-wip-us.apache.org/repos/asf/geode-examples/repo Commit: http://git-wip-us.apache.org/repos/asf/geode-examples/commit/912a3723 Tree: http://git-wip-us.apache.org/repos/asf/geode-examples/tree/912a3723 Diff: http://git-wip-us.apache.org/repos/asf/geode-examples/diff/912a3723 Branch: refs/heads/feature/GEODE-2231 Commit: 912a37234fdccb9a80de9335bab366b5cd3ec6b0 Parents: 17fd0ab Author: Karen Miller <[email protected]> Authored: Wed Feb 22 16:21:06 2017 -0800 Committer: Karen Miller <[email protected]> Committed: Wed Feb 22 16:21:06 2017 -0800 ---------------------------------------------------------------------- partitioned/README.md | 153 +++++++++++++------ .../geode/examples/partitioned/BaseClient.java | 4 + .../geode/examples/partitioned/Consumer.java | 24 ++- .../geode/examples/partitioned/Producer.java | 25 ++- .../examples/partitioned/ConsumerTest.java | 55 +++++++ .../examples/partitioned/ProducerTest.java | 52 +++++++ 6 files changed, 261 insertions(+), 52 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/geode-examples/blob/912a3723/partitioned/README.md ---------------------------------------------------------------------- diff --git a/partitioned/README.md b/partitioned/README.md index 7b2e4b0..0a90284 100644 --- a/partitioned/README.md +++ b/partitioned/README.md @@ -79,6 +79,17 @@ The argument specification identifies the region. INFO: Inserted 10 entries in EmployeeRegion. ``` +1. There are several ways to see the contents of the region. +Run the consumer to get and print all 10 entries in the +```EmployeeRegion```. The argument specification identifies the region. + + ``` + $ ../gradlew run -Pmain=Consumer -Pargs=EmployeeRegion + ... + INFO: 10 entries in EmployeeRegion on the server(s). + ... + ``` + To see contents of the region keys, use a ```gfsh``` query: ``` @@ -86,27 +97,20 @@ The argument specification identifies the region. ... gfsh>connect gfsh>query --query="select e.key from /EmployeeRegion.entries e" + ... ``` + Or, to see contents of the region values, use a ```gfsh``` query: ``` gfsh>query --query="select * from /EmployeeRegion" - ``` - -1. Run the consumer to get and print all 10 entries in each region, the -```EmployeeRegion``` and the ```BadEmployeeRegion```: - - ``` - $ ../gradlew run -Pmain=Consumer + ... ``` Note that the quantity of entries may also be observed with ```gfsh```: ``` - $ $GEODE_HOME/bin/gfsh - ... - gfsh>connect gfsh>describe region --name=EmployeeRegion .......................................................... Name : EmployeeRegion @@ -128,18 +132,16 @@ The argument specification identifies the region. there are for each region on each server by looking at statistics. ``` - gfsh>show metrics --categories=partition --region=/BadEmployeeRegion --member=server1 + gfsh>show metrics --categories=partition --region=/EmployeeRegion --member=server1 ``` Within the output, the result for ```totalBucketSize``` identifies the number of entries hosted on the specified server. - Vary the command to see both ```server1``` and ```server2```, as well as - ```EmployeeRegion``` and ```BadEmployeeRegion```. + Vary the command to see statistics for both ```server1``` and + ```server2```. - Note that for the ```BadEmployeeRegion```, one of the servers will host - all the entries, while the other server will not have any of the entries. - This is due to the bad hash code generated for those keys. -1. Kill one of the servers: +1. The region entries are distributed across both servers. +Kill one of the servers: ``` $ $GEODE_HOME/bin/gfsh @@ -150,38 +152,102 @@ The argument specification identifies the region. ``` 1. Run the consumer a second time, and notice that approximately half of -the entries of the ```EmployeeRegion``` are still available on the remaining server. +the entries of the ```EmployeeRegion``` are still available on the +remaining server. Those hosted by the server that was stopped were lost. -And, depending on which server hosted all the `BadEmployeeRegion` entries and which server was stopped, there will either be 0 entries or all 10 entries on the remaining server. ``` - $ ../gradlew run -Pmain=Consumer + $ ../gradlew run -Pmain=Consumer -Pargs=EmployeeRegion ... ... - INFO: 6 entries in EmployeeRegion on the server(s). + INFO: 5 entries in EmployeeRegion on the server(s). ... - INFO: 0 entries in BadEmployeeRegion on the server(s). ``` - Again, this observation may also be made with ```gfsh```: + +6. Shut down the system: + + ``` + $ scripts/stopAll.sh + ``` + +## Part 2: What Can Go Wrong + +Hashing distributes entries among buckets that reside on servers. +A good hash code is important in order to spread the entries among buckets. + +The ```EmployeeRegion``` used in Part 1 of this example has a good hashing +function. +Region entries are well distributed among buckets +(and therefore, among servers). +The ```BadEmployeeRegion``` used in this part of the example +has a pointedly poor hash code implementation, +to illustrate what can go wrong. +The hash code is so bad that all entries in the +```BadEmployeeRegion``` end up in the same bucket. +Because of this, +one of the servers will host all the entries, +while the other server will not have any of the entries. + +Here are the steps to run through this example again, +using ```BadEmployeeRegion``` instead of ```EmployeeRegion```. +This part assumes that you have already built the JAR (step 2 in Part 1 +of this example). + +1. Set directory ```geode-examples/partitioned``` to be the +current working directory. +Each step in this example specifies paths relative to that directory. + +1. Run a script that starts a locator and two servers. +The built JAR will be placed onto the classpath when the script +starts the servers: + + ``` + $ scripts/startAll.sh + ``` + +1. Run the producer to put 10 entries into the ```BadEmployeeRegion```. +The argument specification identifies the region. + + ``` + $ ../gradlew run -Pmain=Producer -Pargs=BadEmployeeRegion + ... + INFO: Inserted 10 entries in BadEmployeeRegion. + ``` + +1. Run the consumer to get and print all 10 entries in the +`BadEmployeeRegion`. The argument specification identifies the region. +Alternatively, use one of the `gfsh` commands +(given in Part 1 of this example) +to verify that the servers are hosting the 10 entries in the region. + + ``` + $ ../gradlew run -Pmain=Consumer -Pargs=EmployeeRegion + ... + INFO: 10 entries in EmployeeRegion on the server(s). + ... + ``` + +1. Kill one of the servers: ``` $ $GEODE_HOME/bin/gfsh ... gfsh>connect - gfsh>describe region --name=EmployeeRegion - .......................................................... - Name : EmployeeRegion - Data Policy : partition - Hosting Members : server2 - - Non-Default Attributes Shared By Hosting Members + gfsh>stop server --name=server1 + gfsh>quit + ``` - Type | Name | Value - ------ | ----------- | --------- - Region | size | 4 - | data-policy | PARTITION +1. Run the consumer or a `gfsh` query, and notice that either all or +none of the entries of the ```BadEmployeeRegion``` are still available on the +remaining server. +Those hosted by the server that was stopped were lost. - gfsh>quit + ``` + $ ../gradlew run -Pmain=Consumer -Pargs=BadEmployeeRegion + ... + ... + INFO: 5 entries in BadEmployeeRegion on the server(s). + ... ``` 6. Shut down the system: @@ -190,13 +256,14 @@ And, depending on which server hosted all the `BadEmployeeRegion` entries and wh $ scripts/stopAll.sh ``` -## Part 2: What Can Go Wrong +## Partitioning Example Takeaways -Hashing distributes entries among buckets that reside on servers. -A good hash code is important in order to spread the entries among buckets. +1. Partitioned regions distribute region entries across buckets within servers. +A robust system will use redundancy in conjunction with partitioning +in production systems, +so that data is not lost if a server goes down. + +2. A proper hashcode is important for distributing entries across buckets. +Not demonstrated in this example, but a proper equals methods is also +required. -While the ```EmployeeRegion``` used in Part 1 of this example -has a good hashing function, -the ```BadEmployeeRegion``` has a pointedly poor hash code implementation. -The hash code is so bad that all entries in the -```BadEmployeeRegion``` end up in the same bucket. http://git-wip-us.apache.org/repos/asf/geode-examples/blob/912a3723/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java index 5e51a66..5fdc236 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/BaseClient.java @@ -47,6 +47,10 @@ public abstract class BaseClient { this.clientCache = getClientCache(); } + public BaseClient(ClientCache clientCache) { + this.clientCache = clientCache; + } + protected Region getRegion1() { if (region1 == null) { region1 = getClientCache() http://git-wip-us.apache.org/repos/asf/geode-examples/blob/912a3723/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java index 1cef4e9..2e61866 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Consumer.java @@ -22,7 +22,27 @@ public class Consumer extends BaseClient { public static void main(String[] args) { Consumer c = new Consumer(); - c.printRegionContents(); + c.checkAndPrint(args); + } + + public void checkAndPrint(String[] args) { + if (0 == args.length) { + throw new RuntimeException("Expected argument specifying region name."); + } else { + if (args.length > 1) { + throw new RuntimeException("Expected only 1 argument, and received more than 1."); + } else { + if (args[0].equals("EmployeeRegion")) { + this.printRegionContents(); + } else { + if (args[0].equals("BadEmployeeRegion")) { + this.printBadRegionContents(); + } else { + throw new RuntimeException("Unrecognized region name in argument specification."); + } + } + } + } } public Consumer() {} @@ -43,7 +63,9 @@ public class Consumer extends BaseClient { logger.info(r1.get(key).toString()); } } + } + public void printBadRegionContents() { /* Print BadEmployeeRegion size and contents */ Region r2 = this.getRegion2(); Set<EmployeeKey> setOfKeys2 = r2.keySetOnServer(); http://git-wip-us.apache.org/repos/asf/geode-examples/blob/912a3723/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java ---------------------------------------------------------------------- diff --git a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java index 787625a..04fcfc1 100644 --- a/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java +++ b/partitioned/src/main/java/org/apache/geode/examples/partitioned/Producer.java @@ -20,27 +20,36 @@ import org.apache.geode.cache.Region; public class Producer extends BaseClient { public static void main(String[] args) { - Producer p = new Producer(); + p.checkAndPopulate(args); + } + + public void checkAndPopulate(String[] args) { if (0 == args.length) { throw new RuntimeException("Expected argument specifying region name."); } else { - if (args[0].equals("EmployeeRegion")) { - p.populateRegion(); + if (args.length > 1) { + throw new RuntimeException("Expected only 1 argument, and received more than 1."); } else { - if (args[0].equals("BadEmployeeRegion")) { - p.populateBadRegion(); + if (args[0].equals("EmployeeRegion")) { + this.populateRegion(); } else { - throw new RuntimeException("Unrecognized region name in argument specification."); + if (args[0].equals("BadEmployeeRegion")) { + this.populateBadRegion(); + } else { + throw new RuntimeException("Unrecognized region name in argument specification."); + } } } } } - public Producer() {} + public Producer() { + super(); + } public Producer(ClientCache clientCache) { - this.clientCache = clientCache; + super(clientCache); } /* Put 10 entries into the region with the quality hashing function. */ http://git-wip-us.apache.org/repos/asf/geode-examples/blob/912a3723/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java ---------------------------------------------------------------------- diff --git a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java index 1682ab8..e6a550e 100644 --- a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java +++ b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ConsumerTest.java @@ -43,6 +43,12 @@ public class ConsumerTest { private Region region2 = mock(Region.class); private ClientRegionFactory clientRegionFactory = mock(ClientRegionFactory.class); private Set keys = mock(Set.class); + private static final String[] EMPTYARGS = new String[0]; + private static final String[] GOODARGS1 = {"EmployeeRegion"}; + private static final String[] GOODARGS2 = {"BadEmployeeRegion"}; + private static final String[] BADARGS1 = {""}; + private static final String[] BADARGS2 = {"BadRegionName"}; + private static final String[] BADARGS3 = {"BadEmployeeRegion", "2"}; @Before public void setup() { @@ -106,4 +112,53 @@ public class ConsumerTest { consumer.printRegionContents(); } + @Test + public void testPrintBadRegionContents() { + consumer.printBadRegionContents(); + } + + @Test + public void testCheckAndPrint1() { + /* no exception expected */ + consumer.checkAndPrint(GOODARGS1); + } + + @Test + public void testCheckAndPrint2() { + /* no exception expected */ + consumer.checkAndPrint(GOODARGS2); + } + + @Test + public void testCheckAndPrint3() { + /* no arguments specified, array length should be 0 */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Expected argument specifying region name."); + spy(Consumer.class).checkAndPrint(EMPTYARGS); + } + + @Test + public void testCheckAndPrint4() { + /* First argument is the empty string */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Unrecognized region name in argument specification."); + spy(Consumer.class).checkAndPrint(BADARGS1); + } + + @Test + public void testCheckAndPrint5() { + /* Arguments are an invalid region name */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Unrecognized region name in argument specification."); + spy(Consumer.class).checkAndPrint(BADARGS2); + } + + @Test + public void testCheckAndPrint6() { + /* Arguments are an invalid region name and an extra but unused argument */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Expected only 1 argument, and received more than 1."); + spy(Consumer.class).checkAndPrint(BADARGS3); + } + } http://git-wip-us.apache.org/repos/asf/geode-examples/blob/912a3723/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java ---------------------------------------------------------------------- diff --git a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java index 70729bc..350d96d 100644 --- a/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java +++ b/partitioned/src/test/java/org/apache/geode/examples/partitioned/ProducerTest.java @@ -37,11 +37,18 @@ public class ProducerTest { public ExpectedException expectedException = ExpectedException.none(); private Producer producer; + private Producer pMock = mock(Producer.class); private ClientCache clientCache = mock(ClientCache.class); private Region<EmployeeKey, EmployeeData> region1 = mock(Region.class); private Region<BadEmployeeKey, EmployeeData> region2 = mock(Region.class); private ClientRegionFactory clientRegionFactory = mock(ClientRegionFactory.class); private Set keys = mock(Set.class); + private static final String[] EMPTYARGS = new String[0]; + private static final String[] GOODARGS1 = {"EmployeeRegion"}; + private static final String[] GOODARGS2 = {"BadEmployeeRegion"}; + private static final String[] BADARGS1 = {""}; + private static final String[] BADARGS2 = {"BadRegionName"}; + private static final String[] BADARGS3 = {"BadEmployeeRegion", "2"}; @Before @@ -53,6 +60,7 @@ public class ProducerTest { .thenReturn(clientRegionFactory); when(clientRegionFactory.create(Consumer.REGION1_NAME)).thenReturn(region1); when(clientRegionFactory.create(Consumer.REGION2_NAME)).thenReturn(region2); + doNothing().when(pMock).populateRegion(); } @@ -73,6 +81,50 @@ public class ProducerTest { } + @Test + public void testCheckAndPopulate1() { + /* no exception expected */ + pMock.checkAndPopulate(GOODARGS1); + } + + @Test + public void testCheckAndPopulate2() { + /* no exception expected */ + pMock.checkAndPopulate(GOODARGS2); + } + + @Test + public void testCheckAndPopulate3() { + /* no arguments specified, array length should be 0 */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Expected argument specifying region name."); + spy(Producer.class).checkAndPopulate(EMPTYARGS); + } + + @Test + public void testCheckAndPopulate4() { + /* First argument is the empty string */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Unrecognized region name in argument specification."); + spy(Producer.class).checkAndPopulate(BADARGS1); + } + + @Test + public void testCheckAndPopulate5() { + /* Arguments are an invalid region name */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Unrecognized region name in argument specification."); + spy(Producer.class).checkAndPopulate(BADARGS2); + } + + @Test + public void testCheckAndPopulate6() { + /* Arguments are an invalid region name and an extra but unused argument */ + expectedException.expect(Exception.class); + expectedException.expectMessage("Expected only 1 argument, and received more than 1."); + spy(Producer.class).checkAndPopulate(BADARGS3); + } + @After public void tearDown() {
