[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult
[ https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590134#comment-15590134 ] ASF subversion and git services commented on GEODE-538: --- Commit 11ef3ebbe30a8340f57776bf4063684b91ccd0a3 in incubator-geode's branch refs/heads/develop from [~khowe] [ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=11ef3eb ] GEODE-538: Add check for persistent data recovery PartitionedRegion.getNodeForBucketReadOrLoad can return an invalid node if persistent data recovery is in process and a get() targets a bucket that hasn't been recoverd yet. This can result in returning an incorrect value (null) or throwing ConflictingPersistentDataException from a get() or put() on the region. This change adds a check for persistent recovery to be completed before creating the new bucket. If recovery isn't complete then the operation on the region will fail with a PartitionOfflineException. Queries on a region while persistent recovery is in progress can also result in incorrect results so a similar check is added to DefaultQuery.checkQueryOnPR. This closes #264 > PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion > failed if do not wait on getResult > > > Key: GEODE-538 > URL: https://issues.apache.org/jira/browse/GEODE-538 > Project: Geode > Issue Type: Bug > Components: persistence >Reporter: Jinmei Liao >Assignee: Kenneth Howe > Attachments: logs.zip > > > While attempting to work on GEODE-506, I commented out line s1538-1540 like > the following > async1 = vm1.invokeAsync(createChildPR); > async0 = vm0.invokeAsync(createChildPR); > AsyncInvocation async2 = vm2.invokeAsync(createChildPR); > //async0.getResult(MAX_WAIT); -- line 1538 > //async1.getResult(MAX_WAIT); -- line 1539 > //async2.getResult(MAX_WAIT); -- line 1540 > This will reproduce the error similar to the one reported in GEODE506, but it > failed on checking the data on parent region instead of the child region. Log > files attached. > dunit.RMIException: While invoking > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run > in VM 0 running on Host localhost with 4 VMs > at dunit.VM.invoke(VM.java:368) > at dunit.VM.invoke(VM.java:312) > at dunit.VM.invoke(VM.java:266) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at junit.framework.TestCase.runBare(TestCase.java:141) > at junit.framework.TestResult$1.protect(TestResult.java:122) > at junit.framework.TestResult.runProtected(TestResult.java:142) > at junit.framework.TestResult.run(TestResult.java:125) > at junit.framework.TestCase.run(TestCase.java:129) > at junit.framework.TestSuite.runTest(TestSuite.java:252) > at junit.framework.TestSuite.run(TestSuite.java:247) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) > Caused by: junit.framework.AssertionFailedError: For key 1 expected: but > was: > at junit.framework
[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult
[ https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582559#comment-15582559 ] Kenneth Howe commented on GEODE-538: Queries on a persistent colocated PR can also cause this problem. > PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion > failed if do not wait on getResult > > > Key: GEODE-538 > URL: https://issues.apache.org/jira/browse/GEODE-538 > Project: Geode > Issue Type: Bug > Components: persistence >Reporter: Jinmei Liao >Assignee: Kenneth Howe > Attachments: logs.zip > > > While attempting to work on GEODE-506, I commented out line s1538-1540 like > the following > async1 = vm1.invokeAsync(createChildPR); > async0 = vm0.invokeAsync(createChildPR); > AsyncInvocation async2 = vm2.invokeAsync(createChildPR); > //async0.getResult(MAX_WAIT); -- line 1538 > //async1.getResult(MAX_WAIT); -- line 1539 > //async2.getResult(MAX_WAIT); -- line 1540 > This will reproduce the error similar to the one reported in GEODE506, but it > failed on checking the data on parent region instead of the child region. Log > files attached. > dunit.RMIException: While invoking > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run > in VM 0 running on Host localhost with 4 VMs > at dunit.VM.invoke(VM.java:368) > at dunit.VM.invoke(VM.java:312) > at dunit.VM.invoke(VM.java:266) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at junit.framework.TestCase.runBare(TestCase.java:141) > at junit.framework.TestResult$1.protect(TestResult.java:122) > at junit.framework.TestResult.runProtected(TestResult.java:142) > at junit.framework.TestResult.run(TestResult.java:125) > at junit.framework.TestCase.run(TestCase.java:129) > at junit.framework.TestSuite.runTest(TestSuite.java:252) > at junit.framework.TestSuite.run(TestSuite.java:247) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) > Caused by: junit.framework.AssertionFailedError: For key 1 expected: but > was: > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.TestCase.assertEquals(TestCase.java:244) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run(PersistentPartitionedRegionTestBase.java:177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at hydra.MethExecutor.executeObject(MethExecutor.java:267) > at > dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:77) > at sun.reflect.NativeMethodAccessorImpl.
[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult
[ https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15527484#comment-15527484 ] Kenneth Howe commented on GEODE-538: With the async*.getResult(MAX_WAIT) commented out, then checkData() is called while persistent data recovery is still in progress following the child region creation on all the members. The test now (usually) fails because a PartitionedRegion.get() call targets a bucket that hasn't yet been recovered. PR.getNodeForBucketReadOrLoad() ends up creating a new bucket, and the get() ends up returning null. Examining full logs there will also be one or more members that throw ConflictingPersistentDataException when the recovery thread(s) tries to recover the bucket that was erroneously created. put() operations on PRs with incomplete persistent recovery can also cause this failure. > PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion > failed if do not wait on getResult > > > Key: GEODE-538 > URL: https://issues.apache.org/jira/browse/GEODE-538 > Project: Geode > Issue Type: Bug > Components: persistence >Reporter: Jinmei Liao >Assignee: Kenneth Howe > Attachments: logs.zip > > > While attempting to work on GEODE-506, I commented out line s1538-1540 like > the following > async1 = vm1.invokeAsync(createChildPR); > async0 = vm0.invokeAsync(createChildPR); > AsyncInvocation async2 = vm2.invokeAsync(createChildPR); > //async0.getResult(MAX_WAIT); -- line 1538 > //async1.getResult(MAX_WAIT); -- line 1539 > //async2.getResult(MAX_WAIT); -- line 1540 > This will reproduce the error similar to the one reported in GEODE506, but it > failed on checking the data on parent region instead of the child region. Log > files attached. > dunit.RMIException: While invoking > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run > in VM 0 running on Host localhost with 4 VMs > at dunit.VM.invoke(VM.java:368) > at dunit.VM.invoke(VM.java:312) > at dunit.VM.invoke(VM.java:266) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at junit.framework.TestCase.runBare(TestCase.java:141) > at junit.framework.TestResult$1.protect(TestResult.java:122) > at junit.framework.TestResult.runProtected(TestResult.java:142) > at junit.framework.TestResult.run(TestResult.java:125) > at junit.framework.TestCase.run(TestCase.java:129) > at junit.framework.TestSuite.runTest(TestSuite.java:252) > at junit.framework.TestSuite.run(TestSuite.java:247) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) > Caused by: junit.framework.AssertionFailedError: For key 1 expected: but > was: > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.TestCase.assertEquals(TestCase.java:244) > at > com.gemstone.gemfire.internal.cache.
[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult
[ https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423587#comment-15423587 ] Darrel Schneider commented on GEODE-538: In the current source code I think this is lines: 1562-1564. I made commented out these lines like so: async1 = vm1.invokeAsync(createChildPR); async0 = vm0.invokeAsync(createChildPR); AsyncInvocation async2 = vm2.invokeAsync(createChildPR); //async0.getResult(MAX_WAIT); //async1.getResult(MAX_WAIT); //async2.getResult(MAX_WAIT); but the unit test still passes. Ken has reported seeing operations on parent regions that are created but still off-line waiting for a colocated child region hang. > PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion > failed if do not wait on getResult > > > Key: GEODE-538 > URL: https://issues.apache.org/jira/browse/GEODE-538 > Project: Geode > Issue Type: Bug > Components: persistence >Reporter: Jinmei Liao >Assignee: Dan Smith > Attachments: logs.zip > > > While attempting to work on GEODE-506, I commented out line s1538-1540 like > the following > async1 = vm1.invokeAsync(createChildPR); > async0 = vm0.invokeAsync(createChildPR); > AsyncInvocation async2 = vm2.invokeAsync(createChildPR); > //async0.getResult(MAX_WAIT); -- line 1538 > //async1.getResult(MAX_WAIT); -- line 1539 > //async2.getResult(MAX_WAIT); -- line 1540 > This will reproduce the error similar to the one reported in GEODE506, but it > failed on checking the data on parent region instead of the child region. Log > files attached. > dunit.RMIException: While invoking > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run > in VM 0 running on Host localhost with 4 VMs > at dunit.VM.invoke(VM.java:368) > at dunit.VM.invoke(VM.java:312) > at dunit.VM.invoke(VM.java:266) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at junit.framework.TestCase.runBare(TestCase.java:141) > at junit.framework.TestResult$1.protect(TestResult.java:122) > at junit.framework.TestResult.runProtected(TestResult.java:142) > at junit.framework.TestResult.run(TestResult.java:125) > at junit.framework.TestCase.run(TestCase.java:129) > at junit.framework.TestSuite.runTest(TestSuite.java:252) > at junit.framework.TestSuite.run(TestSuite.java:247) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) > Caused by: junit.framework.AssertionFailedError: For key 1 expected: but > was: > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.TestCase.assertEquals(TestCase.java:244) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run(PersistentPartitionedRegionTestBase.java:177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Me
[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult
[ https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007632#comment-15007632 ] Dan Smith commented on GEODE-538: - I was working with Jinmei when we noticed this. I think the basic problem that we are allowing operations to proceed on a region that is not really initialized. What's going on is that a member is restarting a member with two persistent colocated regions. After it creates the parent region, we return a region object from the RegionFactory.create method. However, geode hasn't actually recovered the parent region from disk because it's waiting waiting for the child region to also be created. Unfortunately if someone calls region.get on the parent at this point it looks like geode returns null, rather than throwing some sort of exception. That's bad because it looks like a key is missing when if fact geode hasn't actually recovered the region yet. > PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion > failed if do not wait on getResult > > > Key: GEODE-538 > URL: https://issues.apache.org/jira/browse/GEODE-538 > Project: Geode > Issue Type: Bug > Components: persistence >Reporter: Jinmei Liao >Assignee: Dan Smith > Attachments: logs.zip > > > While attempting to work on GEODE-506, I commented out line s1538-1540 like > the following > async1 = vm1.invokeAsync(createChildPR); > async0 = vm0.invokeAsync(createChildPR); > AsyncInvocation async2 = vm2.invokeAsync(createChildPR); > //async0.getResult(MAX_WAIT); -- line 1538 > //async1.getResult(MAX_WAIT); -- line 1539 > //async2.getResult(MAX_WAIT); -- line 1540 > This will reproduce the error similar to the one reported in GEODE506, but it > failed on checking the data on parent region instead of the child region. Log > files attached. > dunit.RMIException: While invoking > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run > in VM 0 running on Host localhost with 4 VMs > at dunit.VM.invoke(VM.java:368) > at dunit.VM.invoke(VM.java:312) > at dunit.VM.invoke(VM.java:266) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543) > at > com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at junit.framework.TestCase.runBare(TestCase.java:141) > at junit.framework.TestResult$1.protect(TestResult.java:122) > at junit.framework.TestResult.runProtected(TestResult.java:142) > at junit.framework.TestResult.run(TestResult.java:125) > at junit.framework.TestCase.run(TestCase.java:129) > at junit.framework.TestSuite.runTest(TestSuite.java:252) > at junit.framework.TestSuite.run(TestSuite.java:247) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) > Caused by: junit.framework.AssertionFailedError: For key 1 expected: but > was: > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.TestCase.assertEquals(TestCa