[
https://issues.apache.org/jira/browse/GEODE-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Shu updated GEODE-9311:
----------------------------
Description:
Even after the issue (GEODE-9310) is addressed, the retry issue may still occur
if the JedisCluster client is connected to the node is being shut down.
Here is a test run result:
vm2 gets the command:
[vm2] [warn 2021/05/25 16:01:41.479 PDT server-2 <GeodeRedisServer-Command-1>
tid=0x64] Executing Redis command: ZREM key member1-212 member1-211 member1-214
member1-213 member1-210 member1-209 member1-208 member1-205 member1-204
*** command is executed on the primary ***
[vm2] [warn 2021/05/25 16:01:41.503 PDT server-2 <Function Execution
Processor2> tid=0x51] bucket region is
BucketRegion[path='/__PR/_B__REDIS__DATA_123;serial=200;primary=true] key key
[vm2] java.lang.Exception
[vm2] at
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:533)
[vm2] at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
[vm2] at
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1213)
[vm2] at
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:3024)
[vm2] at
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2236)
[vm2] at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
And distributed to replica (vm1):
vm1] [warn 2021/05/25 16:01:41.518 PDT server-1 <P2P message reader for
192.168.0.14(server-2:83136)<v2>:41002 shared ordered sender uid=7 local
port=58643 remote port=53459> tid=0x59] membersRemoveAll invoked
[vm1] java.lang.Exception
[vm1] at
org.apache.geode.redis.internal.data.RedisSortedSet.membersRemoveAll(RedisSortedSet.java:175)
[vm1] at
org.apache.geode.redis.internal.data.RedisSortedSet.applyDelta(RedisSortedSet.java:89)
[vm1] at
org.apache.geode.redis.internal.data.AbstractRedisData.fromDelta(AbstractRedisData.java:193)
[vm1] at
org.apache.geode.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1841)
[vm1] at
org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1696)
[vm1] at
org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1643)
[vm1] at
org.apache.geode.internal.cache.map.RegionMapPut.updateEntry(RegionMapPut.java:485)
[vm1] at
org.apache.geode.internal.cache.map.RegionMapPut.createOrUpdateEntry(RegionMapPut.java:256)
vm2 is bounced:
[vm2] [info 2021/05/25 16:01:41.526 PDT server-2 <RMI TCP
Connection(1)-192.168.0.14> tid=0x14] Got result: 83136
[vm2] from org.apache.geode.test.dunit.VM$$Lambda$370/1141741369.call with 0
args on object: org.apache.geode.test.dunit.VM$$Lambda$370/1141741369@68ecd55a
(took 0 ms)
[info 2021/05/25 16:01:41.527 PDT <pool-1-thread-2> tid=0x36] Bouncing 2 old
pid is 83136 and version is 10240.0.0
The JedisCluster client did not get response back (possibly detects connection
is gone), and it does retry again, as the test failed with following:
java.util.concurrent.ExecutionException:
redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more
cluster attempts left.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed(ZRemDUnitTest.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.apache.geode.test.junit.rules.serializable.SerializableExternalResource$1.evaluate(SerializableExternalResource.java:38)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.apache.geode.test.dunit.rules.ClusterStartupRule$1.evaluate(ClusterStartupRule.java:138)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:54)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No
more cluster attempts left.
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:84)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:25)
at redis.clients.jedis.JedisCluster.zrem(JedisCluster.java:984)
at
org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.doZRem(ZRemDUnitTest.java:157)
at
org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.lambda$zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed$0(ZRemDUnitTest.java:174)
at
org.apache.geode.test.junit.rules.ExecutorServiceRule.lambda$submit$2(ExecutorServiceRule.java:232)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
was:
Even after the issue (GEODE-9310) is addressed, the retry issue may still occur
if the JedisCluster client is connected to the node is being shut down.
Here is a test run result:
vm2 gets the command:
[vm2] [warn 2021/05/25 16:01:41.479 PDT server-2 <GeodeRedisServer-Command-1>
tid=0x64] Executing Redis command: ZREM key member1-212 member1-211 member1-214
member1-213 member1-210 member1-209 member1-208 member1-205 member1-204
*** command is executed on the primary ***
[vm2] [warn 2021/05/25 16:01:41.503 PDT server-2 <Function Execution
Processor2> tid=0x51] bucket region is
BucketRegion[path='/__PR/_B__REDIS__DATA_123;serial=200;primary=true] key key
[vm2] java.lang.Exception
[vm2] at
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:533)
[vm2] at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
[vm2] at
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1213)
[vm2] at
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:3024)
[vm2] at
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2236)
[vm2] at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
And distributed to replica (vm1):
vm1] [warn 2021/05/25 16:01:41.518 PDT server-1 <P2P message reader for
192.168.0.14(server-2:83136)<v2>:41002 shared ordered sender uid=7 local
port=58643 remote port=53459> tid=0x59] membersRemoveAll invoked
[vm1] java.lang.Exception
[vm1] at
org.apache.geode.redis.internal.data.RedisSortedSet.membersRemoveAll(RedisSortedSet.java:175)
[vm1] at
org.apache.geode.redis.internal.data.RedisSortedSet.applyDelta(RedisSortedSet.java:89)
[vm1] at
org.apache.geode.redis.internal.data.AbstractRedisData.fromDelta(AbstractRedisData.java:193)
[vm1] at
org.apache.geode.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1841)
[vm1] at
org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1696)
[vm1] at
org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1643)
[vm1] at
org.apache.geode.internal.cache.map.RegionMapPut.updateEntry(RegionMapPut.java:485)
[vm1] at
org.apache.geode.internal.cache.map.RegionMapPut.createOrUpdateEntry(RegionMapPut.java:256)
vm2 is bounced:
[vm2] [info 2021/05/25 16:01:41.526 PDT server-2 <RMI TCP
Connection(1)-192.168.0.14> tid=0x14] Got result: 83136
[vm2] from org.apache.geode.test.dunit.VM$$Lambda$370/1141741369.call with 0
args on object: org.apache.geode.test.dunit.VM$$Lambda$370/1141741369@68ecd55a
(took 0 ms)
[info 2021/05/25 16:01:41.527 PDT <pool-1-thread-2> tid=0x36] Bouncing 2 old
pid is 83136 and version is 10240.0.0
The JedisCluster client did not get response back (possibly detects connection
is gone), and it does retry again, as the test failed with following:
redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more
cluster attempts left.
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:84)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
at
redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:25)
at redis.clients.jedis.JedisCluster.zscore(JedisCluster.java:1075)
at
org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.verifyDataNotExist(ZRemDUnitTest.java:194)
at
org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed(ZRemDUnitTest.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.apache.geode.test.junit.rules.serializable.SerializableExternalResource$1.evaluate(SerializableExternalResource.java:38)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.apache.geode.test.dunit.rules.ClusterStartupRule$1.evaluate(ClusterStartupRule.java:138)
> It is possible that JedisCluster client may still retry if the Geode Redis
> server it is connected to shuts down
> ---------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-9311
> URL: https://issues.apache.org/jira/browse/GEODE-9311
> Project: Geode
> Issue Type: Bug
> Components: redis
> Reporter: Eric Shu
> Priority: Major
>
> Even after the issue (GEODE-9310) is addressed, the retry issue may still
> occur if the JedisCluster client is connected to the node is being shut down.
> Here is a test run result:
> vm2 gets the command:
> [vm2] [warn 2021/05/25 16:01:41.479 PDT server-2 <GeodeRedisServer-Command-1>
> tid=0x64] Executing Redis command: ZREM key member1-212 member1-211
> member1-214 member1-213 member1-210 member1-209 member1-208 member1-205
> member1-204
> *** command is executed on the primary ***
> [vm2] [warn 2021/05/25 16:01:41.503 PDT server-2 <Function Execution
> Processor2> tid=0x51] bucket region is
> BucketRegion[path='/__PR/_B__REDIS__DATA_123;serial=200;primary=true] key key
> [vm2] java.lang.Exception
> [vm2] at
> org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:533)
> [vm2] at
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
> [vm2] at
> org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1213)
> [vm2] at
> org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:3024)
> [vm2] at
> org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2236)
> [vm2] at
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
> And distributed to replica (vm1):
> vm1] [warn 2021/05/25 16:01:41.518 PDT server-1 <P2P message reader for
> 192.168.0.14(server-2:83136)<v2>:41002 shared ordered sender uid=7 local
> port=58643 remote port=53459> tid=0x59] membersRemoveAll invoked
> [vm1] java.lang.Exception
> [vm1] at
> org.apache.geode.redis.internal.data.RedisSortedSet.membersRemoveAll(RedisSortedSet.java:175)
> [vm1] at
> org.apache.geode.redis.internal.data.RedisSortedSet.applyDelta(RedisSortedSet.java:89)
> [vm1] at
> org.apache.geode.redis.internal.data.AbstractRedisData.fromDelta(AbstractRedisData.java:193)
> [vm1] at
> org.apache.geode.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1841)
> [vm1] at
> org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1696)
> [vm1] at
> org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1643)
> [vm1] at
> org.apache.geode.internal.cache.map.RegionMapPut.updateEntry(RegionMapPut.java:485)
> [vm1] at
> org.apache.geode.internal.cache.map.RegionMapPut.createOrUpdateEntry(RegionMapPut.java:256)
> vm2 is bounced:
> [vm2] [info 2021/05/25 16:01:41.526 PDT server-2 <RMI TCP
> Connection(1)-192.168.0.14> tid=0x14] Got result: 83136
> [vm2] from org.apache.geode.test.dunit.VM$$Lambda$370/1141741369.call with 0
> args on object:
> org.apache.geode.test.dunit.VM$$Lambda$370/1141741369@68ecd55a (took 0 ms)
> [info 2021/05/25 16:01:41.527 PDT <pool-1-thread-2> tid=0x36] Bouncing 2 old
> pid is 83136 and version is 10240.0.0
> The JedisCluster client did not get response back (possibly detects
> connection is gone), and it does retry again, as the test failed with
> following:
> java.util.concurrent.ExecutionException:
> redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more
> cluster attempts left.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed(ZRemDUnitTest.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.apache.geode.test.junit.rules.serializable.SerializableExternalResource$1.evaluate(SerializableExternalResource.java:38)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.apache.geode.test.dunit.rules.ClusterStartupRule$1.evaluate(ClusterStartupRule.java:138)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:54)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException:
> No more cluster attempts left.
> at
> redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:84)
> at
> redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at
> redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at
> redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at
> redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at
> redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at
> redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:25)
> at redis.clients.jedis.JedisCluster.zrem(JedisCluster.java:984)
> at
> org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.doZRem(ZRemDUnitTest.java:157)
> at
> org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.lambda$zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed$0(ZRemDUnitTest.java:174)
> at
> org.apache.geode.test.junit.rules.ExecutorServiceRule.lambda$submit$2(ExecutorServiceRule.java:232)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
