[jira] [Commented] (GEODE-222) RedisDistDUnitTest testConcOps failed due to JedisDataException
[ https://issues.apache.org/jira/browse/GEODE-222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701673#comment-14701673 ] Vitaliy Gavrilov commented on GEODE-222: I cannot reproduce this with several hundred test runs but I may have found a possible code path for this failure RedisDistDUnitTest testConcOps failed due to JedisDataException --- Key: GEODE-222 URL: https://issues.apache.org/jira/browse/GEODE-222 Project: Geode Issue Type: Bug Affects Versions: 1.0.0-incubating Reporter: Kirk Lund Assignee: Vitaliy Gavrilov RedisDistDUnitTest testConcOps failed in nightly build due to JedisDataException: ERR The server had an internal error please try again https://builds.apache.org/view/All/job/Geode-nightly/186/testReport/junit/com.gemstone.gemfire.redis/RedisDistDUnitTest/testConcOps/ {code:java} java.lang.Exception: An exception occured during async invocation at dunit.AsyncInvocation.getResult(AsyncInvocation.java:175) at com.gemstone.gemfire.redis.RedisDistDUnitTest.testConcOps(RedisDistDUnitTest.java:224) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:252) at junit.framework.TestSuite.run(TestSuite.java:247) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50) at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106) at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360) at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: dunit.RMIException: While invoking com.gemstone.gemfire.redis.RedisDistDUnitTest$1ConcOps.call in VM 2 running on Host asf911.gq1.ygridcore.net with 4 VMs at dunit.VM.invoke(VM.java:359) at dunit.VM$2.run(VM.java:211) at java.lang.Thread.run(Thread.java:745) at dunit.AsyncInvocation.run(AsyncInvocation.java:200) Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR The server had an internal error please try again at redis.clients.jedis.Protocol.processError(Protocol.java:117) at
[jira] [Commented] (GEODE-77) Replace JGroups 2.2.9
[ https://issues.apache.org/jira/browse/GEODE-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701686#comment-14701686 ] ASF subversion and git services commented on GEODE-77: -- Commit d4a3f122555ee6daa15c6aea38adea30bc1a057d in incubator-geode's branch refs/heads/feature/GEODE-77 from [~bschuchardt] [ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=d4a3f12 ] GEODE-77 removed shutdown status check. This check was recently added but it is keeping the manager from properly shutting down because the status is set by DirectChannel before the manager is told to shut down. Replace JGroups 2.2.9 - Key: GEODE-77 URL: https://issues.apache.org/jira/browse/GEODE-77 Project: Geode Issue Type: Bug Reporter: Bruce Schuchardt Assignee: Bruce Schuchardt Priority: Blocker Fix For: 1.0.0-incubating Attachments: GEODE-MembershipManagerFunctionalSpecification-130715-1604-29054.pdf The JGroups 2.2.9 sources that are currently included in Geode must be replaced in order for Geode to leave incubation. A wiki document has been created to investigate alternatives. https://cwiki.apache.org/confluence/display/GEODE/Replacing+JGroups+2.2.9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEODE-77) Replace JGroups 2.2.9
[ https://issues.apache.org/jira/browse/GEODE-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702029#comment-14702029 ] ASF subversion and git services commented on GEODE-77: -- Commit c5c8565c654ad681c7171c7fad582e113cb8e720 in incubator-geode's branch refs/heads/feature/GEODE-77 from [~bschuchardt] [ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=c5c8565 ] GEODE-77 fixing a hang caused by wrong DSFID in RemoveMemberMessage also addressing an authentication issue found by Jason with the new GMSJoinLeaveJUnitTest, and adding a test method for removeMember() Replace JGroups 2.2.9 - Key: GEODE-77 URL: https://issues.apache.org/jira/browse/GEODE-77 Project: Geode Issue Type: Bug Reporter: Bruce Schuchardt Assignee: Bruce Schuchardt Priority: Blocker Fix For: 1.0.0-incubating Attachments: GEODE-MembershipManagerFunctionalSpecification-130715-1604-29054.pdf The JGroups 2.2.9 sources that are currently included in Geode must be replaced in order for Geode to leave incubation. A wiki document has been created to investigate alternatives. https://cwiki.apache.org/confluence/display/GEODE/Replacing+JGroups+2.2.9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEODE-223) RedisDistDUnitTest.testConcCreateDestroy NullPointerException
[ https://issues.apache.org/jira/browse/GEODE-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Bawaskar resolved GEODE-223. Resolution: Fixed RedisDistDUnitTest.testConcCreateDestroy NullPointerException - Key: GEODE-223 URL: https://issues.apache.org/jira/browse/GEODE-223 Project: Geode Issue Type: Bug Components: extensions Reporter: Darrel Schneider Assignee: Vitaliy Gavrilov com.gemstone.gemfire.redis.RedisDistDUnitTest.testConcCreateDestroy failed because of this suspect string: Found suspect string in log4j at line 1132 [error 2015/08/11 11:39:31.020 PDT P2P message reader for cc3-rh6(17559)v0:13796 shared ordered uid=236 port=51035 tid=0x1506] Exception occurred in CacheListener java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1124) at com.gemstone.gemfire.internal.redis.RegionProvider.createRemoteRegionLocally(RegionProvider.java:222) at com.gemstone.gemfire.redis.GemFireRedisServer.afterKeyCreate(GemFireRedisServer.java:532) at com.gemstone.gemfire.redis.GemFireRedisServer.access$400(GemFireRedisServer.java:128) at com.gemstone.gemfire.redis.GemFireRedisServer$MetaCacheListener.afterCreate(GemFireRedisServer.java:559) at com.gemstone.gemfire.internal.cache.EnumListenerEvent$AFTER_CREATE.dispatchEvent(EnumListenerEvent.java:97) at com.gemstone.gemfire.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:9251) at com.gemstone.gemfire.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:7700) at com.gemstone.gemfire.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:6466) at com.gemstone.gemfire.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2623) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:157) at com.gemstone.gemfire.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:6316) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:3214) at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:6125) at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:394) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:118) at com.gemstone.gemfire.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:6100) at com.gemstone.gemfire.internal.cache.AbstractUpdateOperation.doPutOrCreate(AbstractUpdateOperation.java:142) at com.gemstone.gemfire.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.basicOperateOnRegion(AbstractUpdateOperation.java:280) I think the bug is in this code from RegionProvider: {code} r = cache.getRegion(key.toString()); if (type == RedisDataType.REDIS_LIST) doInitializeList(key, r); else if (type == RedisDataType.REDIS_SORTEDSET) doInitializeSortedSet(key, r); this.regions.put(key, r); {code} The NPE happens on the last line (the put call) because r is null. I think this code just needs to handle the race in which cache.getRegion returns null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEODE-223) RedisDistDUnitTest.testConcCreateDestroy NullPointerException
[ https://issues.apache.org/jira/browse/GEODE-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702245#comment-14702245 ] ASF subversion and git services commented on GEODE-223: --- Commit ff9b2f05ffa6312e3fcd3abc2b7bc96077870829 in incubator-geode's branch refs/heads/develop from Vito Gavrilov [ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=ff9b2f0 ] GEODE-223: Handle region create/destroy remote event in Redis adpater Ignore events where region creation initiated remotely attempts to create a local region reference when the region has already been destroyed. Also, the destruction of a region may be caught the query engine, so I have accounted for that by handling com.gemstone.gemfire.cache.query.RegionNotFoundException. Finally, a the Jedis client timeout has been increased for RedisDistDunitTest to account for concurrent region creation/destruction and an expected exception has been added to not fail over the log scanning. Sometimes when a region is destroyed the PooledMessage Processor will log a regiondestroyed exception, which is ok, but makes the test fail. closes #16 RedisDistDUnitTest.testConcCreateDestroy NullPointerException - Key: GEODE-223 URL: https://issues.apache.org/jira/browse/GEODE-223 Project: Geode Issue Type: Bug Components: extensions Reporter: Darrel Schneider Assignee: Vitaliy Gavrilov com.gemstone.gemfire.redis.RedisDistDUnitTest.testConcCreateDestroy failed because of this suspect string: Found suspect string in log4j at line 1132 [error 2015/08/11 11:39:31.020 PDT P2P message reader for cc3-rh6(17559)v0:13796 shared ordered uid=236 port=51035 tid=0x1506] Exception occurred in CacheListener java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1124) at com.gemstone.gemfire.internal.redis.RegionProvider.createRemoteRegionLocally(RegionProvider.java:222) at com.gemstone.gemfire.redis.GemFireRedisServer.afterKeyCreate(GemFireRedisServer.java:532) at com.gemstone.gemfire.redis.GemFireRedisServer.access$400(GemFireRedisServer.java:128) at com.gemstone.gemfire.redis.GemFireRedisServer$MetaCacheListener.afterCreate(GemFireRedisServer.java:559) at com.gemstone.gemfire.internal.cache.EnumListenerEvent$AFTER_CREATE.dispatchEvent(EnumListenerEvent.java:97) at com.gemstone.gemfire.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:9251) at com.gemstone.gemfire.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:7700) at com.gemstone.gemfire.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:6466) at com.gemstone.gemfire.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2623) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:157) at com.gemstone.gemfire.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:6316) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:3214) at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:6125) at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:394) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:118) at com.gemstone.gemfire.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:6100) at com.gemstone.gemfire.internal.cache.AbstractUpdateOperation.doPutOrCreate(AbstractUpdateOperation.java:142) at com.gemstone.gemfire.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.basicOperateOnRegion(AbstractUpdateOperation.java:280) I think the bug is in this code from RegionProvider: {code} r = cache.getRegion(key.toString()); if (type == RedisDataType.REDIS_LIST) doInitializeList(key, r); else if (type == RedisDataType.REDIS_SORTEDSET) doInitializeSortedSet(key, r); this.regions.put(key, r); {code} The NPE happens on the last line (the put call) because r is null. I think this code just needs to handle the race in which cache.getRegion returns null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEODE-229) Javadocs for DiskStoreFactory.setCompactionThreshold are incorrect
Dan Smith created GEODE-229: --- Summary: Javadocs for DiskStoreFactory.setCompactionThreshold are incorrect Key: GEODE-229 URL: https://issues.apache.org/jira/browse/GEODE-229 Project: Geode Issue Type: Bug Reporter: Dan Smith Assignee: Dan Smith The javadocs for compaction-threshold imply that we compact as soon as the amount of garbage is equal or greater than the threshold. From DiskStoreFactory?.setCompactionThreshold {noformat} When the amount of garbage in an oplog exceeds this percentage then when a compaction * is done this garbage will be cleaned up freeing up disk space. Garbage is created by * entry destroys, entry updates, and region destroys. {noformat} However, looking at the code, it looks like we actually trigger compaction when the amount of non-garbage data is equal or less than the threshold: Here Oplog.needsCompaction, it looks like its saying the oplog needs compaction if the live count is less than the threshold. {code} long rvHWMtmp = this.totalCount.get(); if (rvHWMtmp 0) { long tlc = this.totalLiveCount.get(); if (tlc 0) { tlc = 0; } double rv = tlc; double rvHWM = rvHWMtmp; if (((rv / rvHWM) * 100) = parent.getCompactionThreshold()) { return true; } } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEODE-77) Replace JGroups 2.2.9
[ https://issues.apache.org/jira/browse/GEODE-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701538#comment-14701538 ] ASF subversion and git services commented on GEODE-77: -- Commit 26fea7a161f2336aa5577b0f4b2126d11c9ae37f in incubator-geode's branch refs/heads/feature/GEODE-77 from [~bschuchardt] [ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=26fea7a ] GEODE-77 added Messenger statistics and removed old JGroups statistics This also fixes a few bugs that I found during testing. GMSMember wasn't serializing correctly in some cases. I also found that gfsh has a showDeadlock command and hooked in the new findDeepestGraph DependencyGraph search. If gfsh can't find a deadlock it will now report on the deepest call chain it can find, which often points to the source of a problem in the distributed system. Replace JGroups 2.2.9 - Key: GEODE-77 URL: https://issues.apache.org/jira/browse/GEODE-77 Project: Geode Issue Type: Bug Reporter: Bruce Schuchardt Assignee: Bruce Schuchardt Priority: Blocker Fix For: 1.0.0-incubating Attachments: GEODE-MembershipManagerFunctionalSpecification-130715-1604-29054.pdf The JGroups 2.2.9 sources that are currently included in Geode must be replaced in order for Geode to leave incubation. A wiki document has been created to investigate alternatives. https://cwiki.apache.org/confluence/display/GEODE/Replacing+JGroups+2.2.9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEODE-227) Extract library versions into a build property file
Anthony Baker created GEODE-227: --- Summary: Extract library versions into a build property file Key: GEODE-227 URL: https://issues.apache.org/jira/browse/GEODE-227 Project: Geode Issue Type: Bug Components: build Reporter: Anthony Baker Assignee: Mark Bretl Priority: Minor We have disabled transitive dependencies and declare individual library versions throughout the build files. We should extract library versions into property files to make it easy to update (or override) library versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEODE-226) JSON seems to loose time portion on getObject
[ https://issues.apache.org/jira/browse/GEODE-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk Lund updated GEODE-226: Assignee: Hitesh Khamesra JSON seems to loose time portion on getObject - Key: GEODE-226 URL: https://issues.apache.org/jira/browse/GEODE-226 Project: Geode Issue Type: Bug Components: core Reporter: Konstantin Ignatyev Assignee: Hitesh Khamesra com.gemstone.gemfire.pdx.internal.PdxInstanceImpl#getObject Date format, in the JSON land it is pretty much settled to be ISO 8661 https://weblog.west-wind.com/posts/2014/Jan/06/JavaScript-JSON-Date-Parsing-and-real-Dates It would be nice to be able to have Geode’s JSON standard compliant, or have this configurable. Otherwise the we will be loosing time portion of date-s public Object getObject() { if (getPdxType().getNoDomainClass()) { //In case of Developer Rest APIs, All PdxInstances converted from Json will have a className =__GEMFIRE_JSON. //Following code added to convert Json/PdxInstance into the Java object. if(this.getClassName().equals(__GEMFIRE_JSON)){ //introspect the JSON, does the @type meta-data exist. String className = extractTypeMetaData(); if(StringUtils.hasText(className)) { try { ObjectMapper mapper = new ObjectMapper(); mapper.setDateFormat(new SimpleDateFormat(MM/dd/)); mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); mapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true); String JSON = JSONFormatter.toJSON(this); Object classInstance = mapper.readValue(JSON, ClassPathLoader.getLatest().forName(className)); return classInstance; }catch(Exception e){ throw new PdxSerializationException(Could not deserialize as java class type could not resolved, e); } } } return this; } Also this method is not that performant, please see #225 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEODE-225) excessive CPU utilization and garbage collection strain for JSON processing
[ https://issues.apache.org/jira/browse/GEODE-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk Lund updated GEODE-225: Assignee: Hitesh Khamesra excessive CPU utilization and garbage collection strain for JSON processing --- Key: GEODE-225 URL: https://issues.apache.org/jira/browse/GEODE-225 Project: Geode Issue Type: Improvement Components: core Affects Versions: 1.0.0-incubating Reporter: Konstantin Ignatyev Assignee: Hitesh Khamesra I have been looking at Geode-s code and come across major performance killer for JSON handling in Geode, namely implementation of com.gemstone.gemfire.pdx.internal.PdxInstanceImpl#getObject as you can see in the snipped below the code creates ObjectMapper every time it needs to convert PDX instance into JSON. According to docs and examples on Jackson’s site instances of ObjectMapper should be shared globally. Creating it for every transaction is quite expensive in terms of CPU and garbage collection. public Object getObject() { if (getPdxType().getNoDomainClass()) { //In case of Developer Rest APIs, All PdxInstances converted from Json will have a className =__GEMFIRE_JSON. //Following code added to convert Json/PdxInstance into the Java object. if(this.getClassName().equals(__GEMFIRE_JSON)){ //introspect the JSON, does the @type meta-data exist. String className = extractTypeMetaData(); if(StringUtils.hasText(className)) { try { ObjectMapper mapper = new ObjectMapper(); mapper.setDateFormat(new SimpleDateFormat(MM/dd/)); mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); mapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true); String JSON = JSONFormatter.toJSON(this); Object classInstance = mapper.readValue(JSON, ClassPathLoader.getLatest().forName(className)); return classInstance; }catch(Exception e){ throw new PdxSerializationException(Could not deserialize as java class type could not resolved, e); } } } return this; } -- This message was sent by Atlassian JIRA (v6.3.4#6332)