Re: [Discussion] - ClassLoaderService RFC proposal
Hi there Alberto, The ClassLoader Isolation work is currently still in development. Nothing is in the main code-branch yet. But you are correct. 2 versions. DefaultClassLoaderService will be the standard Java ClassLoader functionality as we are currently used to. The other, JBoss Modules based. Until the ClassLoader Isolation work is completed, the DefaultClassLoaderService will make sure the system works as-is. No changes required. The JBoss modules based ClassLoaderService will initially opt-in under the flag, until we find it to be stable enough to make it the default. But we really want this swapping of ClassLoaders to be simple and seamless. The reason for this RFC is to lay the ground work into the main code base, that would allow us to plug-in a different ClassLoaderService into the main code base. At this stage, from a prototype we’ve done, this “simple” change will touch in excess of 200 classes. Mostly only tests. But you can imagine if this work would be in the same PR as the ClassLoader Isolation work, the PR review would be the worst. Not only would you have to understand the changes suggested / implemented, but also all the classes that have simple, single changes in them. But that does not say that the PR for this RFC would be simple to review either.. As we’d still have the 200+ classes that are touched. —Udo On Sep 15, 2020, 10:33 PM +1000, Alberto Gomez , wrote: Nice proposal, Udo. Here come some questions: Is the ClassLoader isolation RFC implemented? I have not seen any references to it in the doc or code. To me this RFC seems like a part of the ClassLoader isolation RFC as, without it, the original one would not work completely. Is this right? If I understand correctly, to start with there will be two implementations of the ClassLoaderService, the DefaultClassLoaderService (with the current ClassLoader functionality) and another one with the modular ClassLoader functionality as provided by JBoss modules. The latter one will be used when the --experimental-classloader flag is used in GFSH. Is this right? Thanks, -Alberto G. From: Udo Kohlmeyer Sent: Monday, September 14, 2020 12:42 PM To: geode Subject: [Discussion] - ClassLoaderService RFC proposal Hi there Apache Geode Devs, (try 2) Please find attached a proposal for a ClassLoaderService. Please review and ponder on it. https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FGEODE%2FIntroduction%2Bof%2BClassLoaderService%2Binto%2BGeodedata=02%7C01%7Cudo%40vmware.com%7C0f20f50e1cb14ae271d308d859738aec%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637357700083700083sdata=lFqZF71ZAVNvEo4FytSqZzpwKAcHuYzj7uuBRiCQ10U%3Dreserved=0 All comments are please to be made in this mail thread. —Udo
Re: Colocated regions missing some buckets after restart
Mario, I doubt this has anything to do with the client connections. If it is it should be between server/member to server/member connection; in that case the unresponsive member is kicked out from the cluster. The recommended configuration is to have persistence regions for both parent and co-located regions (and replicated regions)... There could be issues in the stats too...Can you try executing a test/validation code on server side to dump/list primary and secondary buckets. You can do that using helper methods: pr.getDataStore().getAllLocalPrimaryBucketIds(); -Anil On 9/14/20, 12:25 AM, "Mario Kevo" wrote: Hi, This problem is usually seen only on 1 server. The other servers metrics and bucket count looks fine. Another symptom of this issue is that the max-connections limit is reached on the problematic server if we have a client that tries to reconnect after the server restart. Clients simply get no response from the server so they try to close the connection, but the connection close is not acknowledged by the server. On server side we see that the connections are in CLOSE-WAIT state with packets in the socket receiver queue. It’s as if the servers just stopped processing packets on the sockets while waiting for a member with the primary bucket. So in short, each new client connection is “unresponsive”. The client tries to close it a open a new one, but the socket doesn’t get closed on server side and the connection is left “hanging” on the server. Clients will try to do this until max-connections is reached on the servers. This is why we would be unable to add any data to the regions. But IMHO it’s really not dependent on adding data, since this issue happens occasionally (1 out of ~4 restarts) and only on one server. The initial problem was observed with a persistent region A (with 1 key-value pairs inserted) and a non-persistent region B collocated with region A. We did some tests with both regions being persistent. We haven’t observed the same issue yet (although we did only a few restarts), but we observed something that also looks quite worrying. Both servers start up without reporting issues in the logs. But, looking at the server metrics, one server has wrong information about “bucketCount” and is missing primary buckets. E.g: First server: Partition | putLocalRate | 0.0 | putRemoteRate| 0.0 | putRemoteLatency | 0 | putRemoteAvgLatency | 0 | bucketCount | 113 | primaryBucketCount | 57 Second server: Partition | putLocalRate | 0.0 | putRemoteRate| 0.0 | putRemoteLatency | 0 | putRemoteAvgLatency | 0 | bucketCount | 111 | primaryBucketCount | 55 So we are missing a primary bucket without being aware of the issue. BR, Mario Šalje: Anilkumar Gingade Poslano: 11. rujna 2020. 20:34 Prima: dev@geode.apache.org Predmet: Re: Colocated regions missing some buckets after restart Are you seeing no-buckets for persistent regions or non-persistent. The buckets are created dynamically; when data is added to corresponding buckets... When server is restarted, in case of in-memory regions as the data is not there, the bucket region may not have been created (my suspicion). Can you try adding data and see if the co-located bucket region gets created in respective nodes/server. -Anil. On 9/11/20, 9:46 AM, "Mario Kevo" wrote: Hi geode-dev, We have a system with two servers and a few regions. One region is persistent and other are not but they are colocated with this persistent region. After servers restart on some region we can see that they don't have any bucket. gfsh>show metrics --member=server-1 --region=/region1 --categories=partition Metrics for region:/region1 On Member server-1 Category |Metric| Value - | | - partition | putLocalRate | 0.0 | putRemoteRate| 0.0 | putRemoteLatency | 0 | putRemoteAvgLatency | 0 | bucketCount | 0 | primaryBucketCount | 0 | configuredRedundancy | 1 | actualRedundancy | 0 | numBucketsWithoutRedundancy | 113 | totalBucketSize | 0 gfsh>show metrics --member=server-0 --region=/region1 --categories=partition Metrics for region:/region1 On Member server-0 Category |Metric| Value - |
Re: [Discussion] - ClassLoaderService RFC proposal
Nice proposal, Udo. Here come some questions: Is the ClassLoader isolation RFC implemented? I have not seen any references to it in the doc or code. To me this RFC seems like a part of the ClassLoader isolation RFC as, without it, the original one would not work completely. Is this right? If I understand correctly, to start with there will be two implementations of the ClassLoaderService, the DefaultClassLoaderService (with the current ClassLoader functionality) and another one with the modular ClassLoader functionality as provided by JBoss modules. The latter one will be used when the --experimental-classloader flag is used in GFSH. Is this right? Thanks, -Alberto G. From: Udo Kohlmeyer Sent: Monday, September 14, 2020 12:42 PM To: geode Subject: [Discussion] - ClassLoaderService RFC proposal Hi there Apache Geode Devs, (try 2) Please find attached a proposal for a ClassLoaderService. Please review and ponder on it. https://cwiki.apache.org/confluence/display/GEODE/Introduction+of+ClassLoaderService+into+Geode All comments are please to be made in this mail thread. —Udo
Re: [Discussion] - ClassLoaderService RFC proposal
Looks good to me Udo, thanks for putting it together!. On Mon, 14 Sep 2020 at 11:42, Udo Kohlmeyer wrote: > Hi there Apache Geode Devs, (try 2) > > Please find attached a proposal for a ClassLoaderService. Please review > and ponder on it. > > > https://cwiki.apache.org/confluence/display/GEODE/Introduction+of+ClassLoaderService+into+Geode > > All comments are please to be made in this mail thread. > > —Udo > -- Ju@N
