Hi, Ilya, I've tried to reproduce deployment problem in standalone project involving Ingnite.start() in several WAR files. But this test is still passing.
It is still possible deployment problem can be reprdoced only when GridDeploymentClassLoader is used for loading class on server, and several different Web App class loaders are enabled on clients. Do you have standalone reproducer you can share? Best Regards, Dmitry Pavlov пт, 12 мая 2017 г. в 15:58, Ilya <[email protected]>: > Hi all! > > The question was originally asked (but not answered) on SO: > > http://stackoverflow.com/questions/43803402/how-does-peer-classloading-work-in-apache-ignite > > In short, we have "Failed to deploy user message" exceptions under high > load > in our project. > > Here is an overview of our architecture: > - Distributed cache on three nodes, all nodes run on a single workstation > (in this test); > - Workers on each node; > - Messaging between workers is done using IgniteMessaging (topic has the > type of String and I've tried both byte[] and ByteBuffer as a message > class); > - Client connects to the cluster and triggers some business logic, that > causes cross-node messaging, scan queries and MR jobs (using > IgniteCompute::broadcast). All of these may performed concurrently. > > I've tried both SHARED and CONTINUOUS deployment mode, but the result > remains the same. > > I've noticed lots of similar messages in the logs: > /2017-05-05 13:31:28 INFO org.apache.ignite.logger.java.JavaLogger info > Removed undeployed class: GridDeployment [ts=1493980288578, > depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@38815daa, > clsLdrId=36c3828db51-0d65e7d5-77bf-444d-9b8b-d18bde94ad13, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=true, usage=0] > ... > 2017-05-05 13:31:29 INFO org.apache.ignite.logger.java.JavaLogger info > Removed undeployed class: GridDeployment [ts=1493980289125, > depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@355f6680, > clsLdrId=1dd3828db51-1b20df7a-a98d-45a3-8ab6-e5d229945830, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=true, usage=0] > .../ > > This happens when I use ByteBuffer as message type. In case of byte[], > class > B[ is being constantly re-deployed. > > ScanQuery predicate and IgniteCompute caller are also being constantly > re-deployed. > If we disable ScanQueries and IgniteCompute broadcasts - all is fine, there > are no re-deployments. > > For the further testing I've disabled MRs and kept ScanQueries. I've also > added some debug output to a fresh snapshot of Ignite 2.1.0. Messages > "Class > locally deployed: <my ScanQuery predicate>" usually come from the following > call stack: > > /org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.recordDeploy(GridDeploymentLocalStore.java:404) > at > > org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.deploy(GridDeploymentLocalStore.java:333) > at > > org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.getDeployment(GridDeploymentLocalStore.java:201) > at > > org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getLocalDeployment(GridDeploymentManager.java:383) > at > > org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getDeployment(GridDeploymentManager.java:345) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.injectResources(GridCacheQueryManager.java:918) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.scanIterator(GridCacheQueryManager.java:826) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.executeQuery(GridCacheQueryManager.java:611) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.queryResult(GridCacheQueryManager.java:1593) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1164) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:231) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109) > at > > org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107) > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863) > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386) > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308) > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100) > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885) > at > > org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:114) > at > > org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:802) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)/ > > Messages like "removed undeployed class" usually come from the > IgniteMessaging's call stack. > > I've analyzed the Ignite kernel a bit, and got a suspicion that undeploy is > being triggered for all classes in a classloader, when at least one class > that resides in that classloader was re-deployed in some other loader. > > It happens inside > org.apache.ignite.spi.deployment.local.LocalDeploymentSpi#register > > At first, we get a "Map of new resources added for registered class > loader" using LocalDeploymentSpi#addResource. > Then we "Remove resources for all class loaders except {@code > ignoreClsLdr}." using LocalDeploymentSpi#removeResources. Inside this > method, it looks like we add all loaders that contain the old version of > the > new resource to a "doomed" collection. > Finally, we iterate this collection and call onClassLoaderReleased for > each element. The latter action actually causes all the classes to be > undeployed (finally causing the "Removed undeployed class" messages). > > I don't understand this concept. Why are there multiple classloaders? Why > do > we undeploy the whole classloader in such cases? > > I'd be grateful, if someone could explain, how does peer classloading work > in Ignite "under the hood". > > P. S. I'm looking at the sources of a fresh snapshot of Ignite 2.1.0, but > the behavior is the same with the standard Ignite 1.9.0. > > P. P. S. Unfortunately, I've did not manage to reproduce this issue outside > of our project yet. > > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
