Hi Dmitry! Do I understand correct that GridDeploymentClassLoader is always on server used if peer classloading is enabled?
I've did not reproduce the re-deployment issue, but maybe have found another, which is related to deployment. Here is the code: https://github.com/yakupov/IgniteDeploymentTest How to execute: 1) mvn package in the root; 2) execute JUnit test IgnitionRunner#testJetty (it's in the ITest module). The result is: SEVERE: Failed to process message [senderId=98138e20-8fe4-4750-8281-a92b2067fdcb, messageType=class o.a.i.i.processors.cache.query.GridCacheQueryRequest] java.lang.ClassCastException: LocatorBiPredicate cannot be cast to org.apache.ignite.lang.IgniteBiPredicate at org.apache.ignite.internal.processors.cache.query.GridCacheQueryRequest.finishUnmarshal(GridCacheQueryRequest.java:324) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1298) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:364) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850) at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108) at org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428) at java.lang.Thread.run(Thread.java:745) where import org.apache.ignite.lang.IgniteBiPredicate; public class LocatorBiPredicate implements IgniteBiPredicate<String, Locator> {...} On Tue, May 16, 2017 at 12:51 PM, Dmitry Pavlov [via Apache Ignite Users] < [email protected]> wrote: > Hi Ilya, > > I supposed following: GridDeploymentClassLoader usage for loading message > class on receiver side is mandatory condition for reproduction. But for now > it is only hypothesis. > > In my test if all nodes have message payload class in its classpath there > are no problems with messaging under high load (around 10^6 of messages in > 32 threads). Same is true if application class loader is used in for > loading message class (no WAR files & no Jetty). > > Could you please check class loaders used at sender(s) sides and on > receiver side. Is GridDeploymentClassLoader used for loading message > payload class at receiver? > > Sincerely, > Dmitry Pavlov > > > > вт, 16 мая 2017 г. в 12:17, Ilya <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=12881&i=0>>: > >> Hi Dmitry, >> >> Unfortunately, I've did not yet manage to reproduce this issue outside of >> our project. >> >> What do you mean by "GridDeploymentClassLoader is used for loading class >> on server"? How can server classloaders be configured? How does a remote >> node choose, in which classloader to deploy the received class? >> >> My test configuration is as follows: >> >> - Web application that has a single cache and has two IgniteMessaging >> local listeners; >> - It's run from JUnit test code under Jetty; >> - The three instances form a cluster; >> - One client sends messages to one of the topics (to the random node) >> using IgniteMessaging. >> >> All of these works on a single JVM. I suggest that Jetty servers might >> use dedicated classloaders for each web-app. I've tried to launch client >> from both test code and another web-app under Jetty, but that did not >> change anything. >> >> In fact, the failing test on our production application is launched in >> the same manner: two exploded web-apps (client and server), three instances >> of server app and all of these is run under Jettys in a single VM... >> >> On Sun, May 14, 2017 at 2:22 PM, Dmitry Pavlov [via Apache Ignite Users] >> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=12879&i=0>> >> wrote: >> Hi, Ilya, >> >> I've tried to reproduce deployment problem in standalone project >> involving Ingnite.start() in several WAR files. But this test is still >> passing. >> >> It is still possible deployment problem can be reprdoced only >> when GridDeploymentClassLoader is used for loading class on server, and >> several different Web App class loaders are enabled on clients. >> >> Do you have standalone reproducer you can share? >> >> Best Regards, >> Dmitry Pavlov >> >> >> >> пт, 12 мая 2017 г. в 15:58, Ilya <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=12831&i=0>>: > Hi all! > > The question was originally asked (but not answered) on SO: > http://stackoverflow.com/questions/43803402/how-does- > peer-classloading-work-in-apache-ignite > > In short, we have "Failed to deploy user message" exceptions under high > load > in our project. > > Here is an overview of our architecture: > - Distributed cache on three nodes, all nodes run on a single workstation > (in this test); > - Workers on each node; > - Messaging between workers is done using IgniteMessaging (topic has the > type of String and I've tried both byte[] and ByteBuffer as a message > class); > - Client connects to the cluster and triggers some business logic, that > causes cross-node messaging, scan queries and MR jobs (using > IgniteCompute::broadcast). All of these may performed concurrently. > > I've tried both SHARED and CONTINUOUS deployment mode, but the result > remains the same. > > I've noticed lots of similar messages in the logs: > /2017-05-05 13:31:28 INFO org.apache.ignite.logger.java.JavaLogger info > Removed undeployed class: GridDeployment [ts=1493980288578, > depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@38815daa, > clsLdrId=36c3828db51-0d65e7d5-77bf-444d-9b8b-d18bde94ad13, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=true, usage=0] > ... > 2017-05-05 13:31:29 INFO org.apache.ignite.logger.java.JavaLogger info > Removed undeployed class: GridDeployment [ts=1493980289125, > depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@355f6680, > clsLdrId=1dd3828db51-1b20df7a-a98d-45a3-8ab6-e5d229945830, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=true, usage=0] > .../ > > This happens when I use ByteBuffer as message type. In case of byte[], > class > B[ is being constantly re-deployed. > > ScanQuery predicate and IgniteCompute caller are also being constantly > re-deployed. > If we disable ScanQueries and IgniteCompute broadcasts - all is fine, there > are no re-deployments. > > For the further testing I've disabled MRs and kept ScanQueries. I've also > added some debug output to a fresh snapshot of Ignite 2.1.0. Messages > "Class > locally deployed: <my ScanQuery predicate>" usually come from the following > call stack: > /org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore. > recordDeploy(GridDeploymentLocalStore.java:404) > at > org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore. > deploy(GridDeploymentLocalStore.java:333) > at > org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore. > getDeployment(GridDeploymentLocalStore.java:201) > at > org.apache.ignite.internal.managers.deployment.GridDeploymentManager. > getLocalDeployment(GridDeploymentManager.java:383) > at > org.apache.ignite.internal.managers.deployment.GridDeploymentManager. > getDeployment(GridDeploymentManager.java:345) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager. > injectResources(GridCacheQueryManager.java:918) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager. > scanIterator(GridCacheQueryManager.java:826) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager. > executeQuery(GridCacheQueryManager.java:611) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager. > queryResult(GridCacheQueryManager.java:1593) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager. > runQuery(GridCacheQueryManager.java:1164) > at > org.apache.ignite.internal.processors.cache.query. > GridCacheDistributedQueryManager.processQueryRequest( > GridCacheDistributedQueryManager.java:231) > at > org.apache.ignite.internal.processors.cache.query. > GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManag > er.java:109) > at > org.apache.ignite.internal.processors.cache.query. > GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManag > er.java:107) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager. > processMessage(GridCacheIoManager.java:863) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0( > GridCacheIoManager.java:386) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager. > handleMessage(GridCacheIoManager.java:308) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000( > GridCacheIoManager.java:100) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1. > onMessage(GridCacheIoManager.java:253) > at > org.apache.ignite.internal.managers.communication. > GridIoManager.invokeListener(GridIoManager.java:1257) > at > org.apache.ignite.internal.managers.communication.GridIoManager. > processRegularMessage0(GridIoManager.java:885) > at > org.apache.ignite.internal.managers.communication. > GridIoManager.access$2100(GridIoManager.java:114) > at > org.apache.ignite.internal.managers.communication.GridIoManager$7.run( > GridIoManager.java:802) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617)/ > > Messages like "removed undeployed class" usually come from the > IgniteMessaging's call stack. > > I've analyzed the Ignite kernel a bit, and got a suspicion that undeploy is > being triggered for all classes in a classloader, when at least one class > that resides in that classloader was re-deployed in some other loader. > > It happens inside > org.apache.ignite.spi.deployment.local.LocalDeploymentSpi#register > > At first, we get a "Map of new resources added for registered class > loader" using LocalDeploymentSpi#addResource. > Then we "Remove resources for all class loaders except {@code > ignoreClsLdr}." using LocalDeploymentSpi#removeResources. Inside this > method, it looks like we add all loaders that contain the old version of > the > new resource to a "doomed" collection. > Finally, we iterate this collection and call onClassLoaderReleased for > each element. The latter action actually causes all the classes to be > undeployed (finally causing the "Removed undeployed class" messages). > > I don't understand this concept. Why are there multiple classloaders? Why > do > we undeploy the whole classloader in such cases? > > I'd be grateful, if someone could explain, how does peer classloading work > in Ignite "under the hood". > > P. S. I'm looking at the sources of a fresh snapshot of Ignite 2.1.0, but > the behavior is the same with the standard Ignite 1.9.0. > > P. P. S. Unfortunately, I've did not manage to reproduce this issue outside > of our project yet. > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Understanding-the-mechanics- > of-peer-class-loading-tp12661.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-ignite-users.70518.x6.nabble.com/ > Understanding-the-mechanics-of-peer-class-loading-tp12661p12831.html > To unsubscribe from Understanding the mechanics of peer class loading, click > here. > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > ------------------------------ > View this message in context: Re: Understanding the mechanics of peer > class loading > <http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12879.html> > Sent from the Apache Ignite Users mailing list archive > <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-ignite-users.70518.x6.nabble.com/ > Understanding-the-mechanics-of-peer-class-loading-tp12661p12881.html > To unsubscribe from Understanding the mechanics of peer class loading, click > here > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=12661&code=aXlha3Vwb3Y5M0BnbWFpbC5jb218MTI2NjF8LTQ4NzEwMDkzOQ==> > . > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12961.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
