Hi Stan, Thanks for the info! Indeed this was a very big mistake and your explanation made it clear what the error was.
I was passing an instance of the grpc server to the compute job, a part of the code which I had neglected to add in the description above. Cheers! On Mon, Aug 26, 2019 at 7:21 PM Stanislav Lukyanov <[email protected]> wrote: > The message "Failed to deserialize object > [typeName=io.grpc.internal.InternalHandlerRegistry]" > means InternalHandlerRegistry is being sent between the nodes - which it > shouldn't be. > What you need to do is to find where is it being sent. You shouldn't pass > any gRPC to any Ignite configuration, or, I believe, to any Ignite method. > Also, be careful with compute jobs that use lambdas and anonymous classes. > By their nature they catch objects around them too easy, so often you'll > see a compute job accidentally bringing some context over the network. The > best way to avoid it is to use `public static` classes instead of anonymous > classes and lambdas - verbose but bulletproof. > > If you still have an issue with this please share more complete code and > config example. A small reproducer on github would be ideal. > > Stan > > On Mon, Aug 26, 2019 at 4:09 PM Pascoe Scholle <[email protected]> > wrote: > >> Hi Stan, >> >> Thanks for your response. I have tried this, but it has not fixed the >> issue. >> The grpc server class was moved into the service where the interface >> methods "init","execute" and "cancel" perform an initialization of the >> serve, as well start and stopping respectively. But this was already >> implemented. >> >> Does this issue not have something to do with class loading? The second >> node that is started doesn't have any information of the custom classes for >> serialization. I have also moved all JARS of these custom classes into the >> libs folder in ignite but that has also not helped. >> >> On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov < >> [email protected]> wrote: >> >>> Hi, >>> >>> It looks like the issue is that you're ending up sending an instance of >>> your gRPC server inside your service. This approach is generally incorrect. >>> What you should do is >>> - not pass gRPC to the service instance >>> - add an init() method implementation to your service >>> - in your init() start your gRPC server >>> >>> Stan >>> >>> On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <[email protected]> >>> wrote: >>> >>>> Hi there, >>>> >>>> How long does it usually take, for a post to be seen on the forum? >>>> Maybe I made a mistake so I will provide my question here. Excuse me if I >>>> am being impatient: >>>> >>>> >>>> ========================================= >>>> Good afternoon everyone, >>>> >>>> I have recently run into an issue and I think the problem lies in the >>>> server node configuration. I will attache the output of the stack trace at >>>> the bottom, however I first wish to explain what the software does and how >>>> we are using ignite. >>>> >>>> I start multiple server nodes with peerClassEnabled set to true, using >>>> a TcpDiscoveryVmIpFinder and dont set anything other than a port range for >>>> the ipFinder. >>>> >>>> Using the REST protocol a ComputeTaskAdapter task is executed which >>>> starts a service and this in turn starts a grpc server, I have placed some >>>> scala code to show what I mean. >>>> >>>> class StartService extends ComputeTaskAdapter[String, Any]{ >>>> var ignite: Ignite = null; >>>> @IgniteInstanceResource >>>> def setIgnite(someIgnite: Ignite): Unit = { >>>> ignite = someIgnite >>>> } >>>> >>>> def map(...)={ >>>> ... >>>> // port is an integer >>>> val server = new GrpcServer(ignite, port); >>>> >>>> val service = new ServiceImpl(name, server); >>>> /* >>>> within the method execute of the Service interface, server.start() is >>>> called >>>> */ >>>> >>>> val serviceconfig = new ServiceConfiguration(); >>>> serviceconfig.setName(name); >>>> serviceconfig.setTotalCount(1); >>>> serviceconfig.setMaxPerNodeCount(1); >>>> ignite.services().deploy(serviceconfig); >>>> ... >>>> } >>>> >>>> } >>>> >>>> this task returns a map with some non important variables. >>>> >>>> The grpc server takes the ignite instance created within the above >>>> mentioned computeTask as a variable, I am not sure if this could be the >>>> cause of the issue. >>>> >>>> Using grpc protocol, we create a ComputeTask which is executed by the >>>> grpc server some more code below: >>>> >>>> class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc >>>> stuff..{ >>>> >>>> def someGrpcProtocol(request: Message):Future[String]={ >>>> val newTask = new SomeTask(); >>>> >>>> ignite.compute(ignite.cluster()).execute(newTask, someinput); >>>> Future("Request is being processed"); >>>> } >>>> >>>> } >>>> >>>> >>>> If a single server node is started, the program runs without problems. >>>> However, adding more nodes and trying to execute the new tasks on a remote >>>> node or on a node that has a certain attribute gives me a massive stack >>>> trace in the face. >>>> Basically, if I want to execute a task on a node where the service and >>>> grpc server do not reside, the exception happens. >>>> >>>> I have placed all custom classes within a jar that lies in the libs >>>> folder of the ignite-bin project. >>>> We are currently on version 2.7 >>>> >>>> If you require anything else just let me know, ill be on it asap. >>>> >>>> Thanks for any help that may come my way. >>>> >>>> Cheers! >>>> >>>> Here is most of the stack trace: >>>> class org.apache.ignite.binary.BinaryObjectException: Failed to read >>>> field [name=server] >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192) >>>> at >>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188) >>>> at >>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188) >>>> at >>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>>> at >>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313) >>>> at >>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102) >>>> at >>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) >>>> at >>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140) >>>> at >>>> org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440) >>>> at >>>> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119) >>>> at >>>> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923) >>>> at >>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) >>>> at >>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) >>>> at >>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) >>>> at >>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed >>>> to deserialize object [typeName=io.grpc.internal.ServerImpl] >>>> at >>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188) >>>> ... 29 more >>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed >>>> to read field [name=registry] >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192) >>>> at >>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875) >>>> ... 34 more >>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed >>>> to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry] >>>> at >>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) >>>> at >>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703) >>>> at >>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188) >>>> ... 35 more >>>> >>>> ======================================= >>>> >>>> Thanks and kind regards, >>>> Pascoe >>>> >>>
