Re: Problem with Messages after client reconnect
Hi Sebastian,
Thanks for example and detailed steps to reproduce.
Unfortunately, this is a flaw in current binary metadata exchange
algorithm. I have created a ticket for it -
https://issues.apache.org/jira/browse/IGNITE-5794.
To avoid this issue for now, you should either:
- ensure that there is always at least 1 server node alive,
- restart all client nodes after last server node left cluster.
On Thu, Jul 20, 2017 at 9:18 AM, Sebastian Sindelar <
sebastian.sinde...@ibh-ks.de> wrote:
> Hi.
>
>
>
> Just figured out while the example setup doesn’t produce the error. The
> following TestClient will always cause the exception at the server.
>
>
>
> *public* *class* TestClient {
>
>
>
>*public* *static* *void* main(String[] args) {
>
> IgniteConfiguration igniteConfiguration = *new*
> IgniteConfiguration();
>
> igniteConfiguration.setClientMode(*true*);
>
> igniteConfiguration.setMetricsLogFrequency(0);
>
> Ignite ignite = *Ignition.start(**igniteConfiguration**)*;
>
>
>
> Scanner *scanner* = *new* Scanner(System.*in*);
>
> *while** (**true**) {*
>
> *String **message** = **scanner**.nextLine();*
>
> *IgniteMessaging **messaging** = **ignite**.message(*
> *ignite**.cluster());*
>
> *messaging**.send(**"test"**, **new** ComplexMessage(*
> *message**));*
>
> * }*
>
>
>
>
>
>}
>
> }
>
>
>
> Ignite.message(…) is now called before each message is send. In the main
> application it is called only once.
>
> I also noticed I didn’t describe the steps to reproduce the problem very
> detailed yesterday evening.
>
>
>
>1. Start 1 Client and 1 Server
>2. Type in some message in the client console and hit Enter
>3. Check that message is received and logged by the server
>4. Stop the Server
>5. Start the Server and wait for the client to reconnect
>6. Type in another message in the client console and hit Enter
>7. See the Exception at the server log
>
>
>
> Also one additional note. This only happens if all cluster servers are
> offline and the cluster gets restartet. If you have multiple servers there
> is no problem. At least one server must survive to keep the record on how
> to unmarshall the sent class.
>
>
>
> Best regards,
> Sebastian Sindelar
>
> *Von:* Sebastian Sindelar [mailto:sebastian.sinde...@ibh-ks.de]
> *Gesendet:* Mittwoch, 19. Juli 2017 16:53
> *An:* user@ignite.apache.org
> *Betreff:* Problem with Messages after client reconnect
>
>
>
> Hi.
>
>
>
> I just started with Ignite messaging and I encountered the following
> problem:
>
>
>
> I have a simple Setup with 1 server and 1 client. Sending messages from
> the client to the server works fine. But after a server restart I encounter
> the problem that when I send a message with a custom class I get an
> unmarshalling error:
>
>
>
>
>
> *org.apache.ignite.IgniteCheckedException*: null
>
>at org.apache.ignite.internal.util.IgniteUtils.unmarshal(
> *IgniteUtils.java:9893*) ~[ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.GridIoManager$
> GridUserMessageListener.onMessage(*GridIoManager.java:2216*)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(*GridIoManager.java:1257*)
> [ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2000(*GridIoManager.java:114*)
> [ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.GridIoManager$
> GridCommunicationMessageSet.unwind(*GridIoManager.java:2461*)
> [ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.GridIoManager.
> unwindMessageSet(*GridIoManager.java:1217*) [ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2300(*GridIoManager.java:114*)
> [ignite-core-2.0.0.jar:2.0.0]
>
>at org.apache.ignite.internal.managers.communication.
> GridIoManager$8.run(*GridIoManager.java:1186*)
> [ignite-core-2.0.0.jar:2.0.0]
>
>at java.util.concurrent.ThreadPoolExecutor.runWorker(
> *ThreadPoolExecutor.java:1142*) [na:1.8.0_112]
>
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> *ThreadPoolExecutor.java:617*) [na:1.8.0_112]
>
>at java.lang.Thread.run(*Thread.java:745
AW: Problem with Messages after client reconnect
Hi.
Just figured out while the example setup doesn't produce the error. The
following TestClient will always cause the exception at the server.
public class TestClient {
public static void main(String[] args) {
IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();
igniteConfiguration.setClientMode(true);
igniteConfiguration.setMetricsLogFrequency(0);
Ignite ignite = Ignition.start(igniteConfiguration);
Scanner scanner = new Scanner(System.in);
while (true) {
String message = scanner.nextLine();
IgniteMessaging messaging =
ignite.message(ignite.cluster());
messaging.send("test", new ComplexMessage(message));
}
}
}
Ignite.message(...) is now called before each message is send. In the main
application it is called only once.
I also noticed I didn't describe the steps to reproduce the problem very
detailed yesterday evening.
1. Start 1 Client and 1 Server
2. Type in some message in the client console and hit Enter
3. Check that message is received and logged by the server
4. Stop the Server
5. Start the Server and wait for the client to reconnect
6. Type in another message in the client console and hit Enter
7. See the Exception at the server log
Also one additional note. This only happens if all cluster servers are offline
and the cluster gets restartet. If you have multiple servers there is no
problem. At least one server must survive to keep the record on how to
unmarshall the sent class.
Best regards,
Sebastian Sindelar
Von: Sebastian Sindelar [mailto:sebastian.sinde...@ibh-ks.de]
Gesendet: Mittwoch, 19. Juli 2017 16:53
An: user@ignite.apache.org
Betreff: Problem with Messages after client reconnect
Hi.
I just started with Ignite messaging and I encountered the following problem:
I have a simple Setup with 1 server and 1 client. Sending messages from the
client to the server works fine. But after a server restart I encounter the
problem that when I send a message with a custom class I get an unmarshalling
error:
org.apache.ignite.IgniteCheckedException: null
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9893)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$GridUserMessageListener.onMessage(GridIoManager.java:2216)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2000(GridIoManager.java:114)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2461)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1217)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2300(GridIoManager.java:114)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1186)
[ignite-core-2.0.0.jar:2.0.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
Caused by: java.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:492)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.metadata(CacheObjectBinaryProcessorImpl.java:174)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1231)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:1987)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:283)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:182)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:161)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:304)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99)
~[ignite-core-2.0.0
Problem with Messages after client reconnect
Hi.
I just started with Ignite messaging and I encountered the following problem:
I have a simple Setup with 1 server and 1 client. Sending messages from the
client to the server works fine. But after a server restart I encounter the
problem that when I send a message with a custom class I get an unmarshalling
error:
org.apache.ignite.IgniteCheckedException: null
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9893)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$GridUserMessageListener.onMessage(GridIoManager.java:2216)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2000(GridIoManager.java:114)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2461)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1217)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2300(GridIoManager.java:114)
[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1186)
[ignite-core-2.0.0.jar:2.0.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
Caused by: java.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:492)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.metadata(CacheObjectBinaryProcessorImpl.java:174)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1231)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:1987)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:283)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:182)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:161)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:304)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
~[ignite-core-2.0.0.jar:2.0.0]
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9887)
~[ignite-core-2.0.0.jar:2.0.0]
... 10 common frames omitted
Sending String or Integer Messages works fine.
I tried to boil it down into a simple server and client but can't reproduce the
behaivor exactly:
public class TestServer {
public static void main(String[] args) {
IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();
igniteConfiguration.setMetricsLogFrequency(0);
Ignite ignite = Ignition.start(igniteConfiguration);
IgniteMessaging messaging = ignite
.message(ignite.cluster());
messaging.localListen("test", (nodeId, msg) -> {
System.out.println("Message received: '" + msg + "' von '"
+ nodeId + "'");
return true;
});
}
}
public class TestClient {
public static void main(String[] args) {
IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();
igniteConfiguration.setClientMode(true);
igniteConfiguration.setMetricsLogFrequency(0);
Ignite ignite = Ignition.start(igniteConfiguration);
IgniteMessaging messaging = ignite.message(ignite.cluster());
Scanner scanner = new Scanner(System.in);
while (true) {
String message = scanner.nextLine();
messaging.send("test", new ComplexMessage(message));
}
}
}
public class ComplexMessage {
private final String message;
public ComplexMessage(String message) {
super();
