Yixing Zhang created SAMZA-2587:
-----------------------------------
Summary: IntermediateMessageSerde exception handling
Key: SAMZA-2587
URL: https://issues.apache.org/jira/browse/SAMZA-2587
Project: Samza
Issue Type: Improvement
Reporter: Yixing Zhang
Assignee: Yixing Zhang
In IntermediateMessageSerde, we expect the first byte of payload to be the type
of message (message|watermark|EOS). The serde will read the first byte, then
use the corresponding serde to deserialize the rest of bytes. For compatibility
reason, if any exception happens, the serde will try to deserialize all bytes
again.
This caused an issue. In the case when userMessageSerde failed to deserialize
the message with all bytes except for the first byte, it still tried to use
userMessageSerde to deserialize the message with all bytes. Of course it failed
again. But the container died with OOM instead of exception of the
deserialization. The reason is that the size of an array was encoded in the
bytes. In the second time, the serde was trying to construct a huge array which
caused OOM without throwing the root exception, which was causing it super hard
to debug.
We should remove the second deserialization to make the code more clear.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
