tansion created AVRO-3531:
-----------------------------
Summary: GenericDatumReader in multithread lead to infinite loop
cause misused of IdentityHashMap
Key: AVRO-3531
URL: https://issues.apache.org/jira/browse/AVRO-3531
Project: Apache Avro
Issue Type: Bug
Components: java
Affects Versions: 1.11.0
Reporter: tansion
Hi,
I am working on a java project that uses Kafka with Avro
serialization/deserialization in an messaging platform.
In production enrionment, we meet a serious issue on the deserialization
processs. The GenericDatumReader process some how get into a infinite loop
status, and it is happened accationally.
When the issue happens, The thread stack is like this:
{code:java}
"DmqFixedRateConsumer-Thread-17" #453 daemon prio=5 os_prio=0
tid=0x00007f2ae1832800 nid=0xef49 runnable [0x00007f2a743fc000]
java.lang.Thread.State: RUNNABLE
at java.util.IdentityHashMap.get(IdentityHashMap.java:337)
at
org.apache.avro.generic.GenericDatumReader.getStringClass(GenericDatumReader.java:503)
at
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:454)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at com.xxx.xxx.xxx.xxx.xxx.XXX.deserialize(XXX.java:252)
at com.xxx.xxx.xxx.xxx.xxx.ZZZ.deserialize(ZZZ.java:216)
at com.xxx.xxx.xxx.xxx.xxx.SSS.processMessage(SSS.java:152)
at com.xxx.xxx.xxx.xxx.xxx.SSS.loopProcess(SSS.java:127)
at com.xxx.xxx.xxx.xxx.xxx.SSS$$Lambda$172/367082698.run(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
We create 30 threads, and all the threads are the same as above! They all get
stuck in the IdentityHashMap.get() method.
Accroding to this mail [1.7.6 Slow
Deserialization|https://www.mail-archive.com/[email protected]/msg02902.html],
the Reader is thread-safe, But actually, it seems not.
Why?
org.apache.avro.generic.GenericDatumReader#getStringClass
{code:java}
/**
* Called to read strings. Subclasses may override to use a different string
* representation. By default, this calls {@link #readString(Object,Decoder)}.
*/
protected Object readString(Object old, Schema expected, Decoder in) throws
IOException {
Class stringClass = getStringClass(expected);
if (stringClass == String.class) {
return in.readString();
}
if (stringClass == CharSequence.class) {
return readString(old, in);
}
return newInstanceFromString(stringClass, in.readString());
}
private Map<Schema, Class> stringClassCache = new IdentityHashMap<>();
private Class getStringClass(Schema s) {
Class c = stringClassCache.get(s);
if (c == null) {
c = findStringClass(s);
stringClassCache.put(s, c);
}
return c;
}
{code}
The IdentityHashMap is not thread-safe, which is addressed by javadoc clearly!
Like Hashmap infinite loop issue in multithread using, same issue happen to
IdentityHashMap,too.
My question is: Can the class GenericDatumReader fix this issue and act like
real thread-safe? Or we need to avoid use the instance of GenericDatumReader in
multithread?
Thanks a lot,
Xtsong.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
