[ https://issues.apache.org/jira/browse/AVRO-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17562235#comment-17562235 ]
ASF subversion and git services commented on AVRO-3531: ------------------------------------------------------- Commit 820ed6e5ea4417b5735078bfd26c99f1305ea363 in avro's branch refs/heads/master from clesaec [ https://gitbox.apache.org/repos/asf?p=avro.git;h=820ed6e5e ] AVRO-3531: Improve thread safety in GenericDatumReader (#1719) * avro-3531 : make generic datum reader thread safe * avro3531: use reentrant lock * avro-3531: spotless modif * AVRO-3531:use Conccurent hashmap * AVRO-3531: reput newInstanceFromString method for backward compatibility * Reduce visibility of the getReaderCache() method. * Fix typo from github editor Co-authored-by: Ryan Skraba <r...@skraba.com> > GenericDatumReader in multithread lead to infinite loop cause misused of > IdentityHashMap > ---------------------------------------------------------------------------------------- > > Key: AVRO-3531 > URL: https://issues.apache.org/jira/browse/AVRO-3531 > Project: Apache Avro > Issue Type: Bug > Components: java > Affects Versions: 1.11.0 > Reporter: tansion > Assignee: Christophe Le Saec > Priority: Critical > Labels: pull-request-available > Fix For: 1.11.1 > > Time Spent: 2h > Remaining Estimate: 0h > > Hi, > I am working on a java project that uses Kafka with Avro > serialization/deserialization in an messaging platform. > In production enrionment, we meet a serious issue on the deserialization > processs. The GenericDatumReader process some how get into a infinite loop > status, and it is happened accationally. > When the issue happens, The thread stack is like this: > > {code:java} > "DmqFixedRateConsumer-Thread-17" #453 daemon prio=5 os_prio=0 > tid=0x00007f2ae1832800 nid=0xef49 runnable [0x00007f2a743fc000] > java.lang.Thread.State: RUNNABLE > at java.util.IdentityHashMap.get(IdentityHashMap.java:337) > at > org.apache.avro.generic.GenericDatumReader.getStringClass(GenericDatumReader.java:503) > at > org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:454) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) > at > org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) > at > org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) > at > org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) > at > org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) > at > org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) > at > org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) > at > org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) > at > org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) > at > org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) > at > org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) > at > org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > at com.xxx.xxx.xxx.xxx.xxx.XXX.deserialize(XXX.java:252) > at com.xxx.xxx.xxx.xxx.xxx.ZZZ.deserialize(ZZZ.java:216) > at com.xxx.xxx.xxx.xxx.xxx.SSS.processMessage(SSS.java:152) > at com.xxx.xxx.xxx.xxx.xxx.SSS.loopProcess(SSS.java:127) > at com.xxx.xxx.xxx.xxx.xxx.SSS$$Lambda$172/367082698.run(Unknown Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) {code} > We create 30 threads, and all the threads are the same as above! They all get > stuck in the IdentityHashMap.get() method. > > Accroding to this mail [1.7.6 Slow > Deserialization|https://www.mail-archive.com/user@avro.apache.org/msg02902.html], > the Reader is thread-safe, But actually, it seems not. > Why? > org.apache.avro.generic.GenericDatumReader#getStringClass > > {code:java} > /** > * Called to read strings. Subclasses may override to use a different string > * representation. By default, this calls {@link #readString(Object,Decoder)}. > */ > protected Object readString(Object old, Schema expected, Decoder in) throws > IOException { > Class stringClass = getStringClass(expected); > if (stringClass == String.class) { > return in.readString(); > } > if (stringClass == CharSequence.class) { > return readString(old, in); > } > return newInstanceFromString(stringClass, in.readString()); > } > private Map<Schema, Class> stringClassCache = new IdentityHashMap<>(); > private Class getStringClass(Schema s) { > Class c = stringClassCache.get(s); > if (c == null) { > c = findStringClass(s); > stringClassCache.put(s, c); > } > return c; > } > {code} > The IdentityHashMap is not thread-safe, which is addressed by javadoc > clearly! Like Hashmap infinite loop issue in multithread using, same issue > happen to IdentityHashMap,too. > My question is: Can the class GenericDatumReader fix this issue and act like > real thread-safe? Or we need to avoid use the single instance of > GenericDatumReader in multithread? > Thanks a lot, > Xtsong. > -- This message was sent by Atlassian Jira (v8.20.10#820010)