[
https://issues.apache.org/jira/browse/ATLAS-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayub Pathan updated ATLAS-2199:
-------------------------------
Priority: Blocker (was: Critical)
> DeadLock in atlas while producing metadata with hive view > 300 cols.
> ---------------------------------------------------------------------
>
> Key: ATLAS-2199
> URL: https://issues.apache.org/jira/browse/ATLAS-2199
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8.2
> Reporter: Ayub Pathan
> Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: application.log, atlas.20171009-192118.err,
> atlas.20171009-192118.out, thread-dump.txt
>
>
> Steps to reproduce:
> 1. Produce messages using hive hook - create view with 300 columns. Example,
> command below
> {noformat}
> create table t1_300(col0 string, col1 string, col2 string, col3 string, col4
> string, col5 string, col6 string, col7 string, col8 string, col9 string,
> col10 string, col11 string, col12 string, col13 string, col14 string, col15
> string, col16 string, col17 string, col18 string, col19 string, col20 string,
> col21 string, col22 string, col23 string, col24 string, col25 string, col26
> string, col27 string, col28 string, col29 string, col30 string, col31 string,
> col32 string, col33 string, col34 string, col35 string, col36 string, col37
> string, col38 string, col39 string, col40 string, col41 string, col42 string,
> col43 string, col44 string, col45 string, col46 string, col47 string, col48
> string, col49 string, col50 string, col51 string, col52 string, col53 string,
> col54 string, col55 string, col56 string, col57 string, col58 string, col59
> string, col60 string, col61 string, col62 string, col63 string, col64 string,
> col65 string, col66 string, col67 string, col68 string, col69 string, col70
> string, col71 string, col72 string, col73 string, col74 string, col75 string,
> col76 string, col77 string, col78 string, col79 string, col80 string, col81
> string, col82 string, col83 string, col84 string, col85 string, col86 string,
> col87 string, col88 string, col89 string, col90 string, col91 string, col92
> string, col93 string, col94 string, col95 string, col96 string, col97 string,
> col98 string, col99 string, col100 string, col101 string, col102 string,
> col103 string, col104 string, col105 string, col106 string, col107 string,
> col108 string, col109 string, col110 string, col111 string, col112 string,
> col113 string, col114 string, col115 string, col116 string, col117 string,
> col118 string, col119 string, col120 string, col121 string, col122 string,
> col123 string, col124 string, col125 string, col126 string, col127 string,
> col128 string, col129 string, col130 string, col131 string, col132 string,
> col133 string, col134 string, col135 string, col136 string, col137 string,
> col138 string, col139 string, col140 string, col141 string, col142 string,
> col143 string, col144 string, col145 string, col146 string, col147 string,
> col148 string, col149 string, col150 string, col151 string, col152 string,
> col153 string, col154 string, col155 string, col156 string, col157 string,
> col158 string, col159 string, col160 string, col161 string, col162 string,
> col163 string, col164 string, col165 string, col166 string, col167 string,
> col168 string, col169 string, col170 string, col171 string, col172 string,
> col173 string, col174 string, col175 string, col176 string, col177 string,
> col178 string, col179 string, col180 string, col181 string, col182 string,
> col183 string, col184 string, col185 string, col186 string, col187 string,
> col188 string, col189 string, col190 string, col191 string, col192 string,
> col193 string, col194 string, col195 string, col196 string, col197 string,
> col198 string, col199 string, col200 string, col201 string, col202 string,
> col203 string, col204 string, col205 string, col206 string, col207 string,
> col208 string, col209 string, col210 string, col211 string, col212 string,
> col213 string, col214 string, col215 string, col216 string, col217 string,
> col218 string, col219 string, col220 string, col221 string, col222 string,
> col223 string, col224 string, col225 string, col226 string, col227 string,
> col228 string, col229 string, col230 string, col231 string, col232 string,
> col233 string, col234 string, col235 string, col236 string, col237 string,
> col238 string, col239 string, col240 string, col241 string, col242 string,
> col243 string, col244 string, col245 string, col246 string, col247 string,
> col248 string, col249 string, col250 string, col251 string, col252 string,
> col253 string, col254 string, col255 string, col256 string, col257 string,
> col258 string, col259 string, col260 string, col261 string, col262 string,
> col263 string, col264 string, col265 string, col266 string, col267 string,
> col268 string, col269 string, col270 string, col271 string, col272 string,
> col273 string, col274 string, col275 string, col276 string, col277 string,
> col278 string, col279 string, col280 string, col281 string, col282 string,
> col283 string, col284 string, col285 string, col286 string, col287 string,
> col288 string, col289 string, col290 string, col291 string, col292 string,
> col293 string, col294 string, col295 string, col296 string, col297 string,
> col298 string, col299 string);
> create view v_t1_300 as select * from t1_300;
> {noformat}
> 2. With this Atlas throws out of memory exception. Below error log confirming
> the same.
> {noformat}
> [root@ctr-e134-1499953498516-209460-01-000004 atlas]# cat
> atlas.20171009-192118.err
> Exception in thread "pool-2-thread-6" Exception in thread "pool-2-thread-2"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap$KeySet.iterator(HashMap.java:916)
> at java.util.HashSet.iterator(HashSet.java:172)
> at sun.nio.ch.Util$3.iterator(Util.java:324)
> at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-22" Exception in thread "pool-2-thread-3"
> Exception in thread "HashSessionScavenger-0" java.lang.OutOfMemoryError: Java
> heap space
> at java.util.HashMap$KeySet.iterator(HashMap.java:916)
> at java.util.HashSet.iterator(HashSet.java:172)
> at sun.nio.ch.Util$3.iterator(Util.java:324)
> at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> java.lang.OutOfMemoryError: Java heap space
> at
> java.util.concurrent.ConcurrentHashMap$ValuesView.iterator(ConcurrentHashMap.java:4683)
> at
> org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:314)
> at
> org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:285)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap$KeySet.iterator(HashMap.java:916)
> at java.util.HashSet.iterator(HashSet.java:172)
> at sun.nio.ch.Util$3.iterator(Util.java:324)
> at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
> at
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "pool-2-thread-7" java.lang.OutOfMemoryError: Java heap
> space
> Exception in thread "pool-2-thread-1" java.lang.OutOfMemoryError: Java heap
> space
> [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
> {noformat}
> 3. After this step, Atlas became unresponsive, no requests are honoured. All
> requests are timedout.
> 4. JSTACK output confirming the deadlock.
> {noformat}
> [root@ctr-e134-1499953498516-209460-01-000004 atlas]# ps axu | grep Atlas
> atlas 131887 3.6 0.9 20609684 2612572 ? Sl Oct09 31:03
> /usr/lib/jvm/java-openjdk/bin/java -Datlas.log.dir=/grid/0/log/atlas
> -Datlas.log.file=application.log -Datlas.home=/usr/hdp/.3.0-175/atlas
> -Datlas.conf=/usr/hdp/current/atlas-server/conf -Xms2048m -Xmx2048m
> -XX:MaxNewSize=600m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m -server
> -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/grid/0/log/atlas/atlas_server.hprof
> -Xloggc:/grid/0/log/atlas/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails
> -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps
> -Dlog4j.configuration=atlas-log4j.xml
> -Djava..auth.login.config=/usr/hdp/current/atlas-server/conf/atlas_jaas.conf
> -classpath
> /usr/hdp/current/atlas-server/conf:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/classes:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/.3.0-175/atlas/libext/*:/etc/hbase/conf
> org.apache.atlas.Atlas -app /usr/hdp/current/atlas-server/server/webapp/atlas
> root 1666487 0.0 0.0 10632 968 pts/1 S+ 09:34 0:00 grep
> --color=auto Atlas
> [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
> {noformat}
> {noformat}
> [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
> /usr/lib/jvm/java-openjdk/bin/jstack -F -l 131887
> Attaching to process ID 131887, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 25.144-b01
> Deadlock Detection:
> java.lang.RuntimeException: Unable to deduce type of thread from address
> 0x00007f8838013800 (expected type JavaThread, CompilerThread, ServiceThread,
> JvmtiAgentThread, or SurrogateLockerThread)
> at
> sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:169)
> at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153)
> at
> sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> at
> sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> at
> sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
> at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
> at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
> at sun.tools.jstack.JStack.main(JStack.java:106)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for
> type of address 0x00007f8838013800
> at
> sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> at
> sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> at
> sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:165)
> ... 17 more
> Can't print deadlocks:Unable to deduce type of thread from address
> 0x00007f8838013800 (expected type JavaThread, CompilerThread, ServiceThread,
> JvmtiAgentThread, or SurrogateLockerThread)
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
> at sun.tools.jstack.JStack.main(JStack.java:106)
> Caused by: java.lang.InternalError: Metadata does not appear to be polymorphic
> at
> sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:278)
> at
> sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102)
> at sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:68)
> at sun.jvm.hotspot.memory.DictionaryEntry.klass(DictionaryEntry.java:71)
> at
> sun.jvm.hotspot.memory.DictionaryEntry.equals(DictionaryEntry.java:79)
> at sun.jvm.hotspot.memory.Dictionary.getEntry(Dictionary.java:99)
> at sun.jvm.hotspot.memory.Dictionary.find(Dictionary.java:87)
> at
> sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:159)
> at
> sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:151)
> at
> sun.jvm.hotspot.memory.SystemDictionary.getAbstractOwnableSynchronizerKlass(SystemDictionary.java:135)
> at
> sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.fillLocks(ConcurrentLocksPrinter.java:66)
> at
> sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.<init>(ConcurrentLocksPrinter.java:36)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:71)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
> at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
> at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
> ... 6 more
> [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
> {noformat}
> Attaching the thread dump output as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
