Dmitry Lysnichenko created AMBARI-13148:
-------------------------------------------
Summary: Ambari DB schema upgrade command hung on console even
though schema upgrade is success
Key: AMBARI-13148
URL: https://issues.apache.org/jira/browse/AMBARI-13148
Project: Ambari
Issue Type: Bug
Reporter: Dmitry Lysnichenko
Assignee: Dmitry Lysnichenko
Observed this issue on two configs
Config #1 -- Ambari server upgrade from 1.6.1 to 2.1.2-262 on Suse 11 SP3
Config #2 -- Ambari server upgrade from 2.0.2 to 2.1.2-262 on Ubuntu 12
Ambari DB: Embedded Postgres
The upgrade command was hung and did not come out of the console:
vsharma-u21todalm10-re-5:~ # ambari-server upgrade
Using python /usr/bin/python2.6
Upgrading ambari-server
Updating properties in ambari.properties ...
WARNING: Can not find ambari-env.sh.rpmsave file from previous version,
skipping restore of environment settings
Fixing database objects owner
Ambari Server configured for Embedded Postgres. Confirm you have made a backup
of the Ambari Server database [y/n] (y)? y
Upgrading database schema
When looking at ambari-server.log - the upgrade is a success (see attached).
See below for process and jstack output of the upgrade process
vsharma-u21todalm10-re-5:/tmp/dbdumps # ps -ef | grep ambari
root 22879 22756 0 17:23 pts/1 00:00:00 bash /usr/sbin/ambari-server
upgrade
root 22888 22879 0 17:23 pts/1 00:00:00 /usr/bin/python2.6
/usr/sbin/ambari-server.py upgrade
root 23071 22888 4 17:23 pts/1 00:00:33 /usr/jdk64/jdk1.7.0_67/bin/java
-cp /etc/ambari-server/conf:/usr/lib/ambari-server/*
org.apache.ambari.server.upgrade.SchemaUpgradeHelper >
/var/log/ambari-server/ambari-server.out 2>&1
postgres 23085 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48478) idle
postgres 23086 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48479) idle
postgres 23087 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48480) idle
postgres 23088 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48481) idle
postgres 23089 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48482) idle
postgres 23090 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48483) idle
postgres 23091 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48484) idle
postgres 23092 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48485) idle
postgres 23093 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48486) idle
postgres 23094 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48487) idle
postgres 23095 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48488) idle
postgres 23096 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48489) idle
postgres 23097 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48490) idle
postgres 23098 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48491) idle
postgres 23099 6255 0 17:23 ? 00:00:02 postgres: ambari ambari
127.0.0.1(48492) idle
postgres 23100 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48493) idle
postgres 23101 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48494) idle
postgres 23102 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48495) idle
postgres 23103 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48496) idle
postgres 23104 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48497) idle
postgres 23105 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48498) idle
postgres 23108 6255 0 17:23 ? 00:00:00 postgres: ambari ambari
127.0.0.1(48499) idle
root 23214 22790 0 17:36 pts/2 00:00:00 grep ambari
vsharma-u21todalm10-re-5:/tmp/dbdumps # cd /usr/jdk64/jdk1.7.0_67/bin/
vsharma-u21todalm10-re-5:/usr/jdk64/jdk1.7.0_67/bin # ./jstack 23071
2015-09-16 17:37:14
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00000000014ed000 nid=0x5ac5 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" prio=10 tid=0x00007f8fec53f000 nid=0x5a20 waiting on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"InMemoryCredentialStoreService active cleanup timer" prio=10
tid=0x00007f8fecbfd000 nid=0x5a4c waiting on condition [0x00007f8ffa28a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007cfa39908> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"com.google.inject.internal.util.$Finalizer" daemon prio=10
tid=0x00007f8fec18a800 nid=0x5a2b in Object.wait() [0x00007f8ffaa19000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007cfa3a6f8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000007cfa3a6f8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
"Service Thread" daemon prio=10 tid=0x00007f8ff4029800 nid=0x5a29 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f8ff4027000 nid=0x5a28 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f8ff4024000 nid=0x5a27 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f8ff4022000 nid=0x5a26 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000000675000 nid=0x5a25 in Object.wait()
[0x00007f8ffbb5b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000707c613e8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x0000000707c613e8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" daemon prio=10 tid=0x000000000066d800 nid=0x5a24 in
Object.wait() [0x00007f8ffbc5c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000707c614d0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x0000000707c614d0> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x0000000000669000 nid=0x5a23 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000000623800 nid=0x5a21
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000000625800 nid=0x5a22
runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f8ff4034800 nid=0x5a2a waiting on
condition
JNI global references: 388
========================================
After I manually killed the above process and started ambari-server, ambari
server started successfully
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
