[ 
https://issues.apache.org/jira/browse/TRAFODION-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Subbiah updated TRAFODION-1107:
--------------------------------------
    Fix Version/s:     (was: 2.2.0)

> LP Bug: 1438466 - Multiple tdm_arkcmp child processes started after receipt 
> of HBase error
> ------------------------------------------------------------------------------------------
>
>                 Key: TRAFODION-1107
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-1107
>             Project: Apache Trafodion
>          Issue Type: Bug
>          Components: sql-cmp
>            Reporter: Joanie Cooper
>            Assignee: Qifan Chen
>            Priority: Critical
>
> During a fresh test of running the compGeneral regression suite
> while artificially interjecting an error return from the TrxRegionEndpoint
> coprocessor, numerous tdm_arkcmp child processes were started.
> Before the error hit, we seemed to have a normal number of compilers
> [$Z0005MG] 000,00003170 001 GEN  ES--A-- $Z0002KK    $Z0002IV    tdm_arkcmp   
>   
> [$Z0005MG] 000,00003292 001 GEN  ES--A-- $Z0002P2    $Z0002KK    tdm_arkcmp   
>   
> [$Z0005MG] 000,00003816 001 GEN  ES--A-- $Z000341    $Z0002P2    tdm_arkcmp   
>   
> [$Z0005MG] 000,00003886 001 GEN  ES--A-- $Z000361    $Z000341    tdm_arkcmp
> After forcing the error, it looks like we have new compilers being generated,
> all ultimately part of the original tdm_arkcmp parent off of the sqlci 
> session.
> This is a result of a drop statement.  From the sqlci window, the
> statement appears hung, as it never returns.  But, it appears the
> compilers keep generating new children and the query ultimately never returns.
> When I killed the query, it had 174 compilers running.
> I tried a pstack for one of the compilers, I’ve attached it below.
> g4t3037{joaniec}3: sqps
> Processing cluster.conf on local host g4t3037.houston.hp.com
> [$Z000AF9] Shell/shell Version 1.0.1 Release 1.1.0 (Build release [joaniec], 
> date 26Mar15)
> [$Z000AF9] %ps  
> [$Z000AF9] NID,PID(os)  PRI TYPE STATES  NAME        PARENT      PROGRAM
> [$Z000AF9] ------------ --- ---- ------- ----------- ----------- 
> ---------------
> [$Z000AF9] 000,00018562 000 WDG  ES--A-- $WDG000     NONE        sqwatchdog   
>   
> [$Z000AF9] 000,00018563 000 PSD  ES--A-- $PSD000     NONE        pstartd      
>   
> [$Z000AF9] 000,00018592 001 DTM  ES--A-- $TM0        NONE        tm           
>   
> [$Z000AF9] 000,00019243 001 GEN  ES--A-- $ZSC000     NONE        mxsscp       
>   
> [$Z000AF9] 000,00019274 001 SSMP ES--A-- $ZSM000     NONE        mxssmp       
>   
> [$Z000AF9] 000,00020982 001 GEN  ES--A-- $ZLOBSRV0   NONE        mxlobsrvr    
>   
> [$Z000AF9] 000,00007356 001 GEN  ES--A-- $Z000606    NONE        sqlci        
>   
> [$Z000AF9] 000,00007416 001 GEN  ES--A-- $Z00061W    $Z000606    tdm_arkcmp   
>   
> [$Z000AF9] 000,00007960 001 GEN  ES--A-- $Z0006HF    $Z00061W    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008021 001 GEN  ES--A-- $Z0006J6    $Z0006HF    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008079 001 GEN  ES--A-- $Z0006KU    $Z0006J6    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008137 001 GEN  ES--A-- $Z0006MH    $Z0006KU    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008194 001 GEN  ES--A-- $Z0006P4    $Z0006MH    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008252 001 GEN  ES--A-- $Z0006QS    $Z0006P4    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008312 001 GEN  ES--A-- $Z0006SH    $Z0006QS    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008369 001 GEN  ES--A-- $Z0006U4    $Z0006SH    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008427 001 GEN  ES--A-- $Z0006VS    $Z0006U4    tdm_arkcmp   
>   
> [$Z000AF9] 000,00008491 001 GEN  ES--A-- $Z0006XL    $Z0006VS    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009023 001 GEN  ES--A-- $Z0007CT    $Z0006XL    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009081 001 GEN  ES--A-- $Z0007EG    $Z0007CT    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009141 001 GEN  ES--A-- $Z0007G6    $Z0007EG    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009202 001 GEN  ES--A-- $Z0007HX    $Z0007G6    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009262 001 GEN  ES--A-- $Z0007JM    $Z0007HX    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009320 001 GEN  ES--A-- $Z0007LA    $Z0007JM    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009489 001 GEN  ES--A-- $Z0007R4    $Z0007LA    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009547 001 GEN  ES--A-- $Z0007SS    $Z0007R4    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009604 001 GEN  ES--A-- $Z0007UE    $Z0007SS    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009661 001 GEN  ES--A-- $Z0007W1    $Z0007UE    tdm_arkcmp   
>   
> [$Z000AF9] 000,00009728 001 GEN  ES--A-- $Z0007XY    $Z0007W1    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010268 001 GEN  ES--A-- $Z0008DD    $Z0007XY    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010364 001 GEN  ES--A-- $Z0008G4    $Z0008DD    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010421 001 GEN  ES--A-- $Z0008HR    $Z0008G4    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010479 001 GEN  ES--A-- $Z0008JE    $Z0008HR    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010537 001 GEN  ES--A-- $Z0008L2    $Z0008JE    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010620 001 GEN  ES--A-- $Z0008NF    $Z0008L2    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010677 001 GEN  ES--A-- $Z0008Q2    $Z0008NF    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010735 001 GEN  ES--A-- $Z0008RQ    $Z0008Q2    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010796 001 GEN  ES--A-- $Z0008TG    $Z0008RQ    tdm_arkcmp   
>   
> [$Z000AF9] 000,00010857 001 GEN  ES--A-- $Z0008V7    $Z0008TG    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011367 001 GEN  ES--A-- $Z00099S    $Z0008V7    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011437 001 GEN  ES--A-- $Z0009BS    $Z00099S    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011496 001 GEN  ES--A-- $Z0009DG    $Z0009BS    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011558 001 GEN  ES--A-- $Z0009F8    $Z0009DG    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011618 001 GEN  ES--A-- $Z0009GY    $Z0009F8    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011680 001 GEN  ES--A-- $Z0009IQ    $Z0009GY    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011741 001 GEN  ES--A-- $Z0009KG    $Z0009IQ    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011801 001 GEN  ES--A-- $Z0009M6    $Z0009KG    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011862 001 GEN  ES--A-- $Z0009NX    $Z0009M6    tdm_arkcmp   
>   
> [$Z000AF9] 000,00011921 001 GEN  ES--A-- $Z0009QL    $Z0009NX    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012410 001 GEN  ES--A-- $Z000A4K    $Z0009QL    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012509 001 GEN  ES--A-- $Z000A7E    $Z000A4K    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012582 001 GEN  ES--A-- $Z000A9H    $Z000A7E    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012646 001 GEN  ES--A-- $Z000ABB    $Z000A9H    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012703 001 GEN  ES--A-- $Z000ACY    $Z000ABB    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012760 001 GEN  ES--A-- $Z000AEK    $Z000ACY    tdm_arkcmp   
>   
> [$Z000AF9] 000,00012784 001 GEN  ES--A-- $Z000AF9    NONE        shell        
>   
> [$Z000AF9] 001,00018560 000 WDG  ES--A-- $WDG001     NONE        sqwatchdog   
>   
> [$Z000AF9] 001,00018561 000 PSD  ES--A-- $PSD001     NONE        pstartd      
>   
> [$Z000AF9] 001,00018640 001 DTM  ES--A-- $TM1        NONE        tm           
>   
> [$Z000AF9] 001,00019258 001 GEN  ES--A-- $ZSC001     NONE        mxsscp       
>   
> [$Z000AF9] 001,00019289 001 SSMP ES--A-- $ZSM001     NONE        mxssmp       
>   
> [$Z000AF9] 001,00020989 001 GEN  ES--A-- $ZLOBSRV1   NONE        mxlobsrvr    
> [$Z000GHJ] 000,00020170 001 GEN  ES--A-- $Z000GGA    $Z000GEK    tdm_arkcmp   
>   
> [$Z000GHJ] 000,00020214 001 GEN  ES--A-- $Z000GHJ    NONE        shell        
>   
> [$Z000GHJ] 001,00018560 000 WDG  ES--A-- $WDG001     NONE        sqwatchdog   
>   
> [$Z000GHJ] 001,00018561 000 PSD  ES--A-- $PSD001     NONE        pstartd      
>   
> [$Z000GHJ] 001,00018640 001 DTM  ES--A-- $TM1        NONE        tm           
>   
> [$Z000GHJ] 001,00019258 001 GEN  ES--A-- $ZSC001     NONE        mxsscp       
>   
> [$Z000GHJ] 001,00019289 001 SSMP ES--A-- $ZSM001     NONE        mxssmp       
>   
> [$Z000GHJ] 001,00020989 001 GEN  ES--A-- $ZLOBSRV1   NONE        mxlobsrvr    
>   
> g4t3037{joaniec}120: pstack $Z000GGA
> Usage: pstack <process-id>
> g4t3037{joaniec}121: pstack 20110
> Thread 30 (Thread 0x7fffee0a7700 (LWP 20111)):
> #0  0x0000003e3c00e84d in accept () from /lib64/libpthread.so.0
> #1  0x00007fffefec1995 in SB_Trans::Sock_Listener::accept() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #2  0x00007fffefec84e6 in SB_Trans::Sock_Stream_Accept_Thread::run() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #3  0x00007fffefec8213 in sock_stream_accept_thread_fun(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4  0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #5  0x00007fffee540fa7 in thread_fun(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #6  0x00007fffee54421c in sb_thread_sthr_disp(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 29 (Thread 0x7fffed6a6700 (LWP 20112)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007fffee5435e2 in SB_Thread::CV::wait() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #2  0x00007fffee5436be in SB_Thread::CV::wait(bool) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #3  0x00007fffefebac47 in SB_Sig_Queue::remove() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4  0x00007fffefec8977 in SB_Trans::Sock_Stream_Helper_Thread::run() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #5  0x00007fffefec823a in sock_helper_thread_fun(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #6  0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #7  0x00007fffee540fa7 in thread_fun(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #8  0x00007fffee54421c in sb_thread_sthr_disp(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #9  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 28 (Thread 0x7fffead82700 (LWP 20113)):
> #0  0x0000003e3bc3399d in sigtimedwait () from /lib64/libc.so.6
> #1  0x00007fffefe865c4 in local_monitor_reader(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #2  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #3  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 27 (Thread 0x7fffea301700 (LWP 20115)):
> #0  0x0000003e3c00ed2d in nanosleep () from /lib64/libpthread.so.0
> #1  0x00007fffef6032b9 in Sleep(unsigned int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libwin.so
> #2  0x00007ffff7747d27 in memMonitorUpdateThread(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #3  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #4  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 26 (Thread 0x7fffe977e700 (LWP 20116)):
> #0  0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
> #1  0x00007fffefec0cd5 in SB_Trans::Sock_Controller::epoll_wait(char const*, 
> int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #2  0x00007fffefec01f3 in SB_Trans::Sock_Comp_Thread::run() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #3  0x00007fffefec002d in sock_comp_thread_fun(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4  0x00007fffee540b4f in SB_Thread::Thread::disp(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #5  0x00007fffee540fa7 in thread_fun(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #6  0x00007fffee54421c in sb_thread_sthr_disp(void*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 25 (Thread 0x7fffe523e700 (LWP 20117)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 24 (Thread 0x7fffe513d700 (LWP 20118)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 23 (Thread 0x7fffe503c700 (LWP 20119)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 22 (Thread 0x7fffe4f3b700 (LWP 20120)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 21 (Thread 0x7fffe4e3a700 (LWP 20121)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 20 (Thread 0x7fffe4d39700 (LWP 20122)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 19 (Thread 0x7fffe4c38700 (LWP 20123)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 18 (Thread 0x7fffe4b37700 (LWP 20124)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 17 (Thread 0x7fffe4a36700 (LWP 20125)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 16 (Thread 0x7fffe4935700 (LWP 20126)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff151f9e3 in GCTaskManager::get_task(unsigned int) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff15210a8 in GCTaskThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 15 (Thread 0x7fffdfdec700 (LWP 20127)):
> #0  0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e88ef in os::PlatformEvent::park(long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff1970169 in VMThread::loop() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff1970470 in VMThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 14 (Thread 0x7fffdfceb700 (LWP 20128)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff1647378 in JVM_MonitorWait () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007fffe56ba7f8 in ?? ()
> #5  0x00007fffdfcd3390 in ?? ()
> #6  0x0000000000c33ca0 in ?? ()
> #7  0x00007fffdfcd31c0 in ?? ()
> #8  0x00007fffdfcd3158 in ?? ()
> #9  0x0000000000000000 in ?? ()
> Thread 13 (Thread 0x7fffdfbea700 (LWP 20129)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17d7add in ObjectMonitor::wait(long, bool, Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff1647378 in JVM_MonitorWait () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007fffe56ba7f8 in ?? ()
> #5  0x00007fffdfbd22e0 in ?? ()
> #6  0x0000000000c3e000 in ?? ()
> #7  0x0000000000c38210 in ?? ()
> #8  0x00007fffdfbd2268 in ?? ()
> #9  0x0000000000000000 in ?? ()
> Thread 12 (Thread 0x7fffdfae9700 (LWP 20130)):
> #0  0x0000003e3c00d720 in sem_wait () from /lib64/libpthread.so.0
> #1  0x00007ffff17e85ca in check_pending_signals(bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17e1835 in signal_thread_entry(JavaThread*, Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff1922d9f in JavaThread::thread_main_inner() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff1922ea5 in JavaThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #7  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 11 (Thread 0x7fffdf9e8700 (LWP 20131)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff1414ad8 in CompileQueue::get() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff1922d9f in JavaThread::thread_main_inner() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x00007ffff1922ea5 in JavaThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #8  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #9  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 10 (Thread 0x7fffdf8e7700 (LWP 20132)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab2b6 in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff1414ad8 in CompileQueue::get() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff1418bda in CompileBroker::compiler_thread_loop() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff1922d9f in JavaThread::thread_main_inner() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x00007ffff1922ea5 in JavaThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #8  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #9  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #10 0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 9 (Thread 0x7fffdf7e6700 (LWP 20133)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e3243 in os::PlatformEvent::park() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aaaaf in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff1879388 in ServiceThread::service_thread_entry(JavaThread*, 
> Thread*) () from /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff1922d9f in JavaThread::thread_main_inner() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff1922ea5 in JavaThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #8  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #9  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 8 (Thread 0x7fffdf6e5700 (LWP 20134)):
> #0  0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e88ef in os::PlatformEvent::park(long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff17aad5e in Monitor::IWait(Thread*, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007ffff17ab23e in Monitor::wait(bool, long, bool) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #4  0x00007ffff191edc1 in WatcherThread::sleep() const () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #5  0x00007ffff191f2de in WatcherThread::run() () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #6  0x00007ffff17e9988 in java_start(Thread*) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #7  0x0000003e3c007851 in start_thread () from /lib64/libpthread.so.0
> #8  0x0000003e3bce890d in clone () from /lib64/libc.so.6
> Thread 7 (Thread 0x7fffde317700 (LWP 20137)):
> #0  0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e8114 in Parker::park(bool, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff1946a05 in Unsafe_Park () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007fffe56ba7f8 in ?? ()
> #4  0x0000000000bb4968 in ?? ()
> #5  0x0000000000000000 in ?? ()
> Thread 6 (Thread 0x7fffddedb700 (LWP 20141)):
> #0  0x0000003e3bcdf253 in poll () from /lib64/libc.so.6
> #1  0x00007fffde32a2ca in 
> Java_org_apache_hadoop_net_unix_DomainSocketWatcher_doPoll0 () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/sql/local_hadoop/hadoop-2.5.0-cdh5.3.0/lib/native/libhadoop.so.1.0.0
> #2  0x00007fffe56ba7f8 in ?? ()
> #3  0x00000000020a4800 in ?? ()
> #4  0x0000000000000000 in ?? ()
> Thread 5 (Thread 0x7fffde2de700 (LWP 20153)):
> #0  0x0000003e3bce8f03 in epoll_wait () from /lib64/libc.so.6
> #1  0x00007fffde96b707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from 
> /opt/home/tools/jdk1.7.0_67/jre/lib/amd64/libnio.so
> #2  0x00007fffe56ba7f8 in ?? ()
> #3  0x0000000000000000 in ?? ()
> Thread 4 (Thread 0x7fffddbd8700 (LWP 20154)):
> #0  0x0000003e3c00b43c in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e8228 in Parker::park(bool, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff1946a05 in Unsafe_Park () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007fffe56ba7f8 in ?? ()
> #4  0x00007fffe56ae058 in ?? ()
> #5  0x0000000000000000 in ?? ()
> Thread 3 (Thread 0x7fffddfdc700 (LWP 20156)):
> #0  0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e8114 in Parker::park(bool, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff1946a05 in Unsafe_Park () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007fffe56ba7f8 in ?? ()
> #4  0x00007fffe56ae058 in ?? ()
> #5  0x0000000000000000 in ?? ()
> Thread 2 (Thread 0x7fffdddda700 (LWP 20236)):
> #0  0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007ffff17e8114 in Parker::park(bool, long) () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #2  0x00007ffff1946a05 in Unsafe_Park () from 
> /home/tools/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
> #3  0x00007fffe56ba7f8 in ?? ()
> #4  0x00007fffe56ae058 in ?? ()
> #5  0x0000000000000000 in ?? ()
> Thread 1 (Thread 0x7fffee315c00 (LWP 20110)):
> #0  0x0000003e3c00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x00007fffee543778 in SB_Thread::CV::wait(int, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #2  0x00007fffee54383d in SB_Thread::CV::wait(bool, int, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbutil.so
> #3  0x00007fffefe988cb in SB_Ms_Event_Mgr::wait(long) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #4  0x00007fffefeb916e in XWAIT_com(short, int, bool) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #5  0x00007fffefeb8f1f in XWAIT(short, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbms.so
> #6  0x00007fffefc3731a in fs_int_fs_file_awaitiox(short*, void**, int*, 
> long*, int, short*, bool, bool) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
> #7  0x00007fffefc30e15 in BAWAITIOXTS(short*, void**, int*, long*, int, 
> short*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsbfs.so
> #8  0x00007ffff70be1f7 in GuaConnectionToServer::wait(int, unsigned int*, 
> IpcAwaitiox*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #9  0x00007ffff70b51cb in IpcSetOfConnections::waitOnSet(int, int, int*) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #10 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #11 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #12 0x00007ffff70bdf42 in GuaConnectionToServer::tryToStartNewIO() () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #13 0x00007ffff70beb38 in GuaConnectionToServer::send(IpcMessageBuffer*) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #14 0x00007ffff70b72e4 in IpcMessageStream::send(int, long) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #15 0x00007ffff773fad5 in ExSqlComp::sendRequest(CmpMessageObj*, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #16 0x00007ffff773e620 in 
> ExSqlComp::sendRequest(CmpMessageObj::MessageTypeEnum, char const*, unsigned 
> int, int, long*, int, int, char const*, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #17 0x00007ffff775fdd9 in CliStatement::prepare2(char*, ComDiagsArea&, char*, 
> unsigned int, int, int, unsigned int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #18 0x00007ffff7760846 in CliStatement::prepare(char*, ComDiagsArea&, char*, 
> unsigned int, int, int, unsigned int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #19 0x00007ffff77130d4 in SQLCLI_Prepare2 () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #20 0x00007ffff776dc6e in SQL_EXEC_Prepare2 () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcli.so
> #21 0x00007ffff3c819d2 in ExeCliInterface::prepare(char const*, 
> SQLMODULE_ID_t*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, SQLCLI_OBJ_ID*, 
> SQLCLI_OBJ_ID*, char**, Queue*, char**, Queue*, char*, int*, 
> SQL_QUERY_COST_INFO*, SQL_QUERY_COMPILER_STATS_INFO*, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
> #22 0x00007ffff3c83460 in ExeCliInterface::fetchRowsPrologue(char const*, 
> int, int, char*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libexecutor.so
> #23 0x00007ffff7a627fb in 
> CmpSeabaseDDL::getObjectUIDandOwners(ExeCliInterface*, char const*, char 
> const*, char const*, ComObjectType, int&, int&, bool, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #24 0x00007ffff7ae763d in CmpSeabaseDDL::getSeabaseUserTableDesc(NAString 
> const&, NAString const&, NAString const&, ComObjectType, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #25 0x00007ffff7aea687 in CmpSeabaseDDL::getSeabaseTableDesc(NAString const&, 
> NAString const&, NAString const&, ComObjectType, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #26 0x00007ffff4d5d304 in NATableDB::get(CorrName&, BindWA*, desc_struct*) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #27 0x00007ffff4ab2efc in BindWA::getNATable(CorrName&, int, desc_struct*) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #28 0x00007ffff4ad361e in Scan::bindNode(BindWA*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #29 0x00007ffff4a9fe07 in RelExpr::bindChildren(BindWA*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #30 0x00007ffff4adffde in RelRoot::bindNode(BindWA*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/liboptimizer.so
> #31 0x00007ffff7a3f50e in CmpMain::compile(char const*, int, RelExpr*&, 
> char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int, 
> int, int*, timeval*, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #32 0x00007ffff7a4232c in CmpMain::sqlcomp(char const*, int, RelExpr*&, 
> char**, unsigned int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int, 
> int, int*, timeval*, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #33 0x00007ffff7a43630 in CmpMain::sqlcomp(QueryText&, int, char**, unsigned 
> int*, NAMemory*, CmpMain::CompilerPhase, FragmentDir**, int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libsqlcomp.so
> #34 0x00007ffff40dfc08 in CmpStatement::process(CmpMessageSQLText const&) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
> #35 0x00007ffff40ca93d in ExCmpMessage::actOnReceive(IpcConnection*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libarkcmplib.so
> #36 0x00007ffff70a58d6 in 
> IpcMessageStream::internalActOnReceive(IpcMessageBuffer*, IpcConnection*) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #37 0x00007ffff70c02fb in 
> GuaConnectionToClient::acceptBuffer(IpcMessageBuffer*, unsigned int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #38 0x00007ffff70c100f in GuaReceiveControlConnection::wait(int, unsigned 
> int*, IpcAwaitiox*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #39 0x00007ffff70c23b2 in GuaConnectionToClient::wait(int, unsigned int*, 
> IpcAwaitiox*) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #40 0x00007ffff70b54aa in IpcSetOfConnections::waitOnSet(int, int, int*) () 
> from /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #41 0x00007ffff70b633d in IpcMessageStream::waitOnMsgStream(int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #42 0x00007ffff70b6692 in IpcMessageStream::receive(int) () from 
> /opt/home/joaniec/git_ws_HBase98_30/core/sqf/export/lib64/libcommon.so
> #43 0x00000000004064ae in main ()
> One more quick note, even after the sqlci session ended,
> only the initial tdm_arkcmp process was cleaned up.
> The others remained:
> Processing cluster.conf on local host g4t3037.houston.hp.com
> [$Z000IRF] Shell/shell Version 1.0.1 Release 1.1.0 (Build release [joaniec], 
> dat
> e 26Mar15)
> [$Z000IRF] %ps  
> [$Z000IRF] NID,PID(os)  PRI TYPE STATES  NAME        PARENT      PROGRAM
> [$Z000IRF] ------------ --- ---- ------- ----------- ----------- 
> ---------------
> [$Z000IRF] 000,00018562 000 WDG  ES--A-- $WDG000     NONE        sqwatchdog   
>   
> [$Z000IRF] 000,00018563 000 PSD  ES--A-- $PSD000     NONE        pstartd      
>   
> [$Z000IRF] 000,00018592 001 DTM  ES--A-- $TM0        NONE        tm           
>   
> [$Z000IRF] 000,00019243 001 GEN  ES--A-- $ZSC000     NONE        mxsscp       
>   
> [$Z000IRF] 000,00019274 001 SSMP ES--A-- $ZSM000     NONE        mxssmp       
>   
> [$Z000IRF] 000,00020982 001 GEN  ES--A-- $ZLOBSRV0   NONE        mxlobsrvr    
>   
> [$Z000IRF] 000,00007416 001 GEN  ES--A-- $Z00061W    NONE        tdm_arkcmp   
>   
> [$Z000IRF] 000,00007960 001 GEN  ES--A-- $Z0006HF    $Z00061W    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008021 001 GEN  ES--A-- $Z0006J6    $Z0006HF    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008079 001 GEN  ES--A-- $Z0006KU    $Z0006J6    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008137 001 GEN  ES--A-- $Z0006MH    $Z0006KU    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008194 001 GEN  ES--A-- $Z0006P4    $Z0006MH    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008252 001 GEN  ES--A-- $Z0006QS    $Z0006P4    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008312 001 GEN  ES--A-- $Z0006SH    $Z0006QS    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008369 001 GEN  ES--A-- $Z0006U4    $Z0006SH    tdm_arkcmp   
>   
> [$Z000IRF] 000,00008427 001 GEN  ES--A-- $Z0006VS    $Z0006U4    tdm_arkcmp   
>  
> …
> The new memoryUsage threshold and throttle capabilities were used to initially
> see the problem.
> This was on the CDH 5.3 workstation using the install_local_hadoop support
> on Trafidion R1.1.0 build version git032415 or later build.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to