[
https://issues.apache.org/jira/browse/TRAFODION-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Subbiah closed TRAFODION-1076.
-------------------------------------
> LP Bug: 1435479 - Explain queries cause mxosrvr to hang at
> ex_root_tcb::fetchMultiple()
> ---------------------------------------------------------------------------------------
>
> Key: TRAFODION-1076
> URL: https://issues.apache.org/jira/browse/TRAFODION-1076
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-exe
> Reporter: Weishiun Tsai
> Assignee: Anoop Sharma
> Priority: Blocker
> Fix For: 1.1 (pre-incubation)
>
>
> Certain explain queries have caused mxosrvrs to hang at
> ex_root_tcb::fetchMultiple(). When this happens, killing the client trafci
> connection does not release the mxosrvr either. The mxosrvr continues to
> stay in the same looping state and can’t be reused.
> This is seen on the v0320 build installed on a 6-node cluster. The initial
> analysis is that it has something to do with the CQD explain_space_opt, which
> is on by default. An experiment to turn it off does seem to get the same
> test that was stuck previously going again.
> This is a test stopper for QA as a large number of QA SQL regression tests
> use explain queries to verify the plan quality. For now, we will try to set
> this CQD in the defaults table to turn it off manually. But the suggestion
> is to turn it off from the code in the next build until the underlying issue
> is figured out.
> One of the tests that encountered this problem and got mxosrvr to hang for
> 20+ hours was the test in compiler/explain: tcase.test152. The explain query
> is shown here as 'explainit'. The actually query is a query that uses QA's
> tpch2x tables. This problem shows up only as a combination of the actual
> query plan and the explainit query, as the same set of tests have executed
> explainit earlier with other queries and they did not see this problem:
> >>set schema g_tpch2x;
> --- SQL operation complete.
> >>showddl lineitem;
> CREATE TABLE TRAFODION.G_TPCH2X.LINEITEM
> (
> L_ORDERKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_PARTKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_SUPPKEY INT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_LINENUMBER INT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_QUANTITY NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
> DROPPABLE
> , L_EXTENDEDPRICE NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
> DROPPABLE
> , L_DISCOUNT NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
> DROPPABLE
> , L_TAX NUMERIC(12, 2) NO DEFAULT NOT NULL NOT
> DROPPABLE
> , L_RETURNFLAG CHAR(1) CHARACTER SET ISO88591 COLLATE
> DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_LINESTATUS CHAR(1) CHARACTER SET ISO88591 COLLATE
> DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_SHIPDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
> , L_COMMITDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
> , L_RECEIPTDATE DATE NO DEFAULT NOT NULL NOT DROPPABLE
> , L_SHIPINSTRUCT CHAR(25) CHARACTER SET ISO88591 COLLATE
> DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_SHIPMODE CHAR(10) CHARACTER SET ISO88591 COLLATE
> DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
> , L_COMMENT VARCHAR(44) CHARACTER SET ISO88591
> COLLATE
> DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE
> , PRIMARY KEY (L_SHIPDATE ASC, L_ORDERKEY ASC, L_LINENUMBER ASC)
> )
> SALT USING 8 PARTITIONS
> ;
> --- SQL operation complete.
> >>select count(*) from lineitem;
> (EXPR)
> --------------------
> 11997996
> --- 1 row(s) selected.
> prepare explainIt from
> select substring(cast(SEQ_NUM+100 as char(3)),2,2) s,
> cast(LEFT_CHILD_SEQ_NUM as char(2)) lc,
> cast(RIGHT_CHILD_SEQ_NUM as char(2)) rc,
> substring(operator,1,16) operator,
> substring
> (substring(tname from (1+locate('.',tname))),
> (1+locate('.',substring(tname from (1+locate('.',tname))))),
> 10
> ) tab_name,
> cast(cardinality as char(11)) cardinal,
> cast(operator_cost as char(11)) op_cost,
> cast(total_cost as char(11)) tot_cost,
> cast(detail_cost as char(256)) detail_cost -- This is the one that shows all
> the individual cost components in detail.
> from table (explain(NULL,'XX'))
> order by 1 desc;
> prepare xx from
> select [last 0]
> l_returnflag,
> l_linestatus,
> sum(l_quantity) over (order by l_quantity
> rows between unbounded preceding and 10 preceding ) as sum_qty ,
> sum(l_extendedprice) over (order by l_quantity
> rows between unbounded preceding and current row) as sum_base_price ,
> sum(l_extendedprice * (1 - l_discount)) over (order by l_quantity
> rows between unbounded preceding and unbounded following ) as sum_disc_price,
> sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) over (order by
> l_quantity
> rows between current row and current row ) as sum_charge,
> avg(l_quantity) over (order by l_quantity
> rows between current row and unbounded following ) as avg_qty,
> avg(l_extendedprice) over (order by l_quantity
> rows between current row and 5 following)as avg_price,
> avg(l_discount) over (order by l_quantity
> rows between 6 preceding and 4 preceding ) as avg_disc,
> count(*) as count_order
> from
> lineitem
> where
> l_shipdate <= date '1998-12-01' - interval '72' day(2)
> group by
> l_returnflag,
> l_linestatus,
> l_quantity,
> l_discount,
> l_extendedprice,
> l_tax
> order by
> l_returnflag,
> l_linestatus;
> execute explainit;
> ---------------------------------------------
> Here is the pstack of the mxosrvr that got stuck. Thread 34 is the one
> handling the request:
> Thread 40 (Thread 0x7fffead93700 (LWP 12490)):
> #0 0x00007ffff4af64bd in accept () from /lib64/libc.so.6
> #1 0x00007ffff73639b5 in SB_Trans::Sock_Listener::accept (this=0xeaa950) at
> soc
> k.cpp:618
> #2 0x00007ffff736a506 in SB_Trans::Sock_Stream_Accept_Thread::run
> (this=0xeab8f
> 0) at sockstream.cpp:2085
> #3 0x00007ffff736a233 in sock_stream_accept_thread_fun (pp_arg=0xeab8f0) at
> soc
> kstream.cpp:2019
> #4 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xeab8f0,
> pp_arg=0xeab8f
> 0) at thread.cpp:211
> #5 0x00007ffff47efff7 in thread_fun (pp_arg=0xeab8f0) at thread.cpp:307
> #6 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xeaba20) at
> threadl.cpp:2
> 53
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 39 (Thread 0x7fffea392700 (LWP 12491)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff47f2656 in SB_Thread::CV::wait (this=0xeac288) at
> /home/jenkins/w
> orkspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/include/seabed/int/th
> read.inl:552
> #2 0x00007ffff47f2732 in SB_Thread::CV::wait (this=0xeac288, pv_lock=false)
> at
> /home/jenkins/workspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/includ
> e/seabed/int/thread.inl:591
> #3 0x00007ffff735cc67 in SB_Sig_Queue::remove (this=0xeac230) at
> queue.inl:473
> #4 0x00007ffff736a997 in SB_Trans::Sock_Stream_Helper_Thread::run
> (this=0xeabce
> 0) at sockstream.cpp:2179
> #5 0x00007ffff736a25a in sock_helper_thread_fun (pp_arg=0xeabce0) at
> sockstream
> .cpp:2029
> #6 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xeabce0,
> pp_arg=0xeabce
> 0) at thread.cpp:211
> #7 0x00007ffff47efff7 in thread_fun (pp_arg=0xeabce0) at thread.cpp:307
> #8 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xeac3f0) at
> threadl.cpp:2
> 53
> #9 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #10 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 38 (Thread 0x7fffe7a6e700 (LWP 12494)):
> #0 0x00007ffff4a4071d in sigtimedwait () from /lib64/libc.so.6
> #1 0x00007ffff73285e4 in local_monitor_reader (pp_arg=0x18c1) at
> ../../../monit
> or/linux/clio.cxx:130
> #2 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #3 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 37 (Thread 0x7fffe706d700 (LWP 12495)):
> #0 0x00007ffff4aec0d3 in poll () from /lib64/libc.so.6
> #1 0x00007ffff63c4482 in do_io (v=0xeaeec0) at src/mt_adaptor.c:387
> #2 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #3 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 36 (Thread 0x7fffe666c700 (LWP 12496)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff63c426b in do_completion (v=0xeaeec0) at src/mt_adaptor.c:463
> #2 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #3 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 35 (Thread 0x7fffe5c6b700 (LWP 12513)):
> #0 0x00007ffff4a3fb05 in sigwait () from /lib64/libc.so.6
> #1 0x00007ffff7371abf in SB_Timer_Thread::run (this=0xeb6f30) at
> timer.cpp:270
> #2 0x00007ffff73718c7 in sb_timer_thread_fun (pp_arg=0xeb6f30) at
> timer.cpp:233
> #3 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xeb6f30,
> pp_arg=0xeb6f3
> 0) at thread.cpp:211
> #4 0x00007ffff47efff7 in thread_fun (pp_arg=0xeb6f30) at thread.cpp:307
> #5 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xeb0380) at
> threadl.cpp:2
> 53
> #6 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #7 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 34 (Thread 0x7fffe520c700 (LWP 12514)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff47f27ec in SB_Thread::CV::wait (this=0xed6688, pv_sec=60,
> pv_us=2
> 00000) at
> /home/jenkins/workspace/traf-pub-release-ahw2.2/trafodion/core/sqf/exp
> ort/include/seabed/int/thread.inl:649
> #2 0x00007ffff47f28b1 in SB_Thread::CV::wait (this=0xed6688, pv_lock=true,
> pv_s
> ec=60, pv_us=200000) at
> /home/jenkins/workspace/traf-pub-release-ahw2.2/trafodio
> n/core/sqf/export/include/seabed/int/thread.inl:701
> #3 0x00007ffff733a8eb in SB_Ms_Event_Mgr::wait (this=0xed6590,
> pv_us=60200000)
> at mseventmgr.inl:351
> #4 0x00007ffff735b18e in XWAIT_com (pv_mask=1024, pv_time=6020,
> pv_residual=fal
> se) at pctl.cpp:979
> #5 0x00007ffff735af8f in XWAITNO0 (pv_mask=1024, pv_time=6020) at
> pctl.cpp:902
> #6 0x00007ffff3f74daa in IpcSetOfConnections::waitOnSet
> (this=0x7fffe3c07268, t
> imeout=-1, calledByESP=0, timedout=0x7fffe51f3ff8) at ../common/Ipc.cpp:1593
> #7 0x00007ffff2d5af24 in waitOnAll (waitTime=<optimized out>,
> timedout=0x7fffe5
> 1f3ff8, calledByESP=0, timeout=-1, this=0x7fffe3c07200) at
> ../common/Ipc.h:3089
> #8 ex_root_tcb::fetch (this=0x7fffe3d379f8, cliGlobals=<optimized out>,
> glob=0x
> 7fffe3d1dfd8, output_desc=0x7fffe3d4f808, diagsArea=@0x7fffe51f4180: 0x0,
> timeLi
> mit=-1, newOperation=1, closeCursorOnError=@0x7fffe51f418c: 1) at
> ../executor/ex
> _root.cpp:1934
> #9 0x00007ffff2d5c46a in ex_root_tcb::fetchMultiple (this=0x7fffe3d379f8,
> cliGl
> obals=0xee53d0, glob=0x7fffe3d1dfd8, output_desc=0x7fffe3d4f808,
> diagsArea=@0x7f
> ffe51f4180: 0x0, timeLimit=-1, newOperation=1,
> closeCursorOnError=@0x7fffe51f418
> c: 1, eodSeen=@0x7fffe51f4188: 0) at ../executor/ex_root.cpp:2010
> #10 0x00007ffff437709a in CliStatement::fetch (this=0x7fffe3d33258,
> cliGlobals=0
> xee53d0, output_desc=0x7fffe3d4f808, diagsArea=..., newOperation=1) at
> ../cli/St
> atement.cpp:4768
> #11 0x00007ffff431ddff in SQLCLI_PerformTasks(CliGlobals *, ULng32,
> SQLSTMT_ID *
> , SQLDESC_ID *, SQLDESC_ID *, Lng32, Lng32, typedef __va_list_tag
> __va_list_tag
> *, SQLCLI_PTR_PAIRS *, SQLCLI_PTR_PAIRS *) (cliGlobals=0xee53d0, tasks=5924,
> sta
> tement_id=0x6a3c8a8, input_descriptor=0x0, output_descriptor=0x6a3c908,
> num_inpu
> t_ptr_pairs=-472582136, num_output_ptr_pairs=0, ap=0x7fffe51f4320,
> input_ptr_pai
> rs=0x0, output_ptr_pairs=0x0) at ../cli/Cli.cpp:3383
> #12 0x00007ffff431e81d in SQLCLI_Fetch(CliGlobals *, SQLSTMT_ID *, SQLDESC_ID
> *,
> Lng32, typedef __va_list_tag __va_list_tag *, SQLCLI_PTR_PAIRS *)
> (cliGlobals=<
> optimized out>, statement_id=<optimized out>, output_descriptor=<optimized
> out>,
> num_ptr_pairs=<optimized out>, ap=<optimized out>, ptr_pairs=<optimized out>)
> a
> t ../cli/Cli.cpp:3821
> #13 0x00007ffff43833cb in SQL_EXEC_Fetch (statement_id=0x6a3c8a8,
> output_descrip
> tor=0x6a3c908, num_ptr_pairs=0) at ../cli/CliExtern.cpp:2737
> #14 0x00007ffff6870e01 in SRVR::WSQL_EXEC_Fetch (statement_id=0x6a3c8a8,
> output_
> descriptor=0x6a3c908, num_ptr_pairs=0) at SQLWrapper.cpp:514
> #15 0x00007ffff685f64a in SRVR::FETCH2bulk (pSrvrStmt=0x6a3c290) at
> sqlinterface
> .cpp:6154
> #16 0x00007ffff688dbea in odbc_SQLSrvr_FetchPerf_sme_ (objtag_=<optimized
> out>,
> call_id_=<optimized out>, returnCode=0x7fffe51f467c, dialogueId=<optimized
> out>,
> stmtLabel=0x6585cc4 "EXPLAINIT", maxRowCnt=100, maxRowLen=0, sqlAsyncEnable=0,
> queryTimeout=0, rowsAffected=0x7fffe51f4678, outValuesFormat=0x7fffe51f4674,
> out
> putDataValue=0x7fffe51f4650, sqlWarningOrErrorLength=0x7fffe51f4670,
> sqlWarningO
> rError=@0x7fffe51f4668: 0x0) at srvrothers.cpp:5815
> #17 0x00000000004b6e07 in odbc_SQLSrvr_Fetch_ame_ (objtag_=0x69701c0,
> call_id_=0
> x6970218, dialogueId=468739187, operation_id=<optimized out>,
> sqlAsyncEnable=0,
> queryTimeout=0, stmtHandle=0, stmtLabel=0x6585cc4 "EXPLAINIT", maxRowCnt=100,
> ma
> xRowLen=0) at SrvrConnect.cpp:7498
> #18 0x000000000048bd52 in SQLFETCH_IOMessage (objtag_=0x69701c0,
> call_id_=0x6970
> 218, operation_id=3009) at Interface/odbcs_srvr.cpp:1021
> #19 0x000000000049427e in DISPATCH_TCPIPRequest (objtag_=0x69701c0,
> call_id_=0x6
> 970218, operation_id=<optimized out>) at Interface/odbcs_srvr.cpp:1803
> #20 0x0000000000433872 in BUILD_TCPIP_REQUEST (pnode=0x69701c0) at
> ../Common/TCP
> IPSystemSrvr.cpp:603
> #21 0x000000000043420d in PROCESS_TCPIP_REQUEST (pnode=0x69701c0) at
> ../Common/T
> CPIPSystemSrvr.cpp:581
> #22 0x00000000004623f6 in CNSKListenerSrvr::tcpip_listener (arg=0xda0e50) at
> Int
> erface/linux/Listener_srvr_ps.cpp:400
> #23 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xed1b80) at
> threadl.cpp:2
> 53
> #24 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #25 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 33 (Thread 0x7fffe478a700 (LWP 23365)):
> #0 0x00007ffff4ab9a3d in nanosleep () from /lib64/libc.so.6
> #1 0x00007ffff6ee32d9 in Sleep (milliSecs=<optimized out>) at
> traf_misc.cpp:136
> #2 0x00007ffff435bd37 in memMonitorUpdateThread (param=0x7fffe478caf0) at
> ../cl
> i/memorymonitor.cpp:73
> #3 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #4 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 32 (Thread 0x7fffe3c05700 (LWP 23366)):
> #0 0x00007ffff4af5ef3 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007ffff7362cf5 in SB_Trans::Sock_Controller::epoll_wait
> (this=0x7ffff75b
> 3ca0 <gv_sock_ctlr>, pp_where=0x7ffff7385c08 "Sock_Comp_Thread::run",
> pv_timeout
> =-1) at sock.cpp:336
> #2 0x00007ffff7362213 in SB_Trans::Sock_Comp_Thread::run (this=0xee7ad0) at
> soc
> k.cpp:104
> #3 0x00007ffff736204d in sock_comp_thread_fun (pp_arg=0xee7ad0) at
> sock.cpp:74
> #4 0x00007ffff47efb9f in SB_Thread::Thread::disp (this=0xee7ad0,
> pp_arg=0xee7ad
> 0) at thread.cpp:211
> #5 0x00007ffff47efff7 in thread_fun (pp_arg=0xee7ad0) at thread.cpp:307
> #6 0x00007ffff47f3290 in sb_thread_sthr_disp (pp_arg=0xee7bf0) at
> threadl.cpp:2
> 53
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 31 (Thread 0x7fffdf7af700 (LWP 23367)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 30 (Thread 0x7fffdf6ae700 (LWP 23368)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 29 (Thread 0x7fffdf5ad700 (LWP 23369)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 28 (Thread 0x7fffdf4ac700 (LWP 23370)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 27 (Thread 0x7fffdf3ab700 (LWP 23371)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 26 (Thread 0x7fffdf2aa700 (LWP 23372)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 25 (Thread 0x7fffdf1a9700 (LWP 23373)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 24 (Thread 0x7fffdf0a8700 (LWP 23374)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 23 (Thread 0x7fffdefa7700 (LWP 23375)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 22 (Thread 0x7fffdeea6700 (LWP 23376)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5880c63 in GCTaskManager::get_task(unsigned int) () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5882328 in GCTaskThread::run() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 21 (Thread 0x7fffda347700 (LWP 23377)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c9ee in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5cd3eb9 in VMThread::loop() () from
> /usr/java/jdk1.7.0_75/jre/lib
> /amd64/server/libjvm.so
> #5 0x00007ffff5cd41c0 in VMThread::run() () from
> /usr/java/jdk1.7.0_75/jre/lib/
> amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 20 (Thread 0x7fffda246700 (LWP 23378)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b39cbd in ObjectMonitor::wait(long, bool, Thread*) () from
> /usr/
> java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff59a8538 in JVM_MonitorWait () from
> /usr/java/jdk1.7.0_75/jre/lib/
> amd64/server/libjvm.so
> #4 0x00007fffdfd39208 in ?? ()
> #5 0x00000000c0034eb8 in ?? ()
> #6 0x00007fffda22e2d8 in ?? ()
> #7 0x00000000bb7d0170 in ?? ()
> #8 0x0000000000000000 in ?? ()
> Thread 19 (Thread 0x7fffda145700 (LWP 23379)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b39cbd in ObjectMonitor::wait(long, bool, Thread*) () from
> /usr/
> java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff59a8538 in JVM_MonitorWait () from
> /usr/java/jdk1.7.0_75/jre/lib/
> amd64/server/libjvm.so
> #4 0x00007fffdfd39208 in ?? ()
> #5 0x00000000c0035720 in ?? ()
> #6 0x00007ffff592433f in InterpreterRuntime::monitorenter(JavaThread*,
> BasicObj
> ectLock*) () from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #7 0x00007fffdfb36058 in ?? ()
> #8 0x00007fffdfb36058 in ?? ()
> #9 0x0000000000000000 in ?? ()
> Thread 18 (Thread 0x7fffd41b3700 (LWP 23380)):
> #0 0x00007ffff45c5930 in sem_wait () from /lib64/libpthread.so.0
> #1 0x00007ffff5b4a7ca in check_pending_signals(bool) () from
> /usr/java/jdk1.7.0
> _75/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b43a15 in signal_thread_entry(JavaThread*, Thread*) () from
> /usr
> /java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5c85825 in JavaThread::run() () from
> /usr/java/jdk1.7.0_75/jre/li
> b/amd64/server/libjvm.so
> #5 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #6 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #7 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 17 (Thread 0x7fffd40b2700 (LWP 23381)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cf46 in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5775d18 in CompileQueue::get() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #5 0x00007ffff5779e1a in CompileBroker::compiler_thread_loop() () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #7 0x00007ffff5c85825 in JavaThread::run() () from
> /usr/java/jdk1.7.0_75/jre/li
> b/amd64/server/libjvm.so
> #8 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #9 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #10 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 16 (Thread 0x7fffd3fb1700 (LWP 23382)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cf46 in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5775d18 in CompileQueue::get() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #5 0x00007ffff5779e1a in CompileBroker::compiler_thread_loop() () from
> /usr/jav
> a/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #7 0x00007ffff5c85825 in JavaThread::run() () from
> /usr/java/jdk1.7.0_75/jre/li
> b/amd64/server/libjvm.so
> #8 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #9 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #10 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 15 (Thread 0x7fffd3eb0700 (LWP 23383)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b45423 in os::PlatformEvent::park() () from
> /usr/java/jdk1.7.0_7
> 5/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c73f in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5bdbd18 in ServiceThread::service_thread_entry(JavaThread*,
> Threa
> d*) () from /usr/java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5c8571f in JavaThread::thread_main_inner() () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #6 0x00007ffff5c85825 in JavaThread::run() () from
> /usr/java/jdk1.7.0_75/jre/li
> b/amd64/server/libjvm.so
> #7 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #8 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #9 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 14 (Thread 0x7fffd3daf700 (LWP 23384)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b0c9ee in Monitor::IWait(Thread*, long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff5b0cece in Monitor::wait(bool, long, bool) () from
> /usr/java/jdk1
> .7.0_75/jre/lib/amd64/server/libjvm.so
> #4 0x00007ffff5c81741 in WatcherThread::sleep() const () from
> /usr/java/jdk1.7.
> 0_75/jre/lib/amd64/server/libjvm.so
> #5 0x00007ffff5c81c5e in WatcherThread::run() () from
> /usr/java/jdk1.7.0_75/jre
> /lib/amd64/server/libjvm.so
> #6 0x00007ffff5b4bb88 in java_start(Thread*) () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #8 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 13 (Thread 0x7fffd24da700 (LWP 23397)):
> #0 0x00007ffff4aec0d3 in poll () from /lib64/libc.so.6
> #1 0x00007fffd24ef5ea in
> Java_org_apache_hadoop_net_unix_DomainSocketWatcher_do
> Poll0 () from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
> #2 0x00007fffdfb427f8 in ?? ()
> #3 0x0000000000000000 in ?? ()
> Thread 12 (Thread 0x7fffd2afd700 (LWP 23410)):
> #0 0x00007ffff4af5ef3 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007fffd30f1707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from
> /u
> sr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
> #2 0x00007fffdfcdfc52 in ?? ()
> #3 0x00000000c005a850 in ?? ()
> #4 0x00007fffd2ae5058 in ?? ()
> #5 0x00000000bb6afd24 in ?? ()
> #6 0x00007fffd2ae50a8 in ?? ()
> #7 0x00000000bb6b5e28 in ?? ()
> #8 0x00000000bcc04aa0 in ?? ()
> #9 0x00000000bb6afd40 in ?? ()
> #10 0x0000000000000000 in ?? ()
> Thread 11 (Thread 0x7fffd21d7700 (LWP 23411)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b4a428 in Parker::park(bool, long) () from
> /usr/java/jdk1.7.0_75
> /jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5ca93f5 in Unsafe_Park () from
> /usr/java/jdk1.7.0_75/jre/lib/amd6
> 4/server/libjvm.so
> #3 0x00007fffdfb427f8 in ?? ()
> #4 0x00007fffdfb36058 in ?? ()
> #5 0x0000000000000000 in ?? ()
> Thread 10 (Thread 0x7fffd18cc700 (LWP 23427)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00000000004cc162 in get_task (this=0x738b80 <repos_queue>) at
> SrvrConnect.
> cpp:340
> #2 SessionWatchDog (arg=<optimized out>) at SrvrConnect.cpp:499
> #3 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #4 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 9 (Thread 0x7fffd0ecb700 (LWP 23428)):
> #0 0x00007ffff45c6264 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00007ffff45c1523 in _L_lock_892 () from /lib64/libpthread.so.0
> #2 0x00007ffff45c1407 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x00000000004cbb67 in SessionWatchDog (arg=<optimized out>) at
> SrvrConnect.c
> pp:366
> #4 0x00007ffff45bf9d1 in start_thread () from /lib64/libpthread.so.0
> #5 0x00007ffff4af58fd in clone () from /lib64/libc.so.6
> Thread 8 (Thread 0x7fffd1bd1700 (LWP 23542)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b3955b in ObjectMonitor::wait(long, bool, Thread*) () from
> /usr/
> java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff59a8538 in JVM_MonitorWait () from
> /usr/java/jdk1.7.0_75/jre/lib/
> amd64/server/libjvm.so
> #4 0x00007fffdfd39208 in ?? ()
> #5 0x00000000c005e6a8 in ?? ()
> #6 0x0000000000000000 in ?? ()
> Thread 7 (Thread 0x7fffce92c700 (LWP 23543)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4a314 in Parker::park(bool, long) () from
> /usr/java/jdk1.7.0_75
> /jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5ca93f5 in Unsafe_Park () from
> /usr/java/jdk1.7.0_75/jre/lib/amd6
> 4/server/libjvm.so
> #3 0x00007fffdfdb3bca in ?? ()
> #4 0x00000000c0040520 in ?? ()
> #5 0x00007ffff348ce46 in clock_gettime () from /lib64/librt.so.1
> #6 0x00007ffff5b452af in os::javaTimeNanos() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007fffdfedfa38 in ?? ()
> #8 0x00000000c005a650 in ?? ()
> #9 0x00007fffce9142f0 in ?? ()
> #10 0x00000000baf73958 in ?? ()
> #11 0x0000000000000000 in ?? ()
> Thread 6 (Thread 0x7fffd1fd5700 (LWP 23547)):
> #0 0x00007ffff4af5ef3 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007fffd30f1707 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from
> /u
> sr/java/jdk1.7.0_75/jre/lib/amd64/libnio.so
> #2 0x00007fffdfcdfc52 in ?? ()
> #3 0x00000000c004faf8 in ?? ()
> #4 0x00007fffd1fbd158 in ?? ()
> #5 0x00000000bb6afd24 in ?? ()
> #6 0x00007fffd1fbd1a8 in ?? ()
> #7 0x00000000bb6b5e28 in ?? ()
> #8 0x00000000bcc04aa0 in ?? ()
> #9 0x00000000bb6afd40 in ?? ()
> #10 0x0000000000000000 in ?? ()
> Thread 5 (Thread 0x7fffce82b700 (LWP 23548)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff5b4a428 in Parker::park(bool, long) () from
> /usr/java/jdk1.7.0_75
> /jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5ca93f5 in Unsafe_Park () from
> /usr/java/jdk1.7.0_75/jre/lib/amd6
> 4/server/libjvm.so
> #3 0x00007fffdfb427f8 in ?? ()
> #4 0x00007fffdfb36058 in ?? ()
> #5 0x0000000000000000 in ?? ()
> Thread 4 (Thread 0x7fffcdf37700 (LWP 5594)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b4b5d4 in os::sleep(Thread*, long, bool) () from
> /usr/java/jdk1.
> 7.0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff59b10f2 in JVM_Sleep () from
> /usr/java/jdk1.7.0_75/jre/lib/amd64/
> server/libjvm.so
> #4 0x00007fffdfefaf91 in ?? ()
> #5 0x00007fffcdf1f3b0 in ?? ()
> #6 0x00007ffff348ce46 in clock_gettime () from /lib64/librt.so.1
> #7 0x00007ffff5b452af in os::javaTimeNanos() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #8 0x00007fffdfefbcf8 in ?? ()
> #9 0x00000000c00405c0 in ?? ()
> #10 0x00000000c00405c0 in ?? ()
> #11 0x0000000000000007 in ?? ()
> #12 0x0000000000000bb8 in ?? ()
> #13 0x00000000ebe588c0 in ?? ()
> #14 0x00000000ebe588d8 in ?? ()
> #15 0x00000000ebe588e8 in ?? ()
> #16 0x00007fffdfb36058 in ?? ()
> #17 0x0000000f0e245731 in ?? ()
> #18 0x0000000000000000 in ?? ()
> Thread 3 (Thread 0x7fffcf3b3700 (LWP 19892)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4a314 in Parker::park(bool, long) () from
> /usr/java/jdk1.7.0_75
> /jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5ca93f5 in Unsafe_Park () from
> /usr/java/jdk1.7.0_75/jre/lib/amd6
> 4/server/libjvm.so
> #3 0x00007fffdfdb3bca in ?? ()
> #4 0x00000000c0040520 in ?? ()
> #5 0x00007ffff348ce46 in clock_gettime () from /lib64/librt.so.1
> #6 0x00007ffff5b452af in os::javaTimeNanos() () from
> /usr/java/jdk1.7.0_75/jre/
> lib/amd64/server/libjvm.so
> #7 0x00007fffdfee04dc in ?? ()
> #8 0x000593c5d144b2ef in ?? ()
> #9 0x00000000c0040530 in ?? ()
> #10 0x0000000000000001 in ?? ()
> #11 0x0000000000000000 in ?? ()
> Thread 2 (Thread 0x7fffd1cd2700 (LWP 22157)):
> #0 0x00007ffff45c398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/lib
> pthread.so.0
> #1 0x00007ffff5b4aaef in os::PlatformEvent::park(long) () from
> /usr/java/jdk1.7
> .0_75/jre/lib/amd64/server/libjvm.so
> #2 0x00007ffff5b3955b in ObjectMonitor::wait(long, bool, Thread*) () from
> /usr/
> java/jdk1.7.0_75/jre/lib/amd64/server/libjvm.so
> #3 0x00007ffff59a8538 in JVM_MonitorWait () from
> /usr/java/jdk1.7.0_75/jre/lib/
> amd64/server/libjvm.so
> #4 0x00007fffdfd39208 in ?? ()
> #5 0x00000000edaabb50 in ?? ()
> #6 0x0000000000000000 in ?? ()
> Thread 1 (Thread 0x7ffff7fbcae0 (LWP 12285)):
> #0 0x00007ffff45c35bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthre
> ad.so.0
> #1 0x00007ffff47f2656 in SB_Thread::CV::wait (this=0xeaa7a8) at
> /home/jenkins/w
> orkspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/include/seabed/int/th
> read.inl:552
> #2 0x00007ffff47f26d7 in SB_Thread::CV::wait (this=0xeaa7a8, pv_lock=true)
> at /
> home/jenkins/workspace/traf-pub-release-ahw2.2/trafodion/core/sqf/export/include
> /seabed/int/thread.inl:587
> #3 0x00007ffff733a7e6 in SB_Ms_Event_Mgr::wait (this=0xeaa6b0, pv_us=-1) at
> mse
> ventmgr.inl:343
> #4 0x00007ffff735b18e in XWAIT_com (pv_mask=256, pv_time=-1,
> pv_residual=true)
> at pctl.cpp:979
> #5 0x00007ffff735af3f in XWAIT (pv_mask=256, pv_time=-1) at pctl.cpp:875
> #6 0x00007ffff75c733a in fs_int_fs_file_awaitiox (pp_filenum=0x7fffffff00ee,
> pp
> p_buf=0x7ffffffeffe8, pp_xfercount=0x7ffffffeffd4, pp_tag=0x7ffffffeffd8,
> pv_tim
> eout=-1, pp_segid=0x7ffffffeffe6, pv_int=false, pv_ts=false) at fsi.cpp:1421
> #7 0x00007ffff75c0a19 in BAWAITIOX (pp_filenum=0x7fffffff00ee, ppp_buf=0x0,
> pp_
> xfercount=0x7fffffff0068, pp_tag=0x7fffffff00d8, pv_timeout=-1, pp_segid=0x0)
> at
> fs.cpp:558
> #8 0x00007ffff75c3aad in XAWAITIOX (pp_filenum=0x7fffffff00ee, ppp_buf=0x0,
> pp_
> xfercount=0x7fffffff00ea, pp_tag=0x7fffffff00d8, pv_timeout=-1, pp_segid=0x0)
> at
> fs.cpp:1601
> #9 0x0000000000462e0c in CNSKListenerSrvr::runProgram (this=0xda0e50,
> TcpProces
> sName=<optimized out>, port=<optimized out>, TransportTrace=<optimized out>)
> at
> Interface/linux/Listener_srvr_ps.cpp:474
> #10 0x00000000004de8eb in main (argc=31, argv=0x7fffffff32f8, envp=<optimized
> ou
> t>) at SrvrMain.cpp:863
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
