RE: Prod problem, please help!!! Any resolution?
Title: RE: Prod problem, please help!!! Any resolution? Hi, Just wondering if you made any progress? Let us know. Hannah -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!! Any resolution?
Title: RE: Prod problem, please help!!! Any resolution? Glad you asked Hannah because I was wondering the same John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of [EMAIL PROTECTED] Sent: 01 May 2002 13:18 To: Multiple recipients of list ORACLE-L Subject: RE: Prod problem, please help!!! Any resolution? Hi, Just wondering if you made any progress? Let us know. Hannah -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
Tracy, We ran into some similar problems. Oracle will want you to obtain several system state dumps when the system is hung (we were still able to do this connected internally through server manager). With my limited expertise in reading state dumps the symptoms appeared to be severe latch contention to the point that the system appeared hung. Evidently there was significant re-writes in this area from 8.0 to 8.1 and there have been several attempts to resolve these problems (some introducing new problems). If you can issolate the latch from the state dumps, you can try metalink again and specify the latch name. You may have better luck finding a bug similar to the one you are experiencing searching with the latch name and/or specifying performance. There are several for differing latches. Of the ones in metalink, several were either closed as not reproducable, or marked as fixed in one release or another (although some of those were identical to ours). In our situation, the problem did not occur unless we were attempting to do an online consistent export of our PeopleSoft financials database (the method we use for copying the database, not for backups). We got around the problem by kicking everyone out of the database during exports. As we had a viable work around and I did not want to deal with oracle support, we simply did not pursue it with them. Bill --- DENNIS WILLIAMS [EMAIL PROTECTED] wrote: Tracy - Two ideas: On Metalink, get note 61552.1 Diagnosing Database Hanging Issues I would try to run STATSPACK (or utlbstat/utlestat) while the database is hanging. You can also run STATSPACK on a schedule. Dennis Williams DBA Lifetouch, Inc. [EMAIL PROTECTED] -Original Message- Sent: Tuesday, April 30, 2002 2:22 PM To: Multiple recipients of list ORACLE-L I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: DENNIS WILLIAMS INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). __ Do You Yahoo!? Yahoo! Health - your guide to health and wellness http://health.yahoo.com --
RE: Prod problem, please help!!! Any resolution?
Hi, We have applied an AIX patch to our system, based upon Oracle's recommendation (APAR IY22308). We are currently in a wait and see mode. My confidence in this being the solution will be increased greatly if we can make it through next Monday without the problem. (Mondays are a high-load day for us and that is when the issue usually rears its ugly head). Thanks for asking Please respond to [EMAIL PROTECTED] Sent by:[EMAIL PROTECTED] To:Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Hi, Just wondering if you made any progress? Let us know. Hannah -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!! Any resolution?
Tracy - You might still want to get STATSPACK running. It may provide valuable information on what is happening. Dennis Williams DBA Lifetouch, Inc. [EMAIL PROTECTED] -Original Message- Sent: Wednesday, May 01, 2002 11:18 AM To: Multiple recipients of list ORACLE-L Hi, We have applied an AIX patch to our system, based upon Oracle's recommendation (APAR IY22308). We are currently in a wait and see mode. My confidence in this being the solution will be increased greatly if we can make it through next Monday without the problem. (Mondays are a high-load day for us and that is when the issue usually rears its ugly head). Thanks for asking Please respond to [EMAIL PROTECTED] Sent by:[EMAIL PROTECTED] To:Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Hi, Just wondering if you made any progress? Let us know. Hannah -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: DENNIS WILLIAMS INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
As jared says keep querying the session_Wait u will defnetly bound to catch the guy who might be causing the database to kneel down. If the database is hung, the waits will likely show up immediately. I've had similar problems with waits on a latch that hung the database. If a session is hung, the waits should be there the first time you query. It may also be that you can't even query the v$ views, in which case you must resort to the state dumps. Jared Arun Chakrapani [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 01:18 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:RE: Prod problem, please help!!! If nothing works and when u know the database has frozen u can run the following command to take dump of the database so that u can send it to oracle for analyzing or u can analyze it too. As jared says keep querying the session_Wait u will defnetly bound to catch the guy who might be causing the database to kneel down. Are u flushing shared pool quite frequently. check this out(this was one of the cause which was bringing the database down and was fixed in 7.2 or 7.3 i INET: [EMAIL PROTECTED] -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
I am actually having the same probs now there is a database which is is getting froze for a few secs nobody can login but we did not know what time this was occuring. set a recording thro 9i for every sec later we came to know it was due to cache buffer chain. almost 90 to 100 people were waiting on latch free wait issue. Found that all the users are running the same sql with the same data requests. Looks like we have to tune the sql.Yet to analyze the sqls. -Original Message- Sent: Wednesday, May 01, 2002 5:42 PM To: Multiple recipients of list ORACLE-L As jared says keep querying the session_Wait u will defnetly bound to catch the guy who might be causing the database to kneel down. If the database is hung, the waits will likely show up immediately. I've had similar problems with waits on a latch that hung the database. If a session is hung, the waits should be there the first time you query. It may also be that you can't even query the v$ views, in which case you must resort to the state dumps. Jared Arun Chakrapani [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 01:18 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:RE: Prod problem, please help!!! If nothing works and when u know the database has frozen u can run the following command to take dump of the database so that u can send it to oracle for analyzing or u can analyze it too. As jared says keep querying the session_Wait u will defnetly bound to catch the guy who might be causing the database to kneel down. Are u flushing shared pool quite frequently. check this out(this was one of the cause which was bringing the database down and was fixed in 7.2 or 7.3 i INET: [EMAIL PROTECTED] -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Arun Chakrapani INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Prod problem, please help!!!
The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks If you watch the number of avilable proc, shared memory and sempaphore slots in use/available what happens? Might be that or something inside of oracle is running out of resources. -- Steven Lembark 2930 W. Palmer Workhorse Computing Chicago, IL 60647 +1 800 762 1582 -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Prod problem, please help!!!
If your database is in archivelog mode,is your archive_log_dest full? This will stop the database dead until you remedy the situation. Just my $0.02, Ruth - Original Message - To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] Sent: Tuesday, April 30, 2002 3:21 PM I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Ruth Gramolini INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Prod problem, please help!!!
Hi, Please take system states dumps every 5 seconds withing 20-30 seconds. Let's read what's happening or not happening ? Also, You can upload your HANGANALYZE trace files to iOraHangAnalyzer. It's free. regards... Tracy Rahmlow wrote: I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051 San Diego, California -- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Danisment Gazi Unal http://www.ubTools.com
Re: Prod problem, please help!!!
Tracy, You can start by using the query below to determine what the sessions are actually waiting on: select s.username username, e.event event, s.sid, e.p1text, e.p1, e.p2text, e.p2, e.wait_time, e.seconds_in_wait, e.state from v$session s, v$session_wait e where s.username is not null and s.sid = e.sid -- skip sqlnet idle session messages and e.event not like '%message%client' order by s.username, upper(e.event); Jared Tracy Rahmlow [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 12:21 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:Prod problem, please help!!! I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
Tracy - Two ideas: On Metalink, get note 61552.1 Diagnosing Database Hanging Issues I would try to run STATSPACK (or utlbstat/utlestat) while the database is hanging. You can also run STATSPACK on a schedule. Dennis Williams DBA Lifetouch, Inc. [EMAIL PROTECTED] -Original Message- Sent: Tuesday, April 30, 2002 2:22 PM To: Multiple recipients of list ORACLE-L I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: DENNIS WILLIAMS INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
If nothing works and when u know the database has frozen u can run the following command to take dump of the database so that u can send it to oracle for analyzing or u can analyze it too. As jared says keep querying the session_Wait u will defnetly bound to catch the guy who might be causing the database to kneel down. Are u flushing shared pool quite frequently. check this out(this was one of the cause which was bringing the database down and was fixed in 7.2 or 7.3 i think. Best bet is to keep querying the session_wait and catch the event which is causing the probs. IF YOU HAVE 9I OEM CLIENT INSTALLED USE THE RECORDING THRO PERFORMANCE MANAGER WHICH WILL LET U KNOW LATER ON WHAT HAPPENED. Please do check with the other DBA'S in this group before running of what i say Do systemstate dump 3 times in a row. $ svrmgrl connect internal oradebug setmypid(process_id) oradebug unlimit oradebug dump systemstate 10 or oradebug dump errorstack 3 -Original Message- Sent: Tuesday, April 30, 2002 3:39 PM To: Multiple recipients of list ORACLE-L Tracy, You can start by using the query below to determine what the sessions are actually waiting on: select s.username username, e.event event, s.sid, e.p1text, e.p1, e.p2text, e.p2, e.wait_time, e.seconds_in_wait, e.state from v$session s, v$session_wait e where s.username is not null and s.sid = e.sid -- skip sqlnet idle session messages and e.event not like '%message%client' order by s.username, upper(e.event); Jared Tracy Rahmlow [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 12:21 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:Prod problem, please help!!! I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Arun Chakrapani INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing
Re: Prod problem, please help!!!
Hi, I am unable to query from the database when it hangs up. I am only able to execute the shutdown abort command. To:[EMAIL PROTECTED] cc:Tracy Rahmlow@AMEX Tracy, You can start by using the query below to determine what the sessions are actually waiting on: select s.username username, e.event event, s.sid, e.p1text, e.p1, e.p2text, e.p2, e.wait_time, e.seconds_in_wait, e.state from v$session s, v$session_wait e where s.username is not null and s.sid = e.sid -- skip sqlnet idle session messages and e.event not like '%message%client' order by s.username, upper(e.event); Jared Tracy Rahmlow [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 12:21 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:Prod problem, please help!!! I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Prod problem, please help!!!
Hi, It has been confirmed that we are not running out of space due to the archiver. To:Tracy Rahmlow@AMEX cc: My initial stab in the dark is that if this DB is in archive mode, that the archiver process could be halting additional transactions due to lack of space in the archive destination. Tracy Rahmlow wrote: I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Charlie Mengler Maintenance Warehouse [EMAIL PROTECTED] 10641 Scripps Summit Ct. 858-831-2229 San Diego, CA 92131 Am I sure? Of course I'm sure. I could be wrong, but I'm sure for now! -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
Tracy what do the Sys Admin staff say. How long since the last reboot of the server, has there been any kernel changes recently, any rogue processes? I always try and think away from Oracle initially and get other people involved as well. It is easy to assume it is just an Oracle issue (which it may well be of course) Good luck John -Original Message- [EMAIL PROTECTED] Sent: 30 April 2002 20:39 To: Multiple recipients of list ORACLE-L Tracy, You can start by using the query below to determine what the sessions are actually waiting on: select s.username username, e.event event, s.sid, e.p1text, e.p1, e.p2text, e.p2, e.wait_time, e.seconds_in_wait, e.state from v$session s, v$session_wait e where s.username is not null and s.sid = e.sid -- skip sqlnet idle session messages and e.event not like '%message%client' order by s.username, upper(e.event); Jared Tracy Rahmlow [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 12:21 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:Prod problem, please help!!! I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: John Hallas INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: Prod problem, please help!!!
Tracy, How is the CPU and memory usage on the server when the database hangs? It's likely that database might be experiencing severe latch contention, preventing you from running any queries. To diagnose the problem, you should look at the Oracle Wait events statistics periodically before database hangs. You should install statspack, or if that's not done, run utlbstat/utlestat every hour or so and analyze the statistics. I think that database must be waiting on resources before it completely hangs. Look for wait events, system statistics and latch statistics. Jay -Original Message- Sent: Tuesday, April 30, 2002 4:09 PM To: Multiple recipients of list ORACLE-L Hi, I am unable to query from the database when it hangs up. I am only able to execute the shutdown abort command. To:[EMAIL PROTECTED] cc:Tracy Rahmlow@AMEX Tracy, You can start by using the query below to determine what the sessions are actually waiting on: select s.username username, e.event event, s.sid, e.p1text, e.p1, e.p2text, e.p2, e.wait_time, e.seconds_in_wait, e.state from v$session s, v$session_wait e where s.username is not null and s.sid = e.sid -- skip sqlnet idle session messages and e.event not like '%message%client' order by s.username, upper(e.event); Jared Tracy Rahmlow [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 12:21 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:Prod problem, please help!!! I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). This electronic message contains information from CTIS, Inc., which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipients named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have
RE: Prod problem, please help!!!
Check to see if old dedicated processes are still hanging at the unix level. I had those happened to me and we had to kill them manually. I mean old is one day or few hours old. What's the size of the memory? What's the size of SGA? -Original Message- Sent: Tuesday, April 30, 2002 2:04 PM To: Multiple recipients of list ORACLE-L Tracy, How is the CPU and memory usage on the server when the database hangs? It's likely that database might be experiencing severe latch contention, preventing you from running any queries. To diagnose the problem, you should look at the Oracle Wait events statistics periodically before database hangs. You should install statspack, or if that's not done, run utlbstat/utlestat every hour or so and analyze the statistics. I think that database must be waiting on resources before it completely hangs. Look for wait events, system statistics and latch statistics. Jay -Original Message- Sent: Tuesday, April 30, 2002 4:09 PM To: Multiple recipients of list ORACLE-L Hi, I am unable to query from the database when it hangs up. I am only able to execute the shutdown abort command. To:[EMAIL PROTECTED] cc:Tracy Rahmlow@AMEX Tracy, You can start by using the query below to determine what the sessions are actually waiting on: select s.username username, e.event event, s.sid, e.p1text, e.p1, e.p2text, e.p2, e.wait_time, e.seconds_in_wait, e.state from v$session s, v$session_wait e where s.username is not null and s.sid = e.sid -- skip sqlnet idle session messages and e.event not like '%message%client' order by s.username, upper(e.event); Jared Tracy Rahmlow [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 04/30/2002 12:21 PM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L [EMAIL PROTECTED] cc: Subject:Prod problem, please help!!! I have opened a TAR with Oracle and am waiting a resolution. In the meantime, I am wondering if anybody else has any ideas while I wait. Specifics: IBM AIX 4.3 Rdms 8.1.7.3 Database ~75g OLTP database with approximately 500 dedicated connections and 500 shared connections with Oracle's MTS. Problem: The database hangs, and no user is able to connect to the instance, except locally through srvmgrl. Even within svrmgrl, we are unable to select anything from the database without the query hanging. However, we can abort the instance (shutdown abort) and start it up again just fine. This has happened on 4-22, 4-29 4-30 in the early afternoon. Usually, this is also our peak busy rate for the week. We are executing MTS for 4 applications, all other applications connect through dedicated server. The alert log contains a message unable to start a shared server process. This week it was #41 and last week it was #25. Normally, we do not exceed 5 shared servers. Another thing I noticed is that there is no time allocated to any of the newly created shared servers. It is as if, it can not process any work through existing shared servers and decides to allocate another one, until finally it freezes. I am not sure if this is a MTS problem because I would suspect that I should be able to establish a dedicated server connection. And I can not. I think that this is just a symptom of the underlying problem. It would appear to me that we are running out of a resource, however our sysadms do not see any resource problems. Does anybody have any ideas how to debug this? Thanks -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Tracy Rahmlow INET: [EMAIL PROTECTED] Fat City Network Services-- (858) 538-5051 FAX: (858) 538-5051 San Diego, California-- Public Internet access / Mailing Lists To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). This electronic message contains information from CTIS, Inc., which may
