>From https://forums.oracle.com/forums/thread.jspa?threadID=364644
Oracle finally admit there was a bug: BUG 5607984 - ORACLE DOES NOT CLOSE TCP CONNECTIONS. REMAINS IN CLOSE_WAIT STATE. [On Windows 32-bit]. The patch 10 (patch number 5639232) is supposed to solve the problem for 10.2.0.2.0. We applied it monday morning and everything is fine up to now. This bug is also supposed to be solved in the 10.2.0.3.0 patchset that is availlable on the Metalink site. Do you happen to be running Oracle on a Windows 32-bit server? On Thu, Jun 28, 2012 at 11:55 AM, Raj <[email protected]> wrote: > ** Thank everyone for their help. > > Last night we experienced the same issue again. > Added a new field to the form and modified DB name of the 4 fields and hit > save. > System froze and users started getting ARERR 94. > When checked arerror.log, noticed the following entries: > > Wed Jun 27 21:54:13 2012 390635 : Failure while trying to connect to the > SQL database. > > Please ensure the SQL database is running or contact the Database > Administrator for help (ARERR 550) > Wed Jun 27 21:54:13 2012 ORA-12535: TNS:operation timed out > Wed Jun 27 21:54:13 2012 390635 : Cannot initialize contact with SQL > database (ARERR 551) > Wed Jun 27 21:54:13 2012 Thread 63 not handling connection > > DBA mentioned that the processes on DB instances reached the threshold. > The server/DB became unresponsive. When we looked in the DB, the view (for > the form I was trying to modify) was missing in Production. > So, looks like the Save operation on the form never completed. We had to > bounce the DB instances and AR Server. > After that We had to manually copy the view DDL from our QA environment > and modify it for Prod and run it to enable the view to compile and other > related views to compile. > > Could anyone throw some light what might have happened and how we can > avoid this in future to avoid any impact to customers. > > Here's the environment info: > > Server : 5.01.02 Patch 1313 > Hardware : sun4u > OS: SunOS 5.10 > DB: SQL -- Oracle > DB Version : 0.2.0.5.0 - 64bi > > Thank you, Raj > > > On Friday, March 30, 2012 1:14:40 PM UTC-7, Joe Martin D'Souza wrote: >> >> Yes in my previous email I was talking about the /tmp directory on the OS >> and files created there too.. The files created there are done by AR >> process >> that are a result of an operation from a client.. in your case the client >> was a import operation from the DS (the new toy for us adults from >> Remedyville). >> >> I do no think in this case its your TEMP database which we thought it may >> be >> earlier. Your DBA's information directs us to the OS. Running out of >> volume >> space on the where the /tmp is equally critical. >> >> Check this directory to see if there are a lot of files owned by the AR >> System OS user after you shut down the AR System process.. It might be an >> indication of an application level problem.. Without shutting down the >> processes, you will be unable to delete the files that are owned by the >> AR >> System OS user.. These files will be locked if created after the process >> was >> last started.. >> >> Joe >> >> -----Original Message----- >> From: Raj >> Sent: Friday, March 30, 2012 4:04 PM Newsgroups: >> public.remedy.arsystem.general >> To: [email protected] >> Subject: Re: Can remedy code push cause DB crash, etc? >> >> Thanks Joe: >> >> Here some additional details I gathered from our DBA: >> It was DataWarehouse that had issues and it was not complaining about >> TEMP >> tablespace but the errors in the alert log were pertaining the OS >> resource >> problem >> >> Wed Mar 28 08:22:30 UTC 2012 >> Process startup failed, error stack: >> Wed Mar 28 08:22:31 UTC 2012 >> ORA-27300: OS system dependent operation:fork failed with status: 11 >> ORA-27301: OS failure message: Resource temporarily unavailable >> ORA-27302: failure occurred at: skgpspawn3 >> Wed Mar 28 08:22:32 UTC 2012 >> Process q000 died, see its trace file >> Wed Mar 28 08:22:32 UTC 2012 >> .. >> >> Wed Mar 28 08:49:31 UTC 2012 >> Process startup failed, error stack: >> Wed Mar 28 08:49:31 UTC 2012 >> ORA-27300: OS system dependent operation:fork failed with status: 12 >> ORA-27301: OS failure message: Not enough space >> ORA-27302: failure occurred at: ...... >> Wed Mar 28 08:49:31 UTC 2012 >> Process J001 died, see its trace file >> >> So, what are the recommended size, settings? whom should I approach >> at my end to get these settings in place? >> >> >> On Mar 30, 12:50 pm, Joe Martin D'Souza <[email protected]> wrote: >> > The /tmp directory is needed by the AR Server process to process some >> of >> > its >> > internal stuff.. running out of space here is not good too.. >> > >> > Every client connection that is opened up from a client, creates a >> unique >> > temp file in the /tmp directory and this directory can grow pretty >> large >> > as >> > well.. The size of that file is directly directly dependent on the >> > operation >> > performed.. The operation you were performing is an expensive AR System >> > operation so I can see that this file could grow pretty large. After the >> > client connection is closed the file is typically purged.. >> > >> > If a file for whatever reason does not get purged after the operation, >> its >> > usually an indication of some sort of an application problem.. The file >> is >> > locked so cannot be deleted until you terminate the AR System server >> > process >> > in case that happens.. Also when that happens, you can get this >> directory >> > fill up with files that are large, and not deleted. Monitoring this >> > directory once in a while may be a good idea.. >> > >> > In short you can't be having a low volume dedicated to the tmp >> directory >> > on >> > your AR Server.. >> > >> > Joe >> > >> > >> > >> > >> > >> > >> > >> > -----Original Message----- >> > From: Raj >> > Sent: Friday, March 30, 2012 3:30 PM Newsgroups: >> > >> > public.remedy.arsystem.general >> > To: [email protected] >> > Subject: Re: Can remedy code push cause DB crash, etc? >> > >> > Thanks for all your replies, I discussed this with our DBA. Here' the >> > response when I asked to check the following(1. What is the size of the >> DB >> > Temp Space? >> > 2. Is Temp Database set to autoextend and If there is sufficient free >> disk >> > space ?) >> > Response: >> > The problem we have was not the TEMP tablespace it was the /tmp on the >> > unix >> > server got 100% full. Also it was not production server that had >> problem >> > it >> > was the Datawarehouse server and in the alert log of datawarehouse >> server, >> > it did point to TEMP tablespace problem but of resource unable and not >> > enough space on the unix system >> > >> > Any comments? >> > >> > On Mar 30, 12:02 pm, Joe Martin D'Souza <[email protected]> wrote: >> > > Raj, >> > >> > > Turn autoextend on for your temp database, and make sure the disk has >> > > sufficient free disk space to allow similar operations in the future. >> > > I'm >> > > guessing you must have been missing one or both of these? >> > >> > > Joe >> > >> > > -----Original Message----- >> > > From: Raj >> > > Sent: Friday, March 30, 2012 2:52 PM Newsgroups: >> > >> > > public.remedy.arsystem.general >> > > To: [email protected] >> > > Subject: Re: Can remedy code push cause DB crash, etc? >> > >> > > Yup exactly, we noticed the same thing. DB ran out of temp space and >> > > ultimately crashed. >> > > What are your recommendations, how can we avoid this in future? >> > >> > > On Mar 30, 11:47 am, "Shellman, David" <[email protected]> wrote: >> > > > Been there. Done that. In my case, we ran up against not enough >> temp >> > > > space and froze the system. >> > >> > > > Dave >> > >> > > > -----Original Message----- >> > > > From: Action Request System discussion list(ARSList) >> > > > [mailto:[email protected]] On Behalf Of Raj >> > > > Sent: Friday, March 30, 2012 2:36 PM >> > > > To: [email protected] >> > > > Subject: Re: Can remedy code push cause DB crash, etc? >> > >> > > > Typo ; >> > > > DB Version : 10.2.0.5.0 - 64bi >> > >> > > > On Mar 30, 11:33 am, Raj <[email protected]> wrote: >> > > > > Sorry, missed the environment info: >> > >> > > > > Server : 5.01.02 Patch 1313 >> > > > > Hardware : sun4u >> > > > > OS: SunOS 5.10 >> > > > > DB: SQL -- Oracle >> > > > > DB Version : 0.2.0.5.0 - 64bi >> > >> > > > > On Mar 30, 11:30 am, Raj <[email protected]> wrote: >> > >> > > > > > Hi All, >> > >> > > > > > Very recently I performed Remedy code push on our production >> > > > > > server. >> > >> > > > > > This involved importing some new forms and related workflow and >> > > > > > also >> > > > > > modifying existing forms like HelpDesk and Change forms.(Using >> > > > > > ARAdmin >> > > > > > tool) >> > >> > > > > > I tested this same on our Dev and QA Servers and the code push >> was >> > > > > > a >> > > > > > success without any issues. >> > >> > > > > > But when I performed this on our Production server, it caused DB >> > > > > > crash and brought down Remedy. >> > >> > > > > > The outage was caused by the fact that the database became >> > > > > > unresponsive. >> > >> > > > > > After we brought back Remedy, we noticed some of the DB view >> > > > > > needed >> > > > > > to be recompiled. >> > >> > > > > > As we are still investigating if it has to do with issues in non >> > > > > > remedy components but we couldn't find any solid clues and >> > > > > > ultimately brings it back to Remedy code push. >> > >> > > > > > Here are the logs : >> > >> > > > > > ------------------------------ >> > >> > > > > > Wed Mar 28 01:09:59 2012 Distrib : AR System Distributed Server >> > > > > > terminated when a signal was received by the server (ARDSNOTE >> > > > > > 3000) >> > > > > > Wed Mar 28 01:09:59 2012 15 Wed Mar 28 01:09:59 2012 Distrib : >> AR >> > > > > > System Distributed Server terminated when a signal was received >> by >> > > > > > the server (ARDSNOTE 3000) Wed Mar 28 01:09:59 2012 15 Wed Mar >> 28 >> > > > > > 01:11:23 2012 390600 : Requested database table not found. >> > > > > > Please check the spelling (table name is case-sensitive) (ARERR >> > > > > > 481) >> > > > > > Wed Mar 28 01:29:29 2012 390620 : Failure while trying to >> connect >> > > > > > to >> > > > > > the SQL database. >> > > > > > Please ensure the SQL database is running or contact the >> Database >> > > > > > Administrator for help (ARERR 550) Wed Mar 28 01:29:29 2012 >> > > > > > ORA-12535: TNS:operation timed out Wed Mar 28 01:29:29 2012 >> 390620 >> > > > > > : >> > > > > > Cannot initialize contact with SQL database (ARERR 551) Wed Mar >> 28 >> > > > > > 01:29:29 2012 Thread 67 not handling connection >> > >> > > > > > ------------------------------**- >> > >> > > > > > At this point, I would like to understand from experts in here >> to >> > > > > > see if anything like the above mentioned is possible? If yes, >> what >> > > > > > could be the reasons and what can I look into to find the root >> > > > > > cause. >> > >> > > > > > Thanks, >> > >> > > > > > Raj >> >> ______________________________**______________________________** >> ___________________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" >> >> _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
