Thanks Fred for your prompt reply. I checked the Server Ports and Queues settings:
reaching out to DBA to check "how many connections and processes the database (and database O/S for the user the database runs under) is configured for." as you suggested. Will let you know my findings. On Thursday, June 28, 2012 11:02:13 AM UTC-7, Grooms, Frederick W wrote: > > ** > > Let me make sure I have the facts from your earlier email in the chain … > > Server : 5.01.02 Patch 1313 > Hardware : sun4u > OS: SunOS 5.10 > DB: SQL -- Oracle > DB Version : 10.2.0.5.0 - 64bit > > And since it is ARS version 5 you are using the Oracle 8 client. > > Since the DBA said processes reached the threshold you should ask the DBA > how many connections and processes the database (and database O/S for the > user the database runs under) is configured for. > > If the number of connections and processes is less than the max number of > threads you have configured (Fast, List, Escalation, Admin, Private, …) you > will need to increase the number on the database or reduce the max number > on the ARS server. > > > > If I remember correctly ARS always dropped the view and created a new one > when database field names were changed or added (the table uses the field > IDs and there is a database view that has the names). Remedy doesn’t do > anything with the view itself, it is created just to make outside querying > of the data easier. > > > > Fred > > > > > > -----Original Message----- > *From:* Action Request System discussion list(ARSList) [mailto: > [email protected]] *On Behalf Of *Axton > *Sent:* Thursday, June 28, 2012 12:35 PM > *To:* [email protected] > *Subject:* Re: Can remedy code push cause DB crash, etc? > > > > ** From https://forums.oracle.com/forums/thread.jspa?threadID=364644 > > > > Oracle finally admit there was a bug: BUG 5607984 - ORACLE DOES NOT > CLOSE TCP CONNECTIONS. REMAINS IN CLOSE_WAIT STATE. [On Windows 32-bit]. > > > > The patch 10 (patch number 5639232) is supposed to solve the problem for > 10.2.0.2.0. We applied it monday morning and everything is fine up to now. > > > > This bug is also supposed to be solved in the 10.2.0.3.0 patchset that is > availlable on the Metalink site. > > > Do you happen to be running Oracle on a Windows 32-bit server? > > -----Original Message----- > On Thu, Jun 28, 2012 at 11:55 AM, Raj <[email protected]> wrote: > > ** Thank everyone for their help. > > Last night we experienced the same issue again. > Added a new field to the form and modified DB name of the 4 fields and hit > save. > System froze and users started getting ARERR 94. > When checked arerror.log, noticed the following entries: > > Wed Jun 27 21:54:13 2012 390635 : Failure while trying to connect to the > SQL database. > > > Please ensure the SQL database is running or contact the Database > Administrator for help (ARERR 550) > > Wed Jun 27 21:54:13 2012 ORA-12535: TNS:operation timed out > Wed Jun 27 21:54:13 2012 390635 : Cannot initialize contact with SQL > database (ARERR 551) > Wed Jun 27 21:54:13 2012 Thread 63 not handling connection > > DBA mentioned that the processes on DB instances reached the threshold. > The server/DB became unresponsive. When we looked in the DB, the view (for > the form I was trying to modify) was missing in Production. > So, looks like the Save operation on the form never completed. We had to > bounce the DB instances and AR Server. > After that We had to manually copy the view DDL from our QA environment > and modify it for Prod and run it to enable the view to compile and other > related views to compile. > > Could anyone throw some light what might have happened and how we can > avoid this in future to avoid any impact to customers. > > > Here's the environment info: > > > Server : 5.01.02 Patch 1313 > Hardware : sun4u > OS: SunOS 5.10 > DB: SQL -- Oracle > DB Version : 0.2.0.5.0 - 64bi > > Thank you, Raj > > > -----Original Message----- > On Friday, March 30, 2012 1:14:40 PM UTC-7, Joe Martin D'Souza wrote: > > Yes in my previous email I was talking about the /tmp directory on the OS > and files created there too.. The files created there are done by AR > process > that are a result of an operation from a client.. in your case the client > was a import operation from the DS (the new toy for us adults from > Remedyville). > > I do no think in this case its your TEMP database which we thought it may > be > earlier. Your DBA's information directs us to the OS. Running out of > volume > space on the where the /tmp is equally critical. > > Check this directory to see if there are a lot of files owned by the AR > System OS user after you shut down the AR System process.. It might be an > indication of an application level problem.. Without shutting down the > processes, you will be unable to delete the files that are owned by the AR > System OS user.. These files will be locked if created after the process > was > last started.. > > Joe > > -----Original Message----- > From: Raj > Sent: Friday, March 30, 2012 4:04 PM Newsgroups: > public.remedy.arsystem.general > To: [email protected] > Subject: Re: Can remedy code push cause DB crash, etc? > > Thanks Joe: > > Here some additional details I gathered from our DBA: > It was DataWarehouse that had issues and it was not complaining about TEMP > tablespace but the errors in the alert log were pertaining the OS resource > problem > > Wed Mar 28 08:22:30 UTC 2012 > Process startup failed, error stack: > Wed Mar 28 08:22:31 UTC 2012 > ORA-27300: OS system dependent operation:fork failed with status: 11 > ORA-27301: OS failure message: Resource temporarily unavailable > ORA-27302: failure occurred at: skgpspawn3 > Wed Mar 28 08:22:32 UTC 2012 > Process q000 died, see its trace file > Wed Mar 28 08:22:32 UTC 2012 > .. > > Wed Mar 28 08:49:31 UTC 2012 > Process startup failed, error stack: > Wed Mar 28 08:49:31 UTC 2012 > ORA-27300: OS system dependent operation:fork failed with status: 12 > ORA-27301: OS failure message: Not enough space > ORA-27302: failure occurred at: ...... > Wed Mar 28 08:49:31 UTC 2012 > Process J001 died, see its trace file > > So, what are the recommended size, settings? whom should I approach > at my end to get these settings in place? > > -----Original Message----- > On Mar 30, 12:50 pm, Joe Martin D'Souza <[email protected]> wrote: > > The /tmp directory is needed by the AR Server process to process some of > > its > > internal stuff.. running out of space here is not good too.. > > > > Every client connection that is opened up from a client, creates a unique > > temp file in the /tmp directory and this directory can grow pretty large > > as > > well.. The size of that file is directly directly dependent on the > > operation > > performed.. The operation you were performing is an expensive AR System > > operation so I can see that this file could grow pretty large. After the > > client connection is closed the file is typically purged.. > > > > If a file for whatever reason does not get purged after the operation, > its > > usually an indication of some sort of an application problem.. The file > is > > locked so cannot be deleted until you terminate the AR System server > > process > > in case that happens.. Also when that happens, you can get this directory > > fill up with files that are large, and not deleted. Monitoring this > > directory once in a while may be a good idea.. > > > > In short you can't be having a low volume dedicated to the tmp directory > > on > > your AR Server.. > > > > Joe > > > > -----Original Message----- > > From: Raj > > Sent: Friday, March 30, 2012 3:30 PM Newsgroups: > > > > public.remedy.arsystem.general > > To: [email protected] > > Subject: Re: Can remedy code push cause DB crash, etc? > > > > Thanks for all your replies, I discussed this with our DBA. Here' the > > response when I asked to check the following(1. What is the size of the > DB > > Temp Space? > > 2. Is Temp Database set to autoextend and If there is sufficient free > disk > > space ?) > > Response: > > The problem we have was not the TEMP tablespace it was the /tmp on the > > unix > > server got 100% full. Also it was not production server that had > problem > > it > > was the Datawarehouse server and in the alert log of datawarehouse > server, > > it did point to TEMP tablespace problem but of resource unable and not > > enough space on the unix system > > > > Any comments? > > > > On Mar 30, 12:02 pm, Joe Martin D'Souza <[email protected]> wrote: > > > Raj, > > > > > Turn autoextend on for your temp database, and make sure the disk has > > > sufficient free disk space to allow similar operations in the future. > > > I'm > > > guessing you must have been missing one or both of these? > > > > > Joe > > > > > -----Original Message----- > > > From: Raj > > > Sent: Friday, March 30, 2012 2:52 PM Newsgroups: > > > > > public.remedy.arsystem.general > > > To: [email protected] > > > Subject: Re: Can remedy code push cause DB crash, etc? > > > > > Yup exactly, we noticed the same thing. DB ran out of temp space and > > > ultimately crashed. > > > What are your recommendations, how can we avoid this in future? > > > > > On Mar 30, 11:47 am, "Shellman, David" <[email protected]> wrote: > > > > Been there. Done that. In my case, we ran up against not enough > temp > > > > space and froze the system. > > > > > > Dave > > > > > > -----Original Message----- > > > > From: Action Request System discussion list(ARSList) > > > > [mailto:[email protected]] On Behalf Of Raj > > > > Sent: Friday, March 30, 2012 2:36 PM > > > > To: [email protected] > > > > Subject: Re: Can remedy code push cause DB crash, etc? > > > > > > Typo ; > > > > DB Version : 10.2.0.5.0 - 64bi > > > > > > -----Original Message----- > > > > On Mar 30, 11:33 am, Raj <[email protected]> wrote: > > > > > Sorry, missed the environment info: > > > > > > > Server : 5.01.02 Patch 1313 > > > > > Hardware : sun4u > > > > > OS: SunOS 5.10 > > > > > DB: SQL -- Oracle > > > > > DB Version : 0.2.0.5.0 - 64bi > > > > > > > On Mar 30, 11:30 am, Raj <[email protected]> wrote: > > > > > > > > Hi All, > > > > > > > > Very recently I performed Remedy code push on our production > > > > > > server. > > > > > > > > This involved importing some new forms and related workflow and > > > > > > also > > > > > > modifying existing forms like HelpDesk and Change forms.(Using > > > > > > ARAdmin > > > > > > tool) > > > > > > > > I tested this same on our Dev and QA Servers and the code push > was > > > > > > a > > > > > > success without any issues. > > > > > > > > But when I performed this on our Production server, it caused DB > > > > > > crash and brought down Remedy. > > > > > > > > The outage was caused by the fact that the database became > > > > > > unresponsive. > > > > > > > > After we brought back Remedy, we noticed some of the DB view > > > > > > needed > > > > > > to be recompiled. > > > > > > > > As we are still investigating if it has to do with issues in non > > > > > > remedy components but we couldn't find any solid clues and > > > > > > ultimately brings it back to Remedy code push. > > > > > > > > Here are the logs : > > > > > > > > ------------------------------ > > > > > > Wed Mar 28 01:09:59 2012 Distrib : AR System Distributed Server > > > > > > terminated when a signal was received by the server (ARDSNOTE > > > > > > 3000) > > > > > > Wed Mar 28 01:09:59 2012 15 Wed Mar 28 01:09:59 2012 Distrib : AR > > > > > > System Distributed Server terminated when a signal was received > by > > > > > > the server (ARDSNOTE 3000) Wed Mar 28 01:09:59 2012 15 Wed Mar 28 > > > > > > 01:11:23 2012 390600 : Requested database table not found. > > > > > > Please check the spelling (table name is case-sensitive) (ARERR > > > > > > 481) > > > > > > Wed Mar 28 01:29:29 2012 390620 : Failure while trying to > connect > > > > > > to > > > > > > the SQL database. > > > > > > Please ensure the SQL database is running or contact the Database > > > > > > Administrator for help (ARERR 550) Wed Mar 28 01:29:29 2012 > > > > > > ORA-12535: TNS:operation timed out Wed Mar 28 01:29:29 2012 > 390620 > > > > > > : > > > > > > Cannot initialize contact with SQL database (ARERR 551) Wed Mar > 28 > > > > > > 01:29:29 2012 Thread 67 not handling connection > > > > > > > > ------------------------------- > > > > > > > > At this point, I would like to understand from experts in here to > > > > > > see if anything like the above mentioned is possible? If yes, > what > > > > > > could be the reasons and what can I look into to find the root > > > > > > cause. > > > > > > > > Thanks, > > > > > > > > Raj > > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
