Sorry! Screenshot didn't work. Here is the information again: Type/RPC/Min/Max Admin / 390600 / 1/ 1 Fast / 390620 / 20 / 50 List / 390635 / 20 / 50 Private / 390623 / 1 / 1 Private / 390624 / 1 / 1 Private / 390680 / 1 / 1 Private / 390694 / 1 / 1
On Thursday, June 28, 2012 11:29:00 AM UTC-7, Raj wrote: > > ** Thanks Fred for your prompt reply. > I checked the Server Ports and Queues settings: > > reaching out to DBA to check "how many connections and processes the > database (and database O/S for the user the database runs under) is > configured for." as you suggested. > Will let you know my findings. > > On Thursday, June 28, 2012 11:02:13 AM UTC-7, Grooms, Frederick W wrote: >> >> ** >> >> Let me make sure I have the facts from your earlier email in the chain … >> >> >> Server : 5.01.02 Patch 1313 >> Hardware : sun4u >> OS: SunOS 5.10 >> DB: SQL -- Oracle >> DB Version : 10.2.0.5.0 - 64bit >> >> And since it is ARS version 5 you are using the Oracle 8 client. >> >> Since the DBA said processes reached the threshold you should ask the DBA >> how many connections and processes the database (and database O/S for the >> user the database runs under) is configured for. >> >> If the number of connections and processes is less than the max number of >> threads you have configured (Fast, List, Escalation, Admin, Private, …) you >> will need to increase the number on the database or reduce the max number >> on the ARS server. >> >> >> >> If I remember correctly ARS always dropped the view and created a new one >> when database field names were changed or added (the table uses the field >> IDs and there is a database view that has the names). Remedy doesn’t do >> anything with the view itself, it is created just to make outside querying >> of the data easier. >> >> >> >> Fred >> >> >> >> >> >> -----Original Message----- >> *From:* Action Request System discussion list(ARSList) [mailto: >> [email protected]] *On Behalf Of *Axton >> *Sent:* Thursday, June 28, 2012 12:35 PM >> *To:* [email protected] >> *Subject:* Re: Can remedy code push cause DB crash, etc? >> >> >> >> ** From https://forums.oracle.com/forums/thread.jspa?threadID=364644 >> >> >> >> Oracle finally admit there was a bug: BUG 5607984 - ORACLE DOES NOT >> CLOSE TCP CONNECTIONS. REMAINS IN CLOSE_WAIT STATE. [On Windows 32-bit]. >> >> >> >> The patch 10 (patch number 5639232) is supposed to solve the problem for >> 10.2.0.2.0. We applied it monday morning and everything is fine up to now. >> >> >> >> This bug is also supposed to be solved in the 10.2.0.3.0 patchset that is >> availlable on the Metalink site. >> >> >> Do you happen to be running Oracle on a Windows 32-bit server? >> >> -----Original Message----- >> On Thu, Jun 28, 2012 at 11:55 AM, Raj <[email protected]> wrote: >> >> ** Thank everyone for their help. >> >> Last night we experienced the same issue again. >> Added a new field to the form and modified DB name of the 4 fields and >> hit save. >> System froze and users started getting ARERR 94. >> When checked arerror.log, noticed the following entries: >> >> Wed Jun 27 21:54:13 2012 390635 : Failure while trying to connect to >> the SQL database. >> >> >> Please ensure the SQL database is running or contact the Database >> Administrator for help (ARERR 550) >> >> Wed Jun 27 21:54:13 2012 ORA-12535: TNS:operation timed out >> Wed Jun 27 21:54:13 2012 390635 : Cannot initialize contact with SQL >> database (ARERR 551) >> Wed Jun 27 21:54:13 2012 Thread 63 not handling connection >> >> DBA mentioned that the processes on DB instances reached the threshold. >> The server/DB became unresponsive. When we looked in the DB, the view >> (for the form I was trying to modify) was missing in Production. >> So, looks like the Save operation on the form never completed. We had to >> bounce the DB instances and AR Server. >> After that We had to manually copy the view DDL from our QA environment >> and modify it for Prod and run it to enable the view to compile and other >> related views to compile. >> >> Could anyone throw some light what might have happened and how we can >> avoid this in future to avoid any impact to customers. >> >> >> Here's the environment info: >> >> >> Server : 5.01.02 Patch 1313 >> Hardware : sun4u >> OS: SunOS 5.10 >> DB: SQL -- Oracle >> DB Version : 0.2.0.5.0 - 64bi >> >> Thank you, Raj >> >> >> -----Original Message----- >> On Friday, March 30, 2012 1:14:40 PM UTC-7, Joe Martin D'Souza wrote: >> >> Yes in my previous email I was talking about the /tmp directory on the OS >> and files created there too.. The files created there are done by AR >> process >> that are a result of an operation from a client.. in your case the client >> was a import operation from the DS (the new toy for us adults from >> Remedyville). >> >> I do no think in this case its your TEMP database which we thought it may >> be >> earlier. Your DBA's information directs us to the OS. Running out of >> volume >> space on the where the /tmp is equally critical. >> >> Check this directory to see if there are a lot of files owned by the AR >> System OS user after you shut down the AR System process.. It might be an >> indication of an application level problem.. Without shutting down the >> processes, you will be unable to delete the files that are owned by the >> AR >> System OS user.. These files will be locked if created after the process >> was >> last started.. >> >> Joe >> >> -----Original Message----- >> From: Raj >> Sent: Friday, March 30, 2012 4:04 PM Newsgroups: >> public.remedy.arsystem.general >> To: [email protected] >> Subject: Re: Can remedy code push cause DB crash, etc? >> >> Thanks Joe: >> >> Here some additional details I gathered from our DBA: >> It was DataWarehouse that had issues and it was not complaining about >> TEMP >> tablespace but the errors in the alert log were pertaining the OS >> resource >> problem >> >> Wed Mar 28 08:22:30 UTC 2012 >> Process startup failed, error stack: >> Wed Mar 28 08:22:31 UTC 2012 >> ORA-27300: OS system dependent operation:fork failed with status: 11 >> ORA-27301: OS failure message: Resource temporarily unavailable >> ORA-27302: failure occurred at: skgpspawn3 >> Wed Mar 28 08:22:32 UTC 2012 >> Process q000 died, see its trace file >> Wed Mar 28 08:22:32 UTC 2012 >> .. >> >> Wed Mar 28 08:49:31 UTC 2012 >> Process startup failed, error stack: >> Wed Mar 28 08:49:31 UTC 2012 >> ORA-27300: OS system dependent operation:fork failed with status: 12 >> ORA-27301: OS failure message: Not enough space >> ORA-27302: failure occurred at: ...... >> Wed Mar 28 08:49:31 UTC 2012 >> Process J001 died, see its trace file >> >> So, what are the recommended size, settings? whom should I approach >> at my end to get these settings in place? >> >> -----Original Message----- >> On Mar 30, 12:50 pm, Joe Martin D'Souza <[email protected]> wrote: >> > The /tmp directory is needed by the AR Server process to process some >> of >> > its >> > internal stuff.. running out of space here is not good too.. >> > >> > Every client connection that is opened up from a client, creates a >> unique >> > temp file in the /tmp directory and this directory can grow pretty >> large >> > as >> > well.. The size of that file is directly directly dependent on the >> > operation >> > performed.. The operation you were performing is an expensive AR System >> > operation so I can see that this file could grow pretty large. After the >> > client connection is closed the file is typically purged.. >> > >> > If a file for whatever reason does not get purged after the operation, >> its >> > usually an indication of some sort of an application problem.. The file >> is >> > locked so cannot be deleted until you terminate the AR System server >> > process >> > in case that happens.. Also when that happens, you can get this >> directory >> > fill up with files that are large, and not deleted. Monitoring this >> > directory once in a while may be a good idea.. >> > >> > In short you can't be having a low volume dedicated to the tmp >> directory >> > on >> > your AR Server.. >> > >> > Joe >> > >> > -----Original Message----- >> > From: Raj >> > Sent: Friday, March 30, 2012 3:30 PM Newsgroups: >> > >> > public.remedy.arsystem.general >> > To: [email protected] >> > Subject: Re: Can remedy code push cause DB crash, etc? >> > >> > Thanks for all your replies, I discussed this with our DBA. Here' the >> > response when I asked to check the following(1. What is the size of the >> DB >> > Temp Space? >> > 2. Is Temp Database set to autoextend and If there is sufficient free >> disk >> > space ?) >> > Response: >> > The problem we have was not the TEMP tablespace it was the /tmp on the >> > unix >> > server got 100% full. Also it was not production server that had >> problem >> > it >> > was the Datawarehouse server and in the alert log of datawarehouse >> server, >> > it did point to TEMP tablespace problem but of resource unable and not >> > enough space on the unix system >> > >> > Any comments? >> > >> > On Mar 30, 12:02 pm, Joe Martin D'Souza <[email protected]> wrote: >> > > Raj, >> > >> > > Turn autoextend on for your temp database, and make sure the disk has >> > > sufficient free disk space to allow similar operations in the future. >> > > I'm >> > > guessing you must have been missing one or both of these? >> > >> > > Joe >> > >> > > -----Original Message----- >> > > From: Raj >> > > Sent: Friday, March 30, 2012 2:52 PM Newsgroups: >> > >> > > public.remedy.arsystem.general >> > > To: [email protected] >> > > Subject: Re: Can remedy code push cause DB crash, etc? >> > >> > > Yup exactly, we noticed the same thing. DB ran out of temp space and >> > > ultimately crashed. >> > > What are your recommendations, how can we avoid this in future? >> > >> > > On Mar 30, 11:47 am, "Shellman, David" <[email protected]> wrote: >> > > > Been there. Done that. In my case, we ran up against not enough >> temp >> > > > space and froze the system. >> > >> > > > Dave >> > >> > > > -----Original Message----- >> > > > From: Action Request System discussion list(ARSList) >> > > > [mailto:[email protected]] On Behalf Of Raj >> > > > Sent: Friday, March 30, 2012 2:36 PM >> > > > To: [email protected] >> > > > Subject: Re: Can remedy code push cause DB crash, etc? >> > >> > > > Typo ; >> > > > DB Version : 10.2.0.5.0 - 64bi >> > >> > > > -----Original Message----- >> > > > On Mar 30, 11:33 am, Raj <[email protected]> wrote: >> > > > > Sorry, missed the environment info: >> > >> > > > > Server : 5.01.02 Patch 1313 >> > > > > Hardware : sun4u >> > > > > OS: SunOS 5.10 >> > > > > DB: SQL -- Oracle >> > > > > DB Version : 0.2.0.5.0 - 64bi >> > >> > > > > On Mar 30, 11:30 am, Raj <[email protected]> wrote: >> > >> > > > > > Hi All, >> > >> > > > > > Very recently I performed Remedy code push on our production >> > > > > > server. >> > >> > > > > > This involved importing some new forms and related workflow and >> > > > > > also >> > > > > > modifying existing forms like HelpDesk and Change forms.(Using >> > > > > > ARAdmin >> > > > > > tool) >> > >> > > > > > I tested this same on our Dev and QA Servers and the code push >> was >> > > > > > a >> > > > > > success without any issues. >> > >> > > > > > But when I performed this on our Production server, it caused DB >> > > > > > crash and brought down Remedy. >> > >> > > > > > The outage was caused by the fact that the database became >> > > > > > unresponsive. >> > >> > > > > > After we brought back Remedy, we noticed some of the DB view >> > > > > > needed >> > > > > > to be recompiled. >> > >> > > > > > As we are still investigating if it has to do with issues in non >> > > > > > remedy components but we couldn't find any solid clues and >> > > > > > ultimately brings it back to Remedy code push. >> > >> > > > > > Here are the logs : >> > >> > > > > > ------------------------------ >> > > > > > Wed Mar 28 01:09:59 2012 Distrib : AR System Distributed Server >> > > > > > terminated when a signal was received by the server (ARDSNOTE >> > > > > > 3000) >> > > > > > Wed Mar 28 01:09:59 2012 15 Wed Mar 28 01:09:59 2012 Distrib : >> AR >> > > > > > System Distributed Server terminated when a signal was received >> by >> > > > > > the server (ARDSNOTE 3000) Wed Mar 28 01:09:59 2012 15 Wed Mar >> 28 >> > > > > > 01:11:23 2012 390600 : Requested database table not found. >> > > > > > Please check the spelling (table name is case-sensitive) (ARERR >> > > > > > 481) >> > > > > > Wed Mar 28 01:29:29 2012 390620 : Failure while trying to >> connect >> > > > > > to >> > > > > > the SQL database. >> > > > > > Please ensure the SQL database is running or contact the >> Database >> > > > > > Administrator for help (ARERR 550) Wed Mar 28 01:29:29 2012 >> > > > > > ORA-12535: TNS:operation timed out Wed Mar 28 01:29:29 2012 >> 390620 >> > > > > > : >> > > > > > Cannot initialize contact with SQL database (ARERR 551) Wed Mar >> 28 >> > > > > > 01:29:29 2012 Thread 67 not handling connection >> > >> > > > > > ------------------------------- >> > >> > > > > > At this point, I would like to understand from experts in here >> to >> > > > > > see if anything like the above mentioned is possible? If yes, >> what >> > > > > > could be the reasons and what can I look into to find the root >> > > > > > cause. >> > >> > > > > > Thanks, >> > >> > > > > > Raj >> >> >> _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
