Yes in my previous email I was talking about the /tmp directory on the OS
and files created there too.. The files created there are done by AR process
that are a result of an operation from a client.. in your case the client
was a import operation from the DS (the new toy for us adults from
Remedyville).
I do no think in this case its your TEMP database which we thought it may be
earlier. Your DBA's information directs us to the OS. Running out of volume
space on the where the /tmp is equally critical.
Check this directory to see if there are a lot of files owned by the AR
System OS user after you shut down the AR System process.. It might be an
indication of an application level problem.. Without shutting down the
processes, you will be unable to delete the files that are owned by the AR
System OS user.. These files will be locked if created after the process was
last started..
Joe
-----Original Message-----
From: Raj
Sent: Friday, March 30, 2012 4:04 PM Newsgroups:
public.remedy.arsystem.general
To: [email protected]
Subject: Re: Can remedy code push cause DB crash, etc?
Thanks Joe:
Here some additional details I gathered from our DBA:
It was DataWarehouse that had issues and it was not complaining about TEMP
tablespace but the errors in the alert log were pertaining the OS resource
problem
Wed Mar 28 08:22:30 UTC 2012
Process startup failed, error stack:
Wed Mar 28 08:22:31 UTC 2012
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Wed Mar 28 08:22:32 UTC 2012
Process q000 died, see its trace file
Wed Mar 28 08:22:32 UTC 2012
..
Wed Mar 28 08:49:31 UTC 2012
Process startup failed, error stack:
Wed Mar 28 08:49:31 UTC 2012
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: ......
Wed Mar 28 08:49:31 UTC 2012
Process J001 died, see its trace file
So, what are the recommended size, settings? whom should I approach
at my end to get these settings in place?
On Mar 30, 12:50 pm, Joe Martin D'Souza <[email protected]> wrote:
The /tmp directory is needed by the AR Server process to process some of
its
internal stuff.. running out of space here is not good too..
Every client connection that is opened up from a client, creates a unique
temp file in the /tmp directory and this directory can grow pretty large
as
well.. The size of that file is directly directly dependent on the
operation
performed.. The operation you were performing is an expensive AR System
operation so I can see that this file could grow pretty large. After the
client connection is closed the file is typically purged..
If a file for whatever reason does not get purged after the operation, its
usually an indication of some sort of an application problem.. The file is
locked so cannot be deleted until you terminate the AR System server
process
in case that happens.. Also when that happens, you can get this directory
fill up with files that are large, and not deleted. Monitoring this
directory once in a while may be a good idea..
In short you can't be having a low volume dedicated to the tmp directory
on
your AR Server..
Joe
-----Original Message-----
From: Raj
Sent: Friday, March 30, 2012 3:30 PM Newsgroups:
public.remedy.arsystem.general
To: [email protected]
Subject: Re: Can remedy code push cause DB crash, etc?
Thanks for all your replies, I discussed this with our DBA. Here' the
response when I asked to check the following(1. What is the size of the DB
Temp Space?
2. Is Temp Database set to autoextend and If there is sufficient free disk
space ?)
Response:
The problem we have was not the TEMP tablespace it was the /tmp on the
unix
server got 100% full. Also it was not production server that had problem
it
was the Datawarehouse server and in the alert log of datawarehouse server,
it did point to TEMP tablespace problem but of resource unable and not
enough space on the unix system
Any comments?
On Mar 30, 12:02 pm, Joe Martin D'Souza <[email protected]> wrote:
> Raj,
> Turn autoextend on for your temp database, and make sure the disk has
> sufficient free disk space to allow similar operations in the future.
> I'm
> guessing you must have been missing one or both of these?
> Joe
> -----Original Message-----
> From: Raj
> Sent: Friday, March 30, 2012 2:52 PM Newsgroups:
> public.remedy.arsystem.general
> To: [email protected]
> Subject: Re: Can remedy code push cause DB crash, etc?
> Yup exactly, we noticed the same thing. DB ran out of temp space and
> ultimately crashed.
> What are your recommendations, how can we avoid this in future?
> On Mar 30, 11:47 am, "Shellman, David" <[email protected]> wrote:
> > Been there. Done that. In my case, we ran up against not enough temp
> > space and froze the system.
> > Dave
> > -----Original Message-----
> > From: Action Request System discussion list(ARSList)
> > [mailto:[email protected]] On Behalf Of Raj
> > Sent: Friday, March 30, 2012 2:36 PM
> > To: [email protected]
> > Subject: Re: Can remedy code push cause DB crash, etc?
> > Typo ;
> > DB Version : 10.2.0.5.0 - 64bi
> > On Mar 30, 11:33 am, Raj <[email protected]> wrote:
> > > Sorry, missed the environment info:
> > > Server : 5.01.02 Patch 1313
> > > Hardware : sun4u
> > > OS: SunOS 5.10
> > > DB: SQL -- Oracle
> > > DB Version : 0.2.0.5.0 - 64bi
> > > On Mar 30, 11:30 am, Raj <[email protected]> wrote:
> > > > Hi All,
> > > > Very recently I performed Remedy code push on our production
> > > > server.
> > > > This involved importing some new forms and related workflow and
> > > > also
> > > > modifying existing forms like HelpDesk and Change forms.(Using
> > > > ARAdmin
> > > > tool)
> > > > I tested this same on our Dev and QA Servers and the code push was
> > > > a
> > > > success without any issues.
> > > > But when I performed this on our Production server, it caused DB
> > > > crash and brought down Remedy.
> > > > The outage was caused by the fact that the database became
> > > > unresponsive.
> > > > After we brought back Remedy, we noticed some of the DB view
> > > > needed
> > > > to be recompiled.
> > > > As we are still investigating if it has to do with issues in non
> > > > remedy components but we couldn't find any solid clues and
> > > > ultimately brings it back to Remedy code push.
> > > > Here are the logs :
> > > > ------------------------------
> > > > Wed Mar 28 01:09:59 2012 Distrib : AR System Distributed Server
> > > > terminated when a signal was received by the server (ARDSNOTE
> > > > 3000)
> > > > Wed Mar 28 01:09:59 2012 15 Wed Mar 28 01:09:59 2012 Distrib : AR
> > > > System Distributed Server terminated when a signal was received by
> > > > the server (ARDSNOTE 3000) Wed Mar 28 01:09:59 2012 15 Wed Mar 28
> > > > 01:11:23 2012 390600 : Requested database table not found.
> > > > Please check the spelling (table name is case-sensitive) (ARERR
> > > > 481)
> > > > Wed Mar 28 01:29:29 2012 390620 : Failure while trying to connect
> > > > to
> > > > the SQL database.
> > > > Please ensure the SQL database is running or contact the Database
> > > > Administrator for help (ARERR 550) Wed Mar 28 01:29:29 2012
> > > > ORA-12535: TNS:operation timed out Wed Mar 28 01:29:29 2012 390620
> > > > :
> > > > Cannot initialize contact with SQL database (ARERR 551) Wed Mar 28
> > > > 01:29:29 2012 Thread 67 not handling connection
> > > > -------------------------------
> > > > At this point, I would like to understand from experts in here to
> > > > see if anything like the above mentioned is possible? If yes, what
> > > > could be the reasons and what can I look into to find the root
> > > > cause.
> > > > Thanks,
> > > > Raj
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
