Initially I was on P1 for our first try at upgrading. Then for the second try they said go to P2. Now they are saying go to P3. I'm ready for about anything.
They also now are thinking its the 5.1.2 clients doing a query causing the issue. Well with the exception of 5 out of 150 people, they are all 5.1.2. But there is a freak thing happening when the thread-fire concludes and we restart. Some people have not been able to access the Remedy Support form (console form). It is no where to be found. We upgrade them, and there it is. We've tried taking all the cache files out of v5 client, even reinstalling the v5 client and we cannot get the form back. But if we upgrade to 7 and then uninstall and go back to 5, it's there. We're having a problem that was not discovered in v7 testing. Some reports don't run. Ok, not totally shocked because they were created in Crystal 9. But odd we didn't find that in testing with the v7 client. What's really bad is, I made the report a Crystal 11 report and it still won't work in v7 client. Crystal 11 seems to work differently. Which seems to be the theme for this whole experience! Susan On 7/10/07, Axton <[EMAIL PROTECTED]> wrote:
If it's any consolation, I had to roll 7.0.1 patch 2 back to patch 1 after the server crashed constantly. I am on Solaris, but the net effect is the same; something fed the arserver cyanide, then it puked and died a horrible death. Seems EIE and arserver have been drinking from the same glass. Axton Grams On 7/10/07, Susan Palmer <[EMAIL PROTECTED]> wrote: > ** > Since our upgrade from ARS 5.1.2 to 7.0.1P2 on 6/27/07 I have no life. Ok, > that's the end of my rant. > > Everyday, with the exception of Sat/Sun even though there are some users on > the system, we have a thread run wild issue. It is with the LIST threads. > This manifested itself in a arerror94 - database timeout > > Of course everyone points to Remedy but it didn't feel like a Remedy issue. > I felt like network or database. Well after 4 days of denial the other guys > finally started looking at their sides and various changes were made. > Optimizations in oracle and some minor server changes. Yesterday we > actually ran the full high-production hours schedule without timing out. > Thought I was home free. > > Well on the way home the threads started to initiate again. And finally by > 10:30p I had to restart the services because we were at 14 threads and our > max is 15. Some sleep is required even after a v7 upgrade. > > Restarting the services clears the problem. Yesterday we ran most of the > day at 10 threads. I check old statistics from May and that's what we were > running at then. I know, it seems high ... that's not the issue right now. > > We were having 343 errors appear after a restart which didn't seem all that > bad but apparently are VERY bad. Those got cleared away today and we are > not seeing them anymore. > > So today, it's been a horrific day. Four times we pushed the threads to the > limit and had to restart services. > > Over the last week or so I've become intimately familiar with my sql logs > again. Looking for trends, patterns, hints. Sometimes it seems like when > a person logs in it causes a thread to stop and restart. Or maybe it's the > start of the thread buildup. > > But what I do see periodically are the following errors. AND I think I'm > see a pattern that after one of these there is a slow buildup of threads > from 7-11 over 20-30 minutes then 12-15 is faster. We start seeing > hourglasses when I think we're at 14 on the way to 15. > > I'd like to know what the entry below is telling me. I've asked bmc support > but have received no answer. I've looked in the sql logs at this time and > don't see a real pattern to what the person was doing that would drive me in > a certain direction. > > Any knowledge on this you can share would be appreciated. > > Thanks, > Susan Palmer > ShopperTrak > > ARS v7.0.1P1 > Oracle 10g > Windows 2003 > > > Tue Jul 10 14:55:33 2007 390635 : AR System server terminated when a > signal/exception was received by the server (ARNOTE 20) > Tue Jul 10 14:55:33 2007 > Timestamp: Tue Jul 10 2007 14:55:34.1250 > Thread Id: 2664 > Version: 7.0.01 patch 002 200704021644 Apr 2 2007 20:12:39 > ServerName: remsrv > Database: SQL -- Oracle > Hardware: Intel Pentium > OS: Windows NT 5.2 > RPC Id: 13200 > RPC Call: 34 (EXP) > RPC Queue: 390635 > Client: User AdamsT from Remedy User (protocol 9) at IP address 10.0.4.68 > Form: > Logging On: API SQL Thread > Code: c0000005 > Operation: read > Access Addr: 736C6C69 > Stack Begin: > Stack End > > Tue Jul 10 14:55:33 2007 390635 : AR System server terminated when a > signal/exception was received by the server (ARNOTE 20) > Tue Jul 10 14:55:33 2007 0xc0000005 > Tue Jul 10 14:55:33 2007 390635 : AR System server terminated -- fatal > error encountered (ARNOTE 21) > > Thread log that relates to the above error from a timing perspective: > > <THRD> /* Tue Jul 10 2007 14:15:16.3430 */ Thread Id 3108 (thread number > 21) application statistics thread started. > <THRD> /* Tue Jul 10 2007 14:55:34.3430 */ Thread Id 2664 (thread number > 16) on LIST queue died. > <THRD> /* Tue Jul 10 2007 14:55:34.3430 */ Thread Id 4864 (thread number > 16) on LIST queue restarted. > <THRD> /* Tue Jul 10 2007 14:57:39.8750 */ Thread Id 5784 (thread number > 22) on LIST queue started. > <THRD> /* Tue Jul 10 2007 14:59:05.8750 */ Thread Id 5640 (thread number > 23) on LIST queue started. > <THRD> /* Tue Jul 10 2007 14:59:08.0620 */ Thread Id 5568 (thread number > 24) on LIST queue started. > <THRD> /* Tue Jul 10 2007 14:59:27.4530 */ Thread Id 5652 (thread number > 25) on LIST queue started. > <THRD> /* Tue Jul 10 2007 15:02:14.5620 */ Thread Id 2332 (thread number > 26) on LIST queue started. > <THRD> /* Tue Jul 10 2007 15:03: 11.6560 */ Thread Id 5328 (thread number > 27) on LIST queue started. > <THRD> /* Tue Jul 10 2007 15:03:17.9370 */ Thread Id 4908 (thread number > 28) on LIST queue started. > <THRD> /* Tue Jul 10 2007 15:23:11.9370 */ Thread Id 5704 (thread number > 29) on LIST queue started. > <THRD> /* Tue Jul 10 2007 15:26:09.5460 */ Thread Id 964 (thread number 30) > on LIST queue started. > > > > > > > This thread terror occurred today too. First time FAST threads have been > involved: > > <THRD> /* Tue Jul 10 2007 13:11:11.7650 */ Thread Trace Log -- ON > <THRD> /* Tue Jul 10 2007 13:11:11.7810 */ Thread Id 5856 (thread number > 0) Thread Manager started. > <THRD> /* Tue Jul 10 2007 13:11: 11.7810 */ Thread Id 3236 (thread number > 1) timed call thread started. > <THRD> /* Tue Jul 10 2007 13:11:11.7810 */ Thread Id 5236 (thread number > 2) on ADMIN queue started. > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5740 (thread number > 3) on ALERT queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 5384 (thread number > 4) on ESCALATION queue started. > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5880 (thread number > 5) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3256 (thread number > 6) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3432 (thread number > 7) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4812 (thread number > 8) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3692 (thread number > 9) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 904 (thread number 10) > on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4508 (thread number > 11) on LIST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4816 (thread number > 12) on LIST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 5792 (thread number > 13) on LIST queue started. > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 4120 (thread number > 14) on LIST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4216 (thread number > 15) on LIST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 2960 (thread number > 16) on LIST queue started. > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 2832 (thread number > 17) license monitor thread started. > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5104 (thread number > 18) archive thread started. > <THRD> /* Tue Jul 10 2007 13:11:37.7180 */ Thread Id 4960 (thread number > 19) computed group call thread started. > <THRD> /* Tue Jul 10 2007 13:11: 37.7180 */ Thread Id 5340 (thread number > 20) server statistics thread started. > <THRD> /* Tue Jul 10 2007 13:11:37.7180 */ Thread Id 4940 (thread number > 21) application statistics thread started. > <THRD> /* Tue Jul 10 2007 13:40: 36.1400 */ Thread Id 5880 (thread number > 5) on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 5372 (thread number > 5) on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 5372 (thread number > 5) on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 4196 (thread number > 5) on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:40:36.1710 */ Thread Id 4812 (thread number > 8) on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:40:36.1710 */ Thread Id 884 (thread number 8) > on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:40:38.6710 */ Thread Id 3692 (thread number > 9) on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:40:38.6710 */ Thread Id 1996 (thread number > 9) on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:40:59.2500 */ Thread Id 4120 (thread number > 14) on LIST queue died. > <THRD> /* Tue Jul 10 2007 13:40: 59.2500 */ Thread Id 5860 (thread number > 14) on LIST queue restarted. > <THRD> /* Tue Jul 10 2007 13:41:39.4060 */ Thread Id 4216 (thread number > 15) on LIST queue died. > <THRD> /* Tue Jul 10 2007 13:41:39.4060 */ Thread Id 5108 (thread number > 15) on LIST queue restarted. > <THRD> /* Tue Jul 10 2007 13:41:42.9840 */ Thread Id 2960 (thread number > 16) on LIST queue died. > <THRD> /* Tue Jul 10 2007 13:41:42.9840 */ Thread Id 2700 (thread number > 16) on LIST queue restarted. > <THRD> /* Tue Jul 10 2007 13:42:19.5150 */ Thread Id 3256 (thread number > 6) on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:42:19.5150 */ Thread Id 2108 (thread number > 6) on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:42:36.1710 */ Thread Id 3432 (thread number > 7) on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:42:36.1710 */ Thread Id 4064 (thread number > 7) on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:42: 46.6090 */ Thread Id 480 (thread number > 22) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:42:58.3590 */ Thread Id 4700 (thread number > 23) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:43:05.3430 */ Thread Id 4816 (thread number > 12) on LIST queue died. > <THRD> /* Tue Jul 10 2007 13:43:05.3430 */ Thread Id 5756 (thread number > 12) on LIST queue restarted. > <THRD> /* Tue Jul 10 2007 13:43:13.6710 */ Thread Id 356 (thread number 24) > on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:43:13.7180 */ Thread Id 904 (thread number 10) > on FAST queue died. > <THRD> /* Tue Jul 10 2007 13:43:13.7180 */ Thread Id 2812 (thread number > 10) on FAST queue restarted. > <THRD> /* Tue Jul 10 2007 13:43:25.6870 */ Thread Id 6140 (thread number > 25) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:43:31.7180 */ Thread Id 3960 (thread number > 26) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:43: 41.7810 */ Thread Id 2484 (thread number > 27) on FAST queue started. > <THRD> /* Tue Jul 10 2007 13:46:39.1870 */ Thread Id 5384 (thread number > 4) on ESCALATION queue died. > <THRD> /* Tue Jul 10 2007 13:46: 39.1870 */ Thread Id 3704 (thread number > 4) on ESCALATION queue restarted. > <THRD> /* Tue Jul 10 2007 13:48:45.8750 */ Thread Id 5792 (thread number > 13) on LIST queue died. > <THRD> /* Tue Jul 10 2007 13:48: 45.8750 */ Thread Id 824 (thread number > 13) on LIST queue restarted. > > __20060125_______________________This posting was > submitted with HTML in it___ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
_______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
