RE: RAC system Calls .......
Matt,List Does this have a benefit for SINGLE instance Database Server connected to Multiple Application Servers (Client Server Architecture) ? OS = Solaris 9 Application , DB Servers = SF15K machines Network = 1 GBPS interconnects routing thru a switch May I further extend this to ask if you have any advisable Solaris 9 Kernel parameters settings as the %sys Utilization on the DB Server is Very High ( 40 % ) for Hybrid nature of Trans ? Thanks Vivek -Original Message- Sent: Friday, July 11, 2003 2:14 AM To: Multiple recipients of list ORACLE-L Jumbo frames are the use of larger than normal MTU (Maximum Transmission Unit) settings on gigabit Ethernet links. The traditional limit for Ethernet frames is 1500 bytes, which was fine for 10 and 100 megabit Ethernet links. With gigabit, however, since you lose a certain minimum amount of bandwidth to signaling overhead (preamble, postamble, header info, etc.) and that the Ethernet card has to do a certain minimum processing for each Ethernet frame it receives, a huge amount of CPU overhead can be spent on trying to fill a gigabit pipe. The other problem is that if the host(s) are sending/receiving data larger than 1500 bytes, the data packet has to be fragmented into multiple, smaller packets, which then have to be reassembled on the far side. Since this all has to be done on the host CPU rather than the Ethernet card, it increases both bus overhead and CPU time. With jumbo frames, you use a 1500 byte MTU - the exact amount varies by implementation, but they're generally in the 9000-9200 byte range. That's a 6x improvement in the amount of data per ethernet frame, plus there's less reassembly. Unfortunately, Sun never really embraced it as a technology, so unless you're running one of a couple of third-party gigabit cards, I think you're probably out of luck. The specific relevance to RAC, which I somehow managed to mention, is that data blocks being shuttled 'tween nodes (depending on the blocksize) can be placed into a smaller number of ethernet frames, reducing both latency and overhead. Ideally, each block will fit into one ethernet frame, but as always, YMMV. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ravi Kulkarni Sent: Thursday, July 10, 2003 2:49 PM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Matt, What are jumbo frames? Are these assigning private network IPs to cluster_interconnects parameter? -Ravi. --- Matthew Zito [EMAIL PROTECTED] wrote: And are you using jumbo frames on your interconnect? That can make a significant contribution to reducing overhead from a system standpoint. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of K Gopalakrishnan Sent: Thursday, July 10, 2003 11:44 AM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services
RE: RAC system Calls
Just for fun, have you tried using truss to find out which calls are being used? That might help you. Mladen Gogala Oracle DBA Phone:(203) 459-6855 Email:[EMAIL PROTECTED] -Original Message- Sent: Thursday, July 10, 2003 8:14 PM To: Multiple recipients of list ORACLE-L Raj, Yes. Timed_statistics=true. But I see the 'times' even in single instance(non-rac)database but relatively called negligible number of times. -Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: You seem to have timed_statistics=TRUE right ?? in RAC you will ALWAYS see GCS pings ... basically as long as cluster_database=true, even if only one node is running, you will still see Oracle trying to ping other instances. My guess is that's what is happening ... but I could be very well wrong, I have RAC experience, but never tried to truss the top processes. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Thursday, July 10, 2003 12:30 PM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). *This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*1 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Gogala, Mladen INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Title: RE: RAC system Calls That may be because with RAC there is more activity ... GCS traffic and more stats to update ... Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Thursday, July 10, 2003 8:14 PM To: Multiple recipients of list ORACLE-L Raj, Yes. Timed_statistics=true. But I see the 'times' even in single instance(non-rac)database but relatively called negligible number of times. -Ravi. *This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*1
RE: RAC system Calls
Mladen, As mentioned earlier, I have lots of times funtion calls, which (my guess) relate to (60-70%)IPC send completion sync waits I got in statspack. Dont mean to fill your inboxes, but, pasting few truss lines: 26949: read(14, 01 O\0\006\0\0\0\0\011 i.., 2064) = 335 26949: times(0xFFBED228) = 225698576 26949: times(0xFFBED0D8) = 225698576 26949: times(0xFFBED228) = 225698576 26949: times(0xFFBED108) = 225698576 26949: times(0xFFBED108) = 225698576 26949: times(0xFFBED108) = 225698577 26949: times(0xFFBED108) = 225698577 26949: times(0xFFBECA08) = 225698577 26949: times(0xFFBECA08) = 225698577 26949: times(0xFFBECC88) = 225698577 26949: times(0xFFBECC88) = 225698577 26949: times(0xFFBED0D8) = 225698577 Thanks, Ravi. --- Gogala, Mladen [EMAIL PROTECTED] wrote: Just for fun, have you tried using truss to find out which calls are being used? That might help you. Mladen Gogala Oracle DBA Phone:(203) 459-6855 Email:[EMAIL PROTECTED] -Original Message- Sent: Thursday, July 10, 2003 8:14 PM To: Multiple recipients of list ORACLE-L Raj, Yes. Timed_statistics=true. But I see the 'times' even in single instance(non-rac)database but relatively called negligible number of times. -Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: You seem to have timed_statistics=TRUE right ?? in RAC you will ALWAYS see GCS pings ... basically as long as cluster_database=true, even if only one node is running, you will still see Oracle trying to ping other instances. My guess is that's what is happening ... but I could be very well wrong, I have RAC experience, but never tried to truss the top processes. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Thursday, July 10, 2003 12:30 PM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). *This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*1 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services
RE: RAC system Calls
The times call is very cheap and could not possibly consume so much time.
Here is what man page says:
NAME
times - get process and child process times
SYNOPSIS
#include sys/times.h
clock_t times(struct tms *buffer);
DESCRIPTION
times() fills the structure pointed to by buffer with time-accounting
information. The structure defined in sys/times.h is as follows:
struct tms {
clock_t tms_utime; /* user time */
clock_t tms_stime; /* system time */
clock_t tms_cutime; /* user time, children */
clock_t tms_cstime; /* system time, children */
};
This information comes from the calling process and each of its
terminated child processes for which it has executed a wait(),
wait3(), or waitpid(). The times are in units of 1/CLK_TCK seconds,
where CLK_TCK is processor dependent The value of CLK_TCK can be
queried using the sysconf() function (see sysconf(2)).
Your problem is with the preceding read call. Try to figure out where is
attempting to read from, is there a process handling the file/device and
to which process is all that kernel CPU time billed. Try with ps -el and
top. Also, try with netstat -i to see whether you're retransmitting on
the high speed interconnect.
Mladen Gogala
Oracle DBA
Phone:(203) 459-6855
Email:[EMAIL PROTECTED]
-Original Message-
Sent: Friday, July 11, 2003 12:05 PM
To: Multiple recipients of list ORACLE-L
Mladen, As mentioned earlier, I have lots of times
funtion calls, which (my guess) relate to (60-70%)IPC
send completion sync waits I got in statspack. Dont
mean to fill your inboxes, but, pasting few truss
lines:
26949: read(14, 01 O\0\006\0\0\0\0\011 i.., 2064)
= 335
26949: times(0xFFBED228)
= 225698576
26949: times(0xFFBED0D8)
= 225698576
26949: times(0xFFBED228)
= 225698576
26949: times(0xFFBED108)
= 225698576
26949: times(0xFFBED108)
= 225698576
26949: times(0xFFBED108)
= 225698577
26949: times(0xFFBED108)
= 225698577
26949: times(0xFFBECA08)
= 225698577
26949: times(0xFFBECA08)
= 225698577
26949: times(0xFFBECC88)
= 225698577
26949: times(0xFFBECC88)
= 225698577
26949: times(0xFFBED0D8)
= 225698577
Thanks,
Ravi.
--- Gogala, Mladen [EMAIL PROTECTED] wrote:
Just for fun, have you tried using truss to find out
which calls
are being used? That might help you.
Mladen Gogala
Oracle DBA
Phone:(203) 459-6855
Email:[EMAIL PROTECTED]
-Original Message-
Sent: Thursday, July 10, 2003 8:14 PM
To: Multiple recipients of list ORACLE-L
Raj,
Yes. Timed_statistics=true. But I see the 'times'
even
in single instance(non-rac)database but relatively
called negligible number of times.
-Ravi.
--- Jamadagni, Rajendra
[EMAIL PROTECTED] wrote:
You seem to have timed_statistics=TRUE right ?? in
RAC you will ALWAYS see
GCS pings ... basically as long as
cluster_database=true, even if only one
node is running, you will still see Oracle trying
to
ping other instances.
My guess is that's what is happening ... but I
could
be very well wrong, I
have RAC experience, but never tried to truss the
top processes.
Raj
Rajendra dot Jamadagni at nospamespn dot com
All Views expressed in this email are strictly
personal.
QOTD: Any clod can have facts, having an opinion
is
an art !
-Original Message-
Sent: Thursday, July 10, 2003 12:30 PM
To: Multiple recipients of list ORACLE-L
Hello List,
We are running Benchmark tests on Solaris 2-Node
RAC.
Consistently noticed the following :
- Very high Kernel usage (averaging 45%) on TOP
- Statspack has IPC Send Completion sync waits
(70%
Total ela time)
- On trussing top process, found Oracle to be
issuing
huge number of times system calls in addition to
read/writes(which I think are select/inserts).
Has anyone noticed this in your environment. I am
guessing these to be inter-instance pings, but
could
not get any hits in Doc/Metalink to confirm this.
times call is clocking lot of CPU. Is this
normal
?
Any pointers would be helpful ? If this is out of
context, is there a separate list for RAC?
Thanks,
Ravi.
__
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com
--
Please see the official ORACLE-L FAQ:
http://www.orafaq.net
--
Author: Ravi Kulkarni
INET: [EMAIL PROTECTED]
Fat City
RE: RAC system Calls
Title: RE: RAC system Calls Raj: I am discussing this issue with Ravi offline. From the initial IPC dumps, It looks like the private interconnect is not properly configured/used for cache fusion. I am waiting for more details from him.. Best Regards,K Gopalakrishnan -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Jamadagni, RajendraSent: Friday, July 11, 2003 7:34 AMTo: Multiple recipients of list ORACLE-LSubject: RE: RAC system Calls That may be because with RAC there is more activity .. GCS traffic and more stats to update ... Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Thursday, July 10, 2003 8:14 PM To: Multiple recipients of list ORACLE-L Raj, Yes. Timed_statistics=true. But I see the 'times' even in single instance(non-rac)database but relatively called negligible number of times. -Ravi.
RE: RAC system Calls
Title: RE: RAC system Calls Hmmm ... Thanks KG Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message-From: K Gopalakrishnan [mailto:[EMAIL PROTECTED]Sent: Friday, July 11, 2003 1:14 PMTo: Multiple recipients of list ORACLE-LSubject: RE: RAC system Calls Raj: I am discussing this issue with Ravi offline. From the initial IPC dumps, It looks like the private interconnect is not properly configured/used for cache fusion. I am waiting for more details from him.. Best Regards,K Gopalakrishnan This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2
RE: RAC system Calls
Correct. Looks like we need to configure CLUSTER_INTERCONNECTS parameter. Automatic configuration (Sun Cluster3.0/RAC) did not choose the (correct) IP(s) for interconnect. KG noticed it. I have sent netstat -ai, oradebug IPC dump, ping -s info to him. If you are interested, I can paste it here. -Thanks, Ravi. --- K Gopalakrishnan [EMAIL PROTECTED] wrote: RE: RAC system CallsRaj: I am discussing this issue with Ravi offline. From the initial IPC dumps, It looks like the private interconnect is not properly configured/used for cache fusion. I am waiting for more details from him.. Best Regards, K Gopalakrishnan -Original Message- Jamadagni, Rajendra Sent: Friday, July 11, 2003 7:34 AM To: Multiple recipients of list ORACLE-L That may be because with RAC there is more activity ... GCS traffic and more stats to update ... Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Thursday, July 10, 2003 8:14 PM To: Multiple recipients of list ORACLE-L Raj, Yes. Timed_statistics=true. But I see the 'times' even in single instance(non-rac)database but relatively called negligible number of times. -Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Title: RE: RAC system Calls We don't use cluster_interconnects (it is null) ... but for us, the RAC is using the fat private interconnects without any problems. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- From: Ravi Kulkarni [mailto:[EMAIL PROTECTED]] Sent: Friday, July 11, 2003 3:34 PM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Correct. Looks like we need to configure CLUSTER_INTERCONNECTS parameter. Automatic configuration (Sun Cluster3.0/RAC) did not choose the (correct) IP(s) for interconnect. KG noticed it. I have sent netstat -ai, oradebug IPC dump, ping -s info to him. If you are interested, I can paste it here. -Thanks, Ravi. This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2
RE: RAC system Calls
Do you use 1 or 2 interconnects? Does your oradebug ipc dump show both IPs? For me shows: SSKGXPT 0x3066b00 flags info for network 0 socket no 9 IP 172.168.193.2 UDP 41143 sflags SSKGXPT_UPinfo for network 1 socket no 0 IP 0.0.0.0 UDP 0 sflags SSKGXPT_DOWN Even 172.168.193.2 is not one of the (2) IPs we have for interconnect. Thanks, Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: We don't use cluster_interconnects (it is null) ... but for us, the RAC is using the fat private interconnects without any problems. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Friday, July 11, 2003 3:34 PM To: Multiple recipients of list ORACLE-L Correct. Looks like we need to configure CLUSTER_INTERCONNECTS parameter. Automatic configuration (Sun Cluster3.0/RAC) did not choose the (correct) IP(s) for interconnect. KG noticed it. I have sent netstat -ai, oradebug IPC dump, ping -s info to him. If you are interested, I can paste it here. -Thanks, Ravi. This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Title: RE: RAC system Calls Yup, we have 2 private interconnects and one public .. can't display the ips though ... Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- From: Ravi Kulkarni [mailto:[EMAIL PROTECTED]] Sent: Friday, July 11, 2003 4:19 PM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Do you use 1 or 2 interconnects? Does your oradebug ipc dump show both IPs? For me shows: SSKGXPT 0x3066b00 flags info for network 0 socket no 9 IP 172.168.193.2 UDP 41143 sflags SSKGXPT_UP info for network 1 socket no 0 IP 0.0.0.0 UDP 0 sflags SSKGXPT_DOWN Even 172.168.193.2 is not one of the (2) IPs we have for interconnect. Thanks, Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: We don't use cluster_interconnects (it is null) .. but for us, the RAC is using the fat private interconnects without any problems. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Friday, July 11, 2003 3:34 PM To: Multiple recipients of list ORACLE-L Correct. Looks like we need to configure CLUSTER_INTERCONNECTS parameter. Automatic configuration (Sun Cluster3.0/RAC) did not choose the (correct) IP(s) for interconnect. KG noticed it. I have sent netstat -ai, oradebug IPC dump, ping -s info to him. If you are interested, I can paste it here. -Thanks, Ravi. This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2
RE: RAC system Calls
I have tested using cluster_interconnects parameter. Oradebug IPC dump shows it chooses first IP correctly, recognizes the 2nd interconnect, but with a wrong IP - 0.0.0.0. Looks like a bug.. Has anyone seen this earlier? (I understand there is a bug in HP with similar behaviour - we are on Solaris) SSKGXPT 0x3066b00 flags info for network 0 socket no 9 IP 164.22.21.1 UDP 61635 sflags SSKGXPT_UP info for network 1 socket no 12IP 0.0.0.0 UDP 61636 sflags SSKGXPT_UP Is there a workaround to make Oracle choose the correct IP for the interconnect? TIA, Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: Yup, we have 2 private interconnects and one public ... can't display the ips though ... Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Friday, July 11, 2003 4:19 PM To: Multiple recipients of list ORACLE-L Do you use 1 or 2 interconnects? Does your oradebug ipc dump show both IPs? For me shows: SSKGXPT 0x3066b00 flags info for network 0 socket no 9 IP 172.168.193.2 UDP 41143 sflags SSKGXPT_UPinfo for network 1 socket no 0 IP 0.0.0.0 UDP 0 sflags SSKGXPT_DOWN Even 172.168.193.2 is not one of the (2) IPs we have for interconnect. Thanks, Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: We don't use cluster_interconnects (it is null) ... but for us, the RAC is using the fat private interconnects without any problems. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Friday, July 11, 2003 3:34 PM To: Multiple recipients of list ORACLE-L Correct. Looks like we need to configure CLUSTER_INTERCONNECTS parameter. Automatic configuration (Sun Cluster3.0/RAC) did not choose the (correct) IP(s) for interconnect. KG noticed it. I have sent netstat -ai, oradebug IPC dump, ping -s info to him. If you are interested, I can paste it here. -Thanks, Ravi. This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY,
Re: RAC system Calls
I've seen a problem with CLIC interface, which are, for some reason, not used by oracle. It is a bug (Oxford is on HP-UX) and there is a specific one-off patch. This was OPS, not RAC. The symptoms are very similar: one of the processes was spinning like crazy in the kernel mode. However, RAC != OPS and I don't have any experience with RAC. On 2003.07.11 20:24, Ravi Kulkarni wrote: I have tested using cluster_interconnects parameter. Oradebug IPC dump shows it chooses first IP correctly, recognizes the 2nd interconnect, but with a wrong IP - 0.0.0.0. Looks like a bug.. Has anyone seen this earlier? (I understand there is a bug in HP with similar behaviour - we are on Solaris) SSKGXPT 0x3066b00 flags info for network 0 socket no 9 IP 164.22.21.1 UDP 61635 sflags SSKGXPT_UP info for network 1 socket no 12IP 0.0.0.0 UDP 61636 sflags SSKGXPT_UP Is there a workaround to make Oracle choose the correct IP for the interconnect? TIA, Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: Yup, we have 2 private interconnects and one public ... can't display the ips though ... Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Friday, July 11, 2003 4:19 PM To: Multiple recipients of list ORACLE-L Do you use 1 or 2 interconnects? Does your oradebug ipc dump show both IPs? For me shows: SSKGXPT 0x3066b00 flags info for network 0 socket no 9 IP 172.168.193.2 UDP 41143 sflags SSKGXPT_UPinfo for network 1 socket no 0 IP 0.0.0.0 UDP 0 sflags SSKGXPT_DOWN Even 172.168.193.2 is not one of the (2) IPs we have for interconnect. Thanks, Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: We don't use cluster_interconnects (it is null) ... but for us, the RAC is using the fat private interconnects without any problems. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Friday, July 11, 2003 3:34 PM To: Multiple recipients of list ORACLE-L Correct. Looks like we need to configure CLUSTER_INTERCONNECTS parameter. Automatic configuration (Sun Cluster3.0/RAC) did not choose the (correct) IP(s) for interconnect. KG noticed it. I have sent netstat -ai, oradebug IPC dump, ping -s info to him. If you are interested, I can paste it here. -Thanks, Ravi. This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*2 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City
RE: RAC system Calls
Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: K Gopalakrishnan INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: RAC system Calls
I had a similar experience on OPS when we tried with releasable locks. What is the platform? What kind of interconnect do you have? On 2003.07.10 12:44, K Gopalakrishnan wrote: Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: K Gopalakrishnan INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Mladen Gogala Oracle DBA -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Mladen Gogala INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Title: RE: RAC system Calls You seem to have timed_statistics=TRUE right ?? in RAC you will ALWAYS see GCS pings ... basically as long as cluster_database=true, even if only one node is running, you will still see Oracle trying to ping other instances. My guess is that's what is happening ... but I could be very well wrong, I have RAC experience, but never tried to truss the top processes. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- From: Ravi Kulkarni [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 10, 2003 12:30 PM To: Multiple recipients of list ORACLE-L Subject: RAC system Calls Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). *This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*1
RE: RAC system Calls
And are you using jumbo frames on your interconnect? That can make a significant contribution to reducing overhead from a system standpoint. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of K Gopalakrishnan Sent: Thursday, July 10, 2003 11:44 AM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: K Gopalakrishnan INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Matthew Zito INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Matt, What are jumbo frames? Are these assigning private network IPs to cluster_interconnects parameter? -Ravi. --- Matthew Zito [EMAIL PROTECTED] wrote: And are you using jumbo frames on your interconnect? That can make a significant contribution to reducing overhead from a system standpoint. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of K Gopalakrishnan Sent: Thursday, July 10, 2003 11:44 AM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: K Gopalakrishnan INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Matthew Zito INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Jumbo frames are the use of larger than normal MTU (Maximum Transmission Unit) settings on gigabit Ethernet links. The traditional limit for Ethernet frames is 1500 bytes, which was fine for 10 and 100 megabit Ethernet links. With gigabit, however, since you lose a certain minimum amount of bandwidth to signaling overhead (preamble, postamble, header info, etc.) and that the Ethernet card has to do a certain minimum processing for each Ethernet frame it receives, a huge amount of CPU overhead can be spent on trying to fill a gigabit pipe. The other problem is that if the host(s) are sending/receiving data larger than 1500 bytes, the data packet has to be fragmented into multiple, smaller packets, which then have to be reassembled on the far side. Since this all has to be done on the host CPU rather than the Ethernet card, it increases both bus overhead and CPU time. With jumbo frames, you use a 1500 byte MTU - the exact amount varies by implementation, but they're generally in the 9000-9200 byte range. That's a 6x improvement in the amount of data per ethernet frame, plus there's less reassembly. Unfortunately, Sun never really embraced it as a technology, so unless you're running one of a couple of third-party gigabit cards, I think you're probably out of luck. The specific relevance to RAC, which I somehow managed to mention, is that data blocks being shuttled 'tween nodes (depending on the blocksize) can be placed into a smaller number of ethernet frames, reducing both latency and overhead. Ideally, each block will fit into one ethernet frame, but as always, YMMV. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ravi Kulkarni Sent: Thursday, July 10, 2003 2:49 PM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Matt, What are jumbo frames? Are these assigning private network IPs to cluster_interconnects parameter? -Ravi. --- Matthew Zito [EMAIL PROTECTED] wrote: And are you using jumbo frames on your interconnect? That can make a significant contribution to reducing overhead from a system standpoint. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of K Gopalakrishnan Sent: Thursday, July 10, 2003 11:44 AM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: K Gopalakrishnan INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051
RE: RAC system Calls
Hey Matt, Thanks for the useful info. into a smaller number of ethernet frames, reducing both latency and overhead. Ideally, each block will fit into one ethernet frame, We are using 16K blocksize, so we cannot avoid datagram splits with 9k frames - but certainly, sounds better than with 1500 frames. so unless you're running one of a couple of third-party gigabit cards, I think you're probably out of luck. Will check with SysAdmins if the GigE we have supports. Are there any risks/disadvantages of using 9k frames for interconnect? Thanks, -Ravi. --- Matthew Zito [EMAIL PROTECTED] wrote: Jumbo frames are the use of larger than normal MTU (Maximum Transmission Unit) settings on gigabit Ethernet links. The traditional limit for Ethernet frames is 1500 bytes, which was fine for 10 and 100 megabit Ethernet links. With gigabit, however, since you lose a certain minimum amount of bandwidth to signaling overhead (preamble, postamble, header info, etc.) and that the Ethernet card has to do a certain minimum processing for each Ethernet frame it receives, a huge amount of CPU overhead can be spent on trying to fill a gigabit pipe. The other problem is that if the host(s) are sending/receiving data larger than 1500 bytes, the data packet has to be fragmented into multiple, smaller packets, which then have to be reassembled on the far side. Since this all has to be done on the host CPU rather than the Ethernet card, it increases both bus overhead and CPU time. With jumbo frames, you use a 1500 byte MTU - the exact amount varies by implementation, but they're generally in the 9000-9200 byte range. That's a 6x improvement in the amount of data per ethernet frame, plus there's less reassembly. Unfortunately, Sun never really embraced it as a technology, so unless you're running one of a couple of third-party gigabit cards, I think you're probably out of luck. The specific relevance to RAC, which I somehow managed to mention, is that data blocks being shuttled 'tween nodes (depending on the blocksize) can be placed into a smaller number of ethernet frames, reducing both latency and overhead. Ideally, each block will fit into one ethernet frame, but as always, YMMV. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ravi Kulkarni Sent: Thursday, July 10, 2003 2:49 PM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Matt, What are jumbo frames? Are these assigning private network IPs to cluster_interconnects parameter? -Ravi. --- Matthew Zito [EMAIL PROTECTED] wrote: And are you using jumbo frames on your interconnect? That can make a significant contribution to reducing overhead from a system standpoint. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of K Gopalakrishnan Sent: Thursday, July 10, 2003 11:44 AM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Ravi: Do you have a statspack report? I would like to see that. But in any case, 45% kernel is just too much? BTW have you verified the private interconnect is used for cache fusion transfer.. Make sure the cache fusion is not going thru the public network. Best Regards, K Gopalakrishnan -Original Message- Ravi Kulkarni Sent: Thursday, July 10, 2003 9:30 AM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538
RE: RAC system Calls
Raj, Yes. Timed_statistics=true. But I see the 'times' even in single instance(non-rac)database but relatively called negligible number of times. -Ravi. --- Jamadagni, Rajendra [EMAIL PROTECTED] wrote: You seem to have timed_statistics=TRUE right ?? in RAC you will ALWAYS see GCS pings ... basically as long as cluster_database=true, even if only one node is running, you will still see Oracle trying to ping other instances. My guess is that's what is happening ... but I could be very well wrong, I have RAC experience, but never tried to truss the top processes. Raj Rajendra dot Jamadagni at nospamespn dot com All Views expressed in this email are strictly personal. QOTD: Any clod can have facts, having an opinion is an art ! -Original Message- Sent: Thursday, July 10, 2003 12:30 PM To: Multiple recipients of list ORACLE-L Hello List, We are running Benchmark tests on Solaris 2-Node RAC. Consistently noticed the following : - Very high Kernel usage (averaging 45%) on TOP - Statspack has IPC Send Completion sync waits (70% Total ela time) - On trussing top process, found Oracle to be issuing huge number of times system calls in addition to read/writes(which I think are select/inserts). Has anyone noticed this in your environment. I am guessing these to be inter-instance pings, but could not get any hits in Doc/Metalink to confirm this. times call is clocking lot of CPU. Is this normal ? Any pointers would be helpful ? If this is out of context, is there a separate list for RAC? Thanks, Ravi. __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). *This e-mail message is confidential, intended only for the named recipient(s) above and may contain information that is privileged, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please immediately notify corporate MIS at (860) 766-2000 and delete this e-mail message from your computer, Thank you.*1 __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ravi Kulkarni INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
RE: RAC system Calls
Nah - no reason not to use it, especially given how long they've been around. Basically you should enable jumbo frames everywhere you can - whenever your box talks to a host that doesn't support them, it just won't use them, and when it can it'll get the performance advantage (side note: your switch has to support jumbo frames as well, but every major managed switch manufacturer does, so that's not such a big deal). Everyone wins. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ravi Kulkarni Sent: Thursday, July 10, 2003 7:04 PM To: Multiple recipients of list ORACLE-L Subject: RE: RAC system Calls Hey Matt, Thanks for the useful info. into a smaller number of ethernet frames, reducing both latency and overhead. Ideally, each block will fit into one ethernet frame, We are using 16K blocksize, so we cannot avoid datagram splits with 9k frames - but certainly, sounds better than with 1500 frames. so unless you're running one of a couple of third-party gigabit cards, I think you're probably out of luck. Will check with SysAdmins if the GigE we have supports. Are there any risks/disadvantages of using 9k frames for interconnect? -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Matthew Zito INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
