[zones-discuss] extensive packet loss when zones are booted
Hi I have a Sunfire V440 with 2 processors and 10Gbytes of memory I have configured eight zones all using the second ce interface ce1. The port is connected to a nortel buisiness policy switch running at 100Mbs/full duplex. I am seeing unacceptible levels of packet loss to the systems. With all the zones booted and idle we get around 5-6% loss which can climb to 50-60% with activity. One of these zones will run apache2, the rest will be running tomcat. I would appreciate any assitance as I am new to zones. Is this configuration excessive for the hardware?rgds, Alastair NeilHere is the output from ifconfig -a: -bash-3.00# ifconfig -alo0: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 inet 127.0.0.1 netmask ff00lo0:1: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-inst inet 127.0.0.1 netmask ff00lo0:2: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE432 inet 127.0.0.1 netmask ff00 lo0:3: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone hermes-web inet 127.0.0.1 netmask ff00lo0:4: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps inet 127.0.0.1 netmask ff00lo0:5: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE632 inet 127.0.0.1 netmask ff00lo0:6: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE642 inet 127.0.0.1 netmask ff00lo0:7: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE645 inet 127.0.0.1 netmask ff00 lo0:8: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-IT314 inet 127.0.0.1 netmask ff00ce0: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 2 inet 129.174.94.22 netmask ff00 broadcast 129.174.94.255 ether 0:3:ba:74:56:cbce1: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 inet 0.0.0.0 netmask ff00 broadcast 0.255.255.255 ether 0:3:ba:74:56:cbce1:1: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-inst inet 129.174.94.32 netmask ff00 broadcast 129.174.94.255ce1:2: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE432 inet 129.174.94.35 netmask ff00 broadcast 129.174.94.255ce1:3: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone hermes-web inet 129.174.94.33 netmask ff00 broadcast 129.174.94.255ce1:4: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps inet 129.174.94.34 netmask ff00 broadcast 129.174.94.255ce1:5: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE632 inet 129.174.94.36 netmask ff00 broadcast 129.174.94.255ce1:6: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE642 inet 129.174.94.37 netmask ff00 broadcast 129.174.94.255ce1:7: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE645 inet 129.174.94.38 netmask ff00 broadcast 129.174.94.255ce1:8: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-IT314 inet 129.174.94.39 netmask ff00 broadcast 129.174.94.255 ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] extensive packet loss when zones are booted
Hi Alastair, Alastair Neil wrote On 08/22/06 08:59,: Hi I have a Sunfire V440 with 2 processors and 10Gbytes of memory I have configured eight zones all using the second ce interface ce1. The port is connected to a nortel buisiness policy switch running at 100Mbs/full duplex. I am seeing unacceptible levels of packet loss to the systems. With all the zones booted and idle we get around 5-6% loss which can climb to 50-60% with activity. One of these zones will run apache2, the rest will be running tomcat. If things are idle, what data are you loosing? kstat -m ce0 and separately, ce1 might show something. Where and how are you measuring this loss? I would appreciate any assitance as I am new to zones. Is this configuration excessive for the hardware? Theoretically no. I had a customer run 3,000 netscape 2.x instances on an E4000 running 2.5.1 years ago. Business level service with SLAs. So without metrics this configuration seems fine. Steffen Totally unrelated to zones, the first thing that comes to mind is auto-negotation mis-match. But that is pure speculation on my part based on other network performance problems folks have had. rgds, Alastair Neil Here is the output from ifconfig -a: -bash-3.00# ifconfig -a lo0: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:1: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-inst inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:2: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE432 inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:3: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone hermes-web inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:4: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:5: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE632 inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:6: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE642 inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:7: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE645 inet 127.0.0.1 http://127.0.0.1 netmask ff00 lo0:8: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-IT314 inet 127.0.0.1 http://127.0.0.1 netmask ff00 ce0: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 2 inet 129.174.94.22 http://129.174.94.22 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ether 0:3:ba:74:56:cb ce1: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 inet 0.0.0.0 http://0.0.0.0 netmask ff00 broadcast 0.255.255.255 http://0.255.255.255 ether 0:3:ba:74:56:cb ce1:1: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-inst inet 129.174.94.32 http://129.174.94.32 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:2: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE432 inet 129.174.94.35 http://129.174.94.35 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:3: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone hermes-web inet 129.174.94.33 http://129.174.94.33 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:4: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps inet 129.174.94.34 http://129.174.94.34 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:5: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE632 inet 129.174.94.36 http://129.174.94.36 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:6: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE642 inet 129.174.94.37 http://129.174.94.37 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:7: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3 zone apps-SWE645 inet 129.174.94.38 http://129.174.94.38 netmask ff00 broadcast 129.174.94.255 http://129.174.94.255 ce1:8: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 3
Re: [zones-discuss] zones databases
Carisdad wrote: [EMAIL PROTECTED] wrote: On Fri, Aug 18, 2006 at 08:52:29AM +0200, Robert Milkowski wrote: Hello Brian, Wednesday, July 26, 2006, 8:31:06 PM, you wrote: BK With the performance boosts included in recent solaris versions I'm BK told that there's not much of a difference between handing the database BK raw devices vs. using a filesystem anymore. BK To test this out, my customer would like to try both ufs and vxfs BK filesystems in the global zone and lofs mount them to a local zone BK and test the database on that lofs mount. BK Are there any options that should be supplied for the lofs mount and BK are there any options for the ufs and/or vxfs mounts that should be BK employed to assure the performance should be close to raw devices? 1. lofs is probably a bad idea - mount them directly into a zone lofs is the only supported option for vxfs. przemol While lofs is the only officially supported option, mounting directly in the zone can be accomplished with a work-around. see: http://seer.entsupport.symantec.com/docs/276134.htm The customer will only accept officially supported options on this (i.e. lofs). Why exactly would lofs be a bad idea? What is the downside to lofs rather than directly using VxFS? With lofs they get the benefit of the ability to share one filesystem among multiple zones (not at the same mount point in the global zone). But is there anything that would increase/decrease performance by using lofs? Is there a major performance difference to running a database with its storage on lofs vs. VxFS vs. RAW? ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] extensive packet loss when zones are booted
Thanks for al the helpful suggestions,however, It looks like the problem has been resolved. I suspect it was caused by switch misconfiguration. Intially although both ce0 and ce1 were on the same subnet they were on two different switches. I have moved them to the same switch and now I see no packet loss. I suspect spanning-tree was disabled on one of the switches. thanks again for the help.AlastairOn 8/22/06, Steffen Weiberle [EMAIL PROTECTED] wrote:Alastair Neil wrote On 08/22/06 10:49,: On 8/22/06, *Steffen Weiberle* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Hi Alastair, Alastair Neil wrote On 08/22/06 08:59,: Hi I have a Sunfire V440 with 2 processors and 10Gbytes of memory I have configured eight zones all using the second ce interface ce1.The port is connected to a nortel buisiness policy switch running at 100Mbs/full duplex.I am seeing unacceptible levels of packet loss to the systems. With all the zones booted and idle we get around 5-6% loss which can climb to 50-60% with activity.One of these zones will run apache2, the rest will be running tomcat. If things are idle, what data are you loosing? Ok totally idle is perhaps an exaggeration, I noticed the symptoms when typing in an ssh session, periods where the systems appeared hung.I then started mtr from my linux workstation and left it running and noticed that when idle the reported packet loss was averaging 5%, but once I started a ssh session it escalated quickly.Not sure if both connections are to the same interface or IP address. kstat -m ce0 and separately, ce1 might show something. any parameters in particular of interest? It show full duplex for both interfaces, I am having the network guys verify that that is what the switch ports are running too.I don't have access to a CE interface, so I don't know what i'd be looking at 'til I see it.ndd /dev/ce might be good as well. you would need to set the instanceto 0 and then to 1 to get all the data. I am looking for link partnerstatus--is the other end negotiating or not. I should note I see this packet loss on both interfaces, ip_forwarding and ip_strict_dst_multihoming are both 0.These setting should have no effect, both interfaces are on the same subnet. That said, you have local-mac-address?=false in eeprom. Soboth ce0 and ce1 have the same MAC address. Some switches get confusedby that. Not sure how a Nortel switch of your type behaves in thatscenario. But the symptom if there is a problem is that the switch ping-pongs back and forth between the two interfaces as host0:3:ba:74:56:cb, and as such ethernet frames may get dropped on the floor.Easy was to see if that fixes it is to 'ifconfig ce1 ether0:3:ba:74:56:cc', incrementing the lowest octet by 1 (temporarily) Steffen Where and how are you measuring this loss? I would appreciate any assitance as I am new to zones.Is this configuration excessive for the hardware? Theoretically no. I had a customer run 3,000 netscape 2.x instances on an E4000 running 2.5.1 years ago. Business level service with SLAs. So without metrics this configuration seems fine. Steffen Totally unrelated to zones, the first thing that comes to mind is auto-negotation mis-match. But that is pure speculation on my part based on other network performance problems folks have had. rgds, Alastair Neil Here is the output from ifconfig -a: -bash-3.00# ifconfig -a lo0: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:1: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-inst inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:2: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE432 inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:3: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone hermes-web inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:4: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:5: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE632 inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:6: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE642 inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:7: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-SWE645 inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 lo0:8: flags=2001000849UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL mtu 8232 index 1 zone apps-IT314 inet 127.0.0.1 http://127.0.0.1 http://127.0.0.1 netmask ff00 ce0: flags=1000843UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 2 inet 129.174.94.22 http://129.174.94.22 http://129.174.94.22 netmask
Re: [zones-discuss] zones databases
Brian Kolaci wrote: Carisdad wrote: [EMAIL PROTECTED] wrote: On Fri, Aug 18, 2006 at 08:52:29AM +0200, Robert Milkowski wrote: Hello Brian, Wednesday, July 26, 2006, 8:31:06 PM, you wrote: BK With the performance boosts included in recent solaris versions I'm BK told that there's not much of a difference between handing the database BK raw devices vs. using a filesystem anymore. BK To test this out, my customer would like to try both ufs and vxfs BK filesystems in the global zone and lofs mount them to a local zone BK and test the database on that lofs mount. BK Are there any options that should be supplied for the lofs mount and BK are there any options for the ufs and/or vxfs mounts that should be BK employed to assure the performance should be close to raw devices? 1. lofs is probably a bad idea - mount them directly into a zone lofs is the only supported option for vxfs. While lofs is the only officially supported option, mounting directly in the zone can be accomplished with a work-around. see: http://seer.entsupport.symantec.com/docs/276134.htm The customer will only accept officially supported options on this (i.e. lofs). Why exactly would lofs be a bad idea? What is the downside to lofs rather than directly using VxFS? With lofs they get the benefit of the ability to share one filesystem among multiple zones (not at the same mount point in the global zone). But is there anything that would increase/decrease performance by using lofs? Is there a major performance difference to running a database with its storage on lofs vs. VxFS vs. RAW? Not an answer, just a clarification: LOFS essentially re-mounts an existing filesystem onto an additional mount-point. When using LOFS, there is always the original fs type to consider. The comparison would then be a LOFS-mount of a UFS filesystem vs. a UFS filesystem vs. a raw device. Of course, VxFS can be substituted for UFS. -- -- Jeff VICTOR Sun Microsystemsjeff.victor @ sun.com OS AmbassadorSr. Technical Specialist Solaris 10 Zones FAQ:http://www.opensolaris.org/os/community/zones/faq -- ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] zones databases
Jeff Victor wrote: Brian Kolaci wrote: Carisdad wrote: [EMAIL PROTECTED] wrote: On Fri, Aug 18, 2006 at 08:52:29AM +0200, Robert Milkowski wrote: Hello Brian, Wednesday, July 26, 2006, 8:31:06 PM, you wrote: BK With the performance boosts included in recent solaris versions I'm BK told that there's not much of a difference between handing the database BK raw devices vs. using a filesystem anymore. BK To test this out, my customer would like to try both ufs and vxfs BK filesystems in the global zone and lofs mount them to a local zone BK and test the database on that lofs mount. BK Are there any options that should be supplied for the lofs mount and BK are there any options for the ufs and/or vxfs mounts that should be BK employed to assure the performance should be close to raw devices? 1. lofs is probably a bad idea - mount them directly into a zone lofs is the only supported option for vxfs. While lofs is the only officially supported option, mounting directly in the zone can be accomplished with a work-around. see: http://seer.entsupport.symantec.com/docs/276134.htm The customer will only accept officially supported options on this (i.e. lofs). Why exactly would lofs be a bad idea? What is the downside to lofs rather than directly using VxFS? With lofs they get the benefit of the ability to share one filesystem among multiple zones (not at the same mount point in the global zone). But is there anything that would increase/decrease performance by using lofs? Is there a major performance difference to running a database with its storage on lofs vs. VxFS vs. RAW? Not an answer, just a clarification: LOFS essentially re-mounts an existing filesystem onto an additional mount-point. When using LOFS, there is always the original fs type to consider. The comparison would then be a LOFS-mount of a UFS filesystem vs. a UFS filesystem vs. a raw device. Of course, VxFS can be substituted for UFS. I guess the direct question would be a LOFS-mount of a VxFS filesystem with QuickIO enabled. Typically databases want either QuickIO or forcedirectio (on UFS) enabled so that writes aren't buffered. Is that still the case if LOFS is mounted on another filesystem? The DBA's say that writing to a raw device is somewhat equivalent to writing a file on a filesystem with QuickIO enabled. ___ zones-discuss mailing list zones-discuss@opensolaris.org
