RE: Network performance issue
Hello, We had a similar issue that was resolved when we changed the FreeNAS default sharing behavior from sync to async by adding a new Sysctl with the following value, vfs.nfsd.async value to 1 -Original Message- From: Sam Ceylani [mailto:s...@mistercertified.com] Sent: Friday, April 03, 2015 1:19 AM To: users@cloudstack.apache.org Subject: Re: Network performance issue We use Freenas with ZFS also and it is notorious with NFS shares, we created a dataset and shared on NFS and disabled sync option for this dataset from ZFS, I forgot the command but it was something like zfs set sync=disabled /tank/dataset something like that, it will fix it. This has really nothing to do with cloudstack and xenserver which we use both... Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 1:13 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: Still no luck by disabling these flags. Would you please help us for this. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 8:06 PM, Jeronimo Garcia garciaj...@gmail.commailto:garciaj...@gmail.com wrote: mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
RE: Network performance issue
Awesome, that helped me too. Weird I have other few hypervisors using zfs storage they never had problem with sync enabled. I had bond configured for secondary storage traffic, but still speed was super slow. I even thought that SCVMM low config would be issue, but it wasn't. Thanks :) /Sonali -Original Message- From: Sam Ceylani [mailto:s...@mistercertified.com] Sent: Friday, April 3, 2015 2:59 AM To: users@cloudstack.apache.org Subject: Re: Network performance issue np glad it worked,ty Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 5:06 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: Thanks Sam, Its a great help. Now we can able to achieve 50 to 60 MB/s transferring speed. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 10:49 PM, Sam Ceylani s...@mistercertified.commailto:s...@mistercertified.com wrote: We use Freenas with ZFS also and it is notorious with NFS shares, we created a dataset and shared on NFS and disabled sync option for this dataset from ZFS, I forgot the command but it was something like zfs set sync=disabled /tank/dataset something like that, it will fix it. This has really nothing to do with cloudstack and xenserver which we use both... Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 1:13 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.commailto: pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: Still no luck by disabling these flags. Would you please help us for this. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 8:06 PM, Jeronimo Garcia garciaj...@gmail.commailto:garciaj...@gmail.com mailto:garciaj...@gmail.com wrote: mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.com mailto:pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
RE: Network performance issue
Yes, with few more tuning of FreeNAS we are able to achieve quite comparable performance as compared to our xenserver VM. Thanks Regards PankaJ Singh On Apr 4, 2015 1:22 PM, Sonali Jadhav son...@servercentralen.se wrote: Awesome, that helped me too. Weird I have other few hypervisors using zfs storage they never had problem with sync enabled. I had bond configured for secondary storage traffic, but still speed was super slow. I even thought that SCVMM low config would be issue, but it wasn't. Thanks :) /Sonali -Original Message- From: Sam Ceylani [mailto:s...@mistercertified.com] Sent: Friday, April 3, 2015 2:59 AM To: users@cloudstack.apache.org Subject: Re: Network performance issue np glad it worked,ty Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 5:06 PM, Pankaj Singh pank.sin9...@gmail.commailto: pank.sin9...@gmail.com wrote: Thanks Sam, Its a great help. Now we can able to achieve 50 to 60 MB/s transferring speed. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 10:49 PM, Sam Ceylani s...@mistercertified.com mailto:s...@mistercertified.com wrote: We use Freenas with ZFS also and it is notorious with NFS shares, we created a dataset and shared on NFS and disabled sync option for this dataset from ZFS, I forgot the command but it was something like zfs set sync=disabled /tank/dataset something like that, it will fix it. This has really nothing to do with cloudstack and xenserver which we use both... Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com mailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 1:13 PM, Pankaj Singh pank.sin9...@gmail.commailto: pank.sin9...@gmail.commailto: pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: Still no luck by disabling these flags. Would you please help us for this. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 8:06 PM, Jeronimo Garcia garciaj...@gmail.com mailto:garciaj...@gmail.com mailto:garciaj...@gmail.com wrote: mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.com mailto:pank.sin9...@gmail.com mailto:pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
Re: Network performance issue
Thanks Sam, Its a great help. Now we can able to achieve 50 to 60 MB/s transferring speed. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 10:49 PM, Sam Ceylani s...@mistercertified.com wrote: We use Freenas with ZFS also and it is notorious with NFS shares, we created a dataset and shared on NFS and disabled sync option for this dataset from ZFS, I forgot the command but it was something like zfs set sync=disabled /tank/dataset something like that, it will fix it. This has really nothing to do with cloudstack and xenserver which we use both... Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 1:13 PM, Pankaj Singh pank.sin9...@gmail.commailto: pank.sin9...@gmail.com wrote: Still no luck by disabling these flags. Would you please help us for this. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 8:06 PM, Jeronimo Garcia garciaj...@gmail.com mailto:garciaj...@gmail.com wrote: mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.com mailto:pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
Re: Network performance issue
mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
Re: Network performance issue
Still no luck by disabling these flags. Would you please help us for this. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 8:06 PM, Jeronimo Garcia garciaj...@gmail.com wrote: mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
Re: Network performance issue
np glad it worked,ty Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 5:06 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: Thanks Sam, Its a great help. Now we can able to achieve 50 to 60 MB/s transferring speed. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 10:49 PM, Sam Ceylani s...@mistercertified.commailto:s...@mistercertified.com wrote: We use Freenas with ZFS also and it is notorious with NFS shares, we created a dataset and shared on NFS and disabled sync option for this dataset from ZFS, I forgot the command but it was something like zfs set sync=disabled /tank/dataset something like that, it will fix it. This has really nothing to do with cloudstack and xenserver which we use both... Thanks, Sam Ceylani, MBA Computer Engineer MisterCertified Inc. 301 W. Platt St. Suite 447, Tampa, FL 33606x-apple-data-detectors://0/0 P 813tel:813.264.6460.264.6460tel:813.264.6460 M 813tel:813.416.7867.416.7867tel:813.416.7867 F 800tel:800.553.9520.553.9520tel:800.553.9520 E sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.commailto:sam.ceyl...@mistercertified.com On Apr 2, 2015, at 1:13 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.commailto: pank.sin9...@gmail.commailto:pank.sin9...@gmail.com wrote: Still no luck by disabling these flags. Would you please help us for this. Thanks Regards, PankaJ Singh On Thu, Apr 2, 2015 at 8:06 PM, Jeronimo Garcia garciaj...@gmail.commailto:garciaj...@gmail.com mailto:garciaj...@gmail.com wrote: mm i had this problem before i think it was TCP offliading or RX/TX checksumming. I'd say play with ethtool and see if it makes any difference disabling this flags (or enabling them) Im on #cloudstack@freenode if you want to pop in. Thanks On Thu, Apr 2, 2015 at 3:12 PM, Pankaj Singh pank.sin9...@gmail.commailto:pank.sin9...@gmail.com mailto:pank.sin9...@gmail.com wrote: We have two different network on two similar switches. One is our 192.168.1.0/24 and other is 192.168.2.0/24 network. Our switch capacity is 1000Mbps and so does our NIC cards on Host machines. On n/w 192.168.1.0/24, we are using xenserver 6.2 to deploy our Virtual machines and we can able to achieve persistent 75 to 80 MB/s speed while transferring (via scp) one 12GiB file from one VM to another VM on same Xenserver. We are using 192.168.2.0/24 network for our cloudstack setup. We have used following software with their version to establish out private cloud: Cloudstack = v4.4.2 Management Server On = Centos 6.4 x86_64 bit Xenserver as a hypervisor = v6.2 (similar configuration of machine as we have used in 192.168.1.0/24 n/w) FreeNAS as a Primary and Secondary Storage via NFS = v9.3 (filesystem is ZFS based) Virtual Machine OS = Centos 6.5 x86_64 bit We are using basic networking with its all default configurations but we are not able to achieve at least 60 to 65 MB/s of transferring speed. In cloud If we start scp from one machine to another machine then our transferring speed starts from 63 MB/s and then its keep falling till 20 to 25 MB/s. Would you please tell us that what we are missing here. Is ther any kind of global setting we need to configure so that at least we can achieve more than 50 MB/s transferring speed between two guest VM. Thanks and Regards PankaJ Singh
