Re: [Bacula-users] Extremely Slow Performance on VMware
Citando Geert Stappers geert.stapp...@vanadgroup.com: Op 20120928 om 20:38 schreef Rodrigo Abrantes Antunes: Citando Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br: Citando Geert Stappers geert.stapp...@vanadgroup.com: Rodrigo Abrantes Antunes: Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. Yes, and how is the further design? In others words: The provided list can read as One physical computer with fibre channel disk hosts all the VMs. } One physical computer, with fibre channel disk, hosts all the VMs. If it is so, then tell so. Otherwise eloborate the setup, the design. Back to During a backup I can see bacula-sd using 100% cpu, And where did you see the 100%? ( Which tool was used to read that performance valule? ) I would like to see the output of vmstat 2 5 during non-back-up-time and also the output of vmstat 2 5 during back-up-time. Thing I'm interrested in, are the CPU columns. Especial the colums system and wait. screenshot $ vmstat 2 3 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 6712 11360 192584 159120 0 0 5 4 6 11 5 6 90 0 0 0 6712 11344 192584 159120 0 0 0 0 67 342 7 15 79 0 0 0 6712 11344 192584 159120 0 0 0 0 65 340 8 14 78 0 /screenshot And to avoid an extra e-mail exchange: I'm asking for 2 to the power 3, so 8 measurements. So 2 moments (during backup or outside backup) on 2 Bacula compoments ( storage deamon and file deamon ) on the VMs on 2 physical hosts. Yes, that means that I assume the VMware hosts have a 'vmstat' command. That is because I'm not familair with VMware, I'm from the Xen world :-) I have a physical machine that is a Vmware ESX node wich hosts only one vm, the one with bacula-director, bacula-sd and bacula-fd (called bacula-server), this vm has an RDM with the fibre channel storage where the volumes partition is mounted. Then I have all my clients (some are physycal machines and others are vms in other ESX nodes) with bacula-fd that are backed up. When I manually run a job to backup one of these clients in bacula-server I can see (with the command htop) that bacula-sd is using 100% of the cpu, I also noted that the backup starts at around 4MB/s and What I see for the 'htop' over here, is that there seen to be information in the color of cpu usage. It would interresting to see how 100% CPU usage is divide in system, user and I/O wait. after some time it is around 300KB/s. If I simple send the same files to be backed up with scp for example the transfer goes around 100MB/s. The vms don't have vmstat, I use linux own commands. AFAIK is 'vmstat' default installed on every Linux and Unix system. One thing I noted now, in the vm htop says that 100% cpu is used and actually the machine is very slow when backing up so I think this value is accurate but in VSphere Client in the performance chart it says that the vm is using only 400Mhz of the 5000Mhz that were allocated, but the node cpu usage is low so I don't know why it isn't aloccating more MHZ to the bacula-server. How VMware allocates CPU cycles to VMs is beyond my current knowledge (and off-topic on the bacula user mailinglist ) I installed vmstat, I can't do vmstat during non backup time because it is currently backing up my mail server, about 200gb, it is doing this for almost 15h: vmstat 2 5 during backing-up on the bacula-server (director, storagedaemon) procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 3 0 0 16576 15592 1763272 0 0 10 55 2 7 0 8 92 0 1 0 0 15916 15592 1763964 0 0 0 0 204 49 1 50 49 0 2 0 0 16812 15588 1762996 0 0 0 2 172 72 1 78 21 0 2 0 0 17820 15604 1767348 0 0 0 18 193 114 3 66 32 0 1 0 0 16296 15604 1769924 0 0 0 0 277 29 0 55 45 0 That is _not_ 100% CPU usage, there was at least 20% idle time. vmstat 2 5 during backing-up on the physical mail-server (filedaemon) procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 16824 38496 143856 5019424 0 0 45 42 3 4 1 0 98 1 0 0 16824
Re: [Bacula-users] Extremely Slow Performance on VMware
On 9/28/2012 12:15 PM, Rodrigo Abrantes Antunes wrote: I have a physical machine that is a Vmware ESX node wich hosts only one vm, the one with bacula-director, bacula-sd and bacula-fd (called bacula-server), this vm has an RDM with the fibre channel storage where the volumes partition is mounted. Then I have all my clients (some are physycal machines and others are vms in other ESX nodes) with bacula-fd that are backed up. When I manually run a job to backup one of these clients in bacula-server I can see (with the command htop) that bacula-sd is using 100% of the cpu, I also noted that the backup starts at around 4MB/s and after some time it is around 300KB/s. If I simple send the same files to be backed up with scp for example the transfer goes around 100MB/s. The vms don't have vmstat, I use linux own commands. Make sure that the Bacula database storage is not on the same physical disk drive(s) as either the Bacula spool directory or the client partitions being backed up. Bacula does many database writes during a backup, so disk thrashing can (or will) occur if bacula-fd is reading from the same physical drive that bacula-dir or bacula-sd is writing to. -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance on VMware
Op 20120928 om 20:38 schreef Rodrigo Abrantes Antunes: Citando Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br: Citando Geert Stappers geert.stapp...@vanadgroup.com: Rodrigo Abrantes Antunes: Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. Yes, and how is the further design? In others words: The provided list can read as One physical computer with fibre channel disk hosts all the VMs. } One physical computer, with fibre channel disk, hosts all the VMs. If it is so, then tell so. Otherwise eloborate the setup, the design. Back to During a backup I can see bacula-sd using 100% cpu, And where did you see the 100%? ( Which tool was used to read that performance valule? ) I would like to see the output of vmstat 2 5 during non-back-up-time and also the output of vmstat 2 5 during back-up-time. Thing I'm interrested in, are the CPU columns. Especial the colums system and wait. screenshot $ vmstat 2 3 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 6712 11360 192584 15912000 5 46 11 5 6 90 0 0 0 6712 11344 192584 15912000 0 0 67 342 7 15 79 0 0 0 6712 11344 192584 15912000 0 0 65 340 8 14 78 0 /screenshot And to avoid an extra e-mail exchange: I'm asking for 2 to the power 3, so 8 measurements. So 2 moments (during backup or outside backup) on 2 Bacula compoments ( storage deamon and file deamon ) on the VMs on 2 physical hosts. Yes, that means that I assume the VMware hosts have a 'vmstat' command. That is because I'm not familair with VMware, I'm from the Xen world :-) I have a physical machine that is a Vmware ESX node wich hosts only one vm, the one with bacula-director, bacula-sd and bacula-fd (called bacula-server), this vm has an RDM with the fibre channel storage where the volumes partition is mounted. Then I have all my clients (some are physycal machines and others are vms in other ESX nodes) with bacula-fd that are backed up. When I manually run a job to backup one of these clients in bacula-server I can see (with the command htop) that bacula-sd is using 100% of the cpu, I also noted that the backup starts at around 4MB/s and What I see for the 'htop' over here, is that there seen to be information in the color of cpu usage. It would interresting to see how 100% CPU usage is divide in system, user and I/O wait. after some time it is around 300KB/s. If I simple send the same files to be backed up with scp for example the transfer goes around 100MB/s. The vms don't have vmstat, I use linux own commands. AFAIK is 'vmstat' default installed on every Linux and Unix system. One thing I noted now, in the vm htop says that 100% cpu is used and actually the machine is very slow when backing up so I think this value is accurate but in VSphere Client in the performance chart it says that the vm is using only 400Mhz of the 5000Mhz that were allocated, but the node cpu usage is low so I don't know why it isn't aloccating more MHZ to the bacula-server. How VMware allocates CPU cycles to VMs is beyond my current knowledge (and off-topic on the bacula user mailinglist ) I installed vmstat, I can't do vmstat during non backup time because it is currently backing up my mail server, about 200gb, it is doing this for almost 15h: vmstat 2 5 during backing-up on the bacula-server (director, storagedaemon) procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 3 0 0 16576 15592 176327200105527 0 8 92 0 1 0 0 15916 15592 176396400 0 0 204 49 1 50 49 0 2 0 0 16812 15588 176299600 0 2 172 72 1 78 21 0 2 0 0 17820 15604 176734800 018 193 114 3 66 32 0 1 0 0 16296 15604 176992400 0 0 277 29 0 55 45 0 That is _not_ 100% CPU usage, there was at least 20% idle time. vmstat 2 5 during backing-up on the physical mail-server (filedaemon) procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 16824 38496 143856 501942400454234 1 0 98 1 0 0 16824 30928 143868 502000400 336 434 1328 981 1 1 96 2 2 0 16824 35844 143868 502076800 458 116 1121 635 0 1 99 0 0 0 16824 32180 143876
Re: [Bacula-users] Extremely Slow Performance on VMware
Op 20120927 om 21:55 schreef Rodrigo Abrantes Antunes: Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. giving more informations like snip/ will give us more ideas to troubleshoot your problem. Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. Yes, and how is the further design? In others words: The provided list can read as One physical computer with fibre channel disk hosts all the VMs. If it is so, then tell so. Otherwise eloborate the setup, the design. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. There could also be beter disk drivers. The virtio drivers come to mind. Back to During a backup I can see bacula-sd using 100% cpu, And where did you see the 100%? ( Which tool was used to read that performance valule? ) I would like to see the output of vmstat 2 5 during non-back-up-time and also the output of vmstat 2 5 during back-up-time. Thing I'm interrested in, are the CPU columns. Especial the colums system and wait. screenshot $ vmstat 2 3 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 6712 11360 192584 15912000 5 46 11 5 6 90 0 0 0 6712 11344 192584 15912000 0 0 67 342 7 15 79 0 0 0 6712 11344 192584 15912000 0 0 65 340 8 14 78 0 /screenshot And to avoid an extra e-mail exchange: I'm asking for 2 to the power 3, so 8 measurements. So 2 moments (during backup or outside backup) on 2 Bacula compoments ( storage deamon and file deamon ) on the VMs on 2 physical hosts. Yes, that means that I assume the VMware hosts have a 'vmstat' command. That is because I'm not familair with VMware, I'm from the Xen world :-) Cheers Geert Stappers -- http://www.vanadcimplicity.com/ -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
What kind of files are you backing up? I've seen speeds as low as 0,5 MB / sec on a more or less idle host which has a gazillion of small image files on it. In fact I've needed to split up the backup into several sub-jobs to prevent it from running more than 48h or so ;-) Also, I've seen jobs which start off incredibly slow only to pick up speed near the end... very weird. (bacula version 5.2.6 in this case). Have you checked your mysql tables for a corrupt index on the File or Filename tables? All the best, Uwe -- NIONEX --- Ein Unternehmen der Bertelsmann SE Co. KGaA -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
Hello, 2012/9/27 Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. It is about 1MB/s. Where should I start looking? In various places. :) During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 It is fairly old version of Bacula. Is it possible to upgrade? FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. First you have to check if your storage server is able to write data to disk fast enough. You can do a test with a simple dd if=/dev/zero of=/your-storage-filesystem bs=65536. Second you have to check if you set Spool Attributes = yes in your job, if not change it. You'll have to setup a spool directory for your device too. Next check a network speed. What is your RTT between client and storage daemon? I know it is a 1Gbit, but it is possible that you cannot utilize so much. You can test with a simple ftp client. Is it possible that you have a Firewall between both servers? Last you have to check a client utilization and filesystem read speed. How many small files you have to backup? Every new file to backup has its own overhead. Bacula has to find this file, stat it, open it, read data from it (with 64k block) and finally close it. If you have a 10M files of 1kB then your backup will be much slower then 1 file of 10GB. You have to be aware of it. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. It is possible that without vmware tools you are using an emulated storage and network I/O instead of much faster virtualized. best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
Hello, Is the compression enabled? I had this kind of problem because of the compression. Regards Gael Guilmin From: Rodrigo Abrantes Antunes [mailto:rodrigoantu...@pelotas.ifsul.edu.br] Sent: Thursday, September 27, 2012 9:55 PM To: Davide Franco Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Extremely Slow Performance Citando Davide Franco dfra...@dflc.chmailto:dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.brmailto:rodrigoantu...@pelotas.ifsul.edu.br wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.netmailto:Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Hi Rodrigo, giving more informations like * version of the director, storage, fd * which database back-end are you using * did you install the vmware tools ? * which operating system (version) are using ? * on what kind of storage does your DRM point to (sata, iscsi, fc, etc.) * etc. will give us more ideas to troubleshoot your problem. Regards Davide Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. Thanks This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message. -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance on VMware
I have a physical machine that is a Vmware ESX node wich hosts only one vm, the one with bacula-director, bacula-sd and bacula-fd (called bacula-server), this vm has an RDM with the fibre channel storage where the volumes partition is mounted. Then I have all my clients (some are physycal machines and others are vms in other ESX nodes) with bacula-fd that are backed up. When I manually run a job to backup one of these clients in bacula-server I can see (with the command htop) that bacula-sd is using 100% of the cpu, I also noted that the backup starts at around 4MB/s and after some time it is around 300KB/s. If I simple send the same files to be backed up with scp for example the transfer goes around 100MB/s. The vms don't have vmstat, I use linux own commands. Citando Geert Stappers geert.stapp...@vanadgroup.com: Op 20120927 om 21:55 schreef Rodrigo Abrantes Antunes: Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. giving more informations like snip/ will give us more ideas to troubleshoot your problem. Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. Yes, and how is the further design? In others words: The provided list can read as One physical computer with fibre channel disk hosts all the VMs. If it is so, then tell so. Otherwise eloborate the setup, the design. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. There could also be beter disk drivers. The virtio drivers come to mind. Back to During a backup I can see bacula-sd using 100% cpu, And where did you see the 100%? ( Which tool was used to read that performance valule? ) I would like to see the output of vmstat 2 5 during non-back-up-time and also the output of vmstat 2 5 during back-up-time. Thing I'm interrested in, are the CPU columns. Especial the colums system and wait. screenshot $ vmstat 2 3 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 6712 11360 192584 159120 0 0 5 4 6 11 5 6 90 0 0 0 6712 11344 192584 159120 0 0 0 0 67 342 7 15 79 0 0 0 6712 11344 192584 159120 0 0 0 0 65 340 8 14 78 0 /screenshot And to avoid an extra e-mail exchange: I'm asking for 2 to the power 3, so 8 measurements. So 2 moments (during backup or outside backup) on 2 Bacula compoments ( storage deamon and file deamon ) on the VMs on 2 physical hosts. Yes, that means that I assume the VMware hosts have a 'vmstat' command. That is because I'm not familair with VMware, I'm from the Xen world :-) Cheers Geert Stappers -- http://www.vanadcimplicity.com/ -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/bacula-users -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
Citando Uwe Schuerkamp uwe.schuerk...@nionex.net: What kind of files are you backing up? I've seen speeds as low as 0,5 MB / sec on a more or less idle host which has a gazillion of small image files on it. In fact I've needed to split up the backup into several sub-jobs to prevent it from running more than 48h or so ;-) Also, I've seen jobs which start off incredibly slow only to pick up speed near the end... very weird. (bacula version 5.2.6 in this case). Have you checked your mysql tables for a corrupt index on the File or Filename tables? All the best, Uwe --NIONEX --- Ein Unternehmen der Bertelsmann SE Co. KGaA I'm backing up multiple different clients, some with lots of small files like the mail server and others with big files. All of them starts the backup fast and end slow. Here is the show index, I don't know if it's right or not: show index from File; +---++--+--+-+---+-+--++--++-+ | File | 0 | PRIMARY | 1 | FileId | A | 274094 | NULL | NULL | | BTREE | | | File | 1 | JobId | 1 | JobId | A | 31 | NULL | NULL | | BTREE | | | File | 1 | JobId_2 | 1 | JobId | A | 31 | NULL | NULL | | BTREE | | | File | 1 | JobId_2 | 2 | PathId | A | 3559 | NULL | NULL | | BTREE | | | File | � � 1 | JobId_2 | 3 | FilenameId | A | 274094 | NULL | NULL | | BTREE | | +---++--+--+-+---+-+--++--++-+ show index from Filename; +--++--+--+-+---+-+--++--++-+ | Filename | 0 | PRIMARY | 1 | FilenameId | A | 1588498 | NULL | NULL | | BTREE | | | Filename | 1 | Name | 1 | Name | A | 1588498 | 255 | NULL | | BTREE | | +--++--+--+-+---+-+-- ++--++-+ show index from Path; +---++--+--+-+---+-+--++--++-+ | Path | 0 | PRIMARY | 1 | PathId | A | 40618 | NULL | NULL | | BTREE | | | Path | 1 | Path | 1 | Path | A | 40618 | 255 | NULL | | BTREE | | +---++--+--+-+---+-+--++--++-+ -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
It's the version that is in ubuntu 10.04 repos and I can't update the system right now. My storage array with raid6 can write up to 400MB/s and trough the network up to 100MB/s (Gigabit), these speeds were tested and confirmed and I don't use firewall between bacula-server and clients. I din't set spool attributes, can it increase speed? How? Sorry I'm new to bacula. f I simple send the same files to be backed up with scp for example the transfer goes around 100MB/s, so I don't think I'm using an emulated storage and network I/O. Citando Radosław Korzeniewski rados...@korzeniewski.net: Hello, 2012/9/27 Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. It is about 1MB/s. Where should I start looking? In various places. :) During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 It is fairly old version of Bacula. Is it possible to upgrade? FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. First you have to check if your storage server is able to write data to disk fast enough. You can do a test with a simple dd if=/dev/zero of=/your-storage-filesystem bs=65536. Second you have to check if you set Spool Attributes = yes in your job, if not change it. You'll have to setup a spool directory for your device too. Next check a network speed. What is your RTT between client and storage daemon? I know it is a 1Gbit, but it is possible that you cannot utilize so much. You can test with a simple ftp client. Is it possible that you have a Firewall between both servers? Last you have to check a client utilization and filesystem read speed. How many small files you have to backup? Every new file to backup has its own overhead. Bacula has to find this file, stat it, open it, read data from it (with 64k block) and finally close it. If you have a 10M files of 1kB then your backup will be much slower then 1 file of 10GB. You have to be aware of it. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. It is possible that without vmware tools you are using an emulated storage and network I/O instead of much faster virtualized. best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
No compression is enabled. Citando Gael Guilmin gael.guil...@pdgm.com: Hello, Is the compression enabled? I had this kind of problem because of the compression. Regards Gael Guilmin FROM:Rodrigo Abrantes Antunes [mailto:rodrigoantu...@pelotas.ifsul.edu.br] SENT: Thursday, September 27, 2012 9:55 PM TO: Davide Franco CC: bacula-users@lists.sourceforge.net SUBJECT: Re: [Bacula-users] Extremely Slow Performance Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Hi Rodrigo, giving more informations like * version of the director, storage, fd * which database back-end are you using * did you install the vmware tools ? * which operating system (version) are using ? * on what kind of storage does your DRM point to (sata, iscsi, fc, etc.) * etc. will give us more ideas to troubleshoot your problem. Regards Davide Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. Thanks This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message. -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance on VMware
Citando Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br: I have a physical machine that is a Vmware ESX node wich hosts only one vm, the one with bacula-director, bacula-sd and bacula-fd (called bacula-server), this vm has an RDM with the fibre channel storage where the volumes partition is mounted. Then I have all my clients (some are physycal machines and others are vms in other ESX nodes) with bacula-fd that are backed up. When I manually run a job to backup one of these clients in bacula-server I can see (with the command htop) that bacula-sd is using 100% of the cpu, I also noted that the backup starts at around 4MB/s and after some time it is around 300KB/s. If I simple send the same files to be backed up with scp for example the transfer goes around 100MB/s. The vms don't have vmstat, I use linux own commands. Citando Geert Stappers geert.stapp...@vanadgroup.com: Op 20120927 om 21:55 schreef Rodrigo Abrantes Antunes: Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. giving more informations like snip/ will give us more ideas to troubleshoot your problem. Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. Yes, and how is the further design? In others words: The provided list can read as One physical computer with fibre channel disk hosts all the VMs. If it is so, then tell so. Otherwise eloborate the setup, the design. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. There could also be beter disk drivers. The virtio drivers come to mind. Back to During a backup I can see bacula-sd using 100% cpu, And where did you see the 100%? ( Which tool was used to read that performance valule? ) I would like to see the output of vmstat 2 5 during non-back-up-time and also the output of vmstat 2 5 during back-up-time. Thing I'm interrested in, are the CPU columns. Especial the colums system and wait. screenshot $ vmstat 2 3 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 6712 11360 192584 159120 0 0 5 4 6 11 5 6 90 0 0 0 6712 11344 192584 159120 0 0 0 0 67 342 7 15 79 0 0 0 6712 11344 192584 159120 0 0 0 0 65 340 8 14 78 0 /screenshot And to avoid an extra e-mail exchange: I'm asking for 2 to the power 3, so 8 measurements. So 2 moments (during backup or outside backup) on 2 Bacula compoments ( storage deamon and file deamon ) on the VMs on 2 physical hosts. Yes, that means that I assume the VMware hosts have a 'vmstat' command. That is because I'm not familair with VMware, I'm from the Xen world :-) Cheers Geert Stappers -- http://www.vanadcimplicity.com/ -- Got visibility? Most devs has no idea what their production app looks like. Find out how fast your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/bacula-users One thing I noted now, in the vm htop says that 100% cpu is used and actually the machine is very slow when backing up so I think this value is accurate but in VSphere Client in the performance chart it says that the vm is using only 400Mhz of the 5000Mhz that were allocated, but the node cpu usage is low so I don't know why it isn't aloccating more MHZ to the bacula-server. I installed vmstat, I can't do vmstat during non backup time because it is currently backing up my mail server, about 200gb, it is doing this for almost 15h: vmstat 2 5 during backing-up on the bacula-server (director, storagedaemon)
[Bacula-users] Extremely Slow Performance
Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
Citando Davide Franco dfra...@dflc.ch: On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br wrote: Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Hi Rodrigo, giving more informations like * version of the director, storage, fd * which database back-end are you using * did you install the vmware tools ? * which operating system (version) are using ? * on what kind of storage does your DRM point to (sata, iscsi, fc, etc.) * etc. will give us more ideas to troubleshoot your problem. Regards Davide Director: 5.0.1-1ubuntu1 Storage: 5.0.1-1ubuntu1 FD: 5.0.1-1ubuntu1 (some clients have lower version) Database: mysqI OS: Ubuntu 10.04.4 x64 Server FC Storage 4 GBits/s. All my network is Gigabit Ethernet. One question: can vmware tools have any influence in this case? From what I know vmware tools is mainly for better graphics performance isn't it? I din't installed it. Thanks -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Extremely Slow Performance
On Thu, Sep 27, 2012 at 7:04 PM, Rodrigo Abrantes Antunes rodrigoantu...@pelotas.ifsul.edu.br wrote: ** Hi, I'm having extremely slow performance when backing up, a 10gb backup took around 3h. Where should I start looking? During a backup I can see bacula-sd using 100% cpu, memory usage is normal. My system is a vmware virtual machine with a cpu with two 2,5ghz cores and 2gb ram. The vm have a separate partition for the volumes using ext4 and mounted via RDM in a fc storage array with raid6. I tried mapping more cores and more ram for the vm with the same result so I don't think the problem is lack of resources. Thanks. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users Hi Rodrigo, giving more informations like - version of the director, storage, fd - which database back-end are you using - did you install the vmware tools ? - which operating system (version) are using ? - on what kind of storage does your DRM point to (sata, iscsi, fc, etc.) - etc. will give us more ideas to troubleshoot your problem. Regards Davide -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
