Re: [Bacula-users] Performance testing simultaneous spooling and despooling

2024-09-17 Thread B. Smith
This is all running on TrueNAS, so BSD. The HBA is an LSI 9220-8i. The NIC is 10Gb, but not relevant here because the data and the tape library are all on the same system. The disks are SATA. 32GB RAM, but I don't see the system running out of RAM while spooling/despooling. The JBOD enclosures are

Re: [Bacula-users] Performance testing simultaneous spooling and despooling

2024-09-16 Thread Gary R. Schmidt
On 17/09/2024 08:58, B. Smith wrote: Good evening, I have a ZFS pool as a dedicated Bacula spool. The pool contains six 4TB drives, configured as three mirrors of two striped disks. My tape drive is LTO8. All the data is local to the server. When I despool without simultaneously spooling anot

Re: [Bacula-users] Performance testing simultaneous spooling and despooling

2024-09-16 Thread B. Smith
Apologies, I misstated the configuration. I do in fact have striped mirrors. On Mon, Sep 16, 2024, 8:59 PM Phil Stracchino wrote: > On 9/16/24 18:58, B. Smith wrote: > > Good evening, > > > > I have a ZFS pool as a dedicated Bacula spool. The pool contains six 4TB > > drives, configured as three

Re: [Bacula-users] Performance testing simultaneous spooling and despooling

2024-09-16 Thread Phil Stracchino
On 9/16/24 18:58, B. Smith wrote: Good evening, I have a ZFS pool as a dedicated Bacula spool. The pool contains six 4TB drives, configured as three mirrors of two striped disks. OK, just one observation: That's generally considered the wrong way to do it. The normally preferred arrangement

[Bacula-users] Performance testing simultaneous spooling and despooling

2024-09-16 Thread B. Smith
Good evening, I have a ZFS pool as a dedicated Bacula spool. The pool contains six 4TB drives, configured as three mirrors of two striped disks. My tape drive is LTO8. All the data is local to the server. When I despool without simultaneously spooling another job, my despool rate is about 280 MB/s

Re: [Bacula-users] performance&design&configuration challenges

2020-10-07 Thread Žiga Žvan
Thanks Joe for this info. It looks like it is a client issue as it is written in the document (many small files; operations like stat(), fstat() consume 100% cpu on the client). I think that implementing autochanger solves my problems (mutliple clients will write at the same time and utilize b

Re: [Bacula-users] performance&design&configuration challenges

2020-10-07 Thread Joe GREER
Ziga, It is sad to hear your having issues with Bacula. Some of your concerns have been here since 2005. The only thing you can do to speed things up is to spool the whole job to very fast disk(SSD), break up your large job(number of files), make sure your database is on very fast disk(SSD) and ha

Re: [Bacula-users] performance&design&configuration challenges

2020-10-06 Thread Žiga Žvan
Hi, I have done some testing: a) testing storage with dd command (eg: dd if=/dev/zero of=/storage/test1.img bs=1G count=1 oflag=dsync). The results are: -writing to IBM storage (with cloud enabled) shows 300 MB/sec -writing to local SSD storage shows 600 MB/sec. I guess storage is not a bottlene

Re: [Bacula-users] performance&design&configuration challenges

2020-10-06 Thread Josh Fisher
On 10/6/20 3:45 AM, Žiga Žvan wrote: I believe that I have my spooling attributes set correctly on jobdefs (see bellow). Spool attributes = yes; Spool data defaults to no. Any other idea for performance problems? Regard, Ziga The client version is very old. First try updating the client to

Re: [Bacula-users] performance&design&configuration challenges

2020-10-06 Thread Heitor Faria
LatAm mobile1: + 1 909 655-8971 mobile2: + 55 61 98268-4220 América Latina [ http://bacula.lat/] Original Message From: Žiga Žvan Sent: Tuesday, October 6, 2020 03:11 AM To: bacula-users@lists.sourceforge.net Subject: [Bacula-users] performance&design&configuration challen

Re: [Bacula-users] performance&design&configuration challenges

2020-10-06 Thread Žiga Žvan
I believe that I have my spooling attributes set correctly on jobdefs (see bellow). Spool attributes = yes; Spool data defaults to no. Any other idea for performance problems? Regard, Ziga JobDefs {   Name = "bazar2-job"   Type = Backup   Level = Incremental   Client = bazar2.kranj.cetrtapot.

[Bacula-users] performance&design&configuration challenges

2020-10-06 Thread Žiga Žvan
Hi, I'm having some performance challenges. I would appreciate some educated guess from an experienced bacula user. I'm changing old backup sw that writes to tape drive with bacula writing  to disk. The results are: a) windows file server backup from a deduplicated drive (1.700.000 files, 90

Re: [Bacula-users] performance&design&configuration challenges

2020-10-05 Thread Josh Fisher
On 10/5/20 9:20 AM, Žiga Žvan wrote: Hi, I'm having some performance challenges. I would appreciate some educated guess from an experienced bacula user. I'm changing old backup sw that writes to tape drive with bacula writing  to disk. The results are: a) windows file server backup from a d

[Bacula-users] performance&design&configuration challenges

2020-10-05 Thread Žiga Žvan
Hi, I'm having some performance challenges. I would appreciate some educated guess from an experienced bacula user. I'm changing old backup sw that writes to tape drive with bacula writing  to disk. The results are: a) windows file server backup from a deduplicated drive (1.700.000 files, 900

Re: [Bacula-users] Performance settings for large file LTO-6 backup

2015-06-30 Thread John Stoffel
My one comment is that if you really want to keep this data forever, then you should *really* be making multiple copies to tape, and then also re-reading them and comparing them against the master data. I also think that the biggest time sink will be the finding and building of the daily tar fi

Re: [Bacula-users] Performance settings for large file LTO-6 backup

2015-06-27 Thread Josh Fisher
On 6/27/2015 1:37 AM, Andrew Noonan wrote: > On Fri, Jun 26, 2015 at 2:17 PM, Ana Emília M. Arruda > wrote: >> Are you going to generate a .tar of about 250TB every day? Which will >> be the nature of your restores? You´re going to need always the >> restore of the whole data set or occasional

Re: [Bacula-users] Performance settings for large file LTO-6 backup

2015-06-26 Thread Andrew Noonan
On Fri, Jun 26, 2015 at 2:17 PM, Ana Emília M. Arruda wrote: > Hello Andrew, > > On Fri, Jun 19, 2015 at 5:10 PM, Andrew Noonan wrote: >> >> Hi all, >> >> After wrestling with a Dell TL4000 in the thread marked "Dell >> TL4000 labeling timeout", it looks like the autochanger is going to be >

Re: [Bacula-users] Performance settings for large file LTO-6 backup

2015-06-26 Thread Ana Emília M . Arruda
Hello Andrew, On Fri, Jun 19, 2015 at 5:10 PM, Andrew Noonan wrote: > Hi all, > > After wrestling with a Dell TL4000 in the thread marked "Dell > TL4000 labeling timeout", it looks like the autochanger is going to be > fine thanks to the efforts of several people, especially Ana, on this >

[Bacula-users] Performance settings for large file LTO-6 backup

2015-06-19 Thread Andrew Noonan
Hi all, After wrestling with a Dell TL4000 in the thread marked "Dell TL4000 labeling timeout", it looks like the autochanger is going to be fine thanks to the efforts of several people, especially Ana, on this list. Moving forward, I'm about to start running jobs to at first backfill a

Re: [Bacula-users] Performance Issues

2012-10-01 Thread lst_hoe02
Zitat von Rodrigo Abrantes Antunes : > Hi, when restoring, listing files, backing up, purging or pruning mysql > process uses 100% CPU and the machine is unusable, and such operations last > to long. Doing some research I found that this can be related to database > indexes, but I didn't understa

[Bacula-users] Performance Issues

2012-09-24 Thread Rodrigo Abrantes Antunes
Hi, when restoring, listing files, backing up, purging or pruning mysql process uses 100% CPU and the machine is unusable, and such operations last to long. Doing some research I found that this can be related to database indexes, but I didn't understanf well what I need to do .Here is the output

Re: [Bacula-users] performance over WAN links

2011-10-16 Thread James Harper
Disregard. It's flying along now at the expected speeds. I blame sun spots. James > -Original Message- > From: James Harper [mailto:james.har...@bendigoit.com.au] > Sent: Monday, 17 October 2011 4:14 PM > To: bacula-users@lists.sourceforge.net > Subject: [Bacula-use

[Bacula-users] performance over WAN links

2011-10-16 Thread James Harper
I'm revisiting a remote backup, and am troubled by the fact that Bacula appears to be making using of only a fraction of the available bandwidth. iperf tells me there is around 750KBits/second of usable TCP bandwidth in the fd->sd direction, but Bacula only reports a Bytes/sec rate of 30Kbytes/sec

Re: [Bacula-users] Performance with latent networks

2011-10-03 Thread mayak-cq
On Mon, 2011-10-03 at 10:34 +0200, mayak-cq wrote: > > zurich and copenhagen are 22.589 ms apart on a shared 100mbit > connection -- using the bandwidth delay product: > > theoretical > bandwidth delayproductBits bitsPerByte > bytesInWindow > 500 000 000 * .022589 = 11 2

Re: [Bacula-users] Performance with latent networks

2011-10-03 Thread mayak-cq
On Sun, 2011-10-02 at 11:45 +0200, Radosław Korzeniewski wrote: > Hello, > > 2011/9/30 reaper > > sounds like your sysctl.conf needs tweaking? have you tried > something like this on both sides? > > kernel.msgmnb = 65536 > kernel.msgmax = 65536 >

Re: [Bacula-users] Performance with latent networks

2011-10-02 Thread Radosław Korzeniewski
Hello, 2011/9/30 reaper > sounds like your sysctl.conf needs tweaking? have you tried something like > this on both sides? > > kernel.msgmnb = 65536 > kernel.msgmax = 65536 > kernel.shmmax = 68719476736 > kernel.shmall = 4294967296 > > These are the IPC (inter process communication) kernel p

Re: [Bacula-users] Performance with latent networks

2011-10-01 Thread mayak-cq
On Sat, 2011-10-01 at 07:57 -0700, reaper wrote: > i'm going through a similar issue -- zurich, barca, copenhagen ... > > mayak-cq, can you test bacula performance under your conditions? With and > without ssh (or something similar) tunnel. hi reaper, sure -- i can do that tomorrow. are you

[Bacula-users] Performance with latent networks

2011-10-01 Thread reaper
i'm going through a similar issue -- zurich, barca, copenhagen ... mayak-cq, can you test bacula performance under your conditions? With and without ssh (or something similar) tunnel. +-- |This was sent by rea...@lmn.name via Ba

Re: [Bacula-users] Performance with latent networks

2011-09-30 Thread mayak-cq
On Fri, 2011-09-30 at 02:18 -0700, reaper wrote: > if i understand you correctly, bacula is only using a 128k -- way too > small? curious -- have you played with "Maximum Network Buffer Size" ? > does this help? > > Yes, that's correct, bacula can only scale window to 128k that's why > throughput

[Bacula-users] Performance with latent networks

2011-09-30 Thread reaper
if i understand you correctly, bacula is only using a 128k -- way too small? curious -- have you played with "Maximum Network Buffer Size" ? does this help? Yes, that's correct, bacula can only scale window to 128k that's why throughput gets limited to 10Mbit/s. With ssh tunnel between client an

Re: [Bacula-users] Performance with latent networks

2011-09-30 Thread mayak-cq
On Thu, 2011-09-29 at 23:43 -0700, reaper wrote: > i have noticed that scp is not a good measure of throughput -- i do not know > why. i use an openvpn tunnel between sites and loose about 20% of throughput > due to the tunnel. check window size on distant machine (using wireshark) to > verify

[Bacula-users] Performance with latent networks

2011-09-29 Thread reaper
i have noticed that scp is not a good measure of throughput -- i do not know why. i use an openvpn tunnel between sites and loose about 20% of throughput due to the tunnel. check window size on distant machine (using wireshark) to verify that some upstream device is not changing it. No, no, no.

Re: [Bacula-users] Performance with latent networks

2011-09-29 Thread mayak-cq
On Thu, 2011-09-29 at 21:12 -0700, reaper wrote: > sounds like your sysctl.conf needs tweaking? have you tried something like > this on both sides? > > kernel.msgmnb = 65536 > kernel.msgmax = 65536 > kernel.shmmax = 68719476736 > kernel.shmall = 4294967296 > > # long fat pipes > net

[Bacula-users] Performance with latent networks

2011-09-29 Thread reaper
sounds like your sysctl.conf needs tweaking? have you tried something like this on both sides? kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 # long fat pipes net.core.wmem_max = 8388608 net.core.rmem_max = 8388608 Yes, I tried t

Re: [Bacula-users] Performance with latent networks

2011-09-29 Thread mayak-cq
On Thu, 2011-09-29 at 04:20 -0700, reaper wrote: > Hello. > > I saw this post recently > http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg47393.html > and it seems I'm affected by this problem too. Bacula shows extremely low > performance in networks with high rtt. I have sd i

[Bacula-users] Performance with latent networks

2011-09-29 Thread reaper
Hello. I saw this post recently http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg47393.html and it seems I'm affected by this problem too. Bacula shows extremely low performance in networks with high rtt. I have sd in Germany and client in USA. Bacula can make backups on speed

[Bacula-users] performance problem - Windows & TLS

2011-08-25 Thread mariusz
Hi Kamil, 2 days ago I had got the same problem like you. Open client config file for windows and put "Maximum Network Buffer Size = 65536" in FileDaemon :) It will resolve the problem Mariusz. +-- |This was sent by mariusz

[Bacula-users] performance problem - Windows & TLS

2011-08-23 Thread kamilfurman
Hello After enabling TLS, I've noticed significant performance drawback. I've made some tests for both Linux (Fedora 13) and Windows XP clients. I've used 250MB tar archive. One file. No compression. BACKUP: Windows TLS 850 kB/s Windows NO_TLS 8500 kB/s Linux T

Re: [Bacula-users] Performance with many files

2011-08-11 Thread Adrian Reyer
On Fri, Jul 08, 2011 at 08:30:17AM +0200, Adrian Reyer wrote: > Speed improved many many times. My incremental backup finished after > just 10 minutes while it took 2h earlier. This had been the benefit of using InnoDB over MyISAM. However, at 12GB RAM and 8900 File entries (12GB file on disk)

Re: [Bacula-users] performance problem

2011-08-04 Thread Jeff Shanholtz
cula-users@lists.sourceforge.net Subject: [Bacula-users] performance problem I currently have 3 clients doing a full backup (simultaneously). According to "status client" one is getting 300kb/s (this one is my director and storage server machine), one is getting 225kb/s, and one is

[Bacula-users] performance problem

2011-08-03 Thread Jeff Shanholtz
I currently have 3 clients doing a full backup (simultaneously). According to "status client" one is getting 300kb/s (this one is my director and storage server machine), one is getting 225kb/s, and one is getting 50kb/s. I've disabled AV on access scanning for the bacula-fd.exe process. I have sof

Re: [Bacula-users] Performance

2011-07-27 Thread Konstantin Khomoutov
On Tue, 26 Jul 2011 06:18:25 -0700 Steve Ellis wrote: > >> Another point, even with your current config, if you > >> aren't doing data spooling you are probably slowing things down > >> further, as well as wearing out both the tapes and heads on the > >> drive with lots of shoeshining. > > (I'm a

Re: [Bacula-users] Performance

2011-07-26 Thread Steve Ellis
On 7/26/2011 5:04 AM, Konstantin Khomoutov wrote: > On Tue, 26 Jul 2011 00:18:05 -0700 > Steve Ellis wrote: > > [...] >> Another point, even with your current config, if you >> aren't doing data spooling you are probably slowing things down >> further, as well as wearing out both the tapes and hea

Re: [Bacula-users] Performance

2011-07-26 Thread Rickifer Barros
I disabled the Compression and my speed rate boosted from 8.2 MB/s to 40.8 MB/s but I'm still using Bacula encryption. On Mon, Jul 25, 2011 at 10:14 PM, James Harper < james.har...@bendigoit.com.au> wrote: > > 2011/7/25 Rickifer Barros : > > > Hello Guys... > > > > > > This weekend I did a backup

Re: [Bacula-users] Performance

2011-07-26 Thread Konstantin Khomoutov
On Tue, 26 Jul 2011 00:18:05 -0700 Steve Ellis wrote: [...] > Another point, even with your current config, if you > aren't doing data spooling you are probably slowing things down > further, as well as wearing out both the tapes and heads on the drive > with lots of shoeshining. (I'm asking as

Re: [Bacula-users] Performance

2011-07-26 Thread Jeremy Maes
>> I was under the impression that_all_ LTO4 drives implemented encryption >> (though if having the data traversing the LAN encrypted is your goal, >> you'd still have to do something). I don't know enough about it to know >> how good the encryption in LTO4 is, however (or for that matter, how th

Re: [Bacula-users] Performance

2011-07-26 Thread James Harper
> >> Disable software compression. The tape drive will compress much faster > >> than the client. > >> > > If you can find compressible patterns in the encrypted data stream then > > you are not properly encrypting it. The only option would be to compress > > before encryption which means you can't

Re: [Bacula-users] Performance

2011-07-26 Thread Steve Ellis
On 7/25/2011 6:14 PM, James Harper wrote: >> 2011/7/25 Rickifer Barros: >>> Hello Guys... >>> >>> This weekend I did a backup with a size of 41.92 GB that took 1 hour > and 24 >>> minutes with a rate of 8.27 MB/s. >>> >>> My Bacula Server is installed in a IBM server connected in a Tape > Drive LTO

Re: [Bacula-users] Performance

2011-07-25 Thread James Harper
> 2011/7/25 Rickifer Barros : > > Hello Guys... > > > > This weekend I did a backup with a size of 41.92 GB that took 1 hour and 24 > > minutes with a rate of 8.27 MB/s. > > > > My Bacula Server is installed in a IBM server connected in a Tape Drive LTO4 > > (120 MB/s) via SAS connection (3 Gb/s).

Re: [Bacula-users] Performance

2011-07-25 Thread Phil Stracchino
On 07/25/11 11:13, John Drescher wrote: > On Mon, Jul 25, 2011 at 11:06 AM, Rickifer Barros > wrote: >> I did this beforebut, I didn't know how check in Debian if it really is >> being compressed by the tape drive. The only thing that I got was the bacula >> information about the SD and FD Wri

Re: [Bacula-users] Performance

2011-07-25 Thread Rickifer Barros
OK John...I'll test it. On Mon, Jul 25, 2011 at 12:13 PM, John Drescher wrote: > On Mon, Jul 25, 2011 at 11:06 AM, Rickifer Barros > wrote: > > I did this beforebut, I didn't know how check in Debian if it really > is > > being compressed by the tape drive. The only thing that I got was the

Re: [Bacula-users] Performance

2011-07-25 Thread John Drescher
On Mon, Jul 25, 2011 at 11:06 AM, Rickifer Barros wrote: > I did this beforebut, I didn't know how check in Debian if it really is > being compressed by the tape drive. The only thing that I got was the bacula > information about the SD and FD Written and the "mt" command in Linux don't > say

Re: [Bacula-users] Performance

2011-07-25 Thread Rickifer Barros
I did this beforebut, I didn't know how check in Debian if it really is being compressed by the tape drive. The only thing that I got was the bacula information about the SD and FD Written and the "mt" command in Linux don't say me the real data size of the volume, so I chose to trust on the so

Re: [Bacula-users] Performance

2011-07-25 Thread John Drescher
2011/7/25 Rickifer Barros : > Hello Guys... > > This weekend I did a backup with a size of 41.92 GB that took 1 hour and 24 > minutes with a rate of 8.27 MB/s. > > My Bacula Server is installed in a IBM server connected in a Tape Drive LTO4 > (120 MB/s) via SAS connection (3 Gb/s). > > I'm using En

Re: [Bacula-users] Performance

2011-07-25 Thread Rickifer Barros
I forgot to say that the files I backed up are locally in the Bacula Server. On Mon, Jul 25, 2011 at 11:48 AM, Rickifer Barros wrote: > Hello Guys... > > This weekend I did a backup with a size of 41.92 GB that took 1 hour and 24 > minutes with a rate of 8.27 MB/s. > > My Bacula Server is install

[Bacula-users] Performance

2011-07-25 Thread Rickifer Barros
Hello Guys... This weekend I did a backup with a size of 41.92 GB that took 1 hour and 24 minutes with a rate of 8.27 MB/s. My Bacula Server is installed in a IBM server connected in a Tape Drive LTO4 (120 MB/s) via SAS connection (3 Gb/s). I'm using Encryption and Compression Gzip6. I think th

Re: [Bacula-users] Performance with many files

2011-07-07 Thread Adrian Reyer
On Wed, Jul 06, 2011 at 11:08:44AM -0400, Phil Stracchino wrote: > for table in $(mysql -N --batch -e 'select > concat(table_schema,'.',table_name) from information_schema.tables where > engine='MyISAM' and table_schema not in > ('information_schema','mysql')'); do mysql -N --batch -e "alter table

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-07-06 Thread Steve Costaras
mailto:eric.bolleng...@baculasystems.com] Sent: Wednesday, July 6, 2011 11:20 AM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Performance options for single large (100TB) server backup? Hello, On 07/06/2011 04:20 PM, Florian Heigl wrote: > Saving multiple streams is somethi

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-07-06 Thread Eric Bollengier
Hello, On 07/06/2011 04:20 PM, Florian Heigl wrote: > Saving multiple streams is something that has been proven as a > solution for many years, and where that is still too slow NDMP comes > into place. (in case of ZFS NDMP is still at a unusable stage) > > 100TB is a lot, but I wonder if everyone

Re: [Bacula-users] Performance with many files

2011-07-06 Thread Phil Stracchino
On 07/06/11 10:41, Adrian Reyer wrote: > On Wed, Jul 06, 2011 at 10:09:56AM -0400, Phil Stracchino wrote: >> should I use for my tables?" is MyISAM.[1] At this point, wherever >> possible, EVERYONE should be using InnoDB. > > I will, if the current backup ever finishes. For a start on MySQL 5.1 >

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-07-06 Thread Steve Costaras
ving it to a closed source one if that was possible (it's not like I'm a large company here at all). -Original Message- From: Florian Heigl [mailto:florian.he...@gmail.com] Sent: Wednesday, July 6, 2011 09:20 AM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users]

Re: [Bacula-users] Performance with many files

2011-07-06 Thread Adrian Reyer
On Wed, Jul 06, 2011 at 10:09:56AM -0400, Phil Stracchino wrote: > should I use for my tables?" is MyISAM.[1] At this point, wherever > possible, EVERYONE should be using InnoDB. I will, if the current backup ever finishes. For a start on MySQL 5.1 though (Debian squeeze). I am aware InnoDB has a

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-07-06 Thread Florian Heigl
Hi, Breaking the server into multiple file daemons sounds as broken as breaking the stuff amanda users had to do (break your filesystem into something that fits a tape). Saving multiple streams is something that has been proven as a solution for many years, and where that is still too slow NDMP co

Re: [Bacula-users] Performance with many files

2011-07-06 Thread Phil Stracchino
On 07/06/11 08:04, Adrian Reyer wrote: > Hi, > > I am using bacula for a bit more than a month now and the database gets > slower and slower both for selecting stuff and for running backups as > such. > I am using a MySQL database, still myisam tables and I am considering > switching to InnoDB tab

[Bacula-users] Performance with many files

2011-07-06 Thread Adrian Reyer
Hi, I am using bacula for a bit more than a month now and the database gets slower and slower both for selecting stuff and for running backups as such. I am using a MySQL database, still myisam tables and I am considering switching to InnoDB tables or postgresql. Amongst normal fileserver data the

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-07-04 Thread Steve Costaras
t that alone probably will not solve your problem. I think you're going to have to do a lot of different configurations and test which ones work best for your design parameters (i.e. questions like "How long can I go w/o a full backup" and "How long can I stand a complete dis

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Christian Manal
Am 28.06.2011 18:40, schrieb Steve Costaras: > > > How would the the various parts communicate if you're running multiple > instances on different ports? I would think just by creating multiple > jobs would create multiple socket streams and do the same thing. I should have gotten another coff

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Steve Costaras
hink you're going to have to do a lot of different configurations and test which ones work best for your design parameters (i.e. questions like "How long can I go w/o a full backup" and "How long can I stand a complete disaster recovery restore taking"). From: "Ste

[Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Bob Hetzel
nd "How long can I stand a complete disaster recovery restore taking"). > From: "Steve Costaras" > Subject: [Bacula-users] Performance options for single large (100TB) > server backup? > To: bacula-users@lists.sourceforge.net > Message-ID: > Conte

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Roy Sigurd Karlsbakk
Hi Out of curiosity, why do you do such "forklift replacements" when ZFS supports replacing individual drives, letting the pool resilver and then automatically grow to the new size? roy - Original Message - > I have been using Bacula for over a year now and it has been providing > 'passab

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Steve Costaras
Problem is not really just tape I/O speeds but the ability to get data to it. I.e. the SD is running at about 50% cpu overhead right now (single core) so it could possible handle (2) LTO4 drives assuming a new SD is not spawned off per drive? I don't really need 'rait' itself as that wou

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Steve Costaras
How would the the various parts communicate if you're running multiple instances on different ports? I would think just by creating multiple jobs would create multiple socket streams and do the same thing. On 2011-06-28 02:09, Christian Manal wrote: - File daemon is single threaded so

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Josh Fisher
On 6/27/2011 8:43 PM, Steve Costaras wrote: > > > > > - How to stream a single job to multiple tape drives. Couldn't > figure this out so that only one tape drive is being used. > There are hardware RAIT controllers available from Ultera (http://www.ultera.com/tapesolutions.htm). A R

Re: [Bacula-users] Performance options for single large (100TB) server backup?

2011-06-28 Thread Christian Manal
> - File daemon is single threaded so is limiting backup performance. Is there > was a way to start more than one stream at the same time for a single machine > backup? Right now I have all the file systems for a single client in the same > file set. > > - Tied in with above, accurate backups

[Bacula-users] Performance options for single large (100TB) server backup?

2011-06-27 Thread Steve Costaras
I have been using Bacula for over a year now and it has been providing 'passable' service though I think since day one I have been streching it to it's limits or need a paradigm shift in how I am configuring it. Basically, I have a single server which has direct atached disk (~128TB / 112 dri

Re: [Bacula-users] Performance with latent networks

2011-04-11 Thread Peter Hoskin
Hi, > The issue, I imagine with transfer rates is between the bacula-fd and > bacula-sd. Correct > Do we presume the -sd is in Sydney? You don't say what speed > the Sydney ADSL2+ link is (though apparently it can manage at least > 2.2MByte/sec). 24mbit down, 1mbit up > Is tha

Re: [Bacula-users] Performance with latent networks

2011-04-11 Thread Gavin McCullagh
Hi, On Mon, 11 Apr 2011, Peter Hoskin wrote: > I'm using bacula to do backups of some remote servers, over the Internet > encapsulated in OpenVPN (just to make sure things are encrypted and kept off > public address space). > > The bacula-fd is in Montreal Canada with 100mbit Ethernet. I also ha

[Bacula-users] Performance with latent networks

2011-04-10 Thread Peter Hoskin
Hi, I'm using bacula to do backups of some remote servers, over the Internet encapsulated in OpenVPN (just to make sure things are encrypted and kept off public address space). The bacula-fd is in Montreal Canada with 100mbit Ethernet. I also have another bacula-fd in Canberra Australia on

[Bacula-users] Performance with Mac OSX, Encryption and Gzip3

2010-12-20 Thread Hugo Letemplier
Hi I am using Bacula inside an Cent OS openVZ container on a virtualization cluster. I tried with and without compression and/or encryption and the rates are very various. I am backing up a KVM client on the same nodes of the cluster and an external mac client. I can have rates of more than 30 or

Re: [Bacula-users] Performance problem on a Job of a filesystem with a lots of Files

2010-09-23 Thread Alan Brown
On 23/09/10 15:26, Andrés Yacopino wrote: > I think i am getting worst performance because of ramdon disk access > speed, is that true? > Yes. If you use the time command on your tar process you will find it is similarly slow. Actually it's not so much random disk access speed as the fixed tim

Re: [Bacula-users] Performance problem on a Job of a filesystem with a lots of Files

2010-09-23 Thread John Drescher
> I need to improve performance of a Job which backups 150 files (mail > and File Server). > I was compressing the files on disk in some tgz files first (tar and > gzip) ,then backuping then on tape with Bacula, i was getting about: > > Job write elapsed time = 00:32:16, Transfer rate = 44.93 M

[Bacula-users] Performance problem on a Job of a filesystem with a lots of Files

2010-09-23 Thread Andrés Yacopino
I need to improve performance of a Job which backups 150 files (mail and File Server). I was compressing the files on disk in some tgz files first (tar and gzip) ,then backuping then on tape with Bacula, i was getting about: Job write elapsed time = 00:32:16, Transfer rate = 44.93 M Bytes/seco

Re: [Bacula-users] Performance speedup for LTO-4 (only 70MB/s reached)?

2010-09-17 Thread Udo Lembke
Hi, i'm talking about despooling speed (not overall). But it's the same speed without spooling, but it's clear that over the network i can reach max. 90MB/s. I use 32 bit because of the OpenVZ-template but just some minutes ago, i create the same machine as 64bit, and the speed is a little bit

[Bacula-users] Performance speedup for LTO-4 (only 70MB/s reached)?

2010-09-17 Thread Udo Lembke
Hi, i need some tips for a backup-server. On a new backup-server i reach a backup-speed to an LTO-4 drive of 70MB/s only. Here the config: bacula 5.0.3 self compiled on debian squeeze 32bit OpenVZ-vm (Proxmox as virtualisation platform). neo200s jukebox (1 lto-4 SAS-drive) connected via a LSI SA

[Bacula-users] Performance issues after upgrade

2010-03-16 Thread Wouter Verhelst
Hi, On a bacula installation that I originally set up on a machine running Debian "etch", everything was running smoothly. Since etch has been EOL'ed, it was necessary to upgrade it, and we recently did do an upgrade to lenny. This also involved an upgrade of bacula from 1.38 (the version in etch

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-23 Thread Jeronimo Zucco
Citando James Harper : > Does MySQL have a 'profiler' tool like MSSQL does? I spend most of my > time in MSSQL and if some operation is running slow I just attach the > profiler to it and capture the queries and focus on the ones that are > taking most of the time. > > James What is the impact

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-20 Thread Kern Sibbald
On Saturday 20 June 2009 08:51:53 Tom Sommer wrote: > Tom Sommer wrote: > > Mike Holden wrote: > >> Jari Fredriksson wrote: > INSERT INTO Filename( Name ) > SELECT a.Name > FROM ( > > SELECT DISTINCT Name > FROM batch > ) AS a > WHERE NOT > EXISTS ( > >>

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Tom Sommer
Tom Sommer wrote: > Mike Holden wrote: > >> Jari Fredriksson wrote: >> >> INSERT INTO Filename( Name ) SELECT a.Name FROM ( SELECT DISTINCT Name FROM batch ) AS a WHERE NOT EXISTS ( SELECT Name FROM Filename AS f WHERE f.Na

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Bruno Friedmann
Jari Fredriksson wrote: >> James Harper wrote: >>> The subquery returns a very small result set (0 or 1, >>> assuming you use DISTINCT) and so isn't too inefficient. >>> It's when you say 'WHERE NOT EXISTS (SOME QUERY WITH >>> LOTS OF RESULTS)' that you start to really bog down >> True, but if the

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Bruno Friedmann
Jari Fredriksson wrote: >> James Harper wrote: >>> The subquery returns a very small result set (0 or 1, >>> assuming you use DISTINCT) and so isn't too inefficient. >>> It's when you say 'WHERE NOT EXISTS (SOME QUERY WITH >>> LOTS OF RESULTS)' that you start to really bog down >> True, but if the

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Jari Fredriksson
> James Harper wrote: >> The subquery returns a very small result set (0 or 1, >> assuming you use DISTINCT) and so isn't too inefficient. >> It's when you say 'WHERE NOT EXISTS (SOME QUERY WITH >> LOTS OF RESULTS)' that you start to really bog down > > True, but if the outer query contains a very

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread James Harper
> > James Harper wrote: > > The subquery returns a very small result set (0 or 1, assuming you use > > DISTINCT) and so isn't too inefficient. It's when you say 'WHERE NOT > > EXISTS (SOME QUERY WITH LOTS OF RESULTS)' that you start to really bog > > down > > True, but if the outer query contains

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Mike Holden
James Harper wrote: > The subquery returns a very small result set (0 or 1, assuming you use > DISTINCT) and so isn't too inefficient. It's when you say 'WHERE NOT > EXISTS (SOME QUERY WITH LOTS OF RESULTS)' that you start to really bog > down True, but if the outer query contains a very large num

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread James Harper
> > INSERT INTO Filename(Name) > > SELECT DISTINCT Name > > FROM batch AS a > > WHERE NOT EXISTS > > ( > > SELECT Name > > FROM Filename AS f > > WHERE f.Name = a.Name > > ) > > You may also want to consider using a JOIN rather than a subquery with a > NOT EXISTS, something like (untes

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Tom Sommer
Mike Holden wrote: > Jari Fredriksson wrote: > >>> INSERT INTO Filename( Name ) >>> SELECT a.Name >>> FROM ( >>> >>> SELECT DISTINCT Name >>> FROM batch >>> ) AS a >>> WHERE NOT >>> EXISTS ( >>> >>> SELECT Name >>> FROM Filename AS f >>> WHERE f.Name = a.Name >>> ) >>> >>> >> That looks s

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Mike Holden
Jari Fredriksson wrote: >> >> INSERT INTO Filename( Name ) >> SELECT a.Name >> FROM ( >> >> SELECT DISTINCT Name >> FROM batch >> ) AS a >> WHERE NOT >> EXISTS ( >> >> SELECT Name >> FROM Filename AS f >> WHERE f.Name = a.Name >> ) >> > > That looks silly. > > I would write it shorter as > > INSERT

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Martin Simmons
> On Fri, 19 Jun 2009 09:51:20 +0200, Tom Sommer said: > > Martin Simmons wrote: > >> On Thu, 18 Jun 2009 17:11:04 +0200, Michel Meyers said: > >> > >> Martin Simmons wrote: > >> > On Wed, 17 Jun 2009 13:48:58 +0200, Tom Sommer said: >

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Martin Simmons
> On Fri, 19 Jun 2009 03:00:54 +0300, Jari Fredriksson said: > > > > > INSERT INTO Filename( Name ) > > SELECT a.Name > > FROM ( > > > > SELECT DISTINCT Name > > FROM batch > > ) AS a > > WHERE NOT > > EXISTS ( > > > > SELECT Name > > FROM Filename AS f > > WHERE f.Name = a.Name > > ) > >

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Tom Sommer
Martin Simmons wrote: >> On Thu, 18 Jun 2009 17:11:04 +0200, Michel Meyers said: >> >> Martin Simmons wrote: >> On Wed, 17 Jun 2009 13:48:58 +0200, Tom Sommer said: Martin Simmons wrote: >> On Tue, 16 Jun 2009 1

Re: [Bacula-users] Performance with MySQL queries since 3.0.0 (Dir inserting attributes hang)

2009-06-19 Thread Tom Sommer
Jari Fredriksson wrote: >> INSERT INTO Filename( Name ) >> SELECT a.Name >> FROM ( >> >> SELECT DISTINCT Name >> FROM batch >> ) AS a >> WHERE NOT >> EXISTS ( >> >> SELECT Name >> FROM Filename AS f >> WHERE f.Name = a.Name >> ) >> >> > > That looks silly. > > I would write it shorter as > > I

  1   2   >