Re: [Bacula-users] Bacula disk to tape config
On Tue, 27 Sep 2011, René Moser wrote: We are currently using a proprietary backup solution, and we evaluation bacula to replace it. We have some 100 hosts to backup up. The current work flow is like: 1. backup server backups host files over working time to a disk volume on backup server. 2. During night, the disk volume is written to tape, verified, and the disk volume will be purged. The catalogue knows afterwards that the files have to be restored from tapes and no longer exists on disk volume. So during working day, the tape drive is accessible and we are able to restore files form the tapes. How can does be done with bacula? It sounds like what you're looking for could be implemented as: 1. A normal Full/Differential/Incremental job to a disk-based storage device to happen during the day. 2. A Migrate job to tape from disk to happen during the night. It is important to note in the above that at present the disk-based storage device and the tape device must be on the same Storage Daemon for the Migrate to work. This should achieve moreorless what you're looking for I think (though I must say that I haven't used migrate jobs personally). To be honest though, the above work-flow raises a lot of questions for me: 1. You're doing backups of your live servers _during_ working hours. This is the opposite of what most people ordinarily do. Does this not create load on the already busy servers? Are the backups you get always in a consistent state? 2. It is apparently not possible to backup all 100 hosts straight to tape, so you write them to disk first. Why is that? It sounds like you're manually doing Spooling, which Bacula has transparent support for. You could, in Bacula, set up a single job which copied the backup data to disk first and then as that continues, spool them out to tape. This would need mean the two jobs happening at the same time, so you'd have to do it at night -- or else the tape drives would not be available during the day for restores. 3. Do you use Incremental backups at present or full backups each day? Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
On Tue, 27 Sep 2011, René Moser wrote: On Tue, 2011-09-27 at 16:51 +0100, Gavin McCullagh wrote: 2. It is apparently not possible to backup all 100 hosts straight to tape, so you write them to disk first. Why is that? It sounds like you're manually doing Spooling, which Bacula has transparent support for. You could, in Bacula, set up a single job which copied the backup data to disk first and then as that continues, spool them out to tape. Do you have an example, how the spooling like you described can be implemented? We're only starting to use tapes ourselves now, so this is somewhat theoretical still for me. However, this is where the documentation is and I'm pretty sure it's used by a good number of people who use tapes and the feature has been in Bacula for a long time: http://www.bacula.org/5.0.x-manuals/en/main/main/Data_Spooling.html Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula disk to tape config
Hi, On Tue, 27 Sep 2011, René Moser wrote: Okay, it is not _really_ during day it is more like backups over night to disk and backup to tape should be finished in the morning. But this is just a detail. We did not have (yet) really (big) problems about consistency. But as you say these systems are live systems and if a we hit a bad timing, that is always a problem. I'm not sure how your timings work out, but spooling might give you a shorter overall backup window -- so the tapes would free up quicker, which might allow you to run the entire process outside of work hours. It's difficult to tell from here though. This may be a limitation of the current backup software in use. It may be faster this way (we are using a tape changer). We also have had troubles with some tape drives and this way (backup first to disk) was an advantage so the backup was already on disk and we just had to fix the tape changer and backup the disk volume to tape. Not having extensively used tapes and spooling, I can't say if this will be workable here. You'd need to test. I am really open do to it the bacula way if there is a better workflow. I guess you'll need to do some testing and see what works best for you. It sounds somewhat promising anyway. 3. Do you use Incremental backups at present or full backups each day? Actually mostly differentials. Might incrementals be quicker in terms of limiting load on the live server? Another feature you might consider is the ability to run Virtual Full backups to consolidate an old full backup and subsequent incremental or differential backups into a new full backup. We do this using disk-based backups, but I'm not sure it will be so practical on tapes. You need to have two devices too, one to read from and one to write to which you may not have. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] long-term archival of tapes -- growth of the catalog
Hi, On Wed, 21 Sep 2011, Dan Langille wrote: Do other people do this? I do Copy to tape. Be aware: your tape drive must be on the same SD as your disk storage. Copy and migrate jobs can involve only one SD. You cannot copy/migrate from one SD to another. Gah. I had forgotten that, thanks. That limitation is okay for about 80% of our backups, but there is a second SD currently holding about 20% (and ideally we planned it to take a greater share). Is there any sign of development on shifting data between SDs? I know we had a conversation on this list about it about 18 months ago and there was some interest and even some sponsorship. http://adsm.org/lists/html/Bacula-users/2010-01/msg00050.html In the meantime, has anyone got any sensible workaround? I guess we could run all backups to one SD but have it NFS mount some of its disk-based volumes, but this would double the network traffic during a backup so I'm not so keen on that. Maybe we could leave backups as normal but just use an NFS mount and a dedicated device during the COPY job so that the SD with the tape drive also had the disks on a local device? Thanks for any suggestions Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] long-term archival of tapes -- growth of the catalog
Hi, On Thu, 22 Sep 2011, Ben Walton wrote: Not sure if this would work for you (especially as it requires some up front choices) but this is what I'm looking at here. I did some small tests with it and it seemed to work fine. Each host has a dedicated storage pool and storage device (I'm using all disk-based presently) with each pool/storage combo being an individual directory (/backups/pool/$hostname). Each pool has a hostname-based media type. That's pretty much what we do already. To move backups between SD's, I simply rsync'd the current set of volumes to the new SD and modified the host settings for the SD (and updated the path to the storage as appropriate in the device settings). I wouldn't be so keen on that bit. This will be happening on a regular basis and on a schedule, so I'd really rather not have to modify the running config. If we were talking about tapes here, it doesn't seem crazy for Volume-X in Pool-X which is normally seen in Device-X on SD-X to be placed in Device-Y on SD-Y. So what I'm wondering is, if I create an extra device definition on the other SD which looks at the same volumes via an NFS mount, would that be just like me moving a tape from one drive to another and would that do the job. I would then specify the special DeviceY in my scheduled COPY job the volume would be available on the same SD as the tape drive. This is untested and I may be missing something here though, this entire idea may be unworkable. If it is unworkable, I'd love someone to point that out :-) It would probably be a good idea to enable spooling, given that the job is going to be read over the network. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] long-term archival of tapes -- growth of the catalog
Hi, we've been happily using Bacula now for a few years, with a couple of big disk arrays as the storage devices/media. We use something along the lines of what the manual documents for fully automated disk-based backups. This has worked well and is really quick and convenient for doing restores from. It's expensive in hard disks though and none of the data is offline so a particularly nasty online incident might take out the backups as well as the live data. Our plan now is to add a Dell LTO5 drive in and start using a COPY job once per month. Almost everything gets a monthly Full backup to disk, so we'll then copy those jobs to tape and move them off-site. A couple of tapes will be retired from the pool each year and will be stored moreorless indefinitely. Do other people do this? If so, how do you deal with pruning? Do you just let the database grow over time or do you let the data get pruned and use bscan or other low-level volume tools to read them if necessary? Is there another approach I'm missing? Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore performance
On Wed, 21 Sep 2011, Erik P. Olsen wrote: I am running a very smooth bacula 5.0.3 on Fedora 14. Everything seems to be OK except restores which are incredibly slow. How can I debug it to see what's wrong? Start off by telling us what part of the restore process is slow: - building the file tree for selection - the actual restore of files to disk - something else Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore performance
Hi, On Wed, 21 Sep 2011, Gavin McCullagh wrote: On Wed, 21 Sep 2011, Erik P. Olsen wrote: I am running a very smooth bacula 5.0.3 on Fedora 14. Everything seems to be OK except restores which are incredibly slow. How can I debug it to see what's wrong? Start off by telling us what part of the restore process is slow: - building the file tree for selection - the actual restore of files to disk - something else A few minutes after replying I noticed that you asked this in more detail back in February and said that building the tree was the slow part. I have suffered similar problems with MySQL explored in the thread linked below. By logging slow queries I managed to identify a single query which was the primary cause. http://adsm.org/lists/html/Bacula-users/2010-11/msg00112.html http://adsm.org/lists/html/Bacula-users/2010-11/msg00187.html Then a couple of months ago I noticed a new detail in the wiki about _removing_ bogus indexes. It turns out that by adding extra indexes you can slow down MySQL SELECT queries (I think someone said it doesn't always choose the optimal/correct index). http://wiki.bacula.org/doku.php?id=faq http://adsm.org/lists/html/Bacula-users/2011-08/msg4.html http://adsm.org/lists/html/Bacula-users/2011-08/msg7.html It would be worth taking a look at that. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore performance
Hi, On Wed, 21 Sep 2011, Marcio Merlone wrote: Em 21-09-2011 09:29, Gavin McCullagh escreveu: On Wed, 21 Sep 2011, Erik P. Olsen wrote: I am running a very smooth bacula 5.0.3 on Fedora 14. Everything seems to be OK except restores which are incredibly slow. How can I debug it to see what's wrong? - building the file tree for selection +1 on this. I am running bacula 5.0.1-1ubuntu1 on a Ubuntu 10.04.3 LTS server. Database is about 150GB big and any restore takes 2 or 3 minutes to build the tree. A 150GB database. That's pretty large. How many clients have you? Which database are you using (MySQL, Postgresql)? If you haven't seen it already, this might be useful: http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore performance
Hi, On Wed, 21 Sep 2011, Marcio Merlone wrote: Em 21-09-2011 10:29, Gavin McCullagh escreveu: A 150GB database. That's pretty large. How many clients have you? About a dozen clients - some inactive but still with valid backup - File Retention = 6 months, Job Retention = 1 year. Most clients have few or no changes, but my storage server has 2,825,101 files on a full backup and almost 100.000 on an incremental. It's definitely the database that is 150GB, yeah? We have 40 clients, some of which have 5 million files and similar retention times, but our database is 11GB. Which database are you using (MySQL, Postgresql)? MySQL. Ditto. If you haven't seen it already, this might be useful: http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog Yes, I was there when you mail arrived. It made think about going to Postgres. Is there any upgrade path or helper to go from mysql to postgres? I am now optimizing File table and will then run dbcheck. Check that you have the right indexes (and no extra ones). The removal of a couple of wrong indexes made a massive difference for us. Bacula does seem to be better optimised toward Postgresql, but I'm not convinced you can't get MySQL to work. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore performance
On Wed, 21 Sep 2011, Erik P. Olsen wrote: I forgot to mention that my problem is with building the file tree. So converting to postresql may be the best thing to do. Will the following scenario work? 1. Backup the catalogue 2. Remove mysql database 3. Build postresql database 4. Restore the catalogue Or is there more to it than this? There is more to it than that. You need to stop bacula, dump the mysql database, load it into postgresql, install the postgresql version of bacula-sd and bacula-dir and get that up and running. If your database is large this may take some time. It's not entirely trivial to load a mysql dump into postgresql though. http://mtu.net/~jpschewe/blog/2010/06/migrating-bacula-from-mysql-to-postgresql/ http://workaround.org/node/258 http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/bacula-25/process-to-convert-mysql-to-postgres-85413/ Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] My backup schedule overlaps. Can it be fixed?
Hi, On Thu, 01 Sep 2011, Eric Pratt wrote: The incremental will run, but it shouldn't back anything up if nothing changed since the last time the job ran. When it runs, it looks to see if anything changed and if not, exits with OK. There is no redundancy there. Check the byte count of the job or list the files backed up by the job to see if it actually moved anything. If the byte count is 0 and/or there are no files in it, then it's working exactly as you expect. This should be right, assuming your data isn't changing much around the time of the backups. If you absolutely must only run the one live backup per day, you could consider turning on the accurate backup option and using something more like: Run = Level=VirtualFull Pool=Full-Pool Developer-PC on 1,16 at 5:05 Run = Level=Differential Pool=Diff-Pool Developer-PC sun at 0:15 Run = Level=Incremental Pool=Inc-Pool Developer-PC mon-sat at 0:15 This will run live differential and incremental backups on the days you specify , but rather than running a Full backup on the live data, it uses the existing backup data to consolidate the last full backup and the recent differentials and incrementals into a new Full backup. As the live server isn't involved, you can probably be flexible about what time that happens. This also reduces the load and backup window on the live server in most instances. This should be pretty much equivalent to running a full backup at 0:15 on the 1st and 16th. This is pretty easy to do with disk-based backups, but you may need to think a little if you're using tapes, to make sure all of the right tapes are available. Gavin -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] My backup schedule overlaps. Can it be fixed?
On Tue, 06 Sep 2011, Dan Schaefer wrote: This certainly will fix my problem. All my backups are in fact disk-based and I have a separate process backing up to tapes (not Bacula, atm). I like the idea of the virtual full option, because my full backups usually take longer than I would like. Keeping the full backup processing on the director will help out a lot, because the backups are of the live servers and the diffs and incs will all finish after hours. I will try this VirtualFull option and report back. This page is worth a look then http://www.bacula.org/manuals/en/concepts/concepts/New_Features.html#SECTION0031 http://www.bacula.org/manuals/en/concepts/concepts/New_Features.html#SECTION0037 What I tend to do is along the lines of: 1. Enable the accurate option on the backup job. 2. Create two pools for full backups. 3. Create two disk-based storage devices in bacula-sd.conf, both pointed at the same directory. You will read from one while writing to the other. 4. Define both Storage devices in bacula-dir. 5. Define two pools (P1, P2) for Full backups. On each, set NextPool to be the other full pool. 6. Create two schedule entries Run = Level=VirtualFull Pool=P1 on 1 at 5:05 Run = Level=VirtualFull Pool=P2 on 16 at 5:05 This will switch back and forth between writing from P1 to P2 and P2 to P1. This is how I got it to work and seems to work well. It's possible there's a better way though, as it always seems a bit messy to have multiple pools and devices. On the other hand, if this were tapes, two devices would obviously be required. I'm less sure if you can get away with one pool though, with itself as NextPool. Any corrections or improvements to the above would be most welcome. Gavin -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] removing indexes on File table
Hi, On Tue, 02 Aug 2011, Annette Jaekel wrote: Thats amazing, because I searched for reasons of bad performance in the last days both for backup (18 hours for a full of 300 GB with 5 million files) It seems unlikely that adding an index would noticeably improve backup performance, as that is mostly an INSERT/UPDATE heavy procedure (with the possible exception of accurate backups) and indexes generally slow those jobs down. A full backup job is made up of a few different steps. In the main, it's sending the data from FD to SD followed by adding all of the metadata to the catalog. If you can work out how long each part takes (maybe look at the timestamp on the volume if it's a file volume), you may be able to better see which is the slow bit. Optimising the database won't help if the time is all spent on sending data. It might help to create a test job with a smaller fileset which takes a short time but gives the same rate so you can test quickly. It sounds like the rate you're getting overall is about 37Mbit/sec. A few things to check. - Disk access -- is there anything preventing the FD from serving the data faster than this. Fragmented filesystem, slow disks, low on memory, ... An incremental or differential is a bit complex, but you say it's a full backup, so it should be able to pull the data reasonably quickly. - Check the CPU load, I/O load, memory usage and network bandwidth during a backup on the FD and SD. - Software compression or encryption -- if you have either on, the FD's cpu might become a bottleneck. Try turning them off as an experiment. - Network bandwidth -- use iperf to test a TCP connection from FD-SD and make sure the network is capable of taking the speed you expect. Is the entire network path 100Mbit/sec full duplex or better? Is it busy? and in restores (building the corresponding file tree from my postgres database take more than 10 minutes). One of the first hint I found was to create these indicies you just remove. So I create them, but without any influence corresponding to the performance. Can anyone tell me: Should these indicies (PathID, FilenameID) be advantageous for the database accesses or not? Or is this depending from database package (mysql, postgres)? I wouldn't have expected that adding an index would ever slow down a query, but it seems that with MySQL this isn't always the case and it's clear here that the select query is faster with those indexes removed. Bacula does seem to be better optimised for Postgresql. It would be worthwhile to check that your database has been tuned at least away from the bare defaults. In the case of MySQL, you can use this: http://mysqltuner.pl/mysqltuner.pl the key_buffer seems to be one of the most important parameters. There are lots of howtos around for Postgresql too. Gavin -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos much more. Register early save! http://p.sf.net/sfu/rim-blackberry-1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] removing indexes on File table
Hi, I've reported before that we've had very slow restores, due to the time taken to build the file tree for the console -- which is about 20+ minutes. http://adsm.org//lists/html/Bacula-users/2010-11/msg0.html I've been looking at this again, particularly at a bug that had some interesting debugging and at the FAQ page. http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_catalog http://bugs.bacula.org/view.php?id=1472 I don't recall seeing this detail before, but there is a suggestion to removing additional indexes. These are the indexes on our File table. mysql show indexes FROM File; +---++-+--+-+---+-+--++--++-+ | Table | Non_unique | Key_name| Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +---++-+--+-+---+-+--++--++-+ | File | 0 | PRIMARY |1 | FileId | A | 64154092 | NULL | NULL | | BTREE | | | File | 1 | PathId |1 | PathId | A | 812077 | NULL | NULL | | BTREE | | | File | 1 | FilenameId |1 | FilenameId | A | 2673087 | NULL | NULL | | BTREE | | | File | 1 | FilenameId |2 | PathId | A | 12830818 | NULL | NULL | | BTREE | | | File | 1 | JobId |1 | JobId | A | 1484 | NULL | NULL | | BTREE | | | File | 1 | JobId |2 | PathId | A | 3207704 | NULL | NULL | | BTREE | | | File | 1 | JobId |3 | FilenameId | A | 64154092 | NULL | NULL | | BTREE | | | File | 1 | jobid_index |1 | JobId | A | 1484 | NULL | NULL | | BTREE | | +---++-+--+-+---+-+--++--++-+ 8 rows in set (0.01 sec) It looks to me like I should remove the indexes PathId and FilenameID, leaving the PRIMARY, JobId and jobid_index. Does that seem correct? Gavin -- Got Input? Slashdot Needs You. Take our quick survey online. Come on, we don't ask for help often. Plus, you'll get a chance to win $100 to spend on ThinkGeek. http://p.sf.net/sfu/slashdot-survey ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] removing indexes on File table
Hi, On Mon, 01 Aug 2011, Gavin McCullagh wrote: It looks to me like I should remove the indexes PathId and FilenameID, leaving the PRIMARY, JobId and jobid_index. Does that seem correct? Oooh. I think I'm close to shouting Eureka. Having dropped those indexes (indices?), the same restore now takes about 6-7 minutes to build a tree of 6 million files instead of 25 minutes and more. A 1 million file tree takes under 2 minutes. This leaves me a little slower than the graph on the wiki page, but it's manageable nonetheless and is in the same ball park. The MYI file for that database has dropped from 4.1GB to 2.8GB. gavinmc@cuimhne:~$ sudo ls -lh /var/lib/mysql/bacula/File.MYI /var/lib/mysql/bacula/File.MYD -rw-rw 1 mysql mysql 6.7G 2011-08-01 15:50 /var/lib/mysql/bacula/File.MYD -rw-rw 1 mysql mysql 2.8G 2011-08-01 15:59 /var/lib/mysql/bacula/File.MYI http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_catalog It looks like my day is made. Sincere thanks to mnalis who appears to have put that entry into the FAQ and to Eric and the others who worked on researching this :-) Gavin -- Got Input? Slashdot Needs You. Take our quick survey online. Come on, we don't ask for help often. Plus, you'll get a chance to win $100 to spend on ThinkGeek. http://p.sf.net/sfu/slashdot-survey ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] possible memory leak in bacula/ubuntu/64-bit
Hi, On Tue, 12 Jul 2011, Gavin McCullagh wrote: I stopped the file daemon, started it again and its memory usage fell quite dramatically. Initially it looked like this: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 12768 root 20 0 66512 1616 632 S 0.0 0.0 0:00.00 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf 12769 root 20 0 66512 1616 632 S 0.0 0.0 0:00.00 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf Some minutes later: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 12768 root 20 0 74708 1856 860 S 0.0 0.0 0:00.01 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf 12769 root 20 0 74708 1856 860 S 0.0 0.0 0:00.00 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf where it seems pretty stable. One day (and one backup) later: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 12768 root 20 0 91800 2948 1260 S 0.0 0.1 0:00.09 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf 12769 root 20 0 91800 2948 1260 S 0.0 0.1 0:00.03 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf Gavin -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] possible memory leak in bacula/ubuntu/64-bit
Hi, I was looking at the output of htop today and noticed that the bacula-fd process, although entirely idle was the highest memory process on the server. The output was: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 3949 root 20 0 219M 63296 936 S 0.0 1.5 0:01.61 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf 3952 root 20 0 219M 63296 936 S 0.0 1.5 0:00.81 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf I stopped the file daemon, started it again and its memory usage fell quite dramatically. Initially it looked like this: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 12768 root 20 0 66512 1616 632 S 0.0 0.0 0:00.00 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf 12769 root 20 0 66512 1616 632 S 0.0 0.0 0:00.00 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf Some minutes later: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 12768 root 20 0 74708 1856 860 S 0.0 0.0 0:00.01 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf 12769 root 20 0 74708 1856 860 S 0.0 0.0 0:00.00 /usr/sbin/bacula-fd -c /etc/bacula/bacula-fd.conf where it seems pretty stable. The server (and therefore bacula-fd process) had been up a couple of weeks and has daily incrementals, weekly differentials and monthly full backups scheduled with the accurate option on and compression and encryption off. A full backup is just shy of 500GB. The FD listens on both IPv4 and IPv6 and the backups generally happen over IPv6. Any thoughts? Gavin -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Migrating form mysql to postgresql: Duplicate filename entries?
Hi, On Wed, 29 Jun 2011, Martin Simmons wrote: These duplicates in the File table are probably generated by the batch insert code. Since each pair has the same FileIndex, it should be safe to elide them them. Fair enough, thanks. so perhaps this is safe enough. Does anyone know how these duplicates may have arisen and what the best way to proceed is? It is probably safe the remove the duplicates. It isn't clear how the original duplicates in the Filename arose though, but possibly you had table corruption at some point? Was this database created by an old version of Bacula? The database was originally made in Ubuntu Hardy which was v2.x as I recall now. We then upgraded to v3 (ubuntu packages built using the debian package git archive as we needed volume shadow copy support under Win64) and since then moved to v5 using the Lucid packages. So, yes, made by old versions. Am I missing a constraint I should have? Gavin -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Client / laptop backups
Hi, On Tue, 28 Jun 2011, Roy Sigurd Karlsbakk wrote: We're using Bacula for some backups with three SDs so far, and I wonder if it's possible somehow to allow for client / laptop backups in a good manner. As far as I can see, this will need to either be client-initiated, client saying I'm alive! or something, or having a polling process running to check if the client's online for a given period of time. Is something like this possible or in the works, or is Bacula intended only for server backups? We do this in a somewhat manual way. The client computer has a bconsole configured. When the laptop owner wants a backup, they start bconsole, then type runret, yesret, quityes. They then get a confirmation email when the backup completes. Gavin -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Client / laptop backups
Hi, On Tue, 28 Jun 2011, Gavin McCullagh wrote: We do this in a somewhat manual way. The client computer has a bconsole configured. When the laptop owner wants a backup, they start bconsole, then type runret, yesret, quityes. They then get a confirmation email when the backup completes. I should probably spell out exactly what we do: - configure a console on the director specifically for that laptop - configure a corresponding bconsole on the laptop - create ACLs for that console to access only the job, pool, devices, etc. that relate to it - set the default backup to incremental for that job (assuming that's what you want) so that's what runs immediately - configure a monthly virtual full backup to consolidate the incrementals into a new full backup and allow you to rotate the volumes - configure a dedicated messages entry on the director for this job so that the user gets a copy of the backup confirmation This has worked really well for us. It would be nice of course if a backup could magically start on plugin, but then if the user unplugs shortly afterward, you get a broken backup so we kind of feel that the owner deciding to run a backup is not a bad thing. A nice GUI console client for Windows would perhaps be an improvement but the console commands are so simple that it's easy enough to just write these steps up. Gavin -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Migrating form mysql to postgresql: Loading the database takes very long
On Sat, 25 Jun 2011, Gavin McCullagh wrote: It seems you need to drop --quick which is implied in --skip-opt. The resulting command that I'm working with at the moment is: mysqldump -t -n -c --compatible=postgresql --skip-quote-names --quick \ --lock-tables --add-drop-table --add-locks --create-options --disable-keys \ --extended-insert --set-charset -u bacula -ppassword bacula \ | grep -v INSERT INTO Status\ | sed -e 's/-00-00 00:00:00/1970-01-01 00:00:00/g' \ | sed -e 's/\\0//' bacula-backup.sql` I should clarify this slightly. I needed to remove --skip-opt which implies --skip-quick and add --quick: This option is useful for dumping large tables. It forces mysqldump to retrieve rows for a table from the server a row at a time rather than retrieving the entire row set and buffering it in memory before writing it out. --opt (which is on by default) sets all of: --add-drop-table --add-locks --create-options --disable-keys --extended-insert --lock-tables --quick --set-charset --skip-opt disables --opt, so presumably it negates the above. My command above doesn't go quite that far. Things like table locking shouldn't be necessary if the bacula director is actually stopped at the time. Gavin -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Migrating form mysql to postgresql: Duplicate filename entries?
Hi,
I've been experimenting with a migration from MySQL to Postgres.
One problem I've come across is that there are a handful of duplicate files
in the Filename table
mysql select count(*) as filecount, Filename.name from Filename GROUP BY
Filename.Name ORDER BY filecount DESC LIMIT 30;
+---+---+
| filecount | name |
+---+---+
| 2 | 0 |
| 2 | 1 |
| 2 | 2 |
| 2 | tt172.16.17.36-www_gmail_com.html |
mysql select * from Filename WHERE name IN
('0','1','2','tt172.16.17.36-www_gmail_com.html');
++---+
| FilenameId | Name |
++---+
| 9247 | 0 |
| 101380 | 0 |
| 9248 | 1 |
| 101381 | 1 |
| 9249 | 2 |
| 101382 | 2 |
| 575752 | tt172.16.17.36-www_gmail_com.html |
| 625369 | tt172.16.17.36-www_gmail_com.html |
++---+
and there doesn't appear to be any constraint preventing this.
mysql describe Filename;
++--+--+-+-++
| Field | Type | Null | Key | Default | Extra |
++--+--+-+-++
| FilenameId | int(10) unsigned | NO | PRI | NULL| auto_increment |
| Name | blob | NO | MUL | NULL||
++--+--+-+-++
2 rows in set (0.00 sec)
mysql show create table Filename;
+--+-
| Filename | CREATE TABLE `Filename` (
`FilenameId` int(10) unsigned NOT NULL AUTO_INCREMENT,
`Name` blob NOT NULL,
PRIMARY KEY (`FilenameId`),
KEY `Name` (`Name`(255))
) ENGINE=MyISAM AUTO_INCREMENT=4277244 DEFAULT CHARSET=latin1 |
+--+-
However, the postgresql table does have this constraint and complains when
I try to insert the data.
bacula=# COPY Filename FROM '/var/tmp/bacula-backup/Filename.txt';
ERROR: duplicate key value violates unique constraint filename_name_idx
CONTEXT: COPY filename, line 101380: 101380 0
I could of course prune four duplicate lines from the data before
inserting, but I'm afraid of the possible effect on a future restore.
It appears there are duplicate entries in the File database for each time there
is a duplicate in the Filename table:
mysql SELECT * FROM File WHERE FilenameId IN (select FilenameId from Filename
WHERE name IN ('0','1','2','tt172.16.17.36-www_gmail_com.html')) ;
+---+---+---+-+++---++
| FileId| FileIndex | JobId | PathId | FilenameId | MarkId | LStat
| MD5|
+---+---+---+-+++---++
| 245079464 |207178 | 13471 |3870 | 9249 | 0 | gB iwKO IGk B
Po Po A Hg BAA I BNTy22 BKRzfV BKR4jB A A E | R13Gb8BoM3TpapoeZW0FVQ |
| 245079465 |207178 | 13471 |3870 | 101382 | 0 | gB iwKO IGk B
Po Po A Hg BAA I BNTy22 BKRzfV BKR4jB A A E | R13Gb8BoM3TpapoeZW0FVQ |
| 215364675 |173643 | 11277 |3870 | 9249 | 0 | gB iwKO IGk B
Po Po A Hg BAA I BM/DvL BKRzfV BKR4jB A A E | R13Gb8BoM3TpapoeZW0FVQ |
| 215364676 |173643 | 11277 |3870 | 101382 | 0 | gB iwKO IGk B
Po Po A Hg BAA I BM/DvL BKRzfV BKR4jB A A E | R13Gb8BoM3TpapoeZW0FVQ |
| 259000887 |290784 | 14315 | 745797 | 9248 | 0 | A A IH/ B A A
A A A A BBg44b 22f3Y BBg44b A A M | UumkJoY8ZKJd6/HfnwvCDg |
| 259000888 |290784 | 14315 | 745797 | 101381 | 0 | A A IH/ B A A
A A A A BBg44b 22f3Y BBg44b A A M | UumkJoY8ZKJd6/HfnwvCDg |
| 258833500 |123399 | 14315 | 807939 | 9248 | 0 | A A IH/ B A A
A A A A BBamBf 22f3Y BBamBf A A M | G4vlZvkmC8DFxl8y0RsnfA |
| 258833501 |123399 | 14315 | 807939 | 101381 | 0 | A A IH/ B A A
A A A A BBamBf 22f3Y BBamBf A A M | G4vlZvkmC8DFxl8y0RsnfA |
| 244900411 | 28129 | 13471 |1752 | 9249 | 0 | gB GgUm IGk B
Po Po A DH BAA I BNTyw4 BJlpIE BJlpIE A A E | ohnd0rdHuoOlhnsfdNGaAw |
| 244900412 | 28129 | 13471 |1752 | 101382 | 0 | gB GgUm IGk B
Po Po A DH BAA I BNTyw4 BJlpIE BJlpIE A A E | ohnd0rdHuoOlhnsfdNGaAw |
| 224916087 | 41066 | 12104 |
Re: [Bacula-users] Migrating form mysql to postgresql: Loading the database takes very long
On Tue, 21 Jun 2011, Marcus Mülbüsch wrote: I want to migrate my bacula installation from mysql to postgresql, following the guide in http://www.bacula.org/5.0.x-manuals/en/main/main/Installing_Configuring_Post.html#SECTION00445 After dumping and converting the database the sql-file now has 9GiB. One note I'd like to make on the page above is that the mysqldump command includes the --skip-opt command. This is not reliable to use if you have a reasonably large File table as it includes an option to buffer query result in RAM before outputting to file. We have 65 million entries and our 6GB RAM + 2GB swap is not enough to hold the buffer, so mysqldump ends up getting killed every time before it finishes. It seems you need to drop --quick which is implied in --skip-opt. The resulting command that I'm working with at the moment is: mysqldump -t -n -c --compatible=postgresql --skip-quote-names --quick \ --lock-tables --add-drop-table --add-locks --create-options --disable-keys \ --extended-insert --set-charset -u bacula -ppassword bacula \ | grep -v INSERT INTO Status\ | sed -e 's/-00-00 00:00:00/1970-01-01 00:00:00/g' \ | sed -e 's/\\0//' bacula-backup.sql` That being said, this is untested so far -- I haven't actually done the migration -- but this is the plan thus far :-) Feedback/corrections welcome... Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] restore of large data set takes extremely long to build dir tree,
On Fri, 24 Jun 2011, ted wrote: I have an issue when trying a restore of a large data set consisting of about 6.5TB and 35,000,000 files the console takes an extremely long time to build the directory tree, over an hr. After the tree is built I typed mark * and this command ran for about 18 hours before I hit ctrl-c; the mySQL server showed no activity and neither bconsole nor bacula-dir was not using any CPU, so I believe the process just petered out. I was wondering if anyone has been successful in restoring large TB data sets. Either ones containing a large number of individual files or ones that are 6TB or larger with small file lists and contain small number of large files? You are not alone. I have one client in particular for whom building the file tree takes about 25-30 minutes. Like you, I'm using MySQL. There have been suggestions from a number of quarters which suggest that Postgresql doesn't have this issue. This may not necessarily be a criticism of MySQL, so much as that the developers tend to spend more time working with Postgresql and therefore Bacula is better optimised toward Postgresql. There is a discussion here: http://old.nabble.com/Re%3A-Tuning-for-large-%28millions-of-files%29-backups--tt30099042.html#a30259098 We plan migrating to Postgresql as that's the only solution I know of right now. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Network speed between FD and SD
Hi, On Fri, 24 Jun 2011, Craig Van Tassle wrote: I'm trying to increase the network speed between my FD and my SD. My Cacti graphs are showing that my network usage averages 60Mb/s and the connections between all the clients are on 1Gb links. I would think I would be able to pull more speed to my SD then just 60Mb/s. Is there anything I can do to try to speed up the network transfer between the FD and SD. I would like to be able to get to near Gigabit speeds to my SD. I'm also running a couple of jobs at once, I would have though that would have helped me spool the data faster. Also I'm running on a Dell sc1425 for my SD and my Clients are Dell r610's. I guess you need to start off by working out where the bottleneck is so you know what to optimise. Possibilities include: - you're using compression or encryption and the client's CPU can't compress/encrypt any faster. You could turn either off, but if you don't compress, you have more data to ship and more to store, so this may be a false economy. If you need encryption, you need it. - the client disk access is slow (could be due to fragmentation, a busy disk or just a slow disk) - the storage medium is slow to write to (doing a local backup from the storage server to itself would rule this out quite quickly) - the network isn't as quick as you think (an iPerf test between client and server would confirm/disprove this) - these are incremental backups which generally tend to give a slower data rate There are probably other possibilities, but these are off the top of my head. Gavin -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] time since last non-virtual backup
Hi, I'm looking to work out a query or script which quotes me the time since the last live backup (ie I don't want to include virtual full backups) for each of our configured jobs. Not all of our backups are scheduled, some are triggered manually, so I need to produce a list of how long it has been in each case. It would be no harm to also check the scheduled ones, just in case there were some configuration error. Has anyone already been over this ground and care to share a query or solution? Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] time since last non-virtual backup
Hi, On Mon, 20 Jun 2011, Gavin McCullagh wrote: I'm looking to work out a query or script which quotes me the time since the last live backup (ie I don't want to include virtual full backups) for each of our configured jobs. Not all of our backups are scheduled, some are triggered manually, so I need to produce a list of how long it has been in each case. It would be no harm to also check the scheduled ones, just in case there were some configuration error. This seems quite close SELECT Job.Name, MAX(Job.RealEndTime) FROM Job WHERE Job.Type='B' AND Job.JobStatus='T' GROUP BY Job.Name; but we have clients who manually trigger their own incremental (eg once per week) and then there's a scheduled VirtualFull which runs every month. These would always appear to have a backup in the past month, even if it was 6-8 weeks since the incremental last ran, as the VirtualFull would appear there. Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Firewall traversal
Hi, On Mon, 20 Jun 2011, Kevin O'Connor wrote: Bacula Server (DIR, SD) - Firewall/NAT - Server to be backed up (FD) The FD is accessible from anywhere, but the DIR/SD is not (NAT/FW). When I start the backup, the Director connects to the FD without a problem, but then when the Director tells the FD to connect back to the SD it fails because of the NAT. I'm in a situation where I can't get the ports forwarded, but it would seem that there should be a way to have the SD connect out to the FD or something along those lines to get this working. Is there a way to do that that I've missed in the docs or is really the only way to get this working is to expose the SD? As far as I understand it, both backups and restores involve a TCP session opening from the FD to the SD -- not the reverse. What you could do is to have software on the SD open up a channel onto the FD with a port forward. There are different tools for this but one is ssh or autossh. sd-host% ssh -L 9103:localhost:9103 fd-host then have the FD connect to localhost instead of the sd-host (in the director configuration). With autossh if the connection dies it'll restart itself. I've done this in one case and it worked pretty well. With SSH you are adding the extra load of encrypting all of the data in transit which might or might not be a problem, depending on your available CPU cycles and the quantities of data you need to ship. The same tunnel should work for a restore. Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Firewall traversal
Hi, On Mon, 20 Jun 2011, Kevin O'Connor wrote: I understand how it's supposed to work (FD to SD), that's why I'm asking if there was some cryptic config option or something I was missing to make it do the reverse. It exists as Active/Passive in FTP, so it's not too crazy to think something similar might exist for Bacula. I guess not though. I'll look into the SSH tunneling, hopefully it won't be too bad! An alternative, if the hosts aren't windows would be to set up IPv6. You could apply to sixxs or one of the other tunnel vendors for a tunnel on each network and then apply for a /48 for inside the NATed network. That should be less CPU-intensive and more natural than ssh tunnels. If you haven't used IPv6 yet this would be a bit of an adventure and take a little research, but then, these days it's probably quite a timely adventure. Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] time since last non-virtual backup
On Mon, 20 Jun 2011, Gavin McCullagh wrote: This seems quite close SELECT Job.Name, MAX(Job.RealEndTime) FROM Job WHERE Job.Type='B' AND Job.JobStatus='T' GROUP BY Job.Name; but we have clients who manually trigger their own incremental (eg once per week) and then there's a scheduled VirtualFull which runs every month. Ah. It would appear that RealEndTime is the actual time the job ended, but StartTime and EndTime (in the case of a virtual full backup) seem to be the corresponding start and end times of the previous backup. That's at least a partial solution, in that I can use the EndTime. That previous backup could also be a virtual full, but I guess I might settle for that. Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula IPv6 status (unofficially)
Hi James, On Sat, 11 Jun 2011, James Harper wrote: Not really directly bacula related, but one of the concerns I have with switching to IPv6 for LAN scale traffic is the performance of the various offload features in the network adapters. Did you do any throughput testing? I haven't yet had time to look at performance in detail -- except to say that none of the backups that have run since seem noticeably slower. That being said, it's hard to tell with incremental backups and some of our full backups have other bottlenecks. I'm guessing support for IP Checksum Offload, TSO et al will be driver dependant so it may be quite messy to work out which the details of what cards/kernels/OSes do and don't make full use of these in IPv6. The tool to use for testing this is probably iPerf. I must do some investigation myself. Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula IPv6 status (unofficially)
Hi, On Sat, 11 Jun 2011, Kevin Keane wrote: The two problems I had: - The default setting for bacula was wrong. It would only listen on IPv4 but not IPv6 unless you explicitly added a DIRAddresses/FDAddresses/SDAddresses section to the respective config files. This is still true, so we put in explicit IPv6 = hostname entries. - The Windows FD could not be configured to listen on IPv6 at all. I hadn't got to the point of a test on Windows yet, but this does seem to still be a problem alright. If you put in an ipv6 fdaddress entry, the bacula-fd fails to start, complaining about that line. So I guess bacula-fd for Windows doesn't support IPv6 yet. Both issues caused problems if you specified clients or SDs by DNS name rather than IP, and there was an record in DNS. Bacula would use the record and try to connect over IPv6, but that connection timed out and the backup failed. For Linux boxes, the workaround was to explicitly add the IPv6 addresses to the configuration files. For the Windows machines, I had to create a second DNS name for the same machine that only had an A record but not an record. On linux, you need to explicitly put in an ipv6 address before each daemon will listen on IPv6. On Windows, my experience is that the backup doesn't fail if a record exists for the client, there's just a bit of a delay before things get up and running as the initial IPv6 connection times out. Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bacula IPv6 status (unofficially)
Hi,
just a short note to say that I've been testing Bacula's IPv6 support of
late and have generally found it to be good.
We have:
- consoles connecting to the director over IPv6
- director talking to SD and FD over IPv6
- FD talking to SD over IPv6
As you might expect, if you configure Bacula to connect to a FQDN and
there's no record, you just get an IPv4 connection. If there is a
record available, and a suitable IPv6 route, Bacula will generally try
to connect over IPv6 first and then if it gets a TCP reset (the far end
isn't listening on IPv6) or a timeout (maybe a firewall blocking the
connection), Bacula retries using IPv4 and life proceeds as normal.
A couple of things worth noting:
== Daemon Address Config ==
Thus far, we've found the best thing to do is to use the multiple address
configs with an explicit IPv4 and IPv6 record, eg (for the FD):
FDAddresses = {
ipv4 = { addr = my.fq.dn; }
ipv6 = { addr = my.fq.dn; }
}
If you just use ip = {}, I've found that it only binds to the IPv4 address,
you need to explicitly have an ipv6 entry. The same applies to the
director and storage daemon. I'm open to better suggestions.
I think Bacula usually listens on 0.0.0.0 (all IPv4 addresses) by default,
but doesn't listen on :: (all IPv6 addresses). I'm not sure if this is by
design, but I guess perhaps it's arguably sensible to only listen on IPv6
where it's explicitly enabled for now. The main point is that it can be
enabled.
With IPv6 addresses, if you use SLAAC (stateless address auto-config), the
host's IP address is based on its MAC address, so it doesn't change. If
the host uses privacy addresses the address may change. On Windows 7, as I
understand it there are two addresses, one is initially created randomly
when you plug into that network but it stays constant on that network (this
is the one you expect to receive connections on and which usually goes in
the DNS). A second address is for outgoing connections and changes at each
reboot (or every 24 hours on Windows Vista apparently).
== TCP wrappers ==
In my experience at least, if you use TCP Wrappers on Linux, you need to
enter the IPv6 address of the Bacula daemons which will be connecting.
Although a FQDN will allow an IPv4 host in, you seem to need the IPv6
address. At a guess, this probably has more to do with tcp wrappers than
Bacula though.
This is one situation where Bacula will fail over IPv6 and not revert to
IPv4 which might still work. This is because it's not a connection
failure, it's an authentication failure, so that makes some sense.
Gavin
--
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula IPv6 status (unofficially)
On Fri, 10 Jun 2011, Gavin McCullagh wrote: just a short note to say that I've been testing Bacula's IPv6 support of late and have generally found it to be good. PS: well done to all the developers involved :-) Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Database performance issues
Hi, On Wed, 08 Jun 2011, Jérôme Blion wrote: What tool do you use to perform restore ? I had such issues with BAT... With Webacula, I am not able to reproduce this behaviour. Perhaps a bad query which does not use an index. This conversation is not original. We had a discussion about it here: http://www.adsm.org/lists/html/Bacula-users/2010-10/msg00130.html and we pin-pointed the exact query which was taking the time: http://www.adsm.org/lists/html/Bacula-users/2010-11/msg00187.html Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Database performance issues
Hi, On Wed, 08 Jun 2011, Phil Stracchino wrote: The very first thing I would do would be upgrade to MySQL 5.5.[current] (5.5.13, right now) if you're not already using 5.5, making sure it's properly configured (hint: look at the new configuration directive innodb_buffer_pool_instances), then throw as much RAM as possible at the InnoDB buffer pool and convert all of the tables to InnoDB. Then download MySQltuner (http://mysqltuner.com/mysqltuner.pl) and look at its recommendations for some basic tuning. MyISAM, frankly, *SHOULD* be deprecated at this point. There is still a lot of FUD about InnoDB performance out there, most of it from people who don't actually understand the performance implications of the differences between MyISAM and InnoDB, but the truth is there is virtually no use case on a conventional MySQL server[1] for which What primary storage engine should I be using? has any answer other than InnoDB. It's probably not too inaccurate to say that unless you *NEED* either merge tables or full-text indices, you should be using InnoDB. For simplicity of operation and patching, we're using the Ubuntu archive packages which are MySQL 5.1.41. I realise that's quite old now. There are one or two restores in particular which take a long time (like 30 minutes) to build the restore tree. I'm guessing the reason is these tables: -rw-rw 1 mysql mysql 7.4G 2011-06-08 13:24 File.MYD -rw-rw 1 mysql mysql 5.1G 2011-06-08 13:24 File.MYI -rw-rw 1 mysql mysql 233M 2011-06-08 13:24 Filename.MYI -rw-rw 1 mysql mysql 161M 2011-06-08 13:24 Filename.MYD -rw-rw 1 mysql mysql 153M 2011-06-08 13:24 Path.MYI -rw-rw 1 mysql mysql 99M 2011-06-08 13:24 Path.MYD Addressing this with a move to PostgreSQL has been on my list but I might try a move to InnoDB first as it's likely much simpler. Assuming that version of MySQL, do you know if the case for InnoDB vs MyISAM is still as cut and dry? Would we likely see substantial performance improvements? Thanks for any help, Gavin -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Database performance issues
Hi, On Wed, 08 Jun 2011, Jérôme Blion wrote: Assuming that version of MySQL, do you know if the case for InnoDB vs MyISAM is still as cut and dry? Would we likely see substantial performance improvements? You will see performance improvements if you have lot of concurrents updates. Which Bacula version do you have ? Perhaps it's an index issue. Bacula package for Ubuntu v5.0.1-1ubuntu1 Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Database performance issues
On Wed, 08 Jun 2011, Jérôme Blion wrote: You will see performance improvements if you have lot of concurrents updates. I don't imagine concurrent updates are really an issue for us. Our backups run fast enough generally for our purposes . I daresay they could be faster, but they're not causing us a problem. It's the time for a restore to build the file tree that's a problem. The particularly bad restore in question is a Cyrus IMAP server with about 5 million files in a full backup. The worst case would be a monthly full, a weekly differential and 6 days of incrementals to assemble. Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Database performance issues
Hi, On Wed, 08 Jun 2011, Jérôme Blion wrote: What tool do you use to perform restore ? bconsole: restore 5 (Select the most recent backup for a client) choose host then building directory tree takes ages. I had such issues with BAT... With Webacula, I am not able to reproduce this behaviour. Perhaps a bad query which does not use an index. Sounds plausible. Does your issue occur with bconsole? Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Exclude = yes, or not?
On Wed, 01 Jun 2011, Mauro Colorio wrote:
what's wrong?
FileSet {
Name = my Set
Include {
Options {
Ignore Case = yes;
}
File = E:/Shared/
}
Include {
Options {
Exclude = yes;
Ignore Case = yes;
}
File = E:/Shared/foo
File = E:/Shared/bar
}
}
the job using this fileset still tries to backup folders and file
under E:/Shared/foo,
any hint?
Should the stanza it's in not be an Exclude not an Include?
Exclude {
Options {
Ignore Case = yes;
}
File = E:/Shared/foo
File = E:/Shared/bar
}
Gavin
--
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] severity of warnings issued by Bacula
Hi, On Thu, 05 May 2011, Jeremy Maes wrote: All you need to do to suppress those messages is add all the junction points on the given windows system to the exclude list of your filesystem. I appreciate that I can do that, but it seems like having to create a mass of wilddir entries in the fileset is rather ugly, given that all I want to achieve is a simple don't warn me about junction points. Your own fileset is impressive but its also a pretty good example of why I'd really rather not do that. Once you edit your fileset you should be left only with important warnings. Are junction points the only unimportant warnings? Gavin -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] severity of warnings issued by Bacula
Hi, like many people I imagine, we get various warnings from the Bacula daemons, particularly the file daemons. There are some which seem like it would be nice to simply suppress them and some which are severe and I'd actually like more attention drawn to them. To give an example, on a director's laptop, every backup comes with a slew of: 04-May 15:50 yy-fd JobId 14235: c:/Users/Default/SendTo is a junction point or a different filesystem. Will not descend from c:/ into it. 04-May 15:50 yy-fd JobId 14235: c:/Users/Default/Start Menu is a junction point or a different filesystem. Will not descend from c:/ into it. 04-May 15:50 yy-fd JobId 14235: c:/Users/Default/Templates is a junction point or a different filesystem. Will not descend from c:/ into it. 04-May 15:50 yy-fd JobId 14235: c:/Users/Default User is a junction point or a different filesystem. Will not descend from c:/ into it. There are no fewer than 163 of these messages, which makes the resulting backup email very hard to read. These warnings are totally benign and happen on every single backup. It would be great to have a way to suppress them so. By contrast, I recently also noticed a bunch of errors of this form in a Windows server backup: 01-May 05:00 saturnalia-fd JobId 14146: Generate VSS snapshots. Driver=VSS Vista, Drive(s)=E 01-May 05:00 saturnalia-fd JobId 14146: Could not stat e:/g.Avi: ERR=The system cannot find the path specified. 01-May 05:00 saturnalia-fd JobId 14146: Could not stat e:/gx.doc: ERR=The system cannot find the path specified. 01-May 05:00 saturnalia-fd JobId 14146: Could not stat e:/grx.Avi: ERR=The system cannot find the path specified. This, it turns out was a backup seriously failing due to a filesystem issue on that server. Thanks to Bacula reporting it and me actually reading the report email, we got to solve that. I might easily have missed this though. It would be great to have a feature which allowed us to say for specific jobdefaults, jobs or file daemons that warnings or messages matching a certain pattern were harmless, eg. /is a junction point or a different filesystem. Will not descend/ On the other hand, a Could not stat is a message I'd like to have my attention drawn to. It could indicate that a backup has effectively failed. I currently filter all bacula backup reports into a folder if they have a subject header beginning Bacula: Backup OK of . This means that I only get bacula reports in my inbox where the backup fails. If the headers could somehow reflect this more severe error, that would be great. Is there a way to do something like this? It's difficult to catalogue all warnings that might be bad, but one can quite quickly work out what warnings are normal. I guess I'm looking for something like a pedantic mode which gives messages like: Subject: Bacula: Backup OK (Warning!) of . unless the warnings are ones I've already configured it to ignore -- in which case they might even be left out of the body altogether. Sorry to be demanding :-) Gavin -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] incremental backup
Hi, On Thu, 14 Apr 2011, ruslan usifov wrote: I'm new in bacula world so have a question: If i do incremental backup. For example veri big file change only few bytes i it, what bacula do send all file, or only changed part of file? It backs up the whole file each time a single byte or more changes. Gavin -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Pool per client
On Wed, 13 Apr 2011, Pablo Marques wrote:
But I would still have the problem that I need a device tied up backing
up each client. The problem I am facing is that I need to backup lots of
slow clients, and I need to come up with something so I can back them up
all at the _same_ time on one or maybe a few devices, and still have a
Pool per client.
I'm not clear if you're trying to avoid lots of physical devices or lots of
bacula storage device definitions. You could create one Device {} entry
per client in the bacula-sd.conf. These each correspond to a different
directory on some filesystem. You then run each backup to its own file
Device -- these can all happen concurrently.
You should then be able to migrate each one in turn to tape.
Or maybe I've missed something?
Gavin
--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now! http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Pool per client
Hi, On Wed, 13 Apr 2011, Pablo Marques wrote: I guess I could modify bacula-sd an add/remove a file device per client as needed. I am not sure if I can reload bacula-sd.conf without interrupting running backups. My understanding is that you need a restart, which is likely to kill any running backups. When I add a client I have a template client definition with all per client definitions that I need: I replace $CLIENT_NAME $IP_ADDRESS $PORT and generate a new file and then I do a reload on bconsole, and the client is ready to go. The clients, in my application, decide the backup schedule, and they run it from their bconsole client. Each client can only run or restore its own backups. I do something similar actually, but it's not quite as big or dynamic so I manually modify the bacula-sd and don't generally have to worry about backups that are running at the time I restart the bacula-sd. This might seem (or be) crazy, but in principal you could run a bacula-sd for each client with its own bacula-sd instance running on a particular ip:port. That would avoid the issue of restarting bacula-sd. Of course it might be possible to signal bacula-sd to re-read its config, which would be much simpler :-) Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Performance with latent networks
Hi, On Mon, 11 Apr 2011, Peter Hoskin wrote: I'm using bacula to do backups of some remote servers, over the Internet encapsulated in OpenVPN (just to make sure things are encrypted and kept off public address space). The bacula-fd is in Montreal Canada with 100mbit Ethernet. I also have another bacula-fd in Canberra Australia on 100mbit Ethernet. The bacula-director is in Sydney Australia with ADSL2+ at full line sync. The latency to Montreal is about 300ms while the latency to Canberra is about 30ms. The issue, I imagine with transfer rates is between the bacula-fd and bacula-sd. Do we presume the -sd is in Sydney? You don't say what speed the Sydney ADSL2+ link is (though apparently it can manage at least 2.2MByte/sec). Is that 300ms over the VPN? If you run a long ping, is there any noticeable packet loss? The problem I'm encountering. backups from the Montreal box will peak at a transfer rate of 100kb/sec despite my ability to do 2.2mb/sec via http, ftp, ssh, rsync, etc. from the same host. Presumably these tests are between the -fd and -sd hosts along the same VPN. Broadly AIUI, one bulk TCP transfer should be the same as another and the two should be affected by latency in the same way. This suggests to me that there's something else going on. Maybe the bacula-fd host is busy or the bacula-fd itself is doing encryption or compression which is slowing down its send rate? Are these incremental backups or full? Are you using accurate backups (which need data to be sent to the fd from the dir)? Have you checked the disk and cpu load on the bacula fd and sd during these backups? So it appears the problem is network latency. Is there anything I can try to improve backup speeds from Montreal? I may be wrong, but I'm not convinced it's just latency. Gavin -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Feature idea : Advanced status of a current job;
Hi, On Mon, 11 Apr 2011, Hugo Letemplier wrote: I imagine a command like status job jobid= I presume you've looked at status client= It does much of what you want (current job duration, data transferred, rate, num files, current file), but without the predictive information you're looking for (time/data remaining). In principal, if you or bacula ran an estimate beforehand, you could probably work out an estimated time remaining but I don't think that feature is present. Estimates are only available for full backups. *estimate job=CeartgoleorBackups-Job Using Catalog MyCatalog Connecting to Client ceartgoleor-fd at ceartgoleor.:9102 2000 OK estimate files=223,128 bytes=9,724,976,371 Gavin -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
Hi,
On Fri, 01 Apr 2011, Gavin McCullagh wrote:
The above config will makes bacula-fd listen on IPv6 only on my Ubuntu
instance (which makes sense of course). However, if you add the IPv4 in:
FDAddresses = {
ipv4 = { addr = 0.0.0.0; port = 9102; }
ipv6 = { addr = :: }
}
It no longer binds to any IPv6 addresses.
Interestingly also, when I set the above config on by Ubuntu Desktop, the
bacula-fd becomes quite unstable, crashing repeatedly (though not
immediately) with an email to root saying:
--
[Thread debugging using libthread_db enabled]
[New Thread 0xb77fbb70 (LWP 3464)]
0x0070c416 in __kernel_vsyscall ()
$1 = 1918985571
$2 = 135180360
$3 = 135180400
/etc/bacula/scripts/btraceback.gdb:4: Error in sourced command file:
No symbol catalog_db in current context.
--
Once I removed the IPv6 element, all seems stable again.
Gavin
--
Xperia(TM) PLAY
It's a major breakthrough. An authentic gaming
smartphone on the nation's most reliable network.
And it wants your games.
http://p.sf.net/sfu/verizon-sfdev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
I've reported a bug based on this conversation, if anyone would like to add anything to it http://bugs.bacula.org/view.php?id=1719 Gavin -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] RFC on how best to share pools, storage devices etc.
Hi, On Fri, 01 Apr 2011, Gavin McCullagh wrote: When I was starting out, I came across a post somewhere on the lists that said it was a good idea with disk volumes to create a separate storage device for each client as it would avoid concurrency issues, etc. I went a little further with this and created multiple pools (full, inc, diff) for each client, a fileset for each client, a schedule for each client, etc. . I have a bunch of laptops to back up now and I'm thinking maybe I should try and be more disciplined for these and create a single storage device, pool set, jobdefs, schedule and (default) fileset for all. This will allow me to delegate creation of new jobs more easily, as the config will be smaller for junior staff. I will only be able to see the total shared volume sizes and mtimes which is a downside. Are there any other disadvantages? Is this a good idea or should I just keep going how I have been? Should I try to do the same on our servers to reduce the config down? Many thanks in advance for any suggestions, So I guess nobody has any opinion on whether it's better to create lots of storage devices (directories), pools, etc. for each backup or just share one between them? Gavin -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
Hi, On Fri, 01 Apr 2011, Gavin McCullagh wrote: Ah, I just assumed this problem was with Windows only. I never thought to check a linux host. I can confirm this on Ubuntu with packaged bacula-fd 5.0.2. It looks like the same problem applies to bacula-sd and bacula-dir. Gavin -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] RFC on how best to share pools, storage devices etc.
Hi, we have bacula in use with about 35 different FDs now. It works brilliantly for us¹. Our routine for each server is based on the example automated backup routine described in the manual. We're not using tapes which some people may frown on a little, we're using disk-based volumes. The convenience of disks compared to tapes is something we value -- though I guess I would like to introduce tapes at some point for offline archival. When I was starting out, I came across a post somewhere on the lists that said it was a good idea with disk volumes to create a separate storage device for each client as it would avoid concurrency issues, etc. I went a little further with this and created multiple pools (full, inc, diff) for each client, a fileset for each client, a schedule for each client, etc. Probably overkill and I now have over a hundred pools, thirty-something storage devices, schedules, filesets etc. I keep each host's config in a single file and include it into bacula-dir.conf so it's quite easy to navigate but I suspect this isn't how it was meant to be. It's really handy for me to be able to look at each directory on disk and see the backups for that server, see how big they are, mtimes on the volumes, etc. Is this likely to present a problem at some point? Should I urgently mend my ways? I have a bunch of laptops to back up now and I'm thinking maybe I should try and be more disciplined for these and create a single storage device, pool set, jobdefs, schedule and (default) fileset for all. This will allow me to delegate creation of new jobs more easily, as the config will be smaller for junior staff. I will only be able to see the total shared volume sizes and mtimes which is a downside. Are there any other disadvantages? Is this a good idea or should I just keep going how I have been? Should I try to do the same on our servers to reduce the config down? Many thanks in advance for any suggestions, Gavin ¹ Kern: if you're ever in Dublin (Ireland), we owe you a meal. -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
Hi,
thanks for the follow-up.
On Fri, 01 Apr 2011, Matija Nalis wrote:
Indeed, not specifying either FDAddress or FDAddresses should listen on
all available addresses. In the Linux client, it does. For the 3.x
Well, on my Debianu GNU/Linux, bacula-fd 5.0.2, it does not.
Ah, I just assumed this problem was with Windows only. I never thought to
check a linux host. I can confirm this on Ubuntu with packaged bacula-fd
5.0.2.
In absence of specified ipv6 addr in FDAddresses, it will listen only on
0.0.0.0 (all IPv4 addresses).
I must use:
FDAddresses = {
ipv6 = { addr = :: }
}
in order to make it listen to all IPv4 and IPv6 addresses (which is also
probably what OP wanted to do, and what probably should be bacula default
(which it currently isn't) if no FDAddress/FDAddresses are specified.
The above config will makes bacula-fd listen on IPv6 only on my Ubuntu
instance (which makes sense of course). However, if you add the IPv4 in:
FDAddresses = {
ipv4 = { addr = 0.0.0.0; port = 9102; }
ipv6 = { addr = :: }
}
It no longer binds to any IPv6 addresses.
Gavin
--
Gavin McCullagh
Senior System Administrator
IT Services
Griffith College
South Circular Road
Dublin 8
Ireland
Tel: +353 1 4163365
http://www.gcd.ie
http://www.gcd.ie/brochure.pdf
http://www.gcd.ie/opendays
http://www.gcd.ie/ebrochure
This E-mail is from Griffith College.
The E-mail and any files transmitted with it are confidential and may be
privileged and are intended solely for the use of the individual or entity
to whom they are addressed. If you are not the addressee you are prohibited
from disclosing its content, copying it or distributing it otherwise than to
the addressee. If you have received this e-mail in error, please immediately
notify the sender by replying to this e-mail and delete the e-mail from your
computer.
Bellerophon Ltd, trades as Griffith College (registered in Ireland No.
60469) with its registered address as Griffith College Campus, South
Circular Road, Dublin 8, Ireland.
--
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself;
WebMatrix provides all the features you need to develop and
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
Hi,
On Mon, 14 Mar 2011, Josh Fisher wrote:
On 3/12/2011 5:15 AM, Gavin McCullagh wrote:
On Fri, 11 Mar 2011, Joseph L. Casale wrote:
The bacula-fd daemon (according to netstat -na) doesn't appear to be
listening on the IPv6 address.
Force it to listen on whatever address/port you desire w/ FDAddresses =
http://www.bacula.org/5.0.x-manuals/en/main/main/Client_File_daemon_Configur.html
Thanks for the reply. Is this considered a bug? Admittedly not so many
networks use IPv6 yet so I can imagine it might not be very high priority.
It seems you have a host name that resolves to both an IPv4 address and
an IPv6 address, no? So when DNS returns two addresses for that host
name, which one should be used? Should bacula-fd listen on the first
address returned, the second address returned, or both?
I would guess both? That's how dual-stacking is generally done so that
IPv6 capable hosts can use the record but hosts which don't support
IPv6 yet can still connect using the A record.
In the normal way, I haven't tended to specify the name or address for the
FD to listen on. bacula-fd just binds to the available configured
addresses (the v4 ones at any rate). Is that not what people expect?
FDAddress is retained for backward compatibility. I believe you should
use FDAddresses for IPv6 and/or multi-homed machines.
Surely defining the address at all is optional, no?
Try using something like:
FDAddresses {
ip = {
addr = hostname
}
}
The ip keyword of FDAddresses allows selection of IPv4 and IPv6 addresses.
An alternative solution that will work with both Bacula, and any other
software that may not be as configurable as Bacula, is to assign
different host names for the IPv6 network than are assigned for the IPv4
network. That removes the ambiguity, although I still think you will
have to use the FDAddresses directive as opposed to the FDAddress
directive. The older FDAddress directive may not support IPv6.
I'd been using FDAddresses alright, but would prefer to just have bacula-fd
bind to the available addresses. Is that not a good idea? I guess the
above hostname-based directive isn't too bad.
There is a potential race condition of sorts in that where you use router
advertisements, a host may not immediately pick up its IPv6 address at boot
time so when the bacula-fd service is started, the IPv6 address might not
yet be configured. That said, I restarted bacula-fd with the IPv6 address
configured and bacula-fd still didn't bind to it.
Gavin
--
Gavin McCullagh
Senior System Administrator
IT Services
Griffith College
South Circular Road
Dublin 8
Ireland
Tel: +353 1 4163365
http://www.gcd.ie
http://www.gcd.ie/brochure.pdf
http://www.gcd.ie/opendays
http://www.gcd.ie/ebrochure
This E-mail is from Griffith College.
The E-mail and any files transmitted with it are confidential and may be
privileged and are intended solely for the use of the individual or entity
to whom they are addressed. If you are not the addressee you are prohibited
from disclosing its content, copying it or distributing it otherwise than to
the addressee. If you have received this e-mail in error, please immediately
notify the sender by replying to this e-mail and delete the e-mail from your
computer.
Bellerophon Ltd, trades as Griffith College (registered in Ireland No.
60469) with its registered address as Griffith College Campus, South
Circular Road, Dublin 8, Ireland.
--
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
Hi, On Fri, 11 Mar 2011, Joseph L. Casale wrote: The bacula-fd daemon (according to netstat -na) doesn't appear to be listening on the IPv6 address. Force it to listen on whatever address/port you desire w/ FDAddresses = http://www.bacula.org/5.0.x-manuals/en/main/main/Client_File_daemon_Configur.html Thanks for the reply. Is this considered a bug? Admittedly not so many networks use IPv6 yet so I can imagine it might not be very high priority. Hard coding the IP is not an ideal solution. This is a laptop running DHCP without a reservation so while the IPv6 address probably won't change, the v4 address will. I can give this laptop a reservation, but we're heading in the direction of setting up bacula backups for 20-30 laptops, so I'd prefer not to give them all reservations. Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays http://www.gcd.ie/ebrochure This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
Hi, I have a Windows 7 computer running the bacula-fd v5.0.3. The computer has an IPv6 address (as does the bacula director and storage daemon. The bacula-fd daemon (according to netstat -na) doesn't appear to be listening on the IPv6 address. This doesn't kill us in that eventually things time out and try IPv4, but it leads to annoying delays, particularly if you're just running status client=. Have other people seen this? I can't see a way to explicitly turn on ipv6 listening. Thanks in advance, Gavin -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Slow backup even with a dedicated line
On Mon, 10 Jan 2011, Oliver Hoffmann wrote: I did some tests with different gzip levels and with no compression at all. It makes a difference but not as expected. Without compression I still have a rate of only 11346.1 KB/s. Anything else I should try? Are you sure the cross-over connection is operating at 1Gbps? Are you sure that route interface is being used? It just seems coincidental that you're still being capped to almost exactly 100Mbps. Gavin -- Gaining the trust of online customers is vital for the success of any company that requires sensitive data to be transmitted over the Web. Learn how to best implement a security strategy that keeps consumers' information secure and instills the confidence they need to proceed with transactions. http://p.sf.net/sfu/oracle-sfdevnl ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Feature request
On Sun, 12 Dec 2010, Guillaume Valdenaire wrote: Here is attached a feature request for that wonderful Bacula. Thanks in advance Item 1:Implement a functionality that permits to log which files were restored during a restore job (especially when using the Bweb interface). I'd certainly add my +1 to this. Gavin -- Oracle to DB2 Conversion Guide: Learn learn about native support for PL/SQL, new data types, scalar functions, improved concurrency, built-in packages, OCI, SQL*Plus, data movement tools, best practices and more. http://p.sf.net/sfu/oracle-sfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula not compressing some filetypes
Hi, On Sun, 28 Nov 2010, bopfi68 wrote: i have backup jobs for a mashine who have a lot of jpeg , zip and i dont want to compress them so is it possible to configure some filetypes for not compressing zip, jpg ... in a the same fileset ? This prior discussion is pretty relevant... http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg36469.html Gavin -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tuning for large (millions of files) backups?
Hi, On Fri, 12 Nov 2010, Bob Hetzel wrote: I'm starting to think the issue might be linked to some kernels or linux distros. I have two bacula servers here. One system is a year and a half old (12 GB RAM), has with a File table having approx 40 million File records. That system has had the slowness issue (building the directory tree on restores took about an hour) running first Ubuntu 9.04 or 9.10 and now RedHat 6 beta. The kernel currently is at 2.6.32-44.1.el6.x86_64. I haven't tried downgrading, instead I tweaked the source code to use the old 3.0.3 query and recompiled--I don't use Base jobs or Accurate backups so that's safe for me. The other system is 4 yrs or so old, with less memory (8GB), slower cpus, slower hard drives, etc., and in fairness only 35 million File records. This one builds the directory tree in approx 10 seconds, but is running Centos 5.5. The kernel currently is at 2.6.18-194.11.3.el5. That's an interesting thought. It would be interesting to make an exact comparison, something like: - run a restore with the slow query log on, capture the query text - run the query manually in mysql - dump the mysql database - restore the mysql database on the older server - run the query there It sounds like your database is quite large so this might be too awkward in practice? Strictly speaking the freshly sequentially written database might have a slight unfair advantage, but if the results are radically different then that would be useful to know. Gavin -- Centralized Desktop Delivery: Dell and VMware Reference Architecture Simplifying enterprise desktop deployment and management using Dell EqualLogic storage and VMware View: A highly scalable, end-to-end client virtualization framework. Read more! http://p.sf.net/sfu/dell-eql-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tuning for large (millions of files) backups?
Hi, On Fri, 12 Nov 2010, Mikael Fridh wrote: On Thu, Nov 11, 2010 at 3:47 PM, Gavin McCullagh gavin.mccull...@gcd.ie wrote: # Time: 10 14:24:49 # u...@host: bacula[bacula] @ localhost [] # Query_time: 1139.657646 Lock_time: 0.000471 Rows_sent: 4263403 Rows_examined: 50351037 SET timestamp=1289485489; SELECT Path.Path, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 FROM ( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat, MD5 FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId FROM ( SELECT JobTDate, PathId, FilenameId FROM File JOIN Job USING (JobId) WHERE File.JobId IN (9944,9950,9973,9996) UNION ALL SELECT JobTDate, PathId, FilenameId FROM BaseFiles JOIN File USING (FileId) JOIN Job ON (BaseJobId = Job.JobId) WHERE BaseFiles.JobId IN (9944,9950,9973,9996) ) AS tmp GROUP BY PathId, FilenameId ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (9944,9950,9973,9996)) OR Job.JobId IN (9944,9950,9973,9996)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND T1.PathId = File.PathId AND T1.FilenameId = File.FilenameId ) AS Temp JOIN Filename ON (Filename.FilenameId = Temp.FilenameId) JOIN Path ON (Path.PathId = Temp.PathId) WHERE FileIndex 0 ORDER BY Temp.JobId, FileIndex ASC; Could you please do an EXPLAIN on this query? I prefixed the query by the word EXPLAIN and ran it: mysql source bacularestorequery.sql +++++-++-+-+-+-+ | id | select_type| table | type | possible_keys | key| key_len | ref | rows| Extra | +++++-++-+-+-+-+ | 1 | PRIMARY| derived2 | ALL| NULL | NULL | NULL| NULL| 4277605 | Using where; Using filesort | | 1 | PRIMARY| Filename | eq_ref | PRIMARY | PRIMARY| 4 | Temp.FilenameId | 1 | | | 1 | PRIMARY| Path | eq_ref | PRIMARY | PRIMARY| 4 | Temp.PathId | 1 | | | 2 | DERIVED| derived3 | ALL| NULL | NULL | NULL| NULL| 4277605 | | | 2 | DERIVED| File | ref| PathId,FilenameId,JobId,jobid_index | FilenameId | 8 | T1.FilenameId,T1.PathId | 4 | Using where | | 2 | DERIVED| Job| eq_ref | PRIMARY | PRIMARY| 4 | bacula.File.JobId | 1 | Using where | | 6 | DEPENDENT SUBQUERY | NULL | NULL | NULL | NULL | NULL| NULL|NULL | no matching row in const table | | 3 | DERIVED| derived4 | ALL| NULL | NULL | NULL| NULL| 4302683 | Using temporary; Using filesort | | 4 | DERIVED| Job| range | PRIMARY | PRIMARY| 4 | NULL| 4 | Using where | | 4 | DERIVED| File | ref| JobId,jobid_index | JobId | 4 | bacula.Job.JobId| 41816 | Using index | | 5 | UNION | NULL | NULL | NULL | NULL | NULL| NULL|NULL | no matching row in const table | | NULL | UNION RESULT | union4,5 | ALL| NULL | NULL | NULL| NULL|NULL | | +++++-++-+-+-+-+ 12 rows in set (16 min 15.79 sec) I presume that's what you're looking for? Tuning's not going to get any of those 50 million traversed rows disappear. Only a differently optimized query plan will. Well, if the above helps and/or if you'd like me to run an alternative proposed query I'm happy to. I must confess it would take me quite a few hours to actually understand that query. Gavin -- Centralized Desktop Delivery: Dell and VMware Reference Architecture
Re: [Bacula-users] Tuning for large (millions of files) backups?
Hi, On Thu, 11 Nov 2010, Alan Brown wrote: What tuning (if any) have you performed on your my.cnf and how much memory do you have? Thus far I haven't spent much time on this and haven't tuned MySQL. The slow build an annoyance, but not a killer so I've not really got around to it. The server has 6GB RAM (running an X86_64 kernel). Have I done something wrong? As FileId is a primary key, it doesn't seem like I should need an extra index on that one -- is that wrong? It doesn't need an extra index. Grand. You've also got a duplicate pathid indeax which can be deleted. Ah, I didn't spot that, thanks. This kind of thing is why it makes more sense to switch to postgres when mysql databases get large. I see. Well, as long as I'm not missing some simple tweak to make MySQL run quicker I guess I'll plan to do that. Gavin -- Gavin McCullagh Senior System Administrator IT Services Griffith College South Circular Road Dublin 8 Ireland Tel: +353 1 4163365 http://www.gcd.ie http://www.gcd.ie/brochure.pdf http://www.gcd.ie/opendays This E-mail is from Griffith College. The E-mail and any files transmitted with it are confidential and may be privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the addressee you are prohibited from disclosing its content, copying it or distributing it otherwise than to the addressee. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the e-mail from your computer. Bellerophon Ltd, trades as Griffith College (registered in Ireland No. 60469) with its registered address as Griffith College Campus, South Circular Road, Dublin 8, Ireland. -- Centralized Desktop Delivery: Dell and VMware Reference Architecture Simplifying enterprise desktop deployment and management using Dell EqualLogic storage and VMware View: A highly scalable, end-to-end client virtualization framework. Read more! http://p.sf.net/sfu/dell-eql-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tuning for large (millions of files) backups?
On Mon, 08 Nov 2010, Gavin McCullagh wrote: We seem to have the correct indexes on the file table. I've run optimize table and it still takes 14 minutes to build the tree on one of our bigger clients. We have 51 million entries in the file table. I thought I should give some mroe concrete information: I don't suppose this is news to anyone but here's the mysql slow query log to correspond: # Time: 10 14:24:49 # u...@host: bacula[bacula] @ localhost [] # Query_time: 1139.657646 Lock_time: 0.000471 Rows_sent: 4263403 Rows_examined: 50351037 SET timestamp=1289485489; SELECT Path.Path, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 FROM ( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat, MD5 FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId FROM ( SELECT JobTDate, PathId, FilenameId FROM File JOIN Job USING (JobId) WHERE File.JobId IN (9944,9950,9973,9996) UNION ALL SELECT JobTDate, PathId, FilenameId FROM BaseFiles JOIN File USING (FileId) JOIN Job ON(BaseJobId = Job.JobId) WHERE BaseFiles.JobId IN (9944,9950,9973,9996) ) AS tmp GROUP BY PathId, FilenameId ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (9944,9950,9973,9996)) OR Job.JobId IN (9944,9950,9973,9996)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND T1.PathId = File.PathId AND T1.FilenameId = File.FilenameId ) AS Temp JOIN Filename ON (Filename.FilenameId = Temp.FilenameId) JOIN Path ON (Path.PathId = Temp.PathId) WHERE FileIndex 0 ORDER BY Temp.JobId, FileIndex ASC; I've spent some time with the mysqltuner.pl script but to no avail thus far. There's 6GB RAM so it suggests a key buffer size of 4GB which I've set at 4.1GB. This is an Ubuntu Linux server running MySQL v5.1.41. The mysql data is on an MD software RAID 1 array on 7200rpm SATA disks. The tables are MyISAM (which I had understood to be quicker than innodb in low concurrency situations?). The tuner script is suggesting I should disable innodb as we're not using it which I will do though I wouldn't guess that will make a massive difference. There are no fragmented tables currently. Gavin -- Centralized Desktop Delivery: Dell and VMware Reference Architecture Simplifying enterprise desktop deployment and management using Dell EqualLogic storage and VMware View: A highly scalable, end-to-end client virtualization framework. Read more! http://p.sf.net/sfu/dell-eql-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tuning for large (millions of files) backups?
On Tue, 09 Nov 2010, Alan Brown wrote: and it still takes 14 minutes to build the tree on one of our bigger clients. We have 51 million entries in the file table. Add individual indexes for Fileid, Jobid and Pathid Postgres will work with the combined index for individual table queries, but mysql won't. The following are the indexes on the file table: mysql SHOW INDEXES FROM File; +---++--+--+-+---+-+--++--++-+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +---++--+--+-+---+-+--++--++-+ | File | 0 | PRIMARY |1 | FileId | A | 55861148 | NULL | NULL | | BTREE | | | File | 1 | PathId |1 | PathId | A | 735015 | NULL | NULL | | BTREE | | | File | 1 | FilenameId |1 | FilenameId | A | 2539143 | NULL | NULL | | BTREE | | | File | 1 | FilenameId |2 | PathId | A | 13965287 | NULL | NULL | | BTREE | | | File | 1 | JobId|1 | JobId | A | 1324 | NULL | NULL | | BTREE | | | File | 1 | JobId|2 | PathId | A | 2940060 | NULL | NULL | | BTREE | | | File | 1 | JobId|3 | FilenameId | A | 55861148 | NULL | NULL | | BTREE | | | File | 1 | jobid_index |1 | JobId | A | 1324 | NULL | NULL | | BTREE | | | File | 1 | pathid_index |1 | PathId | A | 735015 | NULL | NULL | | BTREE | | +---++--+--+-+---+-+--++--++-+ I added the last two per your instructions. Building the tree took about 14 minutes without these indexes and takes about 17-18 minutes having added them. Have I done something wrong? As FileId is a primary key, it doesn't seem like I should need an extra index on that one -- is that wrong? Thanks Gavin -- The Next 800 Companies to Lead America's Growth: New Video Whitepaper David G. Thomson, author of the best-selling book Blueprint to a Billion shares his insights and actions to help propel your business during the next growth cycle. Listen Now! http://p.sf.net/sfu/SAP-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tuning for large (millions of files) backups?
On Mon, 08 Nov 2010, Alan Brown wrote: Mysql works well - if tuned, but tuning is a major undertaking when things get large/busy and may take several iterations. Some time back there was an issue with Bacula (v5?) which seemed to come down to a particular query associated (I think) with restores taking a very long time with large datasets on MySQL, but taking a reasonable time on Postgresql. From what I read of the conversation, there didn't seem to be any tuning solution on MySQL. The answer from several people was switch to Postgres. We're using MySQL for Bacula right now but for this reason I've had it in mind to move. Is this still the case or is there a solution now to those MySQL issues. When we do restores, building the tree takes a considerable time now. I haven't had a lot of time to look at it, but suspected it might be down to this issue. Gavin -- The Next 800 Companies to Lead America's Growth: New Video Whitepaper David G. Thomson, author of the best-selling book Blueprint to a Billion shares his insights and actions to help propel your business during the next growth cycle. Listen Now! http://p.sf.net/sfu/SAP-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Tuning for large (millions of files) backups?
Hi Alan, On Mon, 08 Nov 2010, Alan Brown wrote: When we do restores, building the tree takes a considerable time now. I haven't had a lot of time to look at it, but suspected it might be down to this issue. That's a classic symptom of not having the right indexes on the File table. This is in the FAQ somewhere and it's the issue you mentioned as affecting mysql but not postgresql (Their index handling is slightly different). Right you are http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog There is still an element of move to postgresql though Moving from MySQL to PostgreSQL should make it work much better due to different (more optimized) queries and different SQL engine. http://bugs.bacula.org/view.php?id=1472http://bugs.bacula.org/view.php?id=1472 We seem to have the correct indexes on the file table. I've run optimize table and it still takes 14 minutes to build the tree on one of our bigger clients. We have 51 million entries in the file table. The other big contributor to long tree builds is insufficient ram. Keep an eye on swap and the bacula-dir process whilst building the tree. If it starts thrashing, you need more memory. I think we should be okay on that score but it's something to watch out for. Thanks, Gavin -- The Next 800 Companies to Lead America's Growth: New Video Whitepaper David G. Thomson, author of the best-selling book Blueprint to a Billion shares his insights and actions to help propel your business during the next growth cycle. Listen Now! http://p.sf.net/sfu/SAP-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What happen if I delete a single incremental between the full and another incremental
On Mon, 18 Oct 2010, Graham Keeling wrote: On Sat, Oct 16, 2010 at 09:33:13AM +0200, Hugo Letemplier wrote: Hi thanks a lot for your answers I have retried with a new test scenario its clear now and deleting an incremental is really dangerous. But I think that a function that enable the administrator to join 2 jobs would be cool. Imagine that one day lots of data manipulation are done on the machine that I want to backup, so there is a great difference between 2 incremental. The jobs are done, and deleting one job is dangerous for the jobs that follows In this case, that would be great to mix 2 jobs. Its quite complicated to explain I know. Take a look at this little scenario, a classical Full with his incremental jobs : the client is typically a big file server I hope that it can be understood more easily than the previous post ! Perhaps a VirtualFull backup is what you are looking for? Or a differential maybe? Gavin -- Download new Adobe(R) Flash(R) Builder(TM) 4 The new Adobe(R) Flex(R) 4 and Flash(R) Builder(TM) 4 (formerly Flex(R) Builder(TM)) enable the development of rich applications that run across multiple browsers and platforms. Download your free trials today! http://p.sf.net/sfu/adobe-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Maildir and mysql plugin
Hi, On Mon, 23 Aug 2010, Radosław Korzeniewski wrote: 2010/8/23 Marcio Merlone marcio.merl...@a1.ind.br I was wondering... would be nice to have a maildir backup plugin, where one could backup a maildir and have its contents indexed and restorable by sender, subject, date, attachments, etc.. I could try to write one myself, but wouldn't have the required skills and time. Has anyone ever though about this? Is it doable? We backup a cyrus imap server with Bacula. It doesn't use Maildir, but uses an MH-like format which also consists of a directory of files for each mail folder. I imagine such a plugin could probably support Cyrus also. It is a good idea. It should be easy. The problem is, who wants to use it? If it were there and worked for cyrus, I would use it and it would be handy to be able to restore specific emails. Right now, the easiest thing to do is usually to restore the full mailbox, reconstruct it and then pick out the emails which one wants over imap. On a very big mailbox (we have users with 5 emails in one mailbox), that could be a pain. Whether that is sufficient utility to justify its implementation and maintenance I'm not sure. We don't do a lot of these single-email restores. Gavin -- Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to do a remote rollback?
Hi, On Sat, 24 Jul 2010, John Drescher wrote: On Sat, Jul 24, 2010 at 4:10 PM, Mister IT Guru misteritg...@gmx.com wrote: What would be a method of restoring a remote linux based server to it last full backup, (or even it's first!). Is such a move possible? I was thinking of a general approach, one that can be run on any linux box - my end result would be something like, run a restore job on the target client, and have it reboot, and it's now how it was when I took it's first backup. It seems a full environment may be needed. A custom livecd with bacula on it and some predefined setup should go pretty far on this. You could probably do something quite nice with PXE booting either. If you are familiar with the LTSP, Clonezilla and (in particular) FOG projects, they do similar things in certain respects. FOG is a network-based cloning solution. You: 1. Set all of your desktops to PXE boot. 2. Set up your DHCP server to point PXE clients at your FOG server. 3. On boot, the FOG server then gets to decide whether each client boots the local hard disk as usual, or goes into some other mode such as uploading an image of itself to the server or downloading a new image to its hard disk (either unicast or multicast). You control FOG itself using a web interface, though that's not necessarily a requirement. It seems plausible that you could have a similar PXE set up with a bacula restore mode. Most of the time the server just boots as normal, unless you tell the PXE boot server that this particular computer should go into Bacula restore mode. It then boots a small linux OS which includes bacula-fd and perhaps bconsole. A script could then format the partitions, trigger a full restore, set up grub and reboot onto the restored image. I'm not saying this would be a small piece of work, but a fair bit of it could probably be lifted from FOG. Generalising the detailed restore process across all of your servers might be difficult and across servers in general might be really tough. You have to deal with issues like which partitions exist, which should be formatted/wiped, setting up software RAID partitions, etc. The generalised bare metal restore conversation has come here before. Gavin -- The Palm PDK Hot Apps Program offers developers who use the Plug-In Development Kit to bring their C/C++ apps to Palm for a share of $1 Million in cash or HP Products. Visit us here for more details: http://ad.doubleclick.net/clk;226879339;13503038;l? http://clk.atdmt.com/CRS/go/247765532/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Compress speed
On Thu, 01 Jul 2010, Derek Harkness wrote: I've seen a very significant slow in backup speed by enabling gzip compress, 32MB/s (without gzip) vs 4MB/s (with gzip). The server I'm backing up has lots of CPU 24x2.6ghz so the compression time shouldn't be a huge factor. Is this normal for bacula or is there an optimization I'm missing. As I understand it, the compression is done on the client (ie the file daemon), not the server, so it's the CPU speed of the client you need to consider, not that of the server. Is the CPU maxed out on the client during the backup? Gavin -- This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Compress speed
Hi, On Thu, 01 Jul 2010, Derek Harkness wrote: Sorry I miss spoke in the original post. I'm backing up a server which has 24x2.6ghz cpus and is barely using any of them. Sorry, on reflection, you were quite clear. I misread :-) Gavin -- This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Installing Bacula 5.0.2 in Ubuntu 10.04 with MySQL 5.1
Hi, On Sat, 26 Jun 2010, Graham Sparks wrote: 26-Jun 18:01 bacula-dir JobId 0: Fatal error: Could not open Catalogue MyCatalog, database bacula. 26-Jun 18:01 bacula-dir JobId 0: Fatal error: mysql.c:194 Unable to connect to MySQL server. Database=bacula User=bacula MySQL connect failed either server not running or your authorisation is incorrect. 26-Jun 18:01 bacula-dir ERROR TERMINATION I'd suggest (if you haven't already) that you lift the exact username, password, host and database name from the config file and try a mysql -u username -p -h hostname databasename give the password at the prompt and make sure . It could be, for example, that the mysql database is listening on the eth0 interface but not the loopback or vice versa. By default, I presume bacula connects on the localhost so if that's not specified you should probably use 127.0.0.1. Gavin -- This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] verifying bacula mysql database schema post upgrade
Hi, I wonder is there a simple way to verify that Bacula's MySQL tables are as expected by Bacula. I was running Ubuntu Hardy on our main backup server but needed v3 (primarily for VSS). I took a cut of the Debian maintainer's git archive and created and installed packages for v3.0.2. Now that the new LTS release, Ubuntu Lucid, is available with Bacula v5 available, I decided to upgrade. The dbconfig-common script gave a complaint during upgrade: mysql said: ERROR 1091 (42000) at line 3: Can't DROP 'Copy'; check that column/key exists So I wasn't sure if the script ran cleanly to upgrade the database tables. ERROR 1091 (42000) at line 3: Can't DROP 'Copy'; check that column/key exists ERROR 1060 (42S21) at line 4: Duplicate column name 'HasCache' ERROR 1060 (42S21) at line 5: Duplicate column name 'Reviewed' ERROR 1060 (42S21) at line 6: Duplicate column name 'Comment' ERROR 1060 (42S21) at line 7: Duplicate column name 'HasCache' ERROR 1060 (42S21) at line 8: Duplicate column name 'Reviewed' ERROR 1060 (42S21) at line 9: Duplicate column name 'Comment' ERROR 1060 (42S21) at line 11: Duplicate column name 'Severity' ERROR 1050 (42S01) at line 19: Table 'PathHierarchy' already exists ERROR 1061 (42000) at line 26: Duplicate key name 'pathhierarchy_ppathid' ERROR 1050 (42S01) at line 29: Table 'PathVisibility' already exists ERROR 1061 (42000) at line 37: Duplicate key name 'pathvisibility_jobid' ERROR 1061 (42000) at line 40: Duplicate key name 'basefiles_jobid_idx' Update of Bacula MySQL tables succeeded. At a guess, these indicate that the script has already run. I'm just keen to verify that everything is now as it should be, rather than find out when I start getting strange errors and problems with Bacula. Is there some straightforward way to check the database schema? Gavin -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bare metal windows server 2003 restore
Hi, On Sat, 12 Jun 2010, James Harper wrote: You really need a windows live CD (eg bartpe) or else you won't get all your NTFS ACL's and other stuff restored properly. Also, certain versions of mkntfs are broken wrt making a partition bootable. That's a real shame. Knoppix et al are so much more flexible than BartPE. It would be really handy if one could restore Windows servers in that way. I wrote down the steps as I did the restore from a knoppix cd and a nice, short succinct list of commands gets you to an NTFS filesystem with all the files restored. Gavin -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bare metal windows server 2003 restore
Hi, On Mon, 14 Jun 2010, Bob Hetzel wrote: I've never been able to get the bare-metal restore to work doing a restore starting from a Live CD. I last tried it over a year ago and people responded a while later saying they got it to work that way and they would update a web page with said info but it appears never to have happened. This is frustrating alright. I've read similar claims but when you go to try and do it or ask how it's done, nobody seems able to answer. Off the topic of bacula for a second, what did you mean by PS MS SQL Server v5 is involved here. ? Did you mean MS SQL 2005 or MySql 5? It is MS SQL, version 2005 -- sorry not sure where I picked up v5. MySQL does not support VSS. MS SQL does (but only in the 2005 and later version I presume) Phew :-) Though we do have sql server doing its own scheduled backups overnight when things are quiet. Thanks, Gavin -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bare metal windows server 2003 restore
Hi, we have a windows server 2003 server here and realised that its disk setup is in such a bad way that we want to reinstall it. Never having done one, we thought it would be nice to try a bare metal restore of the machine from the backups (to spare disks). Both c:\ and d:\ drives are entirely backed up by Bacula using VSS. I was expecting to: 1. Put a linux live cd in the server and boot it. 2. Partition the disk(s) appropriately. Format them appropriately (NTFS). 3. Start a bacula-fd in linux. 4. Tell the bacula-dir to restore that server entirely through the running bacula-fd (probably need to do c:\ and d:\ separately). 5. Restore the MBR somehow (windows recovery cd maybe?) 6. Cross my fingers and reboot. However, when I looked at the wiki, I found this article which seems a little more complex. http://wiki.bacula.org/doku.php?id=windows_bare_metal_recovery:ntbackup Are my steps [1-5] extremely naive? Would that not work? Do I have to go the way the wiki page says? I thought I recalled someone suggesting that [1-6] should work. Many thanks in advance for any info, Gavin PS MS SQL Server v5 is involved here. Should having VSS mean that's okay to just restore directly? We do have database backups if need be, but it would be nice if that wasn't needed. -- ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backups on Nas
Hi, On Thu, 13 May 2010, mario parreño wrote: But I prefer not mounting a unit, I prefer acceding to the nas directly, because I have many folders of different accounts in the nas and then I will have that to mount in Debian so many folders since has in the nas. Bacula's backups (as far as I'm aware), always run between the bacula file daemon (client) and a bacula (storage daemon). Restores do the reverse. If you can't install the bacula file daemon on the machine you wish to back up, you'll need to install it on another computer which mounts the share you want backed up. A cursory glance at the ReadyNAS 1100 suggests it runs Linux which means you may be able to install the bacula file daemon on it. This page in the ReadyNas community suggests that work may already be happening to do that. http://www.readynas.com/forum/viewtopic.php?f=47t=34844 These guys seem to have managed to install a bacula-sd on a readynas: http://www.scriptbits.net/2009/07/review-readynas-1100-nas-appliance/ Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Consoles and ACLs
Hi, we have a number of servers (about 30) being backed up by Bacula now. This is working reasonably well. We use disk-based volumes and a scheme based on the example in the Automated Disk Backup chapter of the manual. For each backup we create a directory on the filesystem and a storage device. We then create pools, a job, schedule etc. This does mean certain things are a little unwieldy -- we have an unreasonable number (100) of pools. But broadly, things are fairly manageable. I've recently set up a backup which we want restricted access to. Most of our staff are allowed access to all backups for the purpose of restore, etc. This one is an exception. I've created a separate console for each user and some of us will have *all* access to jobs, pools, commands, etc. Ideally, I need to create a console for some users with something like JobACL = *all* *except* RestrictedJob ClientACL = *all* *except* RestrictedClient StorageACL = *all* *except* RestrictedStorage ScheduleACL = *all* *except* RestrictedSchedule PoolACL = *all* *except* RestrictedPool FileSetACL = *all* *except* RestrictedFileSet but I'm guessing this is not possible. I tried to list all of the other jobs, clients, etc. but I got errors (which I will post if need be). What would be nice is if one could create a group of jobs and assign an ACL against the group. I don't see a way to do this though. Am I doing something wrong here? Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Windows 2008/2008r2 Server backup
Hi, On Wed, 12 May 2010, Kevin Keane wrote: Because Windows Backup goes down to the sector or block level, it can back up basically anything that is on your hard disk - Exchange, SQL Server, virtual machines, registries, active directory, junction points, case-sensitive files, files with multiple data streams, and all those other pesky things that needed special handling in NTBackup. It can also back up only a few changed blocks from right in the middle of a large file. So I guess the question in my mind is, how does this differ from Bacula, supposing you back up C:\, D:\, etc. using VSS? - It apparently can back up a patch to a changed file which Bacula doesn't currently. - It sounds like the restore method is more streamlined. In Bacula (I believe) you need to manually set up disk partitions, format NTFS and set up a bacula-fd, then restore onto the fresh NTFS partitions and update the MBR. This is tedious and a bit complex. Is there something that Bacula can't do here? Is the Bacula way likely to go wrong in some way? I'd just prefer not to have to deal with multiple different backup methods. One of the great joys of Bacula is that we now have a single backup system which we can teach any sysadmin to use and he knows how to restore files, regardless of the platform. Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula slow, possibly network
Hi, On Tue, 11 May 2010, martinofmoscow wrote: I kicked off a [400Gb, full] bacup at 1am on Saturday and it completed 11 hours later at mid-day. At the risk of getting the sums wrong and looking silly: 400GB in 11 hours ~ 36GB per hour ~ 600MB per minute ~ 10MB per second ~ 82Mbit/sec There's some rough approximation there but it seems like you're not far from saturating that 100Mb link for the full 11 hours. Your bacula reports should include a transfer speed. What speed is reported (it's usually in Mbyte/sec)? Do you have a record of how long Amanda took? Interestingly, I just noticed that the Bacula dir was able to reel off a backup of the fd on localhost of about 750Mb in under 10 seconds, That's about 8MB per second which is a little slower than what you see above. The fd might not be able to write the data as quickly. Conversely, a differential of 6Mb from a client on the local LAN just took a few minutes to complete. That's slow in terms of network bitrate, but with a differential or incremental backup, the bottleneck is usually in the time to find what files need to be transferred, not in the time to transfer the files. Another possibility is spooling. I've set up a spool directory on the director/sd but have never seen it used. Relatedly, the tape drive during backups is very stop/start, with a lot of pauses. I'd say without an upgrade to GigE, you're not likely to massively improve the rate you have. It's unclear at the moment if the bottleneck is the network or Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula slow, possibly network
On Tue, 11 May 2010, martinofmoscow wrote: Anyone have any experience with using compression on the client with Bacula? Yeah, we use it almost everywhere. The only exception is a video store where the files aren't terribly compressible. As you might imagine, it uses considerable CPU cycles on the client but it also reduces the bandwidth required. Assuming the CPU can keep up, it may well relieve the network bottleneck. Gavin McCullagh-2 wrote: Interestingly, I just noticed that the Bacula dir was able to reel off a backup of the fd on localhost of about 750Mb in under 10 seconds, That's about 8MB per second which is a little slower than what you see above. The fd might not be able to write the data as quickly. Although I'm not sure I agree with this! More like 80Mb per second. More like what I'd expect. Ah, sorry yes. I was confusing myself between Mb (usually Megabit) and MB (usually Megabyte). I see what you mean now, the above is 75MB/sec to the localhost. Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Feature idea feeler - bconsole include / grep
On Thu, 06 May 2010, Steve Polyack wrote: These are certainly good points. My thought is just that instead of breaking out of bconsole to perform these tasks can be cumbersome. Personally, I feel that it's something that I'd use a lot, simply to prevent me from constantly breaking in and out of the interactive bconsole session. However, perhaps a solution would be to have bconsole parse the pipe character and feed output into the command that follows which could be any arbitrary *nix command (as opposed to writing/including our own version of grep). For comparison, the Postgresql and MySQL command line interfaces include something similar to this. MySQL offers the pager command which can be set to less or grep , etc. as well as the tee command to also send output to a file. Postgresql offers a single: \o [FILE] send all query results to file or |pipe Neither allow anything as simple as SELECT ... ; | grep blah though. Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VSS and Windows
On Thu, 06 May 2010, Carlo Filippetto wrote: Hi all, I have a problem with VSS, I receive this: Warning: VSS was not initialized properly. VSS support is disabled. ERR=Overlapped I/O operation is in progress. One cause of VSS problems like this is running 32-bit Bacula-fd on a 64-bit Windows install. Might that be the case here? Gavin -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup Strategy for Laptops
Hi, On Fri, 02 Apr 2010, Avarca, Anthony wrote: I'm using bacula to backup desktop and laptop clients. The desktops work well with a schedule, but laptops are another story. Does anyone have a strategy to backup laptops? Is it possible to have the user trigger a backup? It's not the prettiest thing in the world, but we get the user to trigger a backup using bconsole. Firstly, we allow console connections from the client computer to the director, strictly only for the resources which are relevant to that client. When the user decides to run a backup, 1. Start bconsole 2. Type runreturn 3. Type exitreturn 4. The messages are emailed to the user so they know when the job is finished. the default backup triggered in our case is always incremental and accurate. We then run virtual full backups on a schedule. This scheme has solved a long-term problem for us. It's a laptop for which a full backup takes over 5 hours. The user simply can't wait that long. The incremental by contrast takes about 1 hour, which is workable and he can work on it during the backup, albeit a bit slow. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup Strategy for Laptops
Hi, On Fri, 02 Apr 2010, Avi Rozen wrote: Gavin McCullagh wrote: 1. Start bconsole 2. Type runreturn 3. Type exitreturn 4. The messages are emailed to the user so they know when the job is finished. Assuming the laptop is running a Debian based Linux distro: can't this be automated by running a backup script from /etc/networks/interfaces using the post-up directive, or by placing such a script under /etc/networks/post-up.d ? If it were a debian-based distro, perhaps. However, this laptop runs Windows. I daresay you could script something similar in Windows too though. The other disadvantage of that scheme is that I don't really want a backup to trigger every time the user appears on the network (and a backup is due). I want them to be able to decide I'm happy to plug in and leave the laptop on the network for an hour now and I accept it will run a little slowly during the time. In our particular case, if backups ran automatically, we'd have failed backups because the laptop was unplugged midway through. We'd also have complaints that the laptop was running slow at times. Of course, differently people will have different requirements here. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Virtual Backup feature
Hi, On Thu, 01 Apr 2010, XZed wrote: http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/backuppc-21/full-backup-built-from-incrementals-104809/ As it keeps unanswered, i just wanted to have confirmation that Virtual Backup feature of Bacula, is the right one i'm searching to. That's pretty much a description of Bacula's virtual full backup alright. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Dir/SD on Mac or Windows?
On Tue, 23 Mar 2010, Bruce McCarthy wrote: Basically I would like to get the server-side apps (Director Storage Daemon) running on either Windows or Mac OS X. There are binaries available for Windows up to 3.0.2 but everything I've read leads me to shy away from this unsupported implementation. On the Mac side, bacula.org recommends Fink. Their latest package is at 2.x. DarwinPorts has a package for 3.0.3, and MacPorts has one with version 5.0.0. This is a pretty common situation. For the sake of argument, if you were to use a linux distribution such as Ubuntu Hardy, you'd be getting 2.2.8. If you chose ubuntu Karmic (the latest) you'd get 2.4.4. Debian Lenny is 2.4 also. Personally, I don't get that hung up on being at the very major release -- though I do try to be on the latest bug-fixed minor release. We needed to have Win64 support with VSS, so I compiled debian packages for 3.0.2 and our windows clients are all v3.0.3a. At some point we'll move to v5, but I'm not that much of a rush. Hopefully Ubuntu/Debian will catch up with our needs shortly (debian squeeze and ubuntu lucid both have v5) so hopefully I'll be able to delegate the Bacula compilation back to the package archive shortly and let them maintain testing, security fixes, etc. It may mean we're a little late getting v5.1 or v6, but that's okay. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Enhancements Request
Hi, On Tue, 16 Mar 2010, Bob Cousins wrote: - Data staged to disk on the way to tape -- allowing backups to spool faster than tapes and when tape drives are full. I'm pretty sure this already exists. http://www.bacula.org/en/dev-manual/Data_Spooling.html - Explicit capturing of boot and partitioning information for easier bare metal restores. (Bare metal restores should be easy and fast for all OSes, but that is easier said than done.) Do you mean backing up the MBR? That might be nice alright. And a couple of really low priority but nice to have ideas: - Optional system shutdown after backup -- ideal for desktop machines which would otherwise be idle for hours. Can you not just use the hook to run a command after backup with whatever shutdown command the computer uses? Admittedly I guess the FD could do it, but it's not massively difficult. - API for database backups -- backing up MySQL's files while live is bad. Mysqldump output is big. We need a 'better way.' One way is using a snapshot, where the system in question supports it. For example, if you have your MySQL database on an LVM partition you can stop (or quiesce) mysql, snapshot, start mysql, mount the snapshot and begin backing up the data files off the snapshot. ZFS offers similar: http://blogs.digitar.com/jjww/2008/01/snapback-the-joys-of-backing-up-mysql-with-zfs/ Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to use another system disk partition as backup storage?
Hi, On Mar 11, 2010, at 5:12, vishesh bacula-fo...@backupcentral.com wrote: I am new to bacula and implemented it on my RHEl server 5.2. Everything is working great and now i want to use my another linux system disk partition as backup storage. Should i use NFS or similar network protocol or any other way? I presume you mean a disk in another system. Mounting over nfs means all data must be sent to your storage daemon and then sent on to your nfs server. It would probably make more sense to install a bacula storage daemon on the second machine which would allow backups to go direct. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] what strategy for backing up files only once?
Hi, On Fri, 26 Feb 2010, Ralf Gross wrote: I'm still thinking if it would be possible to use bacula for backing up xxx TB of data, instead of a more expensive solution with LAN-less backups and snapshots. Problem is the time window and bandwith. VirtualFull backups be a partial solution to your problem. We have a laptop which we get very short backup time windows for -- never enough time to run a full backup. Instead, we run incrementals (takes about 20% of the time) and then run virtualfull backups to consolidate them. We never need to run real full backups. If there would be something like a incremental forever feature in bacula the problem could be solved. I know the Accurate Backup feature but without deduplication (don't backup/store the same file more than once) it's possible that the space needed for backups will grow (user moves or renames a directory with 10 TB of data...) Accurate Backup will detect this and back up the data again (instead of just pointing to the new location inside the database). Deduplication of this sort is not available as far as I know. I suggested a feature some time back which might help. I'm not sure if there's enough interest in it though. You can see this in the bacula projects file: http://bacula.git.sourceforge.net/git/gitweb.cgi?p=bacula/bacula;a=blob_plain;f=bacula/projects;hb=HEAD Item n: Implement a Migration job type that will create a reverse incremental (or decremental) backup from two existing full backups. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] what strategy for backing up files only once?
On Sat, 27 Feb 2010, Gavin McCullagh wrote: ` VirtualFull backups be a partial solution to your problem. We have a laptop which we get very short backup time windows for -- never enough time to run a full backup. Instead, we run incrementals (takes about 20% of the time) and then run virtualfull backups to consolidate them. We never need to run real full backups. ... apart from the first time of course. Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Using two disks/problems using vchanger, what could be a good strategy?
Hi, On Wed, 24 Feb 2010, Administrator wrote: Is there any other solution, e.g. could i close a volume using a script and then change the disk? Would bacula create silently a new volume on the new disk if the last used volume was marked full before the disk is removed? Could you just use Maximum Volume Jobs = 1 and let it create/recycle a new volume each time? Then set your retention and maximum number of volumes in the pool appropriately? Gavin -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore Dir/Files recursively
Hi, On Wed, 10 Feb 2010, Ken Barclay wrote: $ add /public/share/120 SALES DIVISION/*.* No files marked. $ add /public/share/120 SALES DIVISION/ No files marked. The command prompt you get from the bacula console is a little bit primitive. Off the top of my head, I'd suggest you try mark /public/share/120 SALES DIVISION add /public/share/120 SALES DIVISION Gavin -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore Dir/Files recursively
On Wed, 10 Feb 2010, Ken Barclay wrote: Thanks Gavin, but $ mark /public/share/120 SALES DIVISION No files marked. $ add /public/share/120 SALES DIVISION/ No files marked. Sorry, I wasn't thinking. I'd suggest you do: cd public cd share mark 120 SALES DIVISION Gavin -- SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW http://p.sf.net/sfu/solaris-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Is it possible to configure bacula server to write backup data to another server
Hi, On Mon, 08 Feb 2010, Khalid Pasha wrote: I am new to Bacula, I am planning to install Bacula server on VM machine and storage ( for keeping backup data ) on other server on which we have mapped LUNs from EMC SAN, my question is it possible to configure Bacula in this way, if yes please share procedure of it. There are 4 components to consider (director [aka server], file daemon [aka client], storage daemon and catalog [a SQL database]). Each one can reside on a different machine in a different place. Backups transfer directly between the file daemon and storage daemon, they don't go via the director. http://www.bacula.org/5.0.x-manuals/en/main/main/What_is_Bacula.html#SECTION0022 You may have many storage daemons and there's no need for any of them to run on the same server as the director. Position them where you have suitable storage available. My bacula server is RHEL and clients to backup will be windows machines. That should be fine. The bacula file daemon on Win32 works very well. Be sure to match (or at least not exceed) the file-daemon version with the director version though, eg. a bacula-fd v5 may not work with a bacula-dir v3). Gavin -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] virtual full backups on file-based devices
Hi,
I'd just like to run something by you guys to see am I doing it right.
We have a senior staff member who exclusively works on a laptop and moves
around and travels a lot. A full backup takes 5+ hours due mainly to the
relatively slow disk and large amount of data. That's just not practical.
An incremental takes about an hour, which is workable.
I have created a bacula job for his laptop. We're using file-based volumes
on a reasonably remote server. His assistant starts bconsole and types
run to start an incremental, accurate backup weekly. We schedule a
monthly virtual full to consolidate the incrementals. To facilitate this,
I've created two pools and two devices, all writing to the same directory
on the backup server.
If I'm right, I think I need to alternate the Pool each month in the
schedule, so that the most recent virtual full is used, something like:
Schedule {
Name = X-Schedule
# Create virtual full backups monthly
#Run = VirtualFull Pool=X-Full-Pool 1st sun at 21:00
Run = VirtualFull Pool=X-Full-Pool jan,mar,may,jul,sep,nov 1st sun
at 21:00
Run = VirtualFull Pool=X-Full-Pool-2 feb,apr,jun,aug,oct,dec 1st sun
at 21:00
}
Does that seem about right? Many thanks in advance for any comments,
Gavin
# Default pool definition
Pool {
Name = X-Full-Pool
Pool Type = Backup
Storage = X-Storage-1
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 3 months # one year
Label Format = X-Full-
Maximum Volume Jobs = 1
Maximum Volumes = 3
NextPool = X-Full-Pool-2
}
# Default pool definition
Pool {
Name = X-Full-Pool-2
Pool Type = Backup
Storage = X-Storage-2
Recycle = yes # Bacula can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 3 months # one year
Label Format = X-Full-
Maximum Volume Jobs = 1
Maximum Volumes = 3
NextPool = X-Full-Pool
}
--
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula via NATed connection and Bacula docs
On Wed, 27 Jan 2010, Dirk H. Schulz wrote: Telnetting from external-fd to server-sd using the above mentionened FQDN and the port of the storage daemon (telnet storage.server.sd 9103) outputs exactly the same as telnetting internally to that port. Afaik, that means: bacula-fd on the external client should be able to connect to bacula-sd on the internal server. But it does not. Running a backup job for this client the director is quite a long time waiting for Client ... to connect to Storage ... and eventually gives up. In this instance, I would be inclined to start a tcpdump like that below on both the -fd and -sd, start your backup and see where exactly the -fd tries to connect to. tcpdump -ni ethX tcp port 9103 The first question I suppose is to see what IP address the -fd is actually using to connect. The second is does the tcp handshake happen correctly and if so what happens then. Perhaps the -fd is connecting to the wrong IP, or it could be a firewall issue, or something else...? Gavin -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How it optimize the transfer speed.
Hi, as a relative bacula newbie myself I have a couple of suggestions. On Tue, 26 Jan 2010, Cyril Lavier wrote: Now that my exclude rule works perfectly (thank you guys), I just begin to see a problem. Backups are made on a LAN 100Mbit. But the actual speed of bacula's backup is about 22GB/hour, it's about 50Mbit/second, so it's the half of the actual capacity of the network. The thing you need to establish here I think is where exactly your bottleneck lies. A few tests to narrow things down: - try an iperf test between client and server first, to check what speed the TCP stacks and the network can sustain a TCP connection going at higher than this speed. Depending on your network, an upgrade to GigE may be quite affordable ... or not. - identify how quickly the server can receive and write data to disk/tape. Perhaps a local bacula backup from the SD to itself of some large sequential data might give you a clue (ideally don't read and write from the same disk). - identify how quickly the client can send data. You could try a backup to an SD nearer (even on the client -- though on a different disk). The above may not give absolute certainty but it may be enough to give you a strong indication. I've managed 10.67 M bytes/second (about 85Mb/sec) on 230GB with the client on a 100Mb/sec link -- but I had to change a firewall out to do that as the old firewall couldn't deal with the load. Without the firewall in the way and the same server, I've seen 1Mbyte/sec -- due to the client being too slow (slow disk, fragmented filesystem and slow cpu). Compression (if you're not already using it) might be useful, but it loads the client cpu quite heavily so if the client were the bottleneck, this is unlikely to help much. It would help with either the network or the storage daemon though. If you are already using compression, cpu might be your bottleneck. Now the problem is real, because I have a fileset which is 1.5TB big, and I need to make a full backup once a month (the other backups are incremental ones), and with the transfer speed I have now, the full backup would last for about 3 days. One workaround to this is to use virtualfull backups (which require the accurate backup option). With this method you can use an old full backup and subsequent differential/incremental backups to construct a more recent full backup -- without bothering the server. In principal, this can allow you to avoid running your 1.5TB full backup. Gavin -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] profiling a backup job
Hi, is there any facility of profiling a job in bacula. By that I mean, being able to gather information on the time taken for parts of the backups. I can see a job is taking a long time (on a senior staff member's laptop) and I can look at the status and see that it's spending large amounts of time in certain parts of the disk. However, this is just by running status client=. now and then which is not very accurate. Is there some way to profile a backup to see how long it spends in different parts of the filesystem and tell what's taking so long? We're running incremental backups with the accurate option which may be part of the trouble. Gavin -- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
