Re: [Bacula-users] Slow Backups
Thanks a bundle, Troy, you definitely gave me plenty to start with. I suspect the list guru's probably will want more info as this output doesn't give any real indication of where the problem might lie. I suspect you're right. For starters, I'd provide the output from a completed full job. The output you provided is for an incremental, and indicates that your system took 28 minutes to search your entire server (the size of which we have no idea) to locate just under 30Mb of data that was eligible for backing up. I'm guessing most of this time was taken in the search not the backing up. What rates are you getting for full backups? Hmm...You're probably right about this. I'd noticed that the Full backups got a generally better speed, but I'd never connected it with searching for data to back up. I actually have not run a full backup of any considerable size since I upgraded, so I'll have to get back to you on Sunday for this. The best way to track down bottlenecks is try and narrow the search. Whilst a backup is running use something like 'top' to keep an eye on the director, storage daemon, and database processes and see if any of them seem to take significant CPU hits. Also, check your iowait (It should be shown in the top header lines, probably as 'io%' or similar) - if that's high, it's an indicator your disk writes are the issue. Good advice, I'll take a look at it. Are you backing up multiple jobs at once? This can hammer your disk IO. No. What sort of speeds do you get from backing up the director/storage daemon? If these are faster, you might have networking issues. I'm not doing much backup of the machine with the director/storage daemons; just the catalog. For that, I see speeds around 7.8MB/s whereas for the incremental backups of my file server the speeds range from 10-2000KB/s, depending of the file set. This is probably the search issue you mentioned above. Looking at the previous full backups of the file server (using 2.0.3) I'm seeing speeds of around 6.9MB/s. Is the database server and storage daemon using the same disk at the same time? I don't believe so. I think the database lives on a different disk. Are you using data spooling, or at least attribute spooling. I'm probably wrong, but suspect the new batch insert code only works if the attributes are being spooled first. I've not managed to upgrade yet, and dont have any way to test this theory tho so hopefully someone else on the list can confirm or deny this please. I'm not doing data spooling, and I don't believe I'm doing attribute spooling unless it's turned on by default. And finally, what spec are your servers? Both director and client ones. Oh and Storage and database if they're separate beasts as well. What database server are you using? My server is a Gentoo Linux box with a single 2.8GHz Pentium 4 processor. It's connected to a hardware RAID5 by fiber channel and the network by gigabit ethernet. The storage disk (the RAID is done in hardware) is set up with evms. The client is the same except the disks being backed up are hardware RAID configured as three separate disks through evms. If you provide this information, you have a much better chance of someone on the list being able to help you, or at least of being able to point you in the right direction. Thanks again, Troy, you've been a big help. ~Kyle Marsh - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] problem with connection to remote machine
Hallo Tomasz, sorry, i canot solve the simple problem. I wrote in the iptables three port 9101,9102,9103 on local machine (bacula server) are allowed and 9102 on remote machine (bacula client). But nmap show me that all port are not there. local machine (bacula server) has no static IP HomeLinux:/ # iptables --list Chain INPUT (policy DROP) target prot opt source destination ACCEPT udp -- anywhere anywhereudp dpt:bacula-fd ACCEPT tcp -- anywhere anywheretcp dpt:bacula-fd ACCEPT tcp -- anywhere anywheretcp dpt:bacula-dir ACCEPT tcp -- anywhere anywheretcp dpt:bacula-sd ACCEPT udp -- anywhere anywhereudp dpt:bacula-sd ACCEPT udp -- anywhere anywhereudp dpt:bacula-dir [EMAIL PROTECTED]:~/jboss-4.0.5.GA/server/default/deploy nmap -v HomeLinux Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2007-09-21 09:06 CEST Initiating Connect() Scan against HomeLinux.site (127.0.0.2) [1680 ports] at 09:06 Discovered open port 22/tcp on 127.0.0.2 Discovered open port 3306/tcp on 127.0.0.2 Discovered open port 111/tcp on 127.0.0.2 The Connect() Scan took 0.13s to scan 1680 total ports. Host HomeLinux.site (127.0.0.2) appears to be up ... good. Interesting ports on HomeLinux.site (127.0.0.2): Not shown: 1677 closed ports PORT STATE SERVICE 22/tcp open ssh 111/tcp open rpcbind 3306/tcp open mysql Nmap finished: 1 IP address (1 host up) scanned in 0.245 seconds remote machine (bacula client) has a static IP ponte-net:/home/argonist # iptables --list Chain INPUT (policy DROP) target prot opt source destination ACCEPT tcp -- anywhere anywheretcp dpt:bacula-fd ACCEPT udp -- anywhere anywhereudp dpt:bacula-fd Using nmap from lokal machine to remote machine [EMAIL PROTECTED]:~ nmap -v ponte-net.de Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2007-09-21 09:10 CEST Machine 87.106.25.52 MIGHT actually be listening on probe port 80 DNS resolution of 1 IPs took 0.00s. Initiating Connect() Scan against s15260882.onlinehome-server.info ( 87.106.25.52) [1680 ports] at 09:10 Discovered open port 80/tcp on 87.106.25.52 Discovered open port 443/tcp on 87.106.25.52 Discovered open port 22/tcp on 87.106.25.52 Discovered open port 8443/tcp on 87.106.25.52 The Connect() Scan took 35.40s to scan 1680 total ports. Host s15260882.onlinehome-server.info (87.106.25.52) appears to be up ... good. Interesting ports on s15260882.onlinehome-server.info (87.106.25.52): Not shown: 1672 filtered ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 135/tcp closed msrpc 137/tcp closed netbios-ns 138/tcp closed netbios-dgm 139/tcp closed netbios-ssn 443/tcp open https 8443/tcp open https-alt Nmap finished: 1 IP address (1 host up) scanned in 35.683 seconds Manuel On 9/21/07, tomasz dereszynski [EMAIL PROTECTED] wrote: Manuel Ostendorf wrote: Hi Bruno, I tried with nmap. Right, bacula-fd is not there, that means is closed. I have been tried with iptables -A INPUT -j ACCEPT -p tcp -dport:9102, but it is not working. Is that wrong? its just not all you need you have to set rules for incoming and outgoing connections on both firewalls its quite simple when you know what are you doing. briefly: - you need to allow connection from dir on remote machine with fd to port 9102 guess/hope you dont have rules about outgoing connection then - you need to allow connection from fd on dir/sd machine to port 9101 (dir) and to 9103 (sd) like i said till you dont rule anyhow outgoing connection that will do the job last thing - of course if both servers are in public net (public IPs) Manuel On 9/20/07, *Bruno Friedmann* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Hi Manuel simply try (if you have ) the nmap command nmap -v ip or name of bacula-fd's machine this should show you at least the bacula-fd daemon listen port open if not working : aka no ping you could use -P0 flag. and if no result are shown, this would indicate that some firewall rules block your bacula traffic. Hope this help you a bit ... Manuel Ostendorf wrote: Hello, I want to backup the datas from the remote machine. The bacula is on lokal machine fine, but the bacula server tried to connect bacula-fd on remote machine and the connection is failed. error message: 20-Sep 13:39 HomeLinux-dir: -Console-.2007-09-20_13.35.38 Fatal error: bsock.c:129 Unable to connect to File daemon on ponte-net.de:9102 http://ponte-net.de:9102. ERR=Die Wartezeit für die Verbindung ist abgelaufen local-machine: HomeLinux:/etc/bacula # netstat -l | grep bacula tcp
Re: [Bacula-users] Slow Backups
On 9/20/07, Kyle Marsh [EMAIL PROTECTED] wrote: Howdy, all. I've noticed that my backups are running ridiculously slow and after watching the lists it seems that the solution was to update to 2.2.x, as there is much improved database insertion in the current release. After doing the update on both my client and director, I don't see any notable speed increase. I've included one of the messages which was e-mailed to me when a job finished. Could anyone give me some suggestions? As a result of the database update will only see an improvement if you are backing lots of files. I mean something like 100,000. John - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Compress files on storage daemon and not on client
Hi everyone, I'm just getting used to Bacula and I'm really impressed about it! But one thing bothers me: I'm using the built-in gzip compression. The files to backup are always compressed on the client side so that the client has a lot of cpu usage at that time. As I'm using a dedicated storage server for the director and storage daemon I'd rather like to compress the files on this server. Is this possible with bacula? Matthias - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Compress files on storage daemon and not on client
On 9/21/07, Matthias Kellermann [EMAIL PROTECTED] wrote: Hi everyone, I'm just getting used to Bacula and I'm really impressed about it! But one thing bothers me: I'm using the built-in gzip compression. The files to backup are always compressed on the client side so that the client has a lot of cpu usage at that time. As I'm using a dedicated storage server for the director and storage daemon I'd rather like to compress the files on this server. Is this possible with bacula? Its not yet in bacula but you could disable software compression and store the volume files on a filesystem that does compression. Like FuseCompress, reiser4 or NTFS (on windows). http://miio.net/fusecompress/ John - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] [OT] Postgresql database enconding and heterogeneous clients
Hi to all: I've the bacula catalog under Postgresql and the database was created with UTF8 encoding. Now, trying to protect one 'old' client ( REDHAT 3 ) I get errors from backup job - better said, the errors are raised in the database - claiming for some characters not valid fro UTF8 encoding, example Fatal error: sql_create.c:845 sql_create.c:845 query SELECT FilenameId FROM Filename WHERE Name='Nenúfares.jpg' failed: IIUC, here I have three possible ways taking in mind that the environment to protect could include UTF8 clients, ISO-8859-1, Windows machines -By now I don't know default enconding, etc... 1.- Recreate catalog using SQL_ASCII ( the environment to protect could include UTF8 clients, ISO-8859-1, Windows machines -By now I don't know default enconding, etc... ) 2.- May be, forcing character set in the bacula-fd client processes I get at least a 'buggy chars' but storable in the database ( ? ) ( This can be accomplished in Oracle at the risk of lose this characters if you export/import data ) 3.- Touch database config to force character set from UTF8 to SQL_ASCII, but I don't know how Postgresql stores characters and then, I've no knowledge about implications of this movement. Based in your experience, and if you have get this issue before, what should be the correct way? If you know more ways info is wellcome, of course. Best Regards D. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Backup to disk AND tape
Hi everyone, I want to use Bacula to backup my files to disk AND to tape at the same time. The backup on disk should be available for 3 months or so while the backup on tape should not be overwritten before 12 months. Is this possible with Bacula? How do I accomplish this without the need to run the backup job twice? Matthias - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Unable to mount tape
Hi there, I am getting an error 3909 Error scanning mount command: mount LTO1 as of today. No problems like this until today. I had used 'bextract' on a couple of tapes earlier but i dotn see how that could be connected...? If anyone has any idea how to correct his I would be grateful. -Kris OS: debian 3.1 bacula 1.36.2 - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup to disk AND tape
Hi, 21.09.2007 15:27,, Matthias Kellermann wrote:: Hi everyone, I want to use Bacula to backup my files to disk AND to tape at the same time. The backup on disk should be available for 3 months or so while the backup on tape should not be overwritten before 12 months. Is this possible with Bacula? How do I accomplish this without the need to run the backup job twice? No, that's not possible. There are a number of workarounds possible, but the ideal thing is currently not possible. It's discussed, though, and I think anyone (relevant :-) thinks such athing would be useful. In other words, it will most probably be implemented some day. (If you want to know more, search for the term copy pool and SD mux in this lest and the -devel one.) Before that happens, I assume that job copying will be possible. This is still in the Kern thinks about it-phase, but again, the need and usefulness of such a feature is not seriously disputed. Currently, as far as I know, Kern is not working on these features, though. If you want to see them as soon as possible, you'd have to offer some help developing them, I guess. Arno Matthias - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Arno Lehmann IT-Service Lehmann www.its-lehmann.de - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Backing up to tape -- how many catalog backups to keep?
Hi folks, I recently modified my Bacula config files to stop backing up to DAT tape and start backing up to disk on our new Infrant ReadyNAS device. I essentially followed the advice in the Automated Disk Backup chapter of the manual and created separate pools for full backups, differential backups, incremental backups, and catalog backups. This raised a question for me: how many catalog backups should I keep? My guess is that we don't really need to keep a large number of catalog backups, since frankly they'd be obsolete once the next catalog backup is written. At the current moment I have my catalog pool set with Maximum Volume Jobs = 1 and Maximum Volumes = 7, which (should) limit it to the most seven catalog backup files. Basically, I don't want to use an undue amount of disk space for backup files that might not be particularly useful. I'm curious as to what the general consensus among the Bacula user community is, regarding this subject. I saw a post from Arno from last year saying that he recycles catalog backups quickly by placing them in the same pool as incremental backups, but I thought that perhaps this question could use to be discussed a bit more explicitly. Thanks, Jody - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backing up to tape -- how many catalog backups to keep?
I'm curious as to what the general consensus among the Bacula user community is, regarding this subject. I saw a post from Arno from last year saying that he recycles catalog backups quickly by placing them in the same pool as incremental backups, but I thought that perhaps this question could use to be discussed a bit more explicitly. I will tell you what I do. First I would like to describe my hardware a little. My database, my director and my primary storage are on 3 different 64 bit gentoo linux machines. I backup most of my data to LTO2 tapes (Exabyte Magnum224) on the main storage machine. Each night after the othe backups are completed I backup the backup catalog. For this I use disk volumes on the director machine. These disk volumes have a max vol size of 2GB and there is plenty of space on the director (200GB raid 5 partition for bacula disk volumes). On this array there are currently 8 volumes in the BackupCatalogs Pool and I have the volume retention set to 15 days for the pool. My postgresql database size it is around 12GB as I have 4 million files but after compression the dumps are less than 2GB so I should have between 8 and 12 copies of the catalog stored this way. John - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Mac os X
Ok so iam working on a full DMG file for mac os x with the BAT and bconsole working I got the bconsole working but when I try to compile BAT it says I need qt4 installed well I have qt4 installed how can I tell it where my qt4 is? Its in /opt/qt4-mac/. Thanks Will -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Mac os X
I personally haven't played with this yet, but check the options for ./configure - there may be an option to tell the compiler the path to qt4. There are quite a few other Macophiles lurking in the list that might have more intel for you. Erich On Sep 21, 2007, at 3:36 PM, William McInnis wrote: Ok so iam working on a full DMG file for mac os x with the BAT and bconsole working I got the bconsole working but when I try to compile BAT it says I need qt4 installed well I have qt4 installed how can I tell it where my qt4 is? Its in /opt/qt4-mac/. Thanks Will -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- --- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] can i access remote the user interface
I'm trying to setup a new backup server and one that was recommended is BACULA. The question is can I access the GUI INTERFACE via remote/local network from my WINDOWS XP? I usually setup linux server without GUI. BACULA has a greate user interface but can I access it remotely (e.g. just typing the IP/URL in my browser). Tired of spam? Yahoo! Mail has the best spam protection around http://ph.mail.yahoo.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] can i access remote the user interface
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nelson Serafica wrote: I'm trying to setup a new backup server and one that was recommended is BACULA. The question is can I access the GUI INTERFACE via remote/local network from my WINDOWS XP? I usually setup linux server without GUI. BACULA has a greate user interface but can I access it remotely (e.g. just typing the IP/URL in my browser). You've got a couple of options. You don't say what GUI you're talking about, though. You could install bweb on the server. That is a pretty decent interface. However, if you're talking about 'bat', right now you're limited to compiling it yourself on Windows, or running it via X11 over the network. - -- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer II |$| |__| | | |__/ | \| _| |[EMAIL PROTECTED] - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG9IsMmb+gadEcsb4RArEsAJ9CrZauXhIoXiR1G7MKptOET3usagCZAQVv 6KVpRdSeS5z5FAIdVYouJPM= =kljr -END PGP SIGNATURE- begin:vcard fn:Ryan Novosielski n:Novosielski;Ryan org:UMDNJ;IST/AST adr;dom:MSB C630;;185 South Orange Avenue;Newark;NJ;07103 email;internet:[EMAIL PROTECTED] title:Systems Programmer III tel;work:(973) 972-0922 tel;fax:(973) 972-7412 tel;pager:(866) 20-UMDNJ x-mozilla-html:FALSE version:2.1 end:vcard - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
