Re: [Bacula-users] Deduplication?
In response to Chris Hoogendyk [EMAIL PROTECTED]: Christopher Derr wrote: Greetings, We're thinking of using Bacula as our disk-to-disk solution for backing user and research data. I'm still reading up on it, but I haven't found the answer to the following question. Called pooling in BackupPC and deduplication by industry, I've been trying to find out if Bacula has it. A search of the site for either word brings up nothing relevant. Does the current version of Bacula have the ability to store backups of the same file as one file with links? For example: If Bob and Joan both have the exact same 2 MB PDF in their home directory, a normal backup would store it twice for a total of 4 MB. What deduplication does, is store the file once in a central location, and then store links from the individual backups to the file. If 100 people have this same file, rather than taking up 200 MB of space, it still only takes up 2 MB. Unique, I believe, to disk-to-disk backups. Nope. I'm not aware of any open source backup software that does that. Amanda doesn't do it either. It's non-trivial and has been discussed on the Bacula list a couple of times. Not sure what the key word would be to search for it. BackupPC does it: http://backuppc.sourceforge.net/ But their architecture was designed from the ground up to support it. I expect that what happens is when a file with a duplicate filename is backed up for the first time, a checksum is generated to compare it to files of the same name already in the system. When incrementals are run, if the file is recently modified, the checksums are checked again. I think the first thing that would need to occur for Bacula to do this, is the use of something stronger than MD5. Perhaps SHA256. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deduplication?
In response to Michel Meyers [EMAIL PROTECTED]: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bill Moran wrote: I expect that what happens is when a file with a duplicate filename is backed up for the first time, a checksum is generated to compare it to files of the same name already in the system. When incrementals are run, if the file is recently modified, the checksums are checked again. I think the first thing that would need to occur for Bacula to do this, is the use of something stronger than MD5. Perhaps SHA256. Why would Bacula need to use SHA256? MD5 should be more than sufficient to distinguish 2 different files that happen to have the same name and filesize. I don't have the math background to say one way or another. I mentioned it because I believe I remember reading about rsync using strong hash functions. - From a checksum/hashing standpoint, Bacula should be ready to go. It's the implementation of the duplicate detection and elimination algorythms that requires careful planning and a lot of work to implement everywhere. I think the big thing is that it would require a new protocol between the director and the FD. As I understand it, the director simply tells the fd to back up everything modified after $date. To do incrementals with deduplication, the fd would have to send back filenames with hash values to the director, who would have to look them up to determine whether the file needed to be backed up or not, at which point the director would have to tell the fd what to do. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [OT] Postgres initial database encoding
David Ballester [EMAIL PROTECTED] wrote: Hi to all: Some days before I send a mail to the list asking about advices on problems with postgres database encoding and some bacula-fd in clients with incompatible encoding http://sourceforge.net/mailarchive/message.php?msg_name=1190366372.6121.21.camel%40localhost.localdomain AFAIK, there is no answer. Don't wanna waste your time but this 'silence' is 'cause the answer is very obvious? :) In any way, thanks for all PostgreSQL is extremely particular about SQL encodings, and rejects non-valid strings. If you've found a string that bacula tries to insert that PG considers invalid, then you've either found a bug in PG or in Bacula. (i.e., either bacula is submitting an invalid string, or PostgreSQL is rejecting a valid one) In any event, you _should_ file a bug report. However, you mention old client in your original post, but make no mention of the particular version of Bacula on the director, sd or fd. Is it possible that you're using an older version and this has been fixed in newer versions? -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [OT] Postgres initial database encoding
David Ballester [EMAIL PROTECTED] wrote: 2007/9/26, Bill Moran [EMAIL PROTECTED]: PostgreSQL is extremely particular about SQL encodings, and rejects non-valid strings. If you've found a string that bacula tries to insert that PG considers invalid, then you've either found a bug in PG or in Bacula. (i.e., either bacula is submitting an invalid string, or PostgreSQL is rejecting a valid one) In any event, you _should_ file a bug report. However, you mention old client in your original post, but make no mention of the particular version of Bacula on the director, sd or fd. Is it possible that you're using an older version and this has been fixed in newer versions? -- Bill Moran http://www.potentialtech.com May be I've explained it bad. The 'old' client refers to character set enconding on the system ( today UTF8 is used widely but one host to protect is an old RHLES 3 32bit - Both director and fd has the same version ( last release ) I've no access now to this machine and I can't say what encoding is using the host, but if I'm not wrong, the problem is that the files created in the client host were saved using some character set ( seems that this files were uoploaded vía ftp from windows machines ) not compatible with Postgresql/UTF8 and both bacula-fd and director handles them well, but trying to insert data in the database makes postgresql to reject it, bacula sees some error returned by the database transaction and aborts the backup procedure. I'm not character encoding expert, but my understanding is that UTF8 encompasses all other encodings. I don't think there's any character set out there that should produce strings that aren't valid UTF8. I've been curious as to how Bacula handles this, and it seems as if it does no validation whatsoever, with the result that PG catches the problem. It would seem to me that a reasonable result from Bacula would be to log an error and skip the file, but continue with the backup. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [OT] Postgres initial database encoding
In response to Martin Simmons [EMAIL PROTECTED]: On Wed, 26 Sep 2007 05:38:46 -0400, Bill Moran said: David Ballester [EMAIL PROTECTED] wrote: Hi to all: Some days before I send a mail to the list asking about advices on problems with postgres database encoding and some bacula-fd in clients with incompatible encoding http://sourceforge.net/mailarchive/message.php?msg_name=1190366372.6121.21.camel%40localhost.localdomain AFAIK, there is no answer. Don't wanna waste your time but this 'silence' is 'cause the answer is very obvious? :) In any way, thanks for all PostgreSQL is extremely particular about SQL encodings, and rejects non-valid strings. If you've found a string that bacula tries to insert that PG considers invalid, then you've either found a bug in PG or in Bacula. (i.e., either bacula is submitting an invalid string, or PostgreSQL is rejecting a valid one) In any event, you _should_ file a bug report. However, you mention old client in your original post, but make no mention of the particular version of Bacula on the director, sd or fd. Is it possible that you're using an older version and this has been fixed in newer versions? If this happens when the client machine is running unix, then I don't think it is a bug in Bacula (at worst it is a feature). The problem is that unix filesystems don't know anything about encodings, so can store any sequence of bytes as a filename. Bacula can't do anything about this, so I think it is a mistake to run the database with strict encoding checks. That would qualify as a Bacula bug, as switching to (for example) SQL_ASCII encoding would disable all checks. The question, however, is what is Bacula's intended behaviour. That's what I don't know. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Performance Problem
In response to Rainer Hackel [EMAIL PROTECTED]: Hi all! I have bacula running (version 2.0.2) and in principle everything works = fine. But now (reading some mails from the list) I ask myself why the = backup-speed is that slow. In average it's about 1500 kb/s. The software is running on fedora. The Computer has a fast CPU and 2GB = of RAM. No network backups, just lokal disk. The backup-drive is a lto-1 = hp. What backup-speed coult i expect? How could i find the bottleneck? How fast are your disks/tapes? Are you backing up to tape or disk? Try some dd tests to see how fast you can transfer data raw. Use tar going from disk to tape to see how fast that runs, and/or use dd going from disk to disk. Frequently, in my experience, otherwise fast computers have slow (but reliable) hard drives in them. Since RAM is so cheap, you usually don't notice this until you're moving _lots_ of data around. From there, you have to take into account that the DBMS has to write the catalog records, so you could run some tests to see how fast it can write new records to see if that's slowing you down. In my experience, CPU/memory are usually not the bottleneck when backups are running. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Skip computer when no connexion
Aitor [EMAIL PROTECTED] wrote: I've configured bacula in order to back up the data of 6 clients. Everything is going fine except when one of the computers isn't connected. Then, Bacula tries to connect and if it isn't then it remains waiting. I would like that in the third attempt it skips this computer. Any idea to implement it? Thanks .Aitor. It doesn't wait indefinitely, you just aren't patient enough. Personally, I have found the timeouts unusually long. There are some settings in the director config that help this. Look at the FDConnectTimeout parameter in the director's config. The default is 30 minutes, which I find high. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] NFS or bacula-fd, which one is faster?
In response to Ivan Adzhubey [EMAIL PROTECTED]: Hi, I have a Linux NFS fileserver which has to be backed up to a bacula server on another Linux box. The fileserver in question exports everything that's needed to be backed up so all files are actually accessible on bacula server via NFS as well. Should I run my backups via a remote bacula-fd client on the fileserver or via local client on the bacula box (reading from NFS-mounted tree), which method do you think will work with faster data transfers? I can try both and benchmark them of course but would appreciate if anyone done a similar setup already and can share experience. It's going to depend on where resources are most available. If you run the FD on the NFS server, it will use CPU to do the compression, but will use less network bandwith. If you run the FD on the bacula server and pull the data via NFS, the Bacula server will use all the CPU to compress but more network traffic will be necessary to pull the uncompressed files through NFS. Also, if you use NFS you won't be able to take advantage of things such as filesystem snapshots. Also, depending on your NFS export settings, you may hit permissions problems. So which is best depends on which of those tradeoffs is most important to you. Also, whether or not you actually use software compression will change the balance. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] invalid byte sequence for encoding UTF8
In response to Roland Roberts [EMAIL PROTECTED]: Bill Moran wrote: Connect to the PostgreSQL database using the psql command: psql -U bacula bacula Then enter \l to list the installed databases and their attributes. You should see the bacula database as UTF8. The database is UTF8; all the databases on this host are UTF8. To see what the client encoding is, you can do show client_encoding; from the psql program. If this is not set to UTF8, you _can_ change it on the fly. Simply issue ALTER DATABASE bacula SET client_encoding='UTF8'; Then you'll want to restart Bacula for it to pick up the new setting. Hmm, okay, but it shows UTF8. And shouldn't the bacula client be doing that? Yes, it _should_. I don't know if it _does_. Because I'm going on vacation and don't want to leave home without a backup (especially since the laptop will be used in the field, literally), I ended up just removing the offending packages. I'll try to play around with this again after vacation. Well, based on your response, I don't have an explanation for your problem. Perhaps when you return we can track it down. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] invalid byte sequence for encoding UTF8
*** - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now http://get.splunk.com/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Secunia] Potential Security Issue in Bacula
think under certain conditions it is possible for the user to submit an SQL statement that could trigger this overrun. How he would use it to gain security access, I cannot say. I'm a bit busy right at the moment because we are getting very close to a major release, so unless you can show me this is critical, I would rather not spend too much more time on it. I document everything of importance that I find wrong with Bacula. However, I consider it would be unwise to provide any public documentation on how this might be exploited, if that is in fact possible, as it would only encourage hackers to do damage. What IMO would be much more appropriate is to advise users to upgrade to avoid any potential problems ... -- Sven Krewitt Security Specialist Secunia Hammerensgade 4, 2. floor DK-1267 Copenhagen K Denmark http://secunia.com/ Phone +45 7020 5144 Fax+45 7020 5145 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-beta] [Bacula-devel] Bacula BETA 2.1.26 released toSource Forge
In response to Dan Langille [EMAIL PROTECTED]: On 16 Jul 2007 at 16:30, Dan Langille wrote: On 16 Jul 2007 at 22:19, Francisco Rodrigo Cortinas Ma wrote: - In the fact that you need funding, the right way is to ask for donations. This will remain valid as long as the project remains small. We have asked for donations. In a prevoius email, Kern mentioned this has been about $8000 since 2000. That's about $4 a day. Given the number of users, it's pretty clear that donations don't work. Please ignore my calculations. They are not based on fact. I messed up. My apologies. My point remains: donations are insufficient to fund the project Just in case anyone is keeping score or taking a survey. I think Kern's plan is a good one. My reasons for saying so are: * If you don't want to pay, you can always build your own binaries. Aside from the Win32 binaries, we build most of our own anyway. You can also build your own binaries and redistribute them if you'd like. * The code is still free in both the beer and the speech sense. This means you can build your own binaries and redistribute them if you'd like. Did I already mention that? * I've been trying to move some money toward the Bacula project, but it's been difficult for exactly the reasons Kern mentioned. Having either a registered 501c3 or a incorporated company of some sort would make this easier. So, I think it's a good plan from every angle. Furthermore, I think that anyone who doesn't think it's a good plan either hasn't reviewed it thoroughly, or has some strange axe to grind. As I see it, this will allow big corporations to be more comfortable adopting Bacula. The money coming from the corps will fuel additional development, which will benefit folks who aren't willing to or prefer not to pay. As Kern said, the only losers I see in this are the existing backup software companies who are charging big bucks. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ [EMAIL PROTECTED] Phone: 412-422-3463x4023 - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] FreeBSD and Authochangers
In response to tomasz [EMAIL PROTECTED]: Ken Gunderson wrote: Hello List: I've posted before regarding chio vs mtx-changer and got reply from Arno but I'd love to hear from some FreeBSD users who are using authochangers. So here goes again... Are you using chio or mtx-changer, and why? I've looked at the chio example scripts and seems that chio-bacula simulates barcodes and rc-chio-changer handles real barcodes but that chio-changer ignores them... The latter makes mention of FBSD-5.2 so that puts it a couple years old. rc-chio-changer has a cvs id of 2006-02-23 and looks to me most recent? Or are all of these abandoned in favor of mtx and mtx-changer? i am using freebsd 6.2 with mtx-changer and barcodes and its all fine (the only one problem was with access for bacula to autochenger node in /dev) autochanger is Quantum superloader 3 We're using chio-changer with FreeBSD 6 and a Dell LTO2 drive. Works just dandy, but we're not doing barcodes, so I can't comment on that. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup rate decreasing
In response to Support [EMAIL PROTECTED]: I have Bacula-2.0.3 running on a dual P3 1 GHz linux system with a 2.6.21.1 kernel. I have noticed that the backup rate of a client decreasing during the backup it starts at 3,200 KB/s and slows down to 2,000 KB/s after 30 minutes. I used Status of the Storage daemon to get the rate about every 5 minutes. The backup was 6 GB to disc. I suspect the slow down may be due to the fact other jobs were waiting for this job to finish since I ran another backup of another client and the rate was constant during the whole backup, the difference being there were no jobs queued to run ie waiting for the device. No. Queued jobs do nothing to affect the speed of a backup. Could the slow down be caused by the director polling the system and so impacting on the storage daemon? No. The problem is almost certainly caused by a change in the makeup of the data being backed up. Since status client gives you the speed at the time you run it, that can change over time if any factors change the speed. This could be something like activity on the client contending with Bacula, or network traffic. However, it's most likely (in my experience) that you start out backing up a lot of big files, then the backup moves into directories where a lot of small files are found. 2G composed of 2 1-G files will back up faster than if it's composed of 2000 1-meg files. This is because each file has some overhead associated with writing backup records to the database. If the speed difference is enough to be of concern, you should look at optimizing whatever database backend you're using to speed up inserts. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula memory usage
In response to Joseph Wright [EMAIL PROTECTED]: It seems that the resources used by the database should be taken into account when looking at Bacula's memory usage, because even though the two are separate, the director is the one putting the records there. Maybe I should re-ask the question as: How heavily does Bacula tax the database with 50 or more clients being backed up concurrently, and what kind of hardware is recommended for such a situation? If you've got 50 jobs running simultaneously, you're liable to need a massive DB server. Unfortunately, however, that's not enough information to be sure. The real load on the DB server is (mostly) determined by two factors. 1) The number of files being backed up. 2) The amount of duplication in pathnames and filenames. If each backup job is 1000s of tiny files, you're going to be working the database pretty hard, as each of those files will require that a record be written to the database. If filenames and pathnames change often, that will require that additional database records be written for each file saved. For a file that has a name never used before in a directory never backed up before, you have 3 database records created. On the flip side, if you're filesets consist largely of the same files that are constantly being edited, and your average filesize is large, you'll find that the speed of your storage media will slow you down and the database will have no trouble keeping up. I suspect that's part of the reason that there are no published hardware requirements for Bacula. The hardware requirements aren't really for Bacula, their for your specific workload. Regardless of all that, if you really mean to run 50 jobs in parallel (i.e. at the same time, not just 50 jobs in series) I would recommend that you get a dedicated DB server with about 12 disks arranged in a RAID 10 and a battery-backed cache. If you can run the jobs in series, then just about any reasonably powered server hardware should suffice. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] volume conflict between jobs
In response to Arno Lehmann [EMAIL PROTECTED]:
[snip]
This is how my Pool is defined:
Pool {
Name = dss07-pool
Pool Type = Backup
LabelFormat = dss07-
AutoPrune = yes
Volume Retention = 14 days
Maximum Volume Bytes = 21474836480
Recycle = yes
You probably want to limit the number of volumes in that pool, because
otherwise volumes will not be recycled but Bacula will forever try to
create a new one.
I thought that it would start recycling after the volume retention
period was over.
No, volumes are recycled onle when there is no other legal way to find
new space.
This is not the behaviour I observe. When a volume has been marked Purged,
it is then recycled the next time a volume from that pool is needed.
When a volume's retention time is exceeded, it is purged.
--
Bill Moran
http://www.potentialtech.com
-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] computer drags during backup, win32
Dave [EMAIL PROTECTED] wrote: Hello, I'm running bacula 2.0.3 on windows xpsp2. When doing any kind of backup, i have the entire system set to back up and if i'm using the box it slows down to a crawl. I'm using vss if that matters. On my various Unix boxes this does not happen. I was wondering if there was an equivalent to unix nice that i could set for win32? If not, is there a way of finding out where this slowdown is coming from, a resource limit, network performance, cpu, and how best to deal with it? Thanks. I've seen this problem as well. As best I can tell, Windows job scheduler bites. In my case, I was able to reduce the effect to an acceptable level by using gzip1 for the compression level instead of the default gzip6. Luckily, the extra space required was not a problem. I don't know if there's a feature request out for this, but maybe I should create one. It would be nice if the FD could be programmed with a delay loop during the compression process, configurable by the user. It's difficult to say whether this would actually meet the goal or not, though. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Network Performance
On Mon, 21 May 2007 12:49:12 -0400 Jordan Desroches [EMAIL PROTECTED] wrote: I've done a bunch of testing since my last post, and the slowdown appears to be CPU usage on the client file daemon (bacula-fd). bacula-fd consumes all the cpu resource it can get, and tends to transfers data faster (40 MB/s) on to the director on a machine with better CPU, even though the disk array on the slower machine (28 MB/s) was faster (similar data sets). I'm using MD5 hashes and compression is not turned on (also not explicitly turned off, if there is such a command). Is it normal for the file daemon to consume 100% of a cpu, or am I doing something wrong? Any time I've seen high CPU usage on the FD, it's been the result of compression being turned on and has been solved by either lowering the aggressiveness of compression or turning it off altogether. So my first advice would be to verify whether compression is on or not. Check your job defaults. Is this a POSIX system? Run top(1) and see where the CPU time is going: user? system? io? -- Bill Moran Potential Technologies http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Network Performance
In response to Jordan Desroches [EMAIL PROTECTED]: Compression is not enabled in the FileSet. Can you isolate where the bottleneck is? IO? CPU? Network? There are frequently discussions about throughput being sub-optimal on the lists. In my experience, these fall into a few categories: *) The SQL server is the bottleneck. This kinda falls outside the discussion of Bacula, as it's usually specific to the particular SQL server you use. This problem shows up most often with filesets that contain lots of little files. However, there _has_ been work done to improve how Bacula writes to the database, so which version you're using is important. If you can make any suggestions on how to improve this further, or even provide test cases to show where it's slow, I expect it will be helpful to the developers. *) IO. Often, folks are saturating the IO of their hardware and don't realize it. This can show up when the database is on the same system as file volumes, as that system has to share the IO of database writes as well as file volume writes. *) Mysterious network problems. This is the one that it would be nice to get some real information on. Some folks have claimed that Bacula is unable to send data anywhere near the speed of the network, even when there is no other bottleneck. Unfortunately, this problem has been very difficult to diagnose, as it requires a high level of expertise in networking to track down the cause, and everyone who has report it has been unable and/or unwilling to isolate it well enough for anyone to really do anything to improve it. So, I hope that information is helpful in narrowing down your problem. Michel Meyers wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jordan Desroches wrote: Greetings! First, I apologize if this comes through multiple times. I'm having trouble connecting to the list. I've been trying to bake off AMANDA and Bacula in our environment, and have run up against a Bacula performance snag. Amanda is regularly able to average ~50 MB/s over our network, while I'm getting ~30 MB/s out of Bacula (spooling turned on over gig-e). I like the feature set and usability of Bacula much better than that which AMANDA provides, but the speed difference is an issue. I think the difference may have to do with AMANDA running multiple simultaneous dumpers on the client. I've bumped Maximum Network Buffer Size to 65536 bytes in both the storage daemon and file daemon configurations with little to no change from the 32K buffer. A typical Bacula client status reads: [...] Any ideas what I can do to eek out some more speed? Just a guess/question: Do you have compression enabled in your job? If the client's doing compression, that might throttle its throughput. Greetings, Michel -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) iD8DBQFGSa8b2Vs+MkscAyURAuC+AKDnMf1RGfkBeq6qYmPZzEneCLVZxwCeNJYk YpXNPfBS5fQRAMS/rNEvgcE= =Adyp -END PGP SIGNATURE- - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Goodbye and thanks for all the fish
In response to Jon Pounder [EMAIL PROTECTED]: I've already sent a private email off on this, but on the list I'll leave it at : Kern you should be ashamed of yourself. Well, I can't speak officially for the Bacula project or for Kern, but I think that's hogwash. If you're going to berate someone publicly, you should air the exact nature of your grievances publicly. If you're going to keep the details private, then you should have the good grace to keep your admonishments private as well. To me, what you have done is the equivalent of calling him names. As to this specific incident. I see it all as a misunderstanding. I don't know how interested Kern and Robert are to resolve this, but it seems to me (from watching the public conversation) that it could be resolved if both parties are willing to work it out. If we should be ashamed of our misunderstandings, we'd better all stop communicating, as misunderstandings are inevitable. I have been involved in the Bacula project for a year now. The first 9 months I spent working 6-7 days a week, 8+ hours a day: porting Storage and the Director to Windows rewriting the installer redesigning the Windows build process and contributed improvements to the Unix build fixing numerous Windows-specific as well as general bugs redesigning, simplifying and improving both the Windows and the core code porting the regression tests to Windows migrating the Source Management from CVS to Subversion I followed all the coding conventions, reviewed every significant change with Kern, and did everything possible to comply with all rules (both those outlined in the Developer's Guide as well as those inferred from reading between the lines in Kern's emails). I spent thousands of dollars putting together a test lab for all the tape loader, tape drive and CD changer technologies. During the last eight weeks I've been working on reproducing and fixing a tape drive on Windows bug and writing a new monitor application for Windows. I've watched the email lists for bugs that are specific to the new components I've added as well as helping users with Windows specific install problems. I also worked on a couple of other open source projects with which I'm involved. Oh, and I caught up on some of my work that actually produces income. The one thing I didn't do was update the manual. As a result, my admin privileges for the Bacula project were removed and I must submit all my changes as patches for review. I've done my best to work with Kern and I thought everything was great until the last week when he started threatening me with removing all the software (Windows Server version) I'd just devoted the better part of the last year working on. I'm not sure what I've done to upset him, but I'm not prepared to work in this environment. One of the nice things about Open Source is that if you don't like the rules you can not only take your marbles and go home, you can also take the other guy's marbles too. I'll be looking into providing a supported and compatible version of the software for Windows. Over time as the core code improves and diverges I'll probably also release versions for the other platforms too. I will also make sure that I pick up bug fixes from the Bacula project. - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/___ Bacula-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/bacula-devel Jon Pounder _/_/_/ _/_/ _/ _/_/_/ _/_/ _/_/_/_/ _/_/_/ _/ _/ _/_/_/ _/ _/_/ _/_/ _/_/ _/ _/_/ _/_/ _/ _/_/_/ _/_/ _/_/_/_/ _/_/_/ _/_/ _/_/_/_/ Inline Internet Systems Inc. Thorold, Ontario, Canada Tools to Power Your e-Business Solutions www.inline.net www.ihtml.com www.ihtmlmerchant.com www.opayc.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Bill Moran http://www.potentialtech.com
Re: [Bacula-users] [Bacula-announce] Bacula version 2.2.x for Solaris, FreeBSD, and Windows
On Sun, Apr 15, 2007 at 11:24:58AM +0200, Kern Sibbald wrote: This email is a *BIG* urgent warning that unless the Bacula Solaris and FreeBSD users pull together and organize regression testing, you may find that version 2.2.x will not work on your platform, and it may be difficult to correct problems after production release in a timely fashion since once released, it is not possible to make any significant changes to mutex usage without destabilizing the code. What's your timeline, Kern. I'll do my damndest to get through all the FreeBSD regression tests with enough time for bug fixes to go through. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] What does this error message mean?
In response to hal [EMAIL PROTECTED]: 25-Apr 08:41 abe-sd: Job RestoreFiles.2007-04-24-08.41.05 waiting to reserve a device. What does it mean? It means that no device suitable for that restore job is available. What device? The previous messages should supply context. What machine, server or client? Again -- previous log messages. Why? I believe this is caused by another job using the device, and this job has to wait on it. Connect to the director and do status dir and status storage. Between those two output, you should be able to ascertain what's going on. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] High database load when migrating many small files
In response to Damian Lubosch [EMAIL PROTECTED]: Hello! I am using Bacula 2.0.3 with Mysql5 on Debian Etch with a 2.8GHz Celeron, 1.5GB RAM, LTO2 drive, 2x 500G (Backup) + 1x80 Disks (System/DB). I have a machine to backup with about 1-2 million small files (~1kb). When I run a migration job for about 4 GB of such data the performance is going down. The tape rewinds very often and the overall performance is about 3MB/sec. I found out (with top) that the mysql is taking all the processing power (together with bacula-sd) when migrating many files. On the other hand, when migrating only few large files -backups is running fine with 20-30MB/sec. How can I improve the performance? Are there any tricks I oversaw? This sounds like a database optimization problem to me. When Bacula is dealing with a lot of small files, performance generally bottlenecks at the datbase's ability to write new records, and everything else is held up waiting for the database (thus your tape drive runs out of data to write and has to shoe-shine). I'm not a MySQL expert, so I can't give any DB-specific advice, but you should look at what can be done to improve MySQL's performance. Is MySQL actually blocking waiting for free CPU, or is it waiting on disk IO, for example. I'm making a lot of guesses here, but what kind of HDD subsystem do you have? You may have to invest in faster (i.e. SCSI or SAS) drives to get enough IO throughput. But isolate the problem first. If MySQL is slow, it's slow doing _something_. Find out what that something is and you'll have a good hint as to what you need to do. -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Connect to SD over OpenVPN
In response to Martijn de Munnik [EMAIL PROTECTED]: Hi, I need to backup several laptops which can be in the office but also at home or somewhere else. I trying to make this working using OpenVPN so the laptops are connected to the backup server using a VPN network. [backup server 172.16.0.10] LAN 172.16.0.0/24 [router with portforwarding 1194] INTERNET [router] LAN [laptop] OpenVPN is successfully configured and continuously tries to make a connection with the backup server. The laptop get an IP address based on their certificate name so all VPN IPs are known. The Director on the backup server can connect to the FD on the laptop and start a job but the laptop tries to connect to the SD on IP 172.16.0.10. That's where the problem is because the VPN IP of the backup server is 10.67.0.1 so it should connect to the SD on 10.67.0.1. How can I tell the client to connect to the SD on the right IP? Define the storage daemon that way. I'm confused, however. Is the SD on the same system as the dir? If so, why does it have 2 IPs? -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Connect to SD over OpenVPN
In response to Martijn de Munnik [EMAIL PROTECTED]: On Fri, 20 Apr 2007 08:20:36 -0400, Bill Moran [EMAIL PROTECTED] wrote: In response to Martijn de Munnik [EMAIL PROTECTED]: Hi, I need to backup several laptops which can be in the office but also at home or somewhere else. I trying to make this working using OpenVPN so the laptops are connected to the backup server using a VPN network. [backup server 172.16.0.10] LAN 172.16.0.0/24 [router with portforwarding 1194] INTERNET [router] LAN [laptop] OpenVPN is successfully configured and continuously tries to make a connection with the backup server. The laptop get an IP address based on their certificate name so all VPN IPs are known. The Director on the backup server can connect to the FD on the laptop and start a job but the laptop tries to connect to the SD on IP 172.16.0.10. That's where the problem is because the VPN IP of the backup server is 10.67.0.1 so it should connect to the SD on 10.67.0.1. How can I tell the client to connect to the SD on the right IP? Define the storage daemon that way. I'm confused, however. Is the SD on the same system as the dir? If so, why does it have 2 IPs? Hi the SD is on the same machine as the DIR (172.16.0.10) but when a vpn connection is made the SD and DIR are known as 10.67.0.1. One solution could be to do all backups over a VPN connection (so also for the machines in office). How about configuring routing on that system so attempts to contact 172.16.0.10 get routed to the correct interface? -- Bill Moran http://www.potentialtech.com - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bogus/confusing warning? : Filesystem change prohibited. Will not descend into ...
In response to Marc Schiffbauer [EMAIL PROTECTED]:
Hi all,
I have a backup which finished with the following warning:
---
bart-fd: Filesystem change prohibited. Will not descend into /home/mschiff
---
Filesystem change prohibited is correct as I use default values
here. But Will not descend into /home/mschiff is wrong because
both /home and /home/mschiff are in the fileset which are two
different filesystems. And a list files jobid=xy really shows me
files on /home/mschiff.
This is the FileSet:
-
FileSet {
Name = bart ImportantData FileSet
Include {
Options {
signature = MD5
noatime = yes
aclsupport = yes
}
Options {
exclude = yes
regex = .*/[Cc]ache/.*
}
File = \\/etc/bacula/backup-include.txt
}
Exclude {
# exclude all dirs containing a file called .BACULA_NO_BACKUP
File = \\|sh -c 'for D in /home; do find $D -xdev -name
.BACULA_NO_BACKUP -type f -printf \%h\\n\; done | tee
/root/bacula_excluded_dirs.log'
}
}
-
The clients /etc/bacula/backup-include.txt:
--
/etc
/opt
/root
/usr/local
/home
/home/mschiff
--
(with /, /home and /home/mschiff being different filesystems)
Any hints how I can get rid of the warning?
This is a known shortcoming at this time. The code that prevents filesystem
change does not check to see if the data will be caught via another rule.
There's no way to prevent the warning via the config. This would be a nice
feature to have added.
--
Bill Moran
http://www.potentialtech.com
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] director password...Urgent
In response to Naufal Zamir [EMAIL PROTECTED]: yep true, but its encrypted and i don't know the password because it was set by some one else.. So how do I recover the password now. It's not encrypted. Regards On 3/19/07, John Drescher [EMAIL PROTECTED] wrote: On 3/19/07, Naufal Zamir [EMAIL PROTECTED] wrote: Hi, Please can anyone tell me how to reset or get the director password. I need it for adding new clients and I am not sure what the password is. This password is set in the bacula-dir.conf file on the director. John -- Bill Moran http://www.potentialtech.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] backup tapes at home
Alan Brown [EMAIL PROTECTED] wrote: On Fri, 2 Mar 2007, Bill Moran wrote: A fire safe might fit the bill here. I've always wondered about this. If I put _plastic_ tapes in a fire safe, how long does the safe actually insulate the tapes from the heat that will damage them. Not long. A standards-compliant data safe on the other hand will provide protection agains at least 2 hours at 1500C and 24 hour coooling down period. Most also provide protection against large drops (floor failure in multistory buildings) and longer burn periods. Several data safes are simply firesafes fitted with inserts aimed at keeping internal temperatures below 60C. Does anyone have any direct experience with how well a fire safe protects something like backup tapes? If it's well insulated, it will protect them for a while, but long enough for the fire to be extinguished? Only once, and only a small safe on a customer site, but yes. Our current main fire/data safe is a Phoenx Data commander 4623, which is capable of taking 720 LTO tapes in current configuration (39 per drawer, cased, increasing to 45 uncased) See http://www.phoenixsafeusa.com/ or http://www.phoenixsafeusa.com/us/viewproduct/4620_data_commander.html This cost a shade under US $10,000 with tax and delivery included. Interesting, but I suspect that's a little out of the price range for someone wanting to protect their data at home. I know I wouldn't even know where to put such a thing at my house. It'd be cheaper to just rent a safe-deposit box. -- Bill Moran http://www.potentialtech.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] 400G compressed tape only using 215G
In response to Jason King [EMAIL PROTECTED]: I have an LTO2 drive with tapes that hold 200g native and 400g with hardware compression. I have the hardware compression turned on on my tape drive. I know because I went and manually made sure using the tape tools that come with FreeBSD (The os the storage daemon is running on). Last night my tape filled up with only 215G. I know it filled up because I got a message to change tapes from the daemon. I didn't do anything specific in Bacula to tell it to use hardware compression on the tape drive because I didn't think I had to. I made sure hardware compression was enabled in FreeBSD. My question is, is there a specific way to tell bacula to take advantage of hardware compression? Perhaps a setting in the storage daemon config file or the bacula director config file. What are you backing up? That 400G with compression is just an average. There's no guarantee that you'll get 2x compression. Personally, I find it false advertising, but nobody's listening to me. A few years ago (pre-Bacula for me) I set up a backup system for a company that did a lot of graphics work (lots of jpegs). They were really upset that their 400G tapes only held 200G. Reason is that jpegs don't compress, because they're _already_ compressed. If you're backing up a lot of pre-compressed stuff (like JPEGs, or PDFs or zipfiles, or MPEGs, or MP3s) you're _not_ going to see the 2x compression those tape manufactures advertise. The fact that you managed to get 215G on a 200G tape tells me that compression is working, but you're just not getting the 2x you were hoping for. -- Bill Moran http://www.potentialtech.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] backup tapes at home
In response to Stéphane Lardier [EMAIL PROTECTED]: I have an autoloader with 8 tapes. I am afraid of fire and I want have a full backup of my computers at home. So, I can save on 6 tapes and have a different job to save on my two dedicated tapes. But, may be it will be not possible to restore the two tapes after fire event. Is it the best way to prevent against a fire damage ? Write a _full_ backup to a tape that is stored offsite. This tape can be used to do a full restore and is far enough away to be safe from fire. You can use the other tapes in the changer for incrementals. How much data you're willing to lose determines how often you do full compared to incremental backups. -- Bill Moran http://www.potentialtech.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] backup tapes at home
In response to Ryan Novosielski [EMAIL PROTECTED]: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stéphane Lardier wrote: I have an autoloader with 8 tapes. I am afraid of fire and I want have a full backup of my computers at home. So, I can save on 6 tapes and have a different job to save on my two dedicated tapes. But, may be it will be not possible to restore the two tapes after fire event. Is it the best way to prevent against a fire damage ? A fire safe might fit the bill here. I've always wondered about this. If I put _plastic_ tapes in a fire safe, how long does the safe actually insulate the tapes from the heat that will damage them. Keep in mind that they don't have to burn to be ruined. They don't even have to melt -- if they get hot enough, they'll be useless. Does anyone have any direct experience with how well a fire safe protects something like backup tapes? If it's well insulated, it will protect them for a while, but long enough for the fire to be extinguished? -- Bill Moran http://www.potentialtech.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Sony LTO-2/LTO-3
In response to Fausto Barros de Sá Teles [EMAIL PROTECTED]: Has anyone there ever used Bacula for backin up to Sony LTO-2/LTO-3 tapes? I have to implement Bacula but I'm not sure if it works with this kind of tape. We use LTO2/LTO3 tapes at work with no problems. They're Dell branded -- don't know who the underlying hardware is made by. -- Bill Moran http://www.potentialtech.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula-dir won't start from a script
In response to Beren [EMAIL PROTECTED]: Hi Guys, The bacula startup script won't start the director, however typing bacula-dir -c /etc/bacula/bacula-dir.conf works fine. I tried putting this line into a script and running it: /usr/sbin/bacula-dir -c /etc/bacula/bacula-dir.conf It also didn't work...however, if I type it in the command line letter for letter, it works fine. Try running the script as sh -x script_name and see if the debugging output helps any. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Cleaning up database records
In response to Martin Simmons [EMAIL PROTECTED]: On Mon, 19 Feb 2007 15:10:26 -0500, Bill Moran said: In response to Martin Simmons [EMAIL PROTECTED]: On Mon, 19 Feb 2007 08:55:07 -0500, Bill Moran said: On the flip side, running dbcheck periodically is pretty much a requirement for keeping Bacula's database reasonably sized. I have it run once a month in read-only mode via cron and email us the results. When the extra stuff gets significant, I use it to go in and clean up. I think that is overly pessimistic -- unless you are backing up lots of data where the file names or directory names keep changing in the Filename and Path tables. Backup up a server with a bunch of Maildirs, and you'll see this fast enough. Ah, good point. Even without that, with 6 servers and 20 workstations being backed up, lots of files get created, renamed, deleted, on a constant basis. Every few months, the amount of unreferenced stuff is big enough to warrant cleanup. Personally, I don't think a monthly review of the state of the backup system is overly pessimistic in any way :) bacula= select count(*) from file; count -- 19667790 (1 row) bacula= select count(*) from filename; count 927786 (1 row) bacula= select count(*) from path; count 655714 (1 row) That's an interestingly low ratio of filename/path :-) Never really thought about it, but it does seem a little odd. It's probably an indicator of the fact that we're a devel shop. Most of our users have copies of the software project in their home directory, so lots of new paths, but copies of the same files (filenames) -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Cleaning up database records
In response to Frank Altpeter [EMAIL PROTECTED]: My current bacula system (FreeBSD, bacula-2.0.1, mysql-4.1) has currently some massive performance problems. One of the reasons i think is caused by the massive amount of old and obsolete records. For example, i had a client to backup once, which has dissappeared some time ago. When this machine has been removed, the client has been deleted from the bacula configuration. But this way the database records for these clients remain in the database, thus making the db records more and more unusable. So, i would like to prune such records to reduce unneeded data. Is there any way on archiving this? dbcheck should clean this stuff up. And, for the future, what's the best practice to avoid this? Purge the volumes prior to removing the clients. On the flip side, running dbcheck periodically is pretty much a requirement for keeping Bacula's database reasonably sized. I have it run once a month in read-only mode via cron and email us the results. When the extra stuff gets significant, I use it to go in and clean up. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Cleaning up database records
In response to Martin Simmons [EMAIL PROTECTED]: On Mon, 19 Feb 2007 08:55:07 -0500, Bill Moran said: On the flip side, running dbcheck periodically is pretty much a requirement for keeping Bacula's database reasonably sized. I have it run once a month in read-only mode via cron and email us the results. When the extra stuff gets significant, I use it to go in and clean up. I think that is overly pessimistic -- unless you are backing up lots of data where the file names or directory names keep changing in the Filename and Path tables. Backup up a server with a bunch of Maildirs, and you'll see this fast enough. Even without that, with 6 servers and 20 workstations being backed up, lots of files get created, renamed, deleted, on a constant basis. Every few months, the amount of unreferenced stuff is big enough to warrant cleanup. Personally, I don't think a monthly review of the state of the backup system is overly pessimistic in any way :) bacula= select count(*) from file; count -- 19667790 (1 row) bacula= select count(*) from filename; count 927786 (1 row) bacula= select count(*) from path; count 655714 (1 row) I find that those tables are dwarfed by the File table, but that is cleaned by Bacula's own pruning to shouldn't be a problem (though there was a bug in this in some obsolete version). I agree that the file table gets dwarfs the other two, but it doesn't change the fact that our filename and path tables get full of unreferenced stuff over time. Cleaning it out occasionally keeps database size down. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] back up on demand
In response to [EMAIL PROTECTED]: Is there something I can add to the config that will allow the director, or client to sense that the other has come on-line, and trigger a backup? very often users take their laptops home, and can miss multiple backups. Some sort of on-line trigger would be a great help. The software doesn't support this at this time, although there's a TODO item: http://www.bacula.org/?page=projects Has that page been updated since the vote? The vote results page (http://www.bacula.org/?page=vote) is stale. The standard answer for now seems to be configure your jobs using RescheduleOnError -- see this page: http://www.bacula.org/rel-manual/Configuring_Director.html#JobResource -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] HDD is full because of the Database size
In response to Dominik Jonas [EMAIL PROTECTED]: our HDD has 8,4 GB and now it is full because the database file is at the moment 7,8 GB -rw-rw1 mysqlmysql7.8G Feb 1 10:14 ibdata1 on the device is the sql dum with 577 MB, too. My question is now how to make the file ibdata1 smaller, delete old entries in the db or i dont know. How can I do this? Research the purge command available through bconsole. You can also set up automatic purging to occur when data reaches a certain age. Now that you know how much data you can store before your drive fills up, you should be able to estimate a reasonable length of time to save it for. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] MTEOM
In response to Julien Cigar [EMAIL PROTECTED]:
We've bought a new tape drive here, a Sony SDX-700C (AIT-3), with new
tapes. I'm running Bacula 1.38.11 under (Debian) Linux (2.6.18)
I've run the btape test / btape fill with success. The tape is
initialized correctly (variable block mode + ...) :
phoenix:/home/jcigar# cat /etc/stinit.def
{buffer-writes read-ahead async-writes}
manufacturer=SONY model = SDX-700C {
mode1 blocksize=0 compression=1
}
phoenix:/home/jcigar# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x32 (AIT-3).
Soft error count since last status=0
General status bits on (4101):
BOT ONLINE IM_REP_EN
Unfortunately I got an error this morning :
01-Feb 03:27 phoenix-sd: Volume Full-Tape-0001 previously written,
moving to end of data.
01-Feb 03:27 phoenix-sd: canis-job.2007-02-01_03.00.00 Error: Unable to
position to end of data on device sony SDX-700C (/dev/nst0):
ERR=dev.c:839 ioctl MTEOM error on sony SDX-700C (/dev/nst0).
ERR=Input/output error.
01-Feb 03:27 phoenix-sd: Marking Volume Full-Tape-0001 in Error in
Catalog.
I'm a little unclear on certain aspects of this ...
Have you made other backups successfully? Did you try a different tape?
Sometimes tapes ship from the factory with problems. Is this the same
tape that you used for the tests?
We had a drive go bad on us and the errors that Bacula was spewing out
were the most cryptic things you could imagine. It became obvious when
we visited the datacenter, as the drive had a flashing error code on
the front.
--
Bill Moran
Collaborative Fusion Inc.
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and HP JetDirect (was: (2.0.1) 5 minute 5 seconds problem)
In response to Chris Hoogendyk [EMAIL PROTECTED]: [snip] I passed an earlier message from this thread along to my network expert, because we have had some complaints about recent HP stuff. I'm only casually up on that stuff. I was surprised by his answer, but not by his depth of knowledge. So, I'm passing it along just so you all have it. Please note that lprng and cups are two of the most widely used printing systems. Mac OS X uses cups, and lprng is frequently recommended as a replacement system for people in linux/unix environments who want some added capability. Chris, I would hope someone would point out to the list that ONLY ports 1024 historically are well known ports, and those 4096 are sort of registered. His information is dated. Historically we only use 2 digits for the year as well ... So far as I am aware there is no binding registration for ports 4096. But far worse is the fact that not just HP printers use port 9100. So do almost all printers which use the idea of a service port, and this is supported by default in lprng and in cups. His facts are wrong here. IPP (i.e. CUPS) uses port 631 and lpd/lpr uses 515. Those 910x ports are registered to Bacula officially. There are no internet police and there's no fine or anything that means that the Jetdirect systems are doing anything illegal, but it's _only_ jetdirect cards that use those ports outside of the IANA registration. Facts: *) Bacula is registered with IANA to use the 910x ports. It's official *) _only_ jetdirect cards use the 910x ports. CUPS and LPD do not. If they are, it's because you're using some sort of CUPS-jetdirect driver (which is actually pretty common) but it's not CUPS, it's the jetdirect driver. If bacula is on port 9100 that is a totally bad idea. Well, he's welcome to his opinion, but he doesn't seem to have any facts to back it up. Bacula ought to have gotten a port registered by IANA in the reserved range rather than just grabbing ports that have long been in use for printing. Again, he's working off 10-year old data here. The ports _are_ registered, the registered port range has been expanded in recent years because there's no room left for new applications below 1024. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Don't understand error
In response to Richard White [EMAIL PROTECTED]:
I have added a Winders server to our Bacula backup jobs. It is located
in a DMZ. Our web server (running on SLES 10) is also in the DMZ and we
have been backing it up with Bacula for a month now. I created rules in
the firewall for the Windows server that are identical to those for the
web server, just with a different IP address. The Bacula client is
installed and the bacula-fd.conf file has been edited. The bacula-fd
service starts automatically.
Here is the bacula-fd.conf:
#
# Default Bacula File Daemon Configuration file
#
# For Bacula release 1.38.10 (08 June 2006) -- cygwin
1.5.18(0.132/4/2)
#
# There is not much to change here except perhaps the
# File daemon Name to
#
#
# List Directors who are permitted to contact this File daemon
#
Director {
Name = lbackup-dir
Password = filedaemon
}
#
# Global File daemon configuration specifications
#
FileDaemon { # this is me
Name = ArcIMS-fd
FDport = 9102 # where we listen for the director
WorkingDirectory = c:/bacula/working
Pid Directory = c:/bacula/working
}
# Send all messages except skipped files back to Director
Messages {
Name = Standard
director = lbackup-dir = all, !skipped
}
This is identical to the conf file on systems that work, except for the
filedaemon name.
When I run a backup job for this server, the job runs for two hours
(almost exactly), with the tape drive indicating data being written,
then terminates with an error. The job summary looks like this:
Try turning keepalive on. Sounds like the system runs for a while without
exchanging any data over the network, and your firewall is dropping the
state from the state table.
rant
Stateful firewalls are a stupid idea.
/rant
29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Fatal error:
Network error with FD during Backup: ERR=Connection reset by peer
29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Fatal error: No
Job status returned from FD.
29-Jan 18:07 lbackup-dir: ArcIMS.2007-01-29_16.07.46 Error: Bacula
1.36.3 (22Apr05): 29-Jan-2007 18:07:51
JobId: 1462
Job:ArcIMS.2007-01-29_16.07.46
Backup Level: Full (upgraded from Differential)
Client: ArcIMS-fd
FileSet:ArcIMS 2007-01-19 15:50:02
Pool: Weekend
Storage:Internal
Start time: 29-Jan-2007 16:07:48
End time: 29-Jan-2007 18:07:51
FD Files Written: 0
SD Files Written: 13,339
FD Bytes Written: 0
SD Bytes Written: 9,959,280,437
Rate: 0.0 KB/s
Software Compression: None
Volume name(s): Week_2D
Volume Session Id: 3
Volume Session Time:1170100801
Last Volume Bytes: 19,948,336,363
Non-fatal FD errors:0
SD Errors: 0
FD termination status: Error
SD termination status: OK
Termination:*** Backup Error ***
Status dir looks like this:
Terminated Jobs:
JobId Level Files Bytes Status FinishedName
[other jobs]
1462 Full 0 0 Error29-Jan-07 18:07 ArcIMS
I am puzzled why there are 13,339 SD files and 9,959,280,437 bytes
written, but zero FD files and bytes written.
Tape files are different than actual files. But the fact that you see
neither files nor bytes indicates (to me) that your doing spooling?
--
Bill Moran
Collaborative Fusion Inc.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] slow postgres, index/table does not exist
In response to Bacula User [EMAIL PROTECTED]: I am using freebsd 6.1, bacula 1.38.11_1, and postgresql-server-7.4.13_1. Each night I dump 913,157,728,765 bytes of 5,395,281 files to tape. I spool data and attributes. I will need to almost triple this amount to deploy bacula. Data goes to tape in under 16.5 hours, but attributes continue going into the catalog for over 13 hours afterwards. In other words, I'm still writing the previous night's data to the database when the new day's backup cycle starts. I never get a chance to get an ascii dump of the db before the next backup cycle begins. My untuned postgres is too slow. I'm researching ways to speed it up. If you have pointers, please share. Can you replace 7.4 with 8.2? There are a LOT of performance improvements in 8.2 over 7.4. You should probably join the PostgreSQL performance mailing list: http://www.postgresql.org/community/lists/subscribe There is a good guide here: http://www.varlena.com/GeneralBits/Tidbits/perf.html Generally speaking, you're dealing with lots of inserts. To speed them up you're going to need very fast disks. You'll probably also benefit from increasing checkpoint_segments and checkpoint_timeout (see http://www.postgresql.org/docs/8.2/static/wal-configuration.html) Make sure you vacuum and analyze frequently. Your scenario may benefit from using the autovacuum daemon. Furthermore, each night has many errors like the following. What do these mean and how do I fix them? They are normal operation. Bacula creates a number of temp tables and deletes them when it's done with them. To ensure sane recovery from a crash, Bacula tries to delete them before using them, just in case they were left over from a crash. There are ways to do this in PostgreSQL without producing these errors, but I don't think there's a database- independent way of doing this without causing these errors. The short answer is they're nothing to worry about. postgres[19278]: [324-1] ERROR: index delinx1 does not exist postgres[19278]: [325-1] ERROR: index delinx1 does not exist postgres[19278]: [326-1] ERROR: table delcandidates does not exist postgres[19278]: [327-1] ERROR: index delinx1 does not exist postgres[19278]: [328-1] ERROR: index delinx1 does not exist postgres[19278]: [329-1] ERROR: table delcandidates does not exist postgres[19278]: [330-1] ERROR: index delinx1 does not exist postgres[19278]: [331-1] ERROR: index delinx1 does not exist postgres[19278]: [332-1] ERROR: table delcandidates does not exist postgres[19278]: [333-1] ERROR: index delinx1 does not exist postgres[19278]: [334-1] ERROR: index delinx1 does not exist postgres[19278]: [335-1] ERROR: table delcandidates does not exist postgres[19278]: [336-1] ERROR: index delinx1 does not exist postgres[19278]: [337-1] ERROR: index delinx1 does not exist postgres[19278]: [338-1] ERROR: table delcandidates does not exist postgres[19278]: [339-1] ERROR: index delinx1 does not exist postgres[19278]: [340-1] ERROR: index delinx1 does not exist postgres[19278]: [341-1] ERROR: table delcandidates does not exist postgres[19278]: [342-1] ERROR: index delinx1 does not exist postgres[19278]: [343-1] ERROR: index delinx1 does not exist postgres[19278]: [344-1] ERROR: table delcandidates does not exist postgres[19278]: [345-1] ERROR: index delinx1 does not exist postgres[19278]: [346-1] ERROR: index delinx1 does not exist postgres[19278]: [347-1] ERROR: table delcandidates does not exist postgres[19278]: [348-1] ERROR: index delinx1 does not exist postgres[19278]: [349-1] ERROR: index delinx1 does not exist Thank you for your help. Bacula is well on its way to replacing our current method of madness. I just need to sort a few more things out. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Building directory tree is very slowly
In response to Ondrej PLANKA [EMAIL PROTECTED]:
Alan Brown napsal(a):
On Tue, 9 Jan 2007, Ondrej PLANKA wrote:
It sounds silly, but add individual indexes for JobId, PathId and
FilnameId as well as the 3-way one.
Ok, do you have some tip, how can I add individual indexes? Because I
think, my indexes are OK or not?
ALTER TABLE File ADD INDEX [name] {column} ;
so...
ALTER TABLE File ADD INDEX file_jobid_idx JobId ;
ALTER TABLE File ADD INDEX file_pathid_idx PathId ;
ALTER TABLE File ADD INDEX file_filenameid_idx FilenameId ;
I really done this individual indexes, but building directory tree for
Jobs are still very slowly.
Index problems are probably the most common reason for this, but they're
not the only possible cause.
I didn't see any hardware details -- perhaps you posted it earlier and
I missed them. Bacula will need a lot of RAM to store that large of a
directory tree. Monitor top, iostat, and any other sys monitoring programs
you can during the tree build.
In our case, we decided to add another gig of RAM to backup server and things
are nice and fast now.
--
Bill Moran
Collaborative Fusion Inc.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] speeding up bacula?
Oliver Lehmann [EMAIL PROTECTED] wrote: I wonder how I can speed up bacula? FD Bytes Written: 54,663,990,797 (54.66 GB) SD Bytes Written: 54,667,075,813 (54.66 GB) Rate: 2687.1 KB/s the directories to backup are on the same 3ware-RAID-5 as the storage is located (Media Type = File). I know that this is not the optimum settings, but the 4*250GB SATA drives are taking all the place in the 4HE case so there is no space left for other disks ;) The MySQL DB is located on the same disk. Even with this not optimum setup I wonder why it is so slow. I can read +write a file from and to the same disk with a throughput of ~25MB/sec (writing only is about 50MB/sec) First off, throw your assumptions away. You're assuming that Bacula should be able to run at the same speed as the disk. Turn off compression and see if it comes close. Then watch top and iostat while a backup is running and see what kind of resources the whole process is utilizing. Check your MySQL insert performance and see if that's holding things up. I give it a 50% chance that the CPU is the bottleneck, not the disk. -Bill - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] [CROSSML] Can Bacula fulfil the following requirement?
In response to Eric Leung [EMAIL PROTECTED]: Dear Bill Thank you very much for your reply, it was so helpful, as I am new to Bacula. However, there are few more question I would like to ask. It's generally considered bad form to pester someone personally about a mailing-list response. I've added the list back into the CC. It's also rather difficult to follow the conversation when you top post. Regarding to the following requirements: 05) backup notification email with html template - NOT SURE Your answer: Email notification works. Not sure about HTML templates. Do you think I can use script to reproduce the email notification instead of using the default email notification from Bacula, in order to use a HTML template for email notification, do I have to modify the source code in Bacula, etc. I'm sure that you could do either of those to get what you want. 07) Quick Restore (backup not split into 2G files, i.e. one single file) - NOT SURE As the current backup system I am using need to split the backup to multiple 2G files, that is, I would like to ask how the backup is stored by Bacula? In a tar format or some other format? In addition to this question, is there any limit to the compression file size, as I know some backup systems limit the size of the files in order to be compressed. So I would like to ask is there any limit in Bacula for compression large files (e.g. few hundred GB files) Yes, I have a system with an external drive array that doesn't support files larger than 2G. I just set the max volume size at 2G and Bacula creates new files when needed. 09) Shadow filesystem backups - NOT SURE Your answer: Yes. VSS in Windows, and other systems can be scripted. So you mean if I backup a Windows machine, I need to use VSS, and other systems, I need to write script? The client I need to backup mainly are Window and Linux machine, so what should I do? Window: use VSS. Linux: it depends on the filesystem you're using, and whether that filesystem supports snapshots. For example, I believe Reiser does, but I don't think that ext2 does. For how to write the script, do a little googling. -Original Message- From: Bill Moran [mailto:[EMAIL PROTECTED] Sent: Thursday, 4 January 2007 12:14 AM To: Eric Leung Cc: [EMAIL PROTECTED] Subject: Re: [Bacula-devel] [CROSSML] Can Bacula fulfil the following requirement? In response to Eric Leung [EMAIL PROTECTED]: 01) Able to split backups across multiple disks - OK 02) Incremental backups - OK 03) Improved flexibility for configuring which folders are backed up 04) Encrypted backup support - OK 05) backup notification email with html template - NOT SURE Email notification works. Not sure about HTML templates. 06) ACL support - NOT SURE Yes. 07) Quick Restore (backup not split into 2G files, i.e. one single file) - NOT SURE I don't understand what this means. 08) Scripts to allow specified files to be restored from the backup - OK: base on the catelog in DB 09) Shadow filesystem backups - NOT SURE Yes. VSS in Windows, and other systems can be scripted. 10) Incremental backups to remote fileservers rather than disks - OK I am looking forward to your reply and using Bacula. It will be great if Bacula can fulfil all of the above requirements, as these are the major factor for my decision. -- Bill Moran Collaborative Fusion Inc. -- Bill Moran Collaborative Fusion Inc. [EMAIL PROTECTED] Phone: 412-422-3463x4023 IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list
Re: [Bacula-users] FOSDEM slide presentation
In response to Kern Sibbald [EMAIL PROTECTED]: I've posted to the web site a preliminary copy of the slide show that I plan to present at FOSDEM in Febrary, and would appreciate your feedback on it. I haven't timed the talk yet, but it should be about 30-40 minutes. I would like to have better graphics for the one graphics slide I have, but then I'm not very good as an artist. About the only other thing I am thinking about doing is showing some actual commands and output much like what is in the Brief Tutorial chapter of the manual -- the problem is that such text either is too small to read on a projected screen or requires 3 slides to get it up ... You can view it at: http://www.bacula.org/presentations/Bacula-FOSDEM-talk-24Feb07.pdf Best regards, * Slide 21: I found the indenting a bit confusing. At first it seems as if you had multiple filesets on the slide. Also, I gave a presentation at our local LUG last month on Bacula, and I left myself time at the end to actually show them what it was like to manually kick off backups and restores in bconsole. I run a full Bacula setup on my laptop so it was pretty convenient. The point being that the demonstration at the end was more popular than the presentation itself -- you might want to consider doing the same. Other than that, it looks really good. I didn't think the graphic slide was particularly bad -- it looks better than the OOo graphics I did for my presentation. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula in Windows XP
In response to Rogerio [EMAIL PROTECTED]: I´m testing Bacula in a Windows XP, when a run job i have the following error: Job started. JobId=10 27-Dec 12:01 Debian-dir: No prior Full backup Job record found. 27-Dec 12:01 Debian-dir: No prior or suitable Full backup found. Doing FULL backup. 27-Dec 12:01 Debian-dir: Start Backup JobId 10, Job=Suporte.2006-12-27_12.01.07 27-Dec 15:01 suporte-fd: Suporte.2006-12-27_12.01.07 Fatal error: Failed to connect to Storage daemon: Debian:9103 27-Dec 15:01 suporte-fd: Suporte.2006-12-27_12.01.07 Error: c:\cygwin\home\kern\bacula\k\src\win32\lib\../../lib/bnet.c:768 gethostbyname() for host Debian failed: ERR=Authoritative answer for host not found. 27-Dec 12:01 Debian-dir: Suporte.2006-12-27_12.01.07 Fatal error: Socket error from Filed on Storage command: ERR=No data available 27-Dec 12:01 Debian-dir: Suporte.2006-12-27_12.01.07 Error: Bacula 1.36.2 (28Feb05): 27-Dec-2006 12:01:13 JobId: 10 Job:Suporte.2006-12-27_12.01.07 Backup Level: Full (upgraded from Incremental) Client: suporte-fd FileSet:Suporte 2006-12-27 11:46:42 Pool: Default Storage:File Start time: 27-Dec-2006 12:01:09 End time: 27-Dec-2006 12:01:13 FD Files Written: 0 SD Files Written: 0 FD Bytes Written: 0 SD Bytes Written: 0 Rate: 0.0 KB/s Software Compression: None Volume name(s): Volume Session Id: 4 Volume Session Time:1167227172 Last Volume Bytes: 0 Non-fatal FD errors:0 SD Errors: 0 FD termination status: SD termination status: Waiting on FD Termination:*** Backup Error *** Any help??? Yes. Use host names that actually resolve to IP addresses. From your error messages, the XP machine is being told to connect to a Storage Daemon on host debian, but there is no such host in DNS. Either add the host to DNS, or correct the config so it uses a proper hostname or IP. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] looking at bacula
In response to Arno Lehmann [EMAIL PROTECTED]: That's not too much for Bacula, although OpenBSD as a client is not as easy to set up as other OSes. At least that's what I remember from list mail - there seem to be some difficulties getting the client to compile. AFAIK, it works, though. http://people.collaborativefusion.com/~seklecki/obsd_bacula.html -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] looking at bacula
In response to Oliver Lehmann [EMAIL PROTECTED]: Bill Moran wrote: In response to Arno Lehmann [EMAIL PROTECTED]: That's not too much for Bacula, although OpenBSD as a client is not as easy to set up as other OSes. At least that's what I remember from list mail - there seem to be some difficulties getting the client to compile. AFAIK, it works, though. http://people.collaborativefusion.com/~seklecki/obsd_bacula.html somehow, the tar is broken here: [EMAIL PROTECTED] tmp gtar -xf ../bacula_openbsd_port.tar gtar: Unexpected EOF in archive gtar: Unexpected EOF in archive gtar: Error is not recoverable: exiting now Exit 2 [EMAIL PROTECTED] tmp tar -xf ../bacula_openbsd_port.tar bacula/w-bacula-clientonly/bacula-1.39.30/autoconf/gnome-macros/autogen.sh: (Empty error message) tar: Truncated input file (need to skip 7312 bytes) Exit 1 [EMAIL PROTECTED] tmp Hmmm ... I get a similar error: tar xf bacula_openbsd_port.tar bacula/w-bacula-clientonly/bacula-1.39.30/autoconf/gnome-macros/autogen.sh: (Empty error message) tar: (Empty error message) Although it seems to extract everything in spite of the message. Brian, can you have a look at this? -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Questions regarding bacula from a potential user
In response to Terry Zink [EMAIL PROTECTED]: Currently my company uses a custom backup solution. While effective, we are growing out of the point where it is scalable for what we need and becoming unmanageable. It is also starting to lack features necessary. Essentially, it backs up both linux and windows systems. It does standard dumps (A level 0, then level 1's compared against the level 0 until the level 1 reaches 50% the level0 size) for linux, and for windows we use smbclient to grab and tar up specific windows shares (usually the c$ drive.. yes.. There are many limitations of this heh.) I stumbled across bacula awhile back, and it looks great! Looks like it may do everything I need. (I need to read the documentation a bit more clearly to determine if it does the 0 and 1's as needed, but I'm sure it can.) The question I'm wondering is.. how well does it work in large scale environments? Our current system, is disk(client server) to disk (backup server) to disk (big array to back up the backup servers) and then to tape. The first level of backup servers (what the client server sends the data to) contains about 22 servers. Currently our system means managing each of those individually. You see my dilemma. Now, granted bacula could easily probably just handle them each as 20 or so different SD machines and the director could manage all 20 or so. At least that's my understanding. 22 storage devices? Well we're backing up around 800 servers at least like this. My question is... aside from anything being blatantly wrong in the above statements, can bacula reliably be used to backup 800 different systems or should I look for a different solution? Every testimonial I see mentions 30 or 50 clients max... Wow. That's a pretty big setup. I've never heard of anyone using Bacula for something that size. In theory, it should scale up that big, but you may encounter some scaling issues that nobody else has seen yet. However, Bacula is open source, and (from your email) I get the feeling that you're the type who can help out by submitting helpful bug reports and working with the community -- maybe even submitting patches. The devel team is great. I think if you hit scaling problems putting Bacula into such a huge environment, you'll get great response from the Bacula community to help make it work. Also, I noticed in the current limitations that 4 billion files is the limit per database? (this would I assume be per director then in reality if they used different databases?) Actually, it's per-database catalog. This is because most databases use 32 bit int for IDs. You can have multiple databases (what Bacula calls a catalot) per Bacula director, which allows you to get around this. Each catalog can only have 4 billion files. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Feature Request - Source address control
Item 1: Cause daemons to use a specific IP address to source communications Origin: Bill Moran [EMAIL PROTECTED] Date: 18 Dec 2006 Status: What: Cause Bacula daemons (dir, fd, sd) to always use the ip address specified in the [DIR|DF|SD]Addr directive as the source IP for initiating communication. Why:On complex networks, as well as extremely secure networks, it's not unusual to have multiple possible routes through the network. Often, each of these routes is secured by different policies (effectively, firewalls allow or deny different traffic depending on the source address) Unfortunately, it can sometimes be difficult or impossible to represent this in a system routing table, as the result is excessive subnetting that quickly exhausts available IP space. The best available workaround is to provide multiple IPs to a single machine that are all on the same subnet. In order for this to work properly, applications must support the ability to bind outgoing connections to a specified address, otherwise the operating system will always choose the first IP that matches the required route. Notes: Many other programs support this. For example, the following can be configured in BIND: query-source address 10.0.0.1; transfer-source 10.0.0.2; Which means queries from this server will always come from 10.0.0.1 and zone transfers will always originate from 10.0.0.2. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Questions regarding bacula from a potential user
In response to Eric Bollengier [EMAIL PROTECTED]: Also, I noticed in the current limitations that 4 billion files is the limit per database? (this would I assume be per director then in reality if they used different databases?) Actually, it's per-database catalog. This is because most databases use 32 bit int for IDs. You can use bigint instead of integer and bigserial instead of serial in your make_postgresql_tables. You can have multiple databases (what Bacula calls a catalot) per Bacula director, which allows you to get around this. Each catalog can only have 4 billion files. You can use 64bit FileId_t..., It works with our postgresql database - cats/cats.h:typedef uint32_t FileId_t; - jcr.h: uint32_t FileId; /* Last file id inserted */ + cats/cats.h:typedef uint64_t FileId_t; + jcr.h: uint64_t FileId; /* Last file id inserted */ I'm surprised, but delighted, to find it's that simple. That's a sign of a well-written application. Would this work with MySQL and SQLLite as well? (Don't know if MySQL has a 64-bit int or not?) If not, any reason why this couldn't be implemented with an ifdef for PostgreSQL -- when SQLLite and MySQL catch up, the ifdef can be removed. Personally, I don't think we're in danger of exceeding this too soon: bacula= select max(fileid) from file; max --- 111205419 (1 row) Been running about 6 months. We should probably plan to upgrade to 64-bit when we upgrade to the latest version in the spring. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Postgres slow because of autocommit
In response to Gabriele Bulfon [EMAIL PROTECTED]: Hello, I stumbled upon some pages that confirm a suspect I had about using postgres with bacula: - Postgres has autocommit by default, slowing down a lot any Bacula operation because Bacula does not use any transaction during write. If this is truedo I have any way to set up just the Bacula database (not the other ones I have) to have no autocommit? Is there any pre-script to set the bacula connection without auto commit? There is no way (that I'm aware of) to disable autocommit. By definition, any database that's not being fed transactions is in autocommit mode (PostgreSQL actually calls this unchained mode). It makes sense: without an explicit transaction, how would the DB server know when to commit the data? I don't know of any DB that does it any differently. That being said, the Bacula team has been working hard at adding transaction support and other performance improvements -- if you search the list archives, you'll see lots of discussions. I think some of the improvements are slated to appear in the next version. That being said, PostgreSQL runs just fine even without the transactions. Everything I run uses PostgreSQL (I manage 4 directors). Two of them manage large data sets using beefy hardware (of 25 servers, 1 of them has 1,500,000 files/12G compressed) Another one is my laptop, and the fourth is a medium-grade machine backing up about 20G of servers and desktops. None of them are experiencing any performance problems. -- Bill Moran Collaborative Fusion Inc. [EMAIL PROTECTED] Phone: 412-422-3463x4023 - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backs up Windows, but can't restore
In response to Richard White [EMAIL PROTECTED]: This is Bacula 1.36.3. I have been backing up Linux servers for more than a year. I have done a test backup and restore of my Windows 2000 Pro workstation with no problems (it has, however, FAT32 partitions). As a part of planning for the future (migrating our NetWare servers to OES Linux and adding the inevitable Winders servers), my boss wants me to be sure I could back up and restore a W2K server. Well, it backs up fine -- nearly 3 GB. When I try to restore, Bacula informs me that 14,694 files have been added to the tree and my cwd is /. However, ls shows nothing at all, I can't cd to any known directory and can't mark any file for restore. The manual's caveats for Windows refer to the (ab)use of '\' and to be mindful of the treatment of NTFS naming conventions. Any ideas on where I am going worng? You might want to try some things and cut/paste them into a response email so we can get a better idea of what's going on. The initial problem I had was that Bacula created a c: and d: directory at the root to handle the problem of Windows having drives. I also seem to remember the c: and d: being case-sensitive (perhaps they were C: and D:) ... anyway, that confused me at first, and I don't remember them showing up in an ls ... but I could be wrong. HTH -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backs up Windows, but can't restore
In response to Richard White [EMAIL PROTECTED]:
On 12/12/2006 at 9:08 AM, in message
[EMAIL PROTECTED], Bill Moran
[EMAIL PROTECTED] wrote:
In response to Richard White [EMAIL PROTECTED]:
This is Bacula 1.36.3. I have been backing up Linux servers for
more
than a year. I have done a test backup and restore of my Windows
2000
Pro workstation with no problems (it has, however, FAT32
partitions). As
a part of planning for the future (migrating our NetWare servers to
OES
Linux and adding the inevitable Winders servers), my boss wants me
to be
sure I could back up and restore a W2K server.
Please make an effort to fix your mail client so it doesn't mangle quoted
messages. The above should look like this:
On 12/12/2006 at 9:08 AM, in message
[EMAIL PROTECTED], Bill Moran
[EMAIL PROTECTED] wrote:
In response to Richard White [EMAIL PROTECTED]:
This is Bacula 1.36.3. I have been backing up Linux servers for more
than a year. I have done a test backup and restore of my Windows 2000
Pro workstation with no problems (it has, however, FAT32 partitions). As
a part of planning for the future (migrating our NetWare servers to OES
Linux and adding the inevitable Winders servers), my boss wants me to be
sure I could back up and restore a W2K server.
Well, it backs up fine -- nearly 3 GB. When I try to restore, Bacula
informs me that 14,694 files have been added to the tree and my cwd is
/. However, ls shows nothing at all, I can't cd to any known directory
and can't mark any file for restore.
The manual's caveats for Windows refer to the (ab)use of '\' and to be
mindful of the treatment of NTFS naming conventions.
Any ideas on where I am going worng?
You might want to try some things and cut/paste them into a response
email so we can get a better idea of what's going on.
The initial problem I had was that Bacula created a c: and d: directory
at the root to handle the problem of Windows having drives. I also
seem to remember the c: and d: being case-sensitive (perhaps they were
C: and D:) ... anyway, that confused me at first, and I don't remember
them showing up in an ls ... but I could be wrong.
Here are the pertinent parts of bacula-dir.conf:
Job {
Name = Sophos
Type = backup
Pool = Daily
Full Backup Pool = Weekend
Schedule = Cycle_S
Client = sophos-fd
Level = differential
Storage = Internal
Fileset = Sophos_One
Messages = Standard
Priority = 10
SpoolData = yes
Write Bootstrap = /bacula/bin/working/BackupCatalog.bsr
}
Job {
Name = RestoreSophos
Type = Restore
Client = sophos-fd
FileSet = Sophos_One
Storage = Internal
Pool = Daily
Messages = Standard
}
FileSet {
Name = Sophos_One
Include {
Options {
signature = MD5
}
File = c:/
}
}
Schedule {
Name = Cycle_S
Run = Differential mon-thu at 17:15
}
Client {
Name = sophos-fd
Address = 10.1.0.5
FDPort = 9102
Catalog = MyCatalog
Password = filedaemon
File Retention = 30 days
Job Retention = 6 months
AutoPrune = yes
}
Here is the bacula-fd.conf on the Windows server:
FileDaemon { # this is me
Name = sophos-fd
FDport = 9102 # where we listen for the director
WorkingDirectory = c:/bacula/working
Pid Directory = c:/bacula/working
}
The only difference between this and my exercise is that the backup job
was defined as full with Weekend being the default pool.
When I run Restore and choose the correct job number, it declares that
there are 14,694 files, as I mentioned, but I can neither see nor mark
them.
Please show this part. The configs aren't likely to help, as it seems
your config is OK. I'm suspecting that the process you're trying to
go through to restore is somehow incorrect. If you show us the
process, we might be able to pick out what's wrong.
--
Bill Moran
Collaborative Fusion Inc.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backs up Windows, but can't restore
In response to Richard White [EMAIL PROTECTED]:
Please show this part. The configs aren't likely to help, as it
seems
your config is OK. I'm suspecting that the process you're trying to
go through to restore is somehow incorrect. If you show us the
process, we might be able to pick out what's wrong.
OK, here goes.
In bconsole, type 'restore'.
Select 2 ('Enter list of comma separated JobIDs to select').
Type in the job number.
At this time I see this:
.
.
.
1 Job, 14,694 files inserted into the tree.
You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added,
unless
you used the all keyword on the command line.
Enter done to leave this mode.
cwd is: /
$
Now I type
ls
and get this:
c:/
I know there is a directory called sec20, so I type this:
cd sec20
and get this:
Invalid path given.
cwd is: /
And so on. It has been a couple of months since I did the testing on my
W2K Pro desktop, but I know I restored files in the usual way.
As I mentioned in my earlier response, Bacula has to work around the
broken Windows concept of drives. Bacula doesn't do drives.
In order to work around it, Bacula creates a directory for each
drive, as you can see above in your ls, it created a c: directory
in the backup volume.
So, try cd c: then do another ls. An example for one of my systems:
cwd is: /
$ ls
e:/
$ cd e:
cwd is: e:/
$ ls
RECYCLER/
System Volume Information/
Virtual Machines/
[...]
--
Bill Moran
Collaborative Fusion Inc.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Source Forge abusive commercial ads
In response to Chris Hoogendyk [EMAIL PROTECTED]: Marketing claims always tend to be a bit sketchy, and getting marketing departments to follow academic practices of source citation is rather unlikely. However, it is possible that they do have a larger installed base for historical reasons, aside from what you might see listed in download statistics from Source Forge. I guess a broader question for Source Forge would be whether they have any stated policy or practice regarding targeting ads at competitors pages within Source Forge. Unfortunately, there are rules in marketing. One of the more important rules is: Say anything that will get you customers, as long as the profit from the customers exceeds the cost of any lawsuits. (Oddly enough, if you replace customers with votes, politicians have a similar rule.) I've noticed that this practice has been getting worse in recent years, at least in the U.S. It used to be, if you had a doctor on a commercial, there was fine print at the bottom of the screen that said something like not a real doctor or otherwise identified the person as an actor with no medical training. That doesn't occur anymore. TV ads (in particular) lie outright -- they just do it in a way that their lawyers think it's unlikely they'll ever have a sizable lawsuit occur as a result. The biggest problem, IMHO, is that the American public has ceased to notice and/or do anything about it. I mean, to bring this back to the original topic, if Kern complains (hell, if the entire Bacula community complains) it will probably cause very little to happen aside from a polite apology and explanation of why it is this way from Sourceforge. If, however, Kern were to start publicly researching alternatives to Sourceforge, they might take notice, as Bacula is a pretty important project with a lot of draw. I'm not going to suggest that you _should_ do that, I'm just saying it might be more effective. If Sourceforge is unwilling/unable to force their advertisers to be honest, perhaps some other project hosting service would be a better fit anyway? -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula FreeBSD 6.x
In response to Shaun T. Erickson [EMAIL PROTECTED]: Does the latest stable release of Bacula work ok on FreeBSD 6.x? Is there anything special I need to know about bacula and this version of FreeBSD? Nothing that I've come across. I've got four separate directors running on 6.X and a whole messa FDs. Dan has done an outstanding job of keeping the port up to date. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] windows xp client backup issue
the Alert command only if you have the mtx package loaded
## Alert Command = sh -c 'tapeinfo -f %c |grep TapeAlert|cat'
#}
#
# A FreeBSD tape drive
#
#Device {
# Name = DDS-4
# Description = DDS-4 for FreeBSD
# Media Type = DDS-4
# Archive Device = /dev/nsa1
# AutomaticMount = yes; # when device opened, read it
# AlwaysOpen = yes
# Offline On Unmount = no
# Hardware End of Medium = no
# BSF at EOM = yes
# Backward Space Record = no
# Fast Forward Space File = no
# TWO EOF = yes
#}
#
# A OnStream tape drive.
# You need the kernel osst driver 0.9.14 or later, and
# do mt -f /dev/nosst0 defblksize 32768 once as root.
#
#Device {
# Name = OnStream
# Description = OnStream drive on Linux
# Media Type = OnStream
# Archive Device = /dev/nst0
# AutomaticMount = yes; # when device opened, read it
# AlwaysOpen = yes
# Offline On Unmount = no
## The min/max blocksizes of 32768 are *required*
# Minimum Block Size = 32768
# Maximum Block Size = 32768
#}
#
# A DVD device
#
#Device {
# Name = DVD-Writer
# Media Type = DVD
# Archive Device = /dev/hdc
# LabelMedia = yes; # lets Bacula label unlabeled media
# Random Access = Yes;
# AutomaticMount = yes; # when device opened, read it
# RemovableMedia = yes;
# AlwaysOpen = no;
# MaximumPartSize = 800M;
# RequiresMount = yes;
# MountPoint = /mnt/cdrom;
# MountCommand = /bin/mount -t iso9660 -o ro %a %m;
# UnmountCommand = /bin/umount %m;
# SpoolDirectory = /tmp/backup;
# WritePartCommand = /etc/bacula/dvd-handler %a write %e %v
# FreeSpaceCommand = /etc/bacula/dvd-handler %a free
#}
#
# For OpenBSD OS = 3.6
#
#Device {
# Name = DDS-3
# Media Type = DDS-3
# Archive Device = /dev/nrst0
# Use MTIOCGET= no
# BSF at EOM = yes
# TWO EOF = no
# AutomaticMount = yes;
# AlwaysOpen = yes;
# RemovableMedia = yes;
# RandomAccess = no;
#}
#
# A very old Exabyte with no end of media detection
#
#Device {
# Name = Exabyte 8mm
# Media Type = 8mm
# Archive Device = /dev/nst0
# Hardware end of medium = No;
# AutomaticMount = yes; # when device opened, read it
# AlwaysOpen = Yes;
# RemovableMedia = yes;
# RandomAccess = no;
#}
#
# Send all messages to the Director,
# mount messages also are sent to the email address
#
Messages {
Name = Standard
director = backup-temp-dir = all
}
thanks
--
Bill Moran
Collaborative Fusion Inc.
[EMAIL PROTECTED]
Phone: 412-422-3463x4023
IMPORTANT: This message contains confidential information and is
intended only for the individual named. If the reader of this
message is not an intended recipient (or the individual
responsible for the delivery of this message to an intended
recipient), please be advised that any re-use, dissemination,
distribution or copying of this message is prohibited. Please
notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The
sender therefore does not accept liability for any errors or
omissions in the contents of this message, which arise as a
result of e-mail transmission.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Filesystem change prohibited. Will not descend into
On Thu, 23 Nov 2006 09:09:13 +0100 Manuel Staechele [EMAIL PROTECTED] wrote: Hello List, i do not understand the message: server-fd: Filesystem change prohibited. Will not descend into /home did bacula save this directory or not? It came only this message, the job it self is OK without warnings, so i think it doesn't matter. If bacula did not save the directory, how can i say to do so? No, bacula did not save that directory. Despite the fact that you did not include any config files, I can assume that your backup job includes a File = /. Bacual does not cross mount barriers by default. See the relevent docs on the onefs parameter: http://bacula.org/rel-manual/Configuring_Director.html#SECTION000147000 The quick answer is: 1) Either set onefs = no 2) or add File = /home/ to your FileSet 3) Do _not_ do both or you'll backup /home twice - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] restore takes forever
On Thu, 23 Nov 2006 10:38:41 GMT Martin Simmons [EMAIL PROTECTED] wrote: On Wed, 22 Nov 2006 17:26:47 -0500, Bill Moran said: On Wed, 22 Nov 2006 23:11:59 +0100 Andras Horvai [EMAIL PROTECTED] wrote: What does VolFiles mean? I didn't find it in the documentation. I've been wondering about that. I wish one of the developers would chime in here. I seem to remember that a trick used on tape drives is to write an EOF marker every so often, then when restoring, the drive can quickly seek X EOF markers ahead before it has to slow down to read through the data. If I'm understanding this correctly, there's no reason Bacula can't do the same thing with file volumes. If it writes an EOF marker every 4G (which it seems to, based on your output) it can seek() to the within 4G of the data it needs, then it only needs to read() through a maximum of 4G to get the data. Using EOF markers like that in a file won't work, because there is no fast way of seeking to an EOF marker (unlike on a tape). ?? If Bacula knows it's writing a file marker every 4G, why can't it just use fseek() to skip forward? However, it is easier than that: Bacula could seek directly to the right place in the file. IIRC, there is some code to do that but it was disabled because it didn't quite work. Well, that's an obvious problem ... - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] restore takes forever
On Wed, 22 Nov 2006 23:11:59 +0100 Andras Horvai [EMAIL PROTECTED] wrote: Hi, Well thanks for your answer. I will change my volumes size useing the Max Volume Bytes setting. What settings do you reccomend if you used this feature? If I issue the list volumes command in console I got this: +-+---+---++--+--+-+--+---+-+-+ | MediaId | VolumeName| VolStatus | VolBytes | VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten | +-+---+---++--+--+-+--+---+-+-+ | 3 | ServersDiff01 | Used | 51,590,350,000 | 12 | 432,000 | 1 |0 | 1 | ServersFile | 2006-11-17 00:24:09 | | 4 | ServersDiff02 | Append| 33,334,244,961 |7 | 432,000 | 1 |0 | 1 | ServersFile | 2006-11-22 23:01:41 | +-+---+---++--+--+-+--+---+-+-+ What does VolFiles mean? I didn't find it in the documentation. I've been wondering about that. I wish one of the developers would chime in here. I seem to remember that a trick used on tape drives is to write an EOF marker every so often, then when restoring, the drive can quickly seek X EOF markers ahead before it has to slow down to read through the data. If I'm understanding this correctly, there's no reason Bacula can't do the same thing with file volumes. If it writes an EOF marker every 4G (which it seems to, based on your output) it can seek() to the within 4G of the data it needs, then it only needs to read() through a maximum of 4G to get the data. Again, I wish a developer would chime in and comment on whether I'm correct or not. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Restore: Building directory tree takes forever
On Sat, 18 Nov 2006 11:47:53 +0100 Frank Mueller [EMAIL PROTECTED] wrote: Hi there, memory usage is at 60MB out of 1GB. No swapping at all. By the way. Shouldn't it be possible to edit the database to delete some File entries to make things work? I already took a look at the db with phpmyadmin, but I do not get the db structure, coz I do not know much about mysql and bacula. I know that there's a very huge directory in that contains over 2 Mio SPAM-Mails and I think completely deleteing that directory might be the solution for my prob. Can anyone lead me to the correct entries to delete from the db? Absolutely do a dump of the database before you try anything! However, you may be on to something. Long term you really need to find the performance problem and get it fixed, but you may be able to speed things up some short term. Bacula keeps the pathnames and filenames in seperate tables from the records of what files were backed up. The following procedure should allow you to clean up the excess files: 1) SELECT pathid FROM path WHERE path='/path/to/files/you/want/to/delete' 2) DELETE FROM file WHERE pathid=the result from step 1 I take no responsibility if this deletes more than you want. Take a dump of the database before you try this. Restore the dump when you're done. The table structure is documented: http://bacula.org/developers/Catalog_Services.html#SECTION000101600 Good luck. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Preventing windows bacula-fd from accessing some files
In response to Jaime Ventura [EMAIL PROTECTED]: Hi Bill, thank you for your reply. I know that if install the windows client using default options, it will run as a service using the system user. Since bacula-fd is running using the system user, it will be able to access every file on the pc, rigth? I need to limit the bacula-fd to access to just a few files ( the ones I need to backup). The thing is, the pc user (AAA in my previous email) have personal information on it and he doesn't want bacula to able to access it. So I thought of running bacula-fd as a BBB user having permissions to access only files I need to backup. That way, bacula-fd doesnt have access to all files on the system, but just the ones I need to backup. This should work just fine. But, having bacula-fd running as BBB user, it wont have permissions to put the icon tray on the AAA systemTray when he is logged in. I could solve this problem if bacula had a bacula-monitor for windows, which it does not. So, i don't really know how to solve this :( Once again, thanks I have a feeling that you're on the wrong road for the wrong reasons. First, you need to clearly establish the status of those files: 1) Are they important data? 2) Are they confidential data? If the answer to #1 is yes, then you need to back them up. It sounds to me that the answer to #2 is yes and you're using that as a reason _not_ to back the data up, which is a _VERY_ bad idea. I could go on and on about foolish assumptions such as the assumption that your Windows client is more secure than the Bacula server ... However, I'll cut to the chase: 1) If the data is just personal, then simply exclude those files from the backup fileset using a wildcard if necessary. 2) If the data is confidential, then you're putting the cart before the horse by trying to teach bacula not to back it up. Instead, use some sort of file-level encryption, such as one of pgp's tools. Then, you can even back up the encrypted files if it makes sense to do so, without sacrificing their confidentiality. The approach you're taking is like trying to fix a flat tire by changing the spark plugs. -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Restore: Building directory tree takes forever
In response to Frank Mueller [EMAIL PROTECTED]: I'm having trouble restoring files from a backup job that contains 1,933,009 files. Building the directory tree seems to take forever... at least I had to interupt it after 30 hours of 100% CPU-usage :-( Wow ... Building directory tree for JobId 4868 ... +++ 1 Job, 1,316,538 files inserted into the tree. Took about 1 minute. I didn't see any excessive CPU usage during the build, but disks ran at 100% the whole time (loading the database data off disk). Fast hardware helps. This system is a Dell 1850 with 2G of RAM and SCSI 10,000 RPM disks. Are you sure it was CPU-bound during the build, and if so, what process was CPU bound? The DB server? The director? Also, I'm using PostgreSQL as the DB backend. I've got a bit of experience tuning Postgres, so I've made sure that it uses all the system RAM for caching. Is it possible your DB server would benefit from some tuning? -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Restore: Building directory tree takes forever
In response to Frank Mueller [EMAIL PROTECTED]: Hi there, ok, my machine here is not comparable to your hardware, but it still should work at least within minutes, not hours or even days I think. Some additional information: CPU: P4 1,7 GHZ RAM: 1GB ATA-Harddisks OS: FreeBSD DB: Mysql 4.0.27 bacula: 1.38.11_1 While building the directory tree, top tells me CPU states: 98.9% user, 0.0% nice, 1.1% system, 0.0% interrupt, 0.0% idle 5463 bacula 3 118 0 14540K 12124K RUN 1:14 96.68% bacula-dir 457 mysql 6 20 0 57580K 32720K kserel 5:15 0.29% mysqld Looks like the director is using all of CPU, not the database, while it only takes about 60MB of RAM. I already checked the db indexes and found em all in place... I have no idea what the hell's going wrong on that machine. I'm no MySQL expert, but isn't there something that needs to go in the my.cnf or whatever in order for MySQL to operate efficiently on large data sets? -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Job not appearing in console
In response to Luke Taylor [EMAIL PROTECTED]:
Hi I have made the amendments but still no joy... The file now contains the
following for the two clients not showing up.
Have you tried starting the director in the foreground with debugging?
JobDefs {
Name = Autonet
Type = Backup
Level = Full
Client = jspautonet-fd
FileSet = WebserverSet
Schedule = jspautonet-fd
Storage = File
Messages = Standard
Pool = autonetpool
Priority = 10
}
Job {
Name = AutonetServer
JobDefs = Autonet
Type = Backup
Level = Full
Client = jspautonet-fd
FileSet = WebserverSet
Schedule = jspautonet-fd
Write Bootstrap = /usr/local/bacula/bin/working/jspautonet.bsr
}
FileSet {
Name = WebserverSet
Include {
Options {
}
File = /usr/local/apache/conf/
File = /usr/local/apache/conf/ssl/
File = /usr/local/tomcat/webapps/
File = /usr/local/tomcat/conf/Catalina/localhost/
File = /usr/local/etc/tomcat/
}
}
Schedule {
Name = jspautonet-fd
Run = Level=Incremental Pool=autonetpool Monday at 20:30
Run = Level=Incremental Pool=autonetpool Tuesday at 20:30
Run = Level=Incremental Pool=autonetpool Wednesday at 20:30
Run = Level=Incremental Pool=autonetpool Thursday at 20:30
Run = Level=Incremental Pool=autonetpool Friday at 20:30
}
Client {
Name = jspautonet-fd
Address = 10.210.17.177
FDPort = 9102
Catalog = MyCatalog
Password =
File Retention = 30 days
Job Retention = 6 months
AutoPrune = yes
}
Pool {
Name = autonetpool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Accept Any volume = yes
}
And
JobDefs {
Name = MXdev
Type = Backup
Level = Full
Client = mxdev-fd
FileSet = Mxdev Set
Schedule = mxdev-fd
Storage = File
Messages = Standard
Pool = mxdevpool
Priority = 10
}
Job {
Name = Backupmxdev
JobDefs = MXdev
Type = Backup
Level = Full
Client = mxdev-fd
FileSet = Mxdev Set # Luke desktop
Schedule = mxdev-fd
Write Bootstrap = /usr/local/bacula/bin/working/mxdev.bsr
}
FileSet {
Name = Mxdev Set
Include {
Options {
}
File = /usr/local/luke
}
}
Schedule {
Name = mxdev-fd
Run = Level=Incremental Pool=mxdevpool Monday at 20:15
Run = Level=Incremental Pool=mxdevpool Tuesday at 20:15
Run = Level=Incremental Pool=mxdevpool Wednesday at 20:15
Run = Level=Incremental Pool=mxdevpool Thursday at 20:15
Run = Level=Incremental Pool=mxdevpool Friday at 20:15
}
Client {
Name = mxdev-fd
Address = 10.26.13.112
FDPort = 9102
Catalog = MyCatalog
Password =
File Retention = 30 days# 30 days
Job Retention = 6 months# six months
AutoPrune = yes # Prune expired Jobs/Files
}
Pool {
Name = mxdevpool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Accept Any Volume = yes
}
All that appears in the bconcole is:
*run
Using default Catalog name=MyCatalog DB=bacula
A job name must be specified.
The defined Job resources are:
1: Client1
2: Twiki Server
3: Backupwindows
4: BackupCatalog
5: RestoreFiles
Select Job resource (1-5):
HOWEVER, if I remove the Twiki Server job frrom the director configuration
file the Autonet job appears (there is still no sign of the mxdev job) and
completes with no errors. If I remover the Autonet and Twiki jobs the mxdev
job appears on the list.it looks to me like there is a setting somewhere
on the system that defines how many jobs can be displayed on that list but I
can't find anything anywhere.
Any further assistance would be much appreciated.
Regards
Luke
-Original Message-
From: Bill Moran [mailto:[EMAIL PROTECTED]
Sent: 15 November 2006 14:38
To: [EMAIL PROTECTED]
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Job not appearing in console
In response to Luke Taylor [EMAIL PROTECTED]:
Thanks for replying, FYI the jobs don't have the same name but below
is the directors configuration file excluding passwords.
FYI I have since addded another job resource which also doesn't appear
on the list.the jobs in question are hignlighted in boldbut
just incase it doesn't get displayed I have put the job resource just
below.
Please don't rely on HTML to communicate for you. The bold doesn't display
on my MUA, and I had to do a good bit of fscking around to figure out which
jobs you were having trouble with. Do I have the correct two?
Job {
Name = Twiki Server
JobDefs = Twiki
Type = Backup
Level = Full
Client = twiki-fd
FileSet = Twiki Set
Schedule = twiki-fd
Write Bootstrap = /usr/local/bacula/bin/working/twiki.bsr
}
Job {
Name = Autonet Server
JobDefs = Autonet
Type = Backup
Level = Full
Client = jspautonet
Re: [Bacula-users] Job not appearing in console
In response to Luke Taylor [EMAIL PROTECTED]:
I tried with ./bacula start -f -100 but don't get anything, I am having a
look around to find out how to do it.
I will update further if/when I manage to find instructions.
bacula-dir -f -d100
(You may have to give -c if it doesn't find the config file)
If you do bacula-dir --help, it'll give you a list of options. Not
because it understands --help, but because it does that when it doesn't
understand its options.
-Original Message-
From: Bill Moran [mailto:[EMAIL PROTECTED]
Sent: 16 November 2006 13:58
To: [EMAIL PROTECTED]
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Job not appearing in console
In response to Luke Taylor [EMAIL PROTECTED]:
Hi I have made the amendments but still no joy... The file now
contains the following for the two clients not showing up.
Have you tried starting the director in the foreground with debugging?
JobDefs {
Name = Autonet
Type = Backup
Level = Full
Client = jspautonet-fd
FileSet = WebserverSet
Schedule = jspautonet-fd
Storage = File
Messages = Standard
Pool = autonetpool
Priority = 10
}
Job {
Name = AutonetServer
JobDefs = Autonet
Type = Backup
Level = Full
Client = jspautonet-fd
FileSet = WebserverSet
Schedule = jspautonet-fd
Write Bootstrap = /usr/local/bacula/bin/working/jspautonet.bsr
}
FileSet {
Name = WebserverSet
Include {
Options {
}
File = /usr/local/apache/conf/
File = /usr/local/apache/conf/ssl/
File = /usr/local/tomcat/webapps/
File = /usr/local/tomcat/conf/Catalina/localhost/
File = /usr/local/etc/tomcat/
}
}
Schedule {
Name = jspautonet-fd
Run = Level=Incremental Pool=autonetpool Monday at 20:30
Run = Level=Incremental Pool=autonetpool Tuesday at 20:30
Run = Level=Incremental Pool=autonetpool Wednesday at 20:30
Run = Level=Incremental Pool=autonetpool Thursday at 20:30
Run = Level=Incremental Pool=autonetpool Friday at 20:30 }
Client {
Name = jspautonet-fd
Address = 10.210.17.177
FDPort = 9102
Catalog = MyCatalog
Password =
File Retention = 30 days
Job Retention = 6 months
AutoPrune = yes
}
Pool {
Name = autonetpool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Accept Any volume = yes
}
And
JobDefs {
Name = MXdev
Type = Backup
Level = Full
Client = mxdev-fd
FileSet = Mxdev Set
Schedule = mxdev-fd
Storage = File
Messages = Standard
Pool = mxdevpool
Priority = 10
}
Job {
Name = Backupmxdev
JobDefs = MXdev
Type = Backup
Level = Full
Client = mxdev-fd
FileSet = Mxdev Set # Luke desktop
Schedule = mxdev-fd
Write Bootstrap = /usr/local/bacula/bin/working/mxdev.bsr
}
FileSet {
Name = Mxdev Set
Include {
Options {
}
File = /usr/local/luke
}
}
Schedule {
Name = mxdev-fd
Run = Level=Incremental Pool=mxdevpool Monday at 20:15
Run = Level=Incremental Pool=mxdevpool Tuesday at 20:15
Run = Level=Incremental Pool=mxdevpool Wednesday at 20:15
Run = Level=Incremental Pool=mxdevpool Thursday at 20:15
Run = Level=Incremental Pool=mxdevpool Friday at 20:15 }
Client {
Name = mxdev-fd
Address = 10.26.13.112
FDPort = 9102
Catalog = MyCatalog
Password =
File Retention = 30 days# 30 days
Job Retention = 6 months# six months
AutoPrune = yes # Prune expired Jobs/Files
}
Pool {
Name = mxdevpool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Accept Any Volume = yes
}
All that appears in the bconcole is:
*run
Using default Catalog name=MyCatalog DB=bacula A job name must be
specified.
The defined Job resources are:
1: Client1
2: Twiki Server
3: Backupwindows
4: BackupCatalog
5: RestoreFiles
Select Job resource (1-5):
HOWEVER, if I remove the Twiki Server job frrom the director
configuration file the Autonet job appears (there is still no sign of
the mxdev job) and completes with no errors. If I remover the Autonet
and Twiki jobs the mxdev job appears on the list.it looks to me
like there is a setting somewhere on the system that defines how many
jobs can be displayed on that list but I can't find anything anywhere.
Any further assistance would be much appreciated.
Regards
Luke
-Original Message-
From: Bill Moran [mailto:[EMAIL PROTECTED]
Sent: 15 November 2006 14:38
To: [EMAIL PROTECTED]
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Job not appearing in console
In response to Luke Taylor [EMAIL PROTECTED]:
Thanks for replying, FYI the jobs don't have
Re: [Bacula-users] Job not appearing in console
In response to Luke Taylor [EMAIL PROTECTED]:
Thanks for replying, FYI the jobs don't have the same name but below is the
directors configuration file excluding passwords.
FYI I have since addded another job resource which also doesn't appear on
the list.the jobs in question are hignlighted in boldbut just incase
it doesn't get displayed I have put the job resource just below.
Please don't rely on HTML to communicate for you. The bold doesn't
display on my MUA, and I had to do a good bit of fscking around to
figure out which jobs you were having trouble with. Do I have the
correct two?
Job {
Name = Twiki Server
JobDefs = Twiki
Type = Backup
Level = Full
Client = twiki-fd
FileSet = Twiki Set
Schedule = twiki-fd
Write Bootstrap = /usr/local/bacula/bin/working/twiki.bsr
}
Job {
Name = Autonet Server
JobDefs = Autonet
Type = Backup
Level = Full
Client = jspautonet-fd
FileSet = Webserver Set
Schedule = jspautonet-fd
Write Bootstrap = /usr/local/bacula/bin/working/jspautonet.bsr
}
I looked at this yesterday and couldn't find anything wrong, then looked
at it again this morning and still can see anything obviously wrong.
So I'm guessing. I don't believe that spaces in job names would cause
any problems, but it's the only thing I can see that stands out. Trying
removing the spaces from your various names (JobDefs, Job, FileSet) and
see if the problem persists. Hope this isn't a wild goose chase for
you, but I don't have any other ideas.
Another good idea would be to start the director in the foreground with
debugging turned on and see if it reports anything helpful. See the
docs for the details on how to do this.
--
Bill Moran
Collaborative Fusion Inc.
IMPORTANT: This message contains confidential information and is intended only
for the individual named. If the reader of this message is not an intended
recipient (or the individual responsible for the delivery of this message to an
intended recipient), please be advised that any re-use, dissemination,
distribution or copying of this message is prohibited. Please notify the
sender immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Job not appearing in console
In response to luket [EMAIL PROTECTED]: Good morning I have installed bacula-1.38.11 onto a CentOS server. It is installed and appears to be running fine and backing up to disk with no problems. I have added 3 clients: A Windows client, and 2 linux clients one of which is on a seperate (internal) network with firewall permissions granted for ports 9001, 9002 9003. We I enter bconsole and type run I get the list of clients but one is missing. I have checked through my configuration files and all looks OK to me. When I take out the working linux job and restart bacula the other one appears and runs through fine so I know it isn't a problem with the clients setup. Sounds like those two jobs are cancelling each other out. They don't share the same name, do they? I have looked around and can't find anything but was wondering if there was a setting which specifies how many will appear on the list. I did have Bacula installed on another server for testing purposes, this was on CentOS also with the same version of Bacula, same ./configure command, and everything I can see is fine. Does anybody know of any settings that need to be changed in order for all the jobs to appear on the list? I'm pretty sure bconsole always lists all jobs, but I can guarantee that it will show 26 minimum, as that's how many I have on one server here. I would guess that you've got some obscure error in your config file. Strip the passwords out of it and paste it into a reply to this thread. We probably only need to the director's config to diagnose this. -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Is Bacula for me: Videography business backups
On Wed, 08 Nov 2006 18:49:30 -0600 [EMAIL PROTECTED] wrote: Arno Lehmann [EMAIL PROTECTED] writes: Currently, the windows FD is running stable and reliable IMO. Unfortunately, I have some problems with VSS backups (windows' name for volume snapshots) where files can't be accessed when I do a snapshot but can be accessed without VSS. For backups of data files not in use during backup operations, this should not be a problem. I'm not sure I'm following you here. Are you saying I would not be able to backup files that were in use? Or just not in some special circumstances...? I'm not sure what role VSS plays or when it might be used but I would need to routinely be able to backup files that are in use. Depends on your definition of in use. If the system has the file locked for writing, you'll have to enable VSS to get a backup of it while it's write locked. If the application has it open read-only, you'll get a good backup either way. On the flip side, I've been using VSS to back up Windows workstations for a few months now with no problems, and all files are able to be backed up no matter how they're opened. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Why does bacula not believe that it's week 44, and how do I figure out what's wrong?
Bacula seem not to understand that today is week 44 of the year. It seemed to understand that last week was week 43, as it ran the job that was scheduled for that week. Here's the schedule that applies: Run = Level=Full Pool=OffSite Storage=Ultrium thursday w03,w07,w11,w15,w19,w23,w27,w31,w35,w39,w43,w47,w51 at 23:05 Run = Level=Full Pool=OffSiteRotation Storage=Ultrium thursday w01,w02,w04,w05,w06,w08,w09,w10,w12,w13,w14,w16,w17,w18,w20,w21,w22,w24,w25,w26,w28,w29,w30,w32,w33,w34,w36,w37,w38,w40,w41,w42,w44,w45,w46,w48,w49,w50,w52,w53 at 23:05 Server date/time seems correct: *time 03-Nov-2006 09:16:47 Bacula insists that it needs a volume from the OffSite pool. As a result, 240G of backups failed to run last night, and I'm having to manually run/modify them this morning (because the tape I need to use is part part of the OffSiteRotation pool, in accordance with our schedule). Is there something wrong with my schedule that I'm simply not seeing? A much more important question to me is How do I diagnose this? How can I peek in to Bacula and see what week it thinks it is? bconsole's time command isn't very helpful on this count. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Can't write to the tape
In response to Greg Little [EMAIL PROTECTED]: Here is where I am at now. I can use mt and tar and dump to the tape, but bconsole shows: Device status: Autochanger turing with devices: LTO-2 (/dev/nst0) Device FileStorage (/bacula/FileVol) is not open or does not exist. Device LTO-2 (/dev/nst0) open but no Bacula volume is mounted. Slot 6 is loaded in drive 0. Total Bytes Read=0 Blocks Read=0 Bytes/block=0 Positioned at File=0 Block=0 When I tried to spool to tape, it asks to mount the volume, although the tape is already in there and I can write to it with mt and tar. If you wrote to the tape with tar _after_ labelling it, then you destroyed the Bacula label. You'll have to relabel the media before you can use it. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Irregular backups
In response to Paul Constable [EMAIL PROTECTED]: Hi alll, I have the need to backup a laptop that is connected to the network at irregular intervals, which precludes any form of scheduling. Does anyone have a means of backing up at, say an hour after logging into the network, or of connecting to the server. I have a user who is frequently away from the network and never leaves the laptop connected to the network overnight. Therefore I need to stimulate the backup automatically without the users intervention or mine. The laptop runs windoze, so I need the means of triggering a script, etc. I've managed to work around this with clever scheduling. By setting reschedule on error to 1 hour, setting the backup to start at 9:00 AM and setting the system to reschedule 9 times, we get most of our mobile users most of the time that they are in. The upshot is that it tries to initiate a backup every hour between 9:00 AM and 6:00 PM. Not the perfect solution, but it's working OK for us. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Recycle a volume by deleting the datas ?
In response to ctobini [EMAIL PROTECTED]: Hello, It seems that a volume life time in bacula is in fact its life time in the catalog database. So, I would like to know is there would be possible to fully recycle a volume (restart to the beginning of the volume by deleting the datas). You can set retention times and configure the volumes to automatically recycle and Bacula will take care of it for you. Or you can use the purge or prune commands from bconsole, depending on your config. Some reading on these topics in the Bacula manual should help. If you get confused, ask. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula: backup is slow
In response to Anders Boström [EMAIL PROTECTED]: gzip on this computer, on one CPU, reach about 18 Mbyte/s. bacula with gzip only reach ~7.7 Mbyte/s. This leads me to believe that there are room for improvement. BM Again, the story changes. Above, you indicate that tar+gzip ran about BM 15% faster than bacula with gzip, which seems reasonable. Now you're BM saying that gzip is ~twice as fast as Bacula + gzip. Where did this new BM number come from? Are you taking in to account networking on this new BM test? If I state that gzip on this computer, on one CPU, reach about 18 Mbyte/s, I mean just that, nothing else. To clarify, this means that pure gzip-performance on this computer, using just one gzip-process, is 18 Mbyte/s. If you would be kind enough to humor me ... Please create a file (or use an existing one) of notable size: few hundred meg. Put the file on the disk and time gzipping it. Run it 5 times. Create a memory filesystem and repeate the gzip tests with the file living on the mfs and the gzipped target existing on the mfs. I have a suspicion that your drives are the limiting factor in this. The above tests should confirm or deny that theory. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula: backup is slow
In response to Anders Boström [EMAIL PROTECTED]: BM == Bill Moran [EMAIL PROTECTED] writes: BM In response to Anders Boström [EMAIL PROTECTED]: gzip on this computer, on one CPU, reach about 18 Mbyte/s. bacula with gzip only reach ~7.7 Mbyte/s. This leads me to believe that there are room for improvement. BM Again, the story changes. Above, you indicate that tar+gzip ran about BM 15% faster than bacula with gzip, which seems reasonable. Now you're BM saying that gzip is ~twice as fast as Bacula + gzip. Where did this new BM number come from? Are you taking in to account networking on this new BM test? If I state that gzip on this computer, on one CPU, reach about 18 Mbyte/s, I mean just that, nothing else. To clarify, this means that pure gzip-performance on this computer, using just one gzip-process, is 18 Mbyte/s. BM If you would be kind enough to humor me ... BM Please create a file (or use an existing one) of notable size: few BM hundred meg. BM Put the file on the disk and time gzipping it. Run it 5 times. BM Create a memory filesystem and repeate the gzip tests with the BM file living on the mfs and the gzipped target existing on the mfs. BM I have a suspicion that your drives are the limiting factor in this. BM The above tests should confirm or deny that theory. I've already done this, but did it again. The results are the same, disc and tmpfs gives the same results, about 18 Mbyte/s. And I would have been *very* surprised if they had differed, as the disc-tests are running out of memory in test 2-5 (due to caching). If the system is caching the entire write operation in RAM, then you're set up for a major disaster some day. Even if there's enough RAM to cache the read op, the write op will ALWAYS require disk writes. If those two operations are taking the same amount of time, then either you're testing wrong, or something is really weird with your setup, or you have the fastest hard drives on this planet. Additionally, the point was to _not_ allow it to use cache for the disk ops, which I didn't communicate clearly -- my apologies. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula: backup is slow
In response to Anders Boström [EMAIL PROTECTED]: AL == Arno Lehmann [EMAIL PROTECTED] writes: AL Still the network is being used and that always involves latencies, AL syncronization times, etc. Yes, and that might be the problem. But if it is about latencies and/or synchronization, then it is a bacula performance problem! AL No, what I'm talking about is network fundamentals. Whenever you send AL data across a network that takes time, and it takes more time than AL dividing xMBit/s by the amount of data. Always. Are you talking about the long-fat-pipe problem? This isn't an issue for us... Linux has a very good TCP-implementation, with window-scaling, and we are running over a local GE-network with less than 50us latency. It's not the LFP problem. First off, every TCP packet adds overhead to your data, which means 1G/sec is really only about 600mb/sec in practical measurements. Secondly, that's 50us per packet. Which means there's work to be done on both ends to normalize packet flow, ack, retransmit lossed packets, etc. I believe someone asked you to double-check that all speeds and duplexes are matched, as that's a _very_ common problem (especially, in my experience, with gig HW). If you responded, I missed it, but I recommend that you double-check. However, also read on ... Is bacula limited in performance due to high latency? (Not that we have that problem, but anyway...) Is bacula limited in performance due to synchronization? AL Networks are limited by several factors. That's not something you can AL fix, and network throughput is not normally the most limiting factor in AL a Bacula setup. The specific issue here is about bacula. Has bacula any specific limitations regarding the network? Please be specific, and don't just generalize. Well, we're off on the wrong foot, in my opinion. There's no indication anywhere that networking is a problem. You even said that you were seeing indications in network traces that network traffic was flowing fine. As a result, I'm a bit confused as to why there's so much focus on the network, but I expect it's a result of the _huge_ number of people who assume that everything is fine with the network, then blame the application, only to later find out that there's something seriously hosed with their network. I've fallen into that trap, but I don't think you have in this case. I'm going to address what I _think_ is the problem in a response to your other email ... -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula: backup is slow
In response to Anders Boström [EMAIL PROTECTED]: I did some new performance-tests: All operations are against a directory-tree with 7,255,659,224 bytes data in 98,025 files. | test1 | test2 | test3 | +---+---+---+ bacula-fd, no compression, md5: | 10:25 | 10:42 | 10:15 | bacula-fd, GZIP, md5: | 16:09 | 15:46 | 17:02 | tar, local (1): | 8:37 | 8:53 | 8:54 | tar + nc (2): | 9:48 | 9:52 | 9:43 | tar + gzip + nc (3): | 14:11 | 14:26 | 15:03 | +---+---+---+ OK. This indicates to me that bacula is doing a damn good job. Only 15% overhead to add checksumming and cataloging features to backup. If you ask me, that's a hell of a deal. (1) time /bin/sh -c tar cf - directory | cat /dev/null (2) time /bin/sh -c tar cf - directory | nc -q 0 backup_server 4711 (3) time /bin/sh -c tar czf - directory | nc -q 0 backup_server 4711 This round of tests is more in line with what I expected, and the bacula performance is quite good. The only major difference compared to my previous tests is that the file-server disc-performance is much better. It seems like bacula suffers much more than tar from slow disc-performance on the file-server. backup-server and network performance don't seem to be an issue at all in the tests, even if write to TCP is a bit slower than /dev/null. However, both tar and bacula suffers from quite large slow-down when gzip is used. This is on an Athlon 64 X2 3800+ (2-core), running 50% idle during backup, leading me to believe that there are room for improvement. But part of the problem might be in the linux-kernel (2.6.17.8). At least when tar was running, the gzip process seemed to move from one CPU-core to the other very frequently. Improvement, maybe, but not for Bacula, as far as I can see. If a dual-core system is running at 50%, then 1 core is maxed out. Since the gzip process is serialized, it can only run on one core at a time, which means the CPU is the limiting factor at this time. If you really need more speed, try using GZIP1 or turning compression off. I suppose it'd be possible to make the FD parallelize the compression, but doing that is beyond my skills, and I'm not sure you'd want it anyway. I mean, do you want a backup system that can totally monopolize all your CPU and starve other applications? Anyway the performance of bacula is good enough for us at the moment. Well ... I'm so glad you're please. What was your performance goal anyway? If you actually thought you'd get backup throughput at wire speed on 1g network, that was your first mistake. I don't know of any disks that can feed data that fast. Hell, from your experiment above, those disks can feed data at about 13M/sec, which is closer to 100mb than gig, and that's the absolute fastest you're going to get. I can share one experience with you regarding disc-performance: Both our two Seagate ST3500641AS discs (500Gb Barracuda 7200.9, SATA) never completes the SMART extended self-test. It worked fine for ~6 months, and now both run forever. The drives report 30% remaining of the self-test, then ~2 hours later 10% remaining. After that it goes up to 40% remaining and the cycle repeats. Also when running the SMART extended self-test, the IO-performance is more than 10 times lower than normal (leading to very long backup-time). We have been in contact with Seagate about this, and have upgraded to the latest firmware, without any success. So I guess we will have to RMA the discs. Are these the disks you're using to test Bacula? Please tell me you're not using hardware known to be broken as a test bed. -- Bill Moran Collaborative Fusion Inc. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula: backup is slow
In response to Anders Boström [EMAIL PROTECTED]: BM == Bill Moran [EMAIL PROTECTED] writes: BM In response to Anders Boström [EMAIL PROTECTED]: I did some new performance-tests: All operations are against a directory-tree with 7,255,659,224 bytes data in 98,025 files. | test1 | test2 | test3 | +---+---+---+ bacula-fd, no compression, md5: | 10:25 | 10:42 | 10:15 | bacula-fd, GZIP, md5: | 16:09 | 15:46 | 17:02 | tar, local (1): | 8:37 | 8:53 | 8:54 | tar + nc (2): | 9:48 | 9:52 | 9:43 | tar + gzip + nc (3): | 14:11 | 14:26 | 15:03 | +---+---+---+ BM OK. This indicates to me that bacula is doing a damn good job. Only BM 15% overhead to add checksumming and cataloging features to backup. BM If you ask me, that's a hell of a deal. (1) time /bin/sh -c tar cf - directory | cat /dev/null (2) time /bin/sh -c tar cf - directory | nc -q 0 backup_server 4711 (3) time /bin/sh -c tar czf - directory | nc -q 0 backup_server 4711 This round of tests is more in line with what I expected, and the bacula performance is quite good. The only major difference compared to my previous tests is that the file-server disc-performance is much better. It seems like bacula suffers much more than tar from slow disc-performance on the file-server. backup-server and network performance don't seem to be an issue at all in the tests, even if write to TCP is a bit slower than /dev/null. However, both tar and bacula suffers from quite large slow-down when gzip is used. This is on an Athlon 64 X2 3800+ (2-core), running 50% idle during backup, leading me to believe that there are room for improvement. But part of the problem might be in the linux-kernel (2.6.17.8). At least when tar was running, the gzip process seemed to move from one CPU-core to the other very frequently. BM Improvement, maybe, but not for Bacula, as far as I can see. If a BM dual-core system is running at 50%, then 1 core is maxed out. Since No, this was not the case. Most of the time, both CPU's was idle 30% of the time, according to top. It's hard to help when your facts keep changing. Further up, you mention that the CPU is 50% idle, now you say that each independent core is 30% idle. This is not meant as an attack, but you will get absolutely nowhere in performance optimization if you can't get the details right. Details are terribly important. BM the gzip process is serialized, it can only run on one core at a time, BM which means the CPU is the limiting factor at this time. gzip on this computer, on one CPU, reach about 18 Mbyte/s. bacula with gzip only reach ~7.7 Mbyte/s. This leads me to believe that there are room for improvement. Again, the story changes. Above, you indicate that tar+gzip ran about 15% faster than bacula with gzip, which seems reasonable. Now you're saying that gzip is ~twice as fast as Bacula + gzip. Where did this new number come from? Are you taking in to account networking on this new test? One thing to consider is that network has an impact, even when the physical speed of the network is faster than the gzip process. There is extra processing involved in packing up the packets, and delays (even if minimal) introduced by networking add latency to the process that isn't always intuitive. BM What was your performance goal anyway? If you actually thought you'd BM get backup throughput at wire speed on 1g network, that was your first BM mistake. I don't know of any disks that can feed data that fast. Hell, BM from your experiment above, those disks can feed data at about 13M/sec, BM which is closer to 100mb than gig, and that's the absolute fastest BM you're going to get. No, I don't expect better performance than the disc-performance. In fact, a lot lower than disc-performance is acceptable. The 7.7 Mbyte/s we reach now is OK. However, full backup times are long, but as long as we can run a full backup in less than 48 hours, it's OK. Again, I'm glad it's working for you. However, as someone who would otherwise be interested in researching this, I don't have any motivation to do so. full backup times are long sounds arbitrary enough to disinterest me. I can share one experience with you regarding disc-performance: Both our two Seagate ST3500641AS discs (500Gb Barracuda 7200.9, SATA) never completes the SMART extended self-test. It worked fine for ~6 months, and now both run forever. The drives report 30% remaining of the self-test, then ~2 hours later 10% remaining. After that it goes up to 40% remaining and the cycle repeats. Also when running the SMART extended self-test, the IO-performance is more than 10 times lower than normal (leading to very long backup-time). We have been
Re: [Bacula-users] Poll - What operating systems do you run Bacula on?
In response to Peter L. Buschman [EMAIL PROTECTED]: All: If it isn't too much of an imposition, I'd like to survey the list and ask the question what operating system are you running Bacula on?. I'm interested in which OS distributions, versions and platforms are being deployed as Bacula servers. 4 systems running Bacula on FreeBSD 6.x using PostgreSQL as the backend. If you want to know where FDs are running, that's a bit of a larger topic. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and Softlinks
In response to Janco van der Merwe [EMAIL PROTECTED]: Hi, Before wasting valuable time in testing can someone tell me how Bacula will react with softlinks? Is it worth going that route? To give you an idea on the one server it has 3 partitions and the aim is to backup the last 3 months of data but the twist is that the directories and partitions of where the data is will change from month to month. What we thought is creating a backup dir on the / partition and then a script that creates softlinks to the required directories within the /backup dir which brings me to the above question will Bacula follow the softlinks? Please wrap your lines around 72 characters. Current versions of Bacula don't follow symlinks. There's been some discussion about how to add a follow symlinks config option, but I don't know what the current status is on that. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula: backup is slow
In response to Ryan Novosielski [EMAIL PROTECTED]: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Though there are probably 10-20 performance pitfalls, the two big problems of performance that I have seen are: - Poorly tuned Catalog database -- insertion of Bacula attributes in the database tends to be slow. There are probably 5 or ten reasons leading to poor DB performance. I'll be working on improving this and documenting it over the next 6-9 months. A good part of what you can do is written in the manual (Catalog Maintenance chapter). The rest appeared on this list within the last month. - A switch (mostly 3Com switches in my experience) that run in half-duplex mode, which slows network traffic down by about a factor of 10. Cisco does this just as often, if not more often. A little surprising to find that the top 2 can't seem to compete on the same level as a D-Link switch from Radio Shack. ;) If you read Cisco's docs, they make the claim that these problems are per-spec. My understanding of the argument is that if you manually set the speed and duplex, you have disabled auto-negotiation. If the other end tries to auto-negotiate, it will be able to detect the speed, just by dumb luck of how the protocol works, but it will _consistently_ mis- detect the duplex, again because of dumb luck of the protocol Their argument seems to be that this behaviour is per the specs. If a D-Link does it differently, then D-Link is doing it wrong, even if it's doing it more intuitively. If Cisco is correct, then it would seem as if the spec were written poorly. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula clients and OpenVPN?
[EMAIL PROTECTED] wrote: Hi Frank, Thanks for sharing this with me as it is very useful and will allow me to possibly remove the OpenVPN part from the design that we are considering since Bacula seems to be able to take care of this matter for us as a built in feature. I guess that I still have the major question of being able to traverse firewalls and routers since as you have mentioned that the Bacula server initiates the connections to the clients and the client software does not initiate any connection to the server although I think that would be a nice feature as well. If you're traversing firewalls, it sounds like you need to establish 1 VPN per firewall, then use routing to route the Bacula services through the VPN. Bacula's TLS isn't going to do that for you. Is there some reason why you can't just establish a persistent VPN between the Bacula server and the firewall and run the jobs across it? -- Bill Moran When I point out limitations of one technique as a motivation for another, I do so in the context of specific problems; for different problems or in other contexts, the first technique may indeed be the better choice. Useful software has been constructed using all of the techniques presented here. Bjarne Stroustrup, _The_C++_Programming_Language_ - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Binding to source IP address
In response to James Ray [EMAIL PROTECTED]: We have a concept of a system address (the machine itself) and a service address (one for each service, say .2 == an apache service, .3 == an bacula service, .4 == a dns service, where as .1 is a _system_ address) all the interfaces will be on the same machine. So any communications coming _from_ the bacula service need to come out of .3 and any connections to the bacula service need to head onto .3 also. What I have seen (only on my breif testing) is that the listen address is the .3 address (as I set in DirAddress) but the outcoming connections from the bacula-dir to the bacula-fd across the network originate for the system default address of .1... I shall re-test and post my outcomes. I just want to quickly chime in that this is of interest to us as well, since we take the same approach. (i.e. each machine has a management address and another address for each service it provides) The primary reason for this is to help simplify firewall rules. We're in a high-security situation here, so our firewalls default to deny (even from one server to another). Each communication must be explicitly allowed. A secondary reason for this is planning for expansion. Some servers run many services (DNS, NFS, LDAP, etc). In the event that we should split the aforementioned machine into two systems (i.e. put DNS LDAP on one, and NFS on the second) we don't have to change any firewall rules or configurations on other machines, because the service IPs can follow the services. I haven't had time to investigate whether the [FD|SD|DIR]Address sets both the listening and the outgoing address, but a firewall audit is on the TODO list, and when I finally get to it, I'll have to address this for a number of services, not only Bacula. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Problem starting bacula-dir with Enable VSS
Just a me too. I can verify that I'm having the same problem with
a 1.38.10 director on FreeBSD (installed from ports).
In response to Diego Rozzini Pires [EMAIL PROTECTED]:
I' new with bacula and i'm having some difficults in enable VSS for
Windows 2003.
I'm using Gentoo, kernel 2.6.17-gentoo-r7 and Bacula Version 1.36.3 on
server, and bacula-fd 1.38.10 on Windows 2003 Server.
When I open the tray icon on Windows i have this:
servername-fd Version: 1.38.10 (08 June 2006) VSS Windows Server 2003
MVS NT 5.2.
I already started VSS in the Windows server using: net start vss. And
also restarted the FD.
My FileSet are this:
FileSet {
Name = Promedonserver2
Enable VSS = yes
Include {
Options {
Compression=GZIP6
signature = MD5
}
File = d:/backup
}
}
When i try to start bacula-dir i got this message:
backupserver bacula # /etc/init.d/bacula restart
* Starting bacula storage daemon ...
* Starting bacula file daemon ...
* Starting bacula director ...
19-Sep 18:21 bacula-dir: ERROR TERMINATION at parse_conf.c:821
Config error: Keyword EnableVSS not permitted in this resource.
Perhaps you left the trailing brace off of the previous resource.
: line 184, col 13 of file /etc/bacula/bacula-dir.conf
Enable VSS = yes
I alredy verified my conf looking for some trailing brace open and tried
to use only one FileSet for test but i didn't found anything.
I've looked at google and bacula site but i didn't find anything too. If
i skipped something on google or bacula site, please let me know!
Thanks for help!
--
__
Diego Rozzini Pires
Consultor de Tecnologia Trainee
ZeniSys Tecnologia da Informação
[EMAIL PROTECTED]
Fone : 55 11 4425-2424
Móvel: 55 11 82083168
Visite : http://www.zenisys.com.br
__
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
Bill Moran
Collaborative Fusion Inc.
IMPORTANT: This message contains confidential information and is
intended only for the individual named. If the reader of this
message is not an intended recipient (or the individual
responsible for the delivery of this message to an intended
recipient), please be advised that any re-use, dissemination,
distribution or copying of this message is prohibited. Please
notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The
sender therefore does not accept liability for any errors or
omissions in the contents of this message, which arise as a
result of e-mail transmission.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Problem starting bacula-dir with Enable VSS
In response to Attila Fülöp [EMAIL PROTECTED]:
Enable VSS = yes
works for me. FreeBSD / 1.38.5 from ports.
FileSet {
Name = WindowsXP
Include {
Options {
# Enable VSS = Yes
# compression = GZIP1
}
File = C:/
}
}
If I uncomment the VSS line, I get:
27-Sep 13:38 backup-dir: ERROR in inc_conf.c:330 Config error: Keyword
EnableVSS not permitted in this resource
: line 114, col 18 of file /usr/local/etc/bacula-dir.conf
Enable VSS = Yes
This config:
FileSet {
Name = WindowsXP
Include {
# Enable VSS = Yes
Options {
# compression = GZIP1
}
File = C:/
}
}
Generates the same error. The error is the same whether I use the or
not.
Bill Moran wrote:
Just a me too. I can verify that I'm having the same problem with
a 1.38.10 director on FreeBSD (installed from ports).
In response to Diego Rozzini Pires [EMAIL PROTECTED]:
I' new with bacula and i'm having some difficults in enable VSS for
Windows 2003.
I'm using Gentoo, kernel 2.6.17-gentoo-r7 and Bacula Version 1.36.3 on
server, and bacula-fd 1.38.10 on Windows 2003 Server.
When I open the tray icon on Windows i have this:
servername-fd Version: 1.38.10 (08 June 2006) VSS Windows Server 2003
MVS NT 5.2.
I already started VSS in the Windows server using: net start vss. And
also restarted the FD.
My FileSet are this:
FileSet {
Name = Promedonserver2
Enable VSS = yes
Include {
Options {
Compression=GZIP6
signature = MD5
}
File = d:/backup
}
}
When i try to start bacula-dir i got this message:
backupserver bacula # /etc/init.d/bacula restart
* Starting bacula storage daemon ...
* Starting bacula file daemon ...
* Starting bacula director ...
19-Sep 18:21 bacula-dir: ERROR TERMINATION at parse_conf.c:821
Config error: Keyword EnableVSS not permitted in this resource.
Perhaps you left the trailing brace off of the previous resource.
: line 184, col 13 of file /etc/bacula/bacula-dir.conf
Enable VSS = yes
I alredy verified my conf looking for some trailing brace open and tried
to use only one FileSet for test but i didn't found anything.
I've looked at google and bacula site but i didn't find anything too. If
i skipped something on google or bacula site, please let me know!
Thanks for help!
--
__
Diego Rozzini Pires
Consultor de Tecnologia Trainee
ZeniSys Tecnologia da Informação
[EMAIL PROTECTED]
Fone : 55 11 4425-2424
Móvel: 55 11 82083168
Visite : http://www.zenisys.com.br
__
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
Bill Moran
Collaborative Fusion Inc.
IMPORTANT: This message contains confidential information and is
intended only for the individual named. If the reader of this
message is not an intended recipient (or the individual
responsible for the delivery of this message to an intended
recipient), please be advised that any re-use, dissemination,
distribution or copying of this message is prohibited. Please
notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The
sender therefore does not accept liability for any errors or
omissions in the contents of this message, which arise as a
result of e-mail transmission.
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business
Re: [Bacula-users] confused about Full vrs. Incremental backups pools
In response to Arno Lehmann [EMAIL PROTECTED]: [Problem] Here's the problem...as incremental backups expire and are purged, bacula will often promote the next incremental to a full backup, since it correctly determines that there are no full backups for a particular client in the Incremental pool. Writing a full backup can be disruptive to both the client and backup server, as some backups are over 2TB, with clients on a slow network. I want to avoid unscheduled full backups as a result of promoted incrementals, and I don't want to be doing full backups every 2 weeks to satisfy the retention period of the Incremental pool. Is there anyway to avoid this behavior? Keep the incremental backups for more than a month. You need an unbroken chain of incrementals, i.e. from the last full backup to the current date no incremental backup can be pruned. Not exactly true. Differentials can be used to consolidate incrementals. Assuming you make incrementals 6 days a week, and Sunday is for fulls and differentials, set retention on your incrementals to 6 days, differentials to 3 weeks. Then you'll always have enough data to perform an incremental without building a new full. That gives you the standard decreasing granularity with increasing age scheme that most people want. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula GZip question
In response to Vadim A. Umanski [EMAIL PROTECTED]: Здравствуйте, bacula-users. Dear colleagues, an advice needed. I've got a mail server, the BackUp solution is Bacula 1.38.9. I've got a lot of data on Accounts disk slice, and the mail server's CPU is not too strong (UltraSPARC II 450 MHz) and Full backup takes 3-4 times more when GZip option is on than when it's off. It seems to me that sending more data to network is easier for the server than compressing them and then sending less data. But the mail server data can be compressed approx. 50% and I wouldn't like to spend unneeded space ... Full backup would take about 80-90 GB of raw data. Can I configure backup so that data compression would be performed by Bacula server (CPU is quite strong and quite little loaded) and NOT by Bacula client? Please give me an advice! Current versions of Bacula don't support this. I don't know whether or not the upcoming release does or not. Have you tried using GZIP1 instead of GZIP? In my own experiments, I've noticed that GZIP1 uses noticeably less CPU than GZIP (which is actually GZIP6) and still accomplishes good compression. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and Postgres 8.1
In response to Magnus Hagander [EMAIL PROTECTED]: Sorry for not resending the needed informations, I already sent them when I first encountered the problem. But then I could solve this by moving the Director together with the DB (I had another Solaris machine in that situation). I'm appending the original infos here. On postgres, I just find this debug, after getting the error from bconsole: LOG: unexpected EOF on client connection This seems to indicate that you have a networking issue. That message alone, not necessarily. It will be logged by PosgreSQL anytime a connection goes away without getting the close message. So a client program exiting without doing PQfinish() will cause this, for example (and this is way way common, unfortunately). Or a client program crashing, of course. It *can* be a sign of network issues, absolutely. But it can also be something else. Agreed. I gues the point I was trying to make was that, based on the messages from the server and the timeout messages on the client, my best guess at this point is that it's a network problem. But it wouldn't be the first time if I were wrong. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and Postgres 8.1
This foray has gotten very frustrating for me. Please excuse if the following replies seem inflammatory, but it comes from an honest desire to _really_ help. In response to Gabriele Bulfon [EMAIL PROTECTED]: That postgres db is being used on the network all day long. - I use it from web apps through jdbc - I use it from pgadmin on a windows laptop - I use it from custom softwares on desktops This problem arises only when I use bconsole on Bacula and the Director is separated from the DB. Everything works fine only if the Director works on localhost DB. In the logs, I did not put the Director's startup traces because they were a lot and they all went ok. Please put some effort in to formatting your emails. I'm tired of fixing your line wrapping. 72 chars is the standard. Now. I'm going to assume 1 of two things: 1) You're a super expert on both Bacula and PostgreSQL. This is evident because you can look at logs and _know_ that certain parts are not important because you _know_ that those sections show normal behaviour. You're so smart that you refuse to send those sections of the logs in spite of multiple requests for them. Obviously, there's absolutely no chance that you could be mistaken, since the system is running so smoothly. 2) There's something in the logs that you find embarrassing. If #1 is the case, then just fix the problem yourself. If #2 is the case, then I have no advice. I really suspect this is something coming out only when I use bconsole. While this could be true, I find it unlikely and not indicated by the evidence you're providing. bconsole talks to the director and the director talks to the database. It appears as if bconsole is able to contact the director, and the errors seem to indicate a connection problem between the director and the database. The fact that this only occurs when you use the director is a silly benchmark, since the _only_ other thing you've managed to get the director to do is start. I can't make a real test now, but because Director starts up very happy with the DB, I'm pretty sure that it would also run the scheduled backups happily. Those are some outlandish assumptions you're making there. If the director fails on a status storage then I find it difficult to imagine that it could succeed in backing anything up. Because I cannot use bconsole, then I cannot create volumes, and cannot verify this. Yes, you can. You could create autolabeled disk volumes in the config, create a test backup job, schedule it to run frequently and you'd find out very quickly. Someone can explain me what happens when I use list volumes on bconsole? there must be some difference in the way the DB is accessed at startup and when I use bconsole No. bconsole - director, then director - db in the same way it does anything else. You've still yet to provide any kind of network debugging. For crying out loud, do a tcpdump -s0 -w /some/file port 5432 during the failure and post the resultant file somewhere. And, quite frankly, the last several emails have amounted to you politely saying, I don't agree with your advice, and therefore, won't take it. Based on this, I'm tempted to stop giving it. If you know so much about how bacula works, why are you still stuck? If you want any _real_ help, you're going to have to post some serious information somewhere. It's amazing that the open source community is generally willing to wade through such data and provide assistance for free. But I find it even more amazing that you repeated _refuse_ to take advantage of such free debugging. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] useful indexes (was Re: dbcheck needs very long time...)
In response to Birger Blixt [EMAIL PROTECTED]: Frank Sweetser wrote: On Thu, Sep 07, 2006 at 10:58:49AM +0200, Masopust, Christian wrote: Hello Frank, thanks a lot for this info! but :-))) could anybody give the complete info (maybe also modify the wiki-page) how these indexes should be created? The best place to find that is in the documentation for your database. That way you can make sure you're getting accurate information, as opposed to something for a previous version or the like. Amazing, my dbcheck at home was searching for orphaned Filename records the whole day, and I finaly did mysaladmin kill on the query. Then I did: mysql bacula create index Job_FilesetId_idx on Job(FileSetId); create index Job_ClientId_idx on Job(ClientId); I find it hard to believe that these indexes help much, unless you've got a boatload of jobs in the system. I doubt they'll hurt anything, though. create index File_PathId_idx on File(PathId); create index File_FilenameId_idx on File(FilenameId); One of these two is redundant. There's already an index: file_fp_idx btree (filenameid, pathid) CLUSTER which should be usable by queries searching on filenameid (this is PostgreSQL, but the idea is the same) Creating another index on filenameid is just bloat and will slow down inserts. Personally, I don't recommend that anyone blindly create these indexes. As you mentioned, it took over 30 minutes to create them, which seems to indicate that their existence will have a negative impact on inserts and updates. In my case, none of these indexes made a significant improvement to dbcheck time. dbcheck took 5-1/2 minutes to run with and without the above indexes (I have about 9,000,000 rows in the file table and 500,000 in the filename table). Try them out ... if they make a significant improvement, use them. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] 25-hour backup job
In response to David Hatcher [EMAIL PROTECTED]: Hi folks, New user here. I find the following peculiar and wonder if big backup jobs take longer to complete than running several consecutive smaller jobs? Here's my story... Server = bacula-fd Version: 1.38.9, OS=Linux Fedora Core 4 Client = labssrv-fd Version: 1.38.4, OS=Windows NT 4.0 For my first full backup on my client (labssrv), I setup the bacula-dir.conf file (see attached) to backup everything on the C, F, G, and H drives. Below is the summary, in particular, the job took 13 hours to backup 173 GB and resulted in 430 non-fatal FD errors, most of which were permission errors. This sounds fairly reasonable to me. JobId: 1150 Job:Labssrv.2006-09-08_19.00.03 Backup Level: Full Client: labssrv-fd Windows NT 4.0,MVS,NT 4.0.1381 FileSet:Labssrv FileSet 2006-09-08 22:52:20 Pool: Weekly Storage:SDLT Scheduled time: 08-Sep-2006 19:00:02 Start time: 08-Sep-2006 22:52:23 End time: 09-Sep-2006 12:39:33 Elapsed time: 13 hours 47 mins 10 secs Priority: 11 FD Files Written: 296,222 SD Files Written: 296,222 FD Bytes Written: 173,119,126,494 (173.1 GB) SD Bytes Written: 173,182,590,769 (173.1 GB) Rate: 3488.2 KB/s Software Compression: None Volume name(s): 000103|77 Volume Session Id: 5 Volume Session Time:1157757233 Last Volume Bytes: 25,817,913,037 (25.81 GB) Non-fatal FD errors:430 SD Errors: 0 FD termination status: OK SD termination status: OK Termination:Backup OK -- with warnings Then I went through the details of the permission errors and granted access to the respective files and directories on my labssrv machine. In addition, I excluded some old archived data that doesn't need to be backed up. Following is the new summary. The job took 25 hours to backup 188 GB and resulted in 0 non-fatal FD errors I'm surprised it took 12 additional hours to backup 15 GB of data, as if there's an exponential problem somewhere. This doesn't make sense to me. I'm thinking about splitting this job into two separate jobs (job one for drives C and F, job two for drives G and H) to see if it will complete in under 25 hours. I have other clients that I back up that run fairly quick backup jobs, although the backups are typically less than 100 GB. JobId: 1225 Job:Labssrv.2006-09-15_19.00.03 Backup Level: Full Client: labssrv-fd Windows NT 4.0,MVS,NT 4.0.1381 FileSet:Labssrv FileSet 2006-09-15 22:50:24 Pool: Weekly Storage:SDLT Scheduled time: 15-Sep-2006 19:00:02 Start time: 15-Sep-2006 22:50:27 End time: 16-Sep-2006 23:34:34 Elapsed time: 1 day 44 mins 7 secs Priority: 11 FD Files Written: 309,962 SD Files Written: 309,962 FD Bytes Written: 187,971,236,795 (187.9 GB) SD Bytes Written: 188,037,056,495 (188.0 GB) Rate: 2110.9 KB/s Software Compression: None Volume name(s): 82|000100 Volume Session Id: 5 Volume Session Time:1158352255 Last Volume Bytes: 39,196,049,239 (39.19 GB) Non-fatal FD errors:0 SD Errors: 0 FD termination status: OK SD termination status: OK Termination:Backup OK Lots of information missing here -- difficult to help much without some additional diagnosis. What DB are you using? Is it possible that you've pushed the db server past some limit where performance starts to degrade? i.e. created enough records that inserts have become expensive? If you monitor CPU and IO usage during the backup, where is the holdup and which program (DBserver? director? storage daemon? file daemon?) is using that resource? Is it possible that you hit something on the DLT tape that caused it to have to rewind or do a bunch of seeking or something else that put the whole job in wait mode for a long time? What is the nature of the data in that directory? Is it possible that the FD is contested for access to those files and is spending a lot of time waiting for them to free up when it tries to grab them? Mostly guesses here, but hopefully something will be helpful. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID
Re: [Bacula-users] Bacula and Postgres 8.1
In response to Gabriele Bulfon [EMAIL PROTECTED]: Sorry for not resending the needed informations, I already sent them when I first encountered the problem. But then I could solve this by moving the Director together with the DB (I had another Solaris machine in that situation). I'm appending the original infos here. On postgres, I just find this debug, after getting the error from bconsole: LOG: unexpected EOF on client connection This seems to indicate that you have a networking issue. Any network control devices between these two systems? (firewalls, routers, managed switches?) Ensure all your speed and duplexing match up. Grab some tcpdumps off the network while this is happening, and see if anything looks fishy. Do it on both ends of the connection and see if packets are getting lost. Can you use psql on the system running the director to connect manually to the PostgreSQL database? Can you run a long running query without getting disconnected (SELECT COUNT(*) FROM some big table will generally take a while) Hope you can help. Gabriele. == I rised debug to 500. Before the I issue the list volumes on bconsole, I see a LOT LOT of debug about postgres selects going through very fine. That's funny, because I don't see any in your attached logs. Once I run the bconsole command I find this: iserver-dir: scheduler.c:253 enter find_runs() iserver-dir: scheduler.c:289 Got job: Enterprise Backup iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:289 Got job: Backup Aliseo iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:289 Got job: Backup ZetaFax iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:289 Got job: Backup Centralino iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:289 Got job: BackupCatalog iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:332 [EMAIL PROTECTED]: run_now=0 run_nh=0 iserver-dir: scheduler.c:355 Leave find_runs() iserver-dir: bsys.c:70 pthread_cond_timedwait sec=60 usec=0 iserver-dir: ua_cmds.c:1615 Open database iserver-dir: postgresql.c:77 db_open first time iserver-dir: bsys.c:70 pthread_cond_timedwait sec=5 usec=0 iserver-dir: bsys.c:77 pthread_cond_timedwait stat=145 ERR=Connection timed out Gabriele Bulfon - Sonicle S.r.l. Tel +39 028246016 Int. 30 - Fax +39 028243880 Via Felice Cavallotti 16 - 20089, Rozzano - Milano - ITALY http://www.sonicle.com -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] useful indexes (was Re: dbcheck needs very long time...)
In response to Birger Blixt [EMAIL PROTECTED]: On 2006-09-20 15:44, Bill Moran wrote: In response to Birger Blixt [EMAIL PROTECTED]: Frank Sweetser wrote: On Thu, Sep 07, 2006 at 10:58:49AM +0200, Masopust, Christian wrote: Hello Frank, thanks a lot for this info! but :-))) could anybody give the complete info (maybe also modify the wiki-page) how these indexes should be created? The best place to find that is in the documentation for your database. That way you can make sure you're getting accurate information, as opposed to something for a previous version or the like. Amazing, my dbcheck at home was searching for orphaned Filename records the whole day, and I finaly did mysaladmin kill on the query. Then I did: mysql bacula create index Job_FilesetId_idx on Job(FileSetId); create index Job_ClientId_idx on Job(ClientId); I find it hard to believe that these indexes help much, unless you've got a boatload of jobs in the system. I doubt they'll hurt anything, though. create index File_PathId_idx on File(PathId); create index File_FilenameId_idx on File(FilenameId); One of these two is redundant. There's already an index: file_fp_idx btree (filenameid, pathid) CLUSTER which should be usable by queries searching on filenameid (this is PostgreSQL, but the idea is the same) Creating another index on filenameid is just bloat and will slow down inserts. Personally, I don't recommend that anyone blindly create these indexes. As you mentioned, it took over 30 minutes to create them, which seems to indicate that their existence will have a negative impact on inserts and updates. In my case, none of these indexes made a significant improvement to dbcheck time. dbcheck took 5-1/2 minutes to run with and without the above indexes (I have about 9,000,000 rows in the file table and 500,000 in the filename table). Try them out ... if they make a significant improvement, use them. Well, I admit, I don't know what I'm doing in this case, but ,,, the search query _did_ take forever, and I canceled it at the end. With new indexes it goes fast, so something did happen. I was checking the script http://www.aha.com/bacula/recover.pl There I found: In order for this program to have a chance of not being painfully slow, the following indexs should be added to your database. CREATE INDEX file_pathid_idx on file(pathid); CREATE INDEX file_filenameid_idx on file(filenameid); I took that as an example, and added them, and the 2 Job indexes as a test, there is always a drop index on table command to remove them if it don't works. I should enjoy if someone that really knows how to tune mysql could send an output from show index from File , so I can optimize the table (that goes for the table Job too I guess ) File 0 PRIMARY 1 FileId 4541609 BTREE File 1 JobId1 JobId 576 BTREE File 1 JobId2 PathId 378467 BTREE File 1 JobId3 FilenameId 4541609 BTREE File 1 JobId_2 1 JobId 576 BTREE File 1 File_PathId_idx 1 PathId 105618 BTREE File 1 File_FilenameId_idx 1 FilenameId 504623 BTREE Maybe I can drop something here, who knows ? One source for information can be the make_mysql_tables script The problem is that depending on your data and usage of the database, YMMV. Optimizations that work for one person might not benefit another. For example: adding indexes _usually_ speeds query performance, but it also _usually_ slows insert/update performance, because every change to the table has to update all the indexes. If you're interested in very fast backups and rarely do restores, it might be a bad idea to add any indexes to improve looks in the database if it slows down the updates. On the other hand, if getting at your data quickly is important, it might be worth slower backups to be able to search the data quickly. People who have lots of jobs vs. folks with only a few jobs. A few large files vs. lots of small files. One big server vs. many servers. Each of these scenarios will distribute the data differently in the database, and cause different optimizations to be worthwhile. Additionally, the hardware on which the DB runs makes a difference: fast disk and low RAM vs. lots of RAM and slow disks. I recommend trying out indexes. It's pretty simple to remove them if they don't work out. But don't forget to test backup performance as well, since adding an index may speed dbcheck, but hurt your backup speed. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http
Re: [Bacula-users] useful indexes (was Re: dbcheck needs very long time...)
In response to Frank Sweetser [EMAIL PROTECTED]: On Wed, Sep 20, 2006 at 09:44:24AM -0400, Bill Moran wrote: Try them out ... if they make a significant improvement, use them. As an aid to experimenting with indexes, and to help people give more hard data, perhaps it might be usefull to add an option to dbcheck to output the results of running explain (mysql, I assume postgresql has an equivalent) on the queries used. This info is really useful to find hotspots that would benefit the most from adding an index or two. Well, I have statement logging and query duration logging turned on on PostgreSQL on our production system. Running dbcheck shows a bunch of intense queries, Here are some of the killers: duration: 153056.519 ms statement: SELECT Filename.FilenameId, File.FilenameId FROM Filename LEFT OUTER JOIN File ON (Filename.FilenameId=File.FilenameId) WHERE File.FilenameId IS NULL bacula= explain SELECT Filename.FilenameId,File.FilenameId FROM Filename LEFT OUTER JOIN File ON (Filename.FilenameId=File.FilenameId) WHERE File.FilenameId IS NULL; QUERY PLAN - Merge Left Join (cost=0.00..824877.11 rows=506718 width=8) Merge Cond: (outer.filenameid = inner.filenameid) Filter: (inner.filenameid IS NULL) - Index Scan using filename_pkey on filename (cost=0.00..23530.19 rows=506718 width=4) - Index Scan using file_fp_idx on file (cost=0.00..676093.57 rows=9918925 width=4) (5 rows) Notice that it's using existing indexes, it's the join that's rough. This one is rough as well: duration: 185632.013 ms statement: SELECT DISTINCT Path.PathId, File.PathId FROM Path LEFT OUTER JOIN File ON (Path.PathId=File.PathId) WHERE File.PathId IS NULL QUERY PLAN - Unique (cost=1819887.33..1824163.31 rows=570131 width=8) - Sort (cost=1819887.33..1821312.65 rows=570131 width=8) Sort Key: path.pathid, file.pathid - Merge Left Join (cost=1587495.88..1760367.99 rows=570131 width=8) Merge Cond: (outer.pathid = inner.pathid) Filter: (inner.pathid IS NULL) - Index Scan using path_pkey on path (cost=0.00..22662.91 rows=570131 width=4) - Sort (cost=1587495.88..1612293.20 rows=9918925 width=4) Sort Key: file.pathid - Seq Scan on file (cost=0.00..278755.25 rows=9918925 width=4) The query plan on this seems to indicate that an index on pathid would be helpful. It's interesting that I didn't see any measurable improvement. -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula and Postgres 8.1
In response to Gabriele Bulfon [EMAIL PROTECTED]: Hello, I open again the question about Bacula working on Postgres. This time, I really need to have Director and Postgres run on two different machines. The reason: I only have one Solaris machine and one Windows machine, so I cannot move the Director to the Windows machine together with Postgres (as far as I know, the Director is still not supported on Windows). What happens is that once I add the needed entries to the bacula-dir.conf to connect to the new machine (DB Address and DB Port), Director starts (and with log enabled to 500, I can see a lot of queries correctly handled), but when I run bconsole on the Director, the Director initially responds with the prompt, then any command that needs DB access (such as list volumes) hangs for some seconds and then fails saying that it cannot connect to the DB. If I look at the log, I can see some timeout failures. How can I investigate more this? Why the first queries go through correctly, and then they fail with bconsole? Please help...I really need to move the DB on the other machine Help: 1) Provide the logs you captured, not your interpretation of them. Nobody is going to be able to diagnose I can see some timeout failures 2) Bump up connection logging on the PostgreSQL server. You'll want to provide those logs as well. 3) Put some effort into formatting your emails so they're readable. -- Bill Moran Collaborative Fusion Inc. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula-dir problems with -HUP? (was Re: [Bacula-devel] Backtrace Attached: bacula-dir lockups on FreeBSD)
In response to Bill Moran [EMAIL PROTECTED]: My plan at this point: 1) Upgrading to RELENG_6_1 tonight 2) See if the problem repeats this Saturday 3) If it does, grab another backtrace and put it out on freebsd-hackers@ to see if anyone has any suggestions. Followup posting. I upgraded the system to FreeBSD 6.1p5. There were some other circumstances that interfered with getting any reasonable data from the system, so it took a little longer than I expected to do the remaining steps. However, it occurred to me that the log rotation occurs at about the same time this freeze would occur. This involves sending Bacula a HUP signal. I moved the log rotation to later in the day (after the jobs are finished) and the whole set of backups ran without a hitch. It would appear as if sending bacula-dir a HUP signal in the middle of a backup causes this problem. Note that -HUPping the director while it is idle doesn't seem to cause any trouble. Note also that I experimented a bit, and hupping doesn't seem to cause the problem all the time. Can anyone verify this on another platform? I want to set up some test scenarios here, but I'm dogged for time, and it may be a while before I can do any more serious work on this issue. On a specific note: changing the time that logs rotate is only a workaround for us, and one with a dubious future. We're running backups more and more frequently, and a lot of them run rather dynamically (using reschedule on error and the like) so it's going to get increasingly more difficult to find a time when I can be sure that no backup is running. -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula-dir problems with -HUP? (was Re: [Bacula-devel] Backtrace Attached: bacula-dir lockups on FreeBSD)
In response to Kern Sibbald [EMAIL PROTECTED]: However, it occurred to me that the log rotation occurs at about the same time this freeze would occur. This involves sending Bacula a HUP signal. Sending Bacula an HUP signal causes it to exit as rapidly as possible. It attempts to cancel and cleanup jobs, but it does so in a rather brutal fashion. Bottom line: make sure no root program sends HUP signals to Bacula while it is running a job, and preferably never send such signals unless you want to shutdown Bacula. This is new to me. And not intuitive IMHO. Most programs interpret a HUP to mean restart gracefully. I got the impression that it would be about the same as issuing reload from bconsole. Most daemons require a HUP signal to tell them to close and reopen their log files. I seem to remember reading a HOWTO that suggested hupping the director after log rotation, but I can't find it now -- perhaps my memory is flawed. I don't understand why any process is sending HUP signals to Bacula when it is cycling logs. This seems to me to be a configuration error. That would be _my_ error. I put the command to HUP bacula-dir in newsyslog.conf. Personally, I find the director's handling of HUP atypical by comparison to other Unix daemons. From the docs, it's not clear whether the director requires a signal to tell it to start logging to a new file, I'm assuming (since HUP is a bad idea) that it does not? Additionally, this still seems to be a bug (even if it won't be affecting me any more) since the director does not exit when hupped, but instead freezes. I've done several searches in the past on the director's handling of signals, and found sparse data. I repeated those searches just now, and still haven't found any information. At least I know that HUP is a bad idea ... -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Bacula-dir problems with -HUP? (was Re: Backtrace Attached: bacula-dir lockups on FreeBSD)
In response to Kern Sibbald [EMAIL PROTECTED]: On Tuesday 19 September 2006 17:53, Bill Moran wrote: [snip] This is new to me. And not intuitive IMHO. Most programs interpret a HUP to mean restart gracefully. I got the impression that it would be about the same as issuing reload from bconsole. Most daemons require a HUP signal to tell them to close and reopen their log files. I seem to remember reading a HOWTO that suggested hupping the director after log rotation, but I can't find it now -- perhaps my memory is flawed. Yes, you are right, I was confusing a SIGHUP with a SIGTERM. SIGHUPing Bacula is never a very good idea. Depending on what version of Bacula it is, it will most likely crash. If I am not mistaken, so far there are no reported crashes against version 1.38.11. Well, that still tells me what to do to make the the problem go away, although you can consider this a report of lockup when sending the director a HUP ;) [snip] Personally, I find the director's handling of HUP atypical by comparison to other Unix daemons. From the docs, it's not clear whether the director requires a signal to tell it to start logging to a new file, I'm assuming (since HUP is a bad idea) that it does not? If you are talking about cycling a Job report log file as defined in the Messages resource, then there is probably no need to SIGHUP Bacula as it opens and closes the log file on a Job by Job basis, if I am not mistaken. That's what I meant. Although I'm still a bit concerned. What happens if the log file is rotated while a job is in progress? Will bacula attempt to append to the end of (the now compressed) previously opened file descriptor? Or will it reconnect to the new file with the correct filename. Job reports split between two log files is an annoyance, but not terrible. Corruption of old log data would be worse. [snip] Yes, even if I did mistake SIGHUP and SIGTERM, it is not a good idea to SIGHUP Bacula. I've adjusted the newsyslog config. Thanks for the feedback, as always. -- Bill Moran Collaborative Fusion Inc. IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
