Re: [Bacula-users] Write Retry on Network Error without Restarting the Job

[Pablo Marques]

Here it is, in 7.2: 

http://www.bacula.org/7.2.x-manuals/en/main/New_Features_in_7_2_0.html#SECTION003213000

Pablo
- Original Message -
From: "Kern Sibbald" 
To: "Duane Webber" , "bacula-users" 

Sent: Friday, February 26, 2016 1:04:31 AM
Subject: Re: [Bacula-users] Write Retry on Network Error without Restarting the 
Job

No this feature does not exist.  However, in a recent version of Bacula 
(7.2.0 I think, but maybe 7.4.0) you can restart failed jobs and it will 
start roughly from the point the backup failed.  Look in the New 
Features section of the document for Incomplete Jobs.

Best regards,
Kern

On 02/26/2016 02:58 AM, Duane Webber wrote:
> Is there an option to have Bacula retry write operations without
> restarting the job?   A search shows the last time this question was
> asked was in 2010 (the answer was no) and I'm hoping there is a change
> since then.  I have remote backups that can take days, if not a week
> plus and it is not uncommon to run into a network error where Bacula
> aborts the backup job altogether.
>
> If Bacula does not have this capability, is there a workaround for the
> same such that the end result is that the remote backups are visible to
> Bacula directly.
>
> I appreciate any suggestions.
>
> Thanks,
>
> Duane
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
>
>
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


-- 

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Write Retry on Network Error without Restarting the Job

[Pablo Marques]

I believe this feature is now available in version 7.4 


From: "Clark, Patti"  
To: "Duane Webber" , "bacula-users" 
 
Sent: Thursday, February 25, 2016 3:09:40 PM 
Subject: Re: [Bacula-users] Write Retry on Network Error without Restarting the 
Job 

Bacula Enterprise has this feature where a job can be restarted. There is no 
work around other than breaking up the jobs into smaller chunks. 

Patti Clark 
Linux System Administrator 
R Systems Support Oak Ridge National Laboratory 

From: Duane Webber < duane.web...@comcast.net > 
Date: Thursday, February 25, 2016 at 11:58 AM 
To: " bacula-users@lists.sourceforge.net " < bacula-users@lists.sourceforge.net 
> 
Subject: [Bacula-users] Write Retry on Network Error without Restarting the Job 

Is there an option to have Bacula retry write operations without restarting the 
job? A search shows the last time this question was asked was in 2010 (the 
answer was no) and I'm hoping there is a change since then. I have remote 
backups that can take days, if not a week plus and it is not uncommon to run 
into a network error where Bacula aborts the backup job altogether. 

If Bacula does not have this capability, is there a workaround for the same 
such that the end result is that the remote backups are visible to Bacula 
directly. 

I appreciate any suggestions. 

Thanks, 

Duane 


Spam 
Phish/Fraud 
Not spam 
Forget previous vote 

-- 
Site24x7 APM Insight: Get Deep Visibility into Application Performance 
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month 
Monitor end-to-end web transactions and take corrective actions now 
Troubleshoot faster and improve end-user experience. Signup Now! 
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140 
___ 
Bacula-users mailing list 
Bacula-users@lists.sourceforge.net 
https://lists.sourceforge.net/lists/listinfo/bacula-users 
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Incomplete Jobs

[Pablo Marques]

Hi 

Is the feature Incomplete Jobs/stop/restart
from Bacula Enterprise 6.0 going to be available
in the (free) version 5 series?

We could definitely use this feature on some
of our Windows users that reboot their machines
or lose their network connection in the middle of a backup.

Regards

Pablo Marques

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Backups increased to 500GB after adding to IPA domain

[Pablo Marques]

Abdullah:

Make sure you have this in your fileset definition

Sparse = yes

Also you can do this in bconsole:

estimate job=client_job_whatever listing
 
it will print the list of files to be backed up.
Look for big files in the list.


Pablo
- Original Message -
From: Abdullah Sofizada asofiz...@emedia.cv.net
To: bacula-users@lists.sourceforge.net
Sent: Thursday, April 5, 2012 11:21:37 AM
Subject: [Bacula-users] Backups increased to 500GB after adding to IPA domain

Hi guys, this is a very weird one. I been trying to tackle this for the 
past two weeks or so to no avail...

My director runs on Redhat Rhel 5.5 running bacula 5.0.2. My clients are 
Redhat Rhel 5.5 running bacula 5.0.2.

Each of the bacula clients are less than 15 GB of data. Backups of each 
client were fine. But two weeks ago the backups for each of these 
clients ballooned to 550GB each!!

When I do a df -h... the servers only show 15GB of space used. The one 
difference I noticed in the past two weeks is...I added these servers to 
our new IPA domain. Which in essence is an ldap server using kerberos 
authentication for identity management. This server runs on Rhel 6.2.

I have many other clients which are not part of the IPA domain that are 
backing up just fine. So I'm sure it has something to do with this. I 
have even tried to remove my bacula clients out of the IPA domain, than 
ran a backup. But it still reports at 550GB of data being backed up.

I appreciate the help...



-- 
-Abdullah


--
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Question

[Pablo Marques]

Juan, try upgrading to 5.2.6 and use these settings:
==
FileSet {
  Name = Root-set
  Include {
Options {
  signature = MD5
  compression = GZIP
  Sparse = yes
  aclsupport = yes
  xattrsupport = yes
 }
File = /
  }
}

Job {
  Name = client-base-fd-data
  JobDefs = jobbaculadefs
  Client = stor-server3-fd
  FileSet = Root-set
  Storage = File
  Pool = File
  Level = Base
}

Job {
  Name = client-fd-data
  JobDefs = jobbaculadefs
  Client = stor-server3-fd
  FileSet = Root-set
  Base = client-base-fd-data
  Storage = File
  Pool = File
  Accurate = yes
}
===

You may need to insert some extra settings in the jobs above.

# Reload bacula and run the base job:
run job=client-base-fd-data
# Let it finish and run the FULL
run job=client-fd-data level=FULL
# Let it finish, check to make sure that this FULL used the Base job in the 
report.
# Now run a differential:
run job=client-fd-data level=Differential
# The report should come back with almost no changes, the differential is 
really a incremental from the last full.

Let me know.

Pablo
- Original Message -
From: Juan Diaz juan.d...@insa-lyon.fr
To: Pablo Marques pmarq...@miamilinux.net
Cc: José Frederico jose.freder...@callide.fr, 
bacula-users@lists.sourceforge.net
Sent: Wednesday, March 28, 2012 9:44:37 AM
Subject: Re: [Bacula-users] Question

Pablo,

I have tried all day with your configuration and it doesn't work. Could you 
please send me the complete configuration for search the differences between 
the two of them.

We work with Bacula 5.0.2 and linux debian 2.6.32-5-amd64.

Also I tried deduplication with the same machine I did first base after full 
and lately differential and the differential was smaller. But when I use that 
base with another machine the differential is bigger than full backup and that 
doesn't make sense.

I CC the group in case there's someone who had the same trouble.

Thank you very much four your help,


Juan David DIAZ
INSA de Lyon
Universidad ICESI
+33 6 49 59 23 86


- Mail original -
De: Juan Diaz juan.d...@insa-lyon.fr
À: Pablo Marques pmarq...@miamilinux.net
Cc: José Frederico jose.freder...@callide.fr, 
bacula-users@lists.sourceforge.net
Envoyé: Mercredi 28 Mars 2012 09:47:23
Objet: Re: [Bacula-users] Question

Thank you Pablo but it didn't worked. The incremental or differential continues 
being bigger than the full backup.

Juan David DIAZ
INSA Lyon
Universidad ICESI
+33 6 49 59 23 86


- Mail original -
De: Pablo Marques pmarq...@miamilinux.net
À: Juan Diaz juan.d...@insa-lyon.fr
Cc: José Frederico jose.freder...@callide.fr, 
bacula-users@lists.sourceforge.net
Envoyé: Mardi 27 Mars 2012 17:40:41
Objet: Re: [Bacula-users] Question

Juan:

That is no happening to us and the only obvious difference between our set up 
and yours is in the fileset definition.

Can you remove these lines in your fileset and give it a try?
BaseJob = pmugcs5
Accurate = mcs
Verify = pin5

This is our fileset:

FileSet {
  Name = Root-set
  Include {
Options {
  signature = MD5
  compression = LZO
  Sparse = yes
  aclsupport = yes
  xattrsupport = yes
 }
File = /
  }
}

Good luck

Pablo

- Original Message -
From: Juan Diaz juan.d...@insa-lyon.fr
To: bacula-users@lists.sourceforge.net
Cc: José Frederico jose.freder...@callide.fr
Sent: Tuesday, March 27, 2012 9:35:27 AM
Subject: [Bacula-users] Question

Hello,

Right now we're evaluating and deciding which is going to be our Backup 
software, Bacula is the most optioned so far but we're having troubles with the 
implementation of the file DE duplication.

When we make a base job of the machine virtual that is going to be the base, 
his size is 553 Mb. After that when we make a full backup of a machine that was 
created from the machine base. His size is 242 Mb that’s ok because it only 
has to make backup of the files that are different of files of the base.

The problem comes when we make a differential backup right after the full 
backup. The result must be 0 Mb or something really small, but we're getting 
backups of 462 Mb.

We used one configuration similar to the one we found in the Bacula’s manual. 
http://www.bacula.org/5.2.x-manuals/en/main/main.pdf Chapter 34.

We have tried removing BaseJob, Accurate and Verify from FileSet but it didn’t 
worked. We have always the same sizes that don't make sense.

We would like to know if you have a solution for our problem, because it worked 
when we don't use DE duplication (just with full and Differential), but we are 
very interested in using it, that would save us a lot of space on ours hard 
drives.

We attached here the job’s configuration:

ob {
  Name = BackupBasse
  Type = Backup
  Level = Base
  Client = basse-fd
  FileSet = TOUT
 Schedule = programme2
  Storage = File
  Messages = Standard
  Pool = File
  Priority = 10
  Write Bootstrap

Re: [Bacula-users] Question

[Pablo Marques]

Juan:

That is no happening to us and the only obvious difference between our set up 
and yours is in the fileset definition.

Can you remove these lines in your fileset and give it a try?
BaseJob = pmugcs5
Accurate = mcs
Verify = pin5

This is our fileset:

FileSet {
  Name = Root-set
  Include {
Options {
  signature = MD5
  compression = LZO
  Sparse = yes
  aclsupport = yes
  xattrsupport = yes
 }
File = /
  }
}

Good luck

Pablo

- Original Message -
From: Juan Diaz juan.d...@insa-lyon.fr
To: bacula-users@lists.sourceforge.net
Cc: José Frederico jose.freder...@callide.fr
Sent: Tuesday, March 27, 2012 9:35:27 AM
Subject: [Bacula-users] Question

Hello, 

Right now we're evaluating and deciding which is going to be our Backup 
software, Bacula is the most optioned so far but we're having troubles with the 
implementation of the file DE duplication.

When we make a base job of the machine virtual that is going to be the base, 
his size is 553 Mb. After that when we make a full backup of a machine that was 
created from the machine base. His size is 242 Mb that’s ok because it only 
has to make backup of the files that are different of files of the base.

The problem comes when we make a differential backup right after the full 
backup. The result must be 0 Mb or something really small, but we're getting 
backups of 462 Mb.

We used one configuration similar to the one we found in the Bacula’s manual. 
http://www.bacula.org/5.2.x-manuals/en/main/main.pdf Chapter 34.

We have tried removing BaseJob, Accurate and Verify from FileSet but it didn’t 
worked. We have always the same sizes that don't make sense.

We would like to know if you have a solution for our problem, because it worked 
when we don't use DE duplication (just with full and Differential), but we are 
very interested in using it, that would save us a lot of space on ours hard 
drives.

We attached here the job’s configuration:

ob {
  Name = BackupBasse
  Type = Backup
  Level = Base
  Client = basse-fd
  FileSet = TOUT
 Schedule = programme2
  Storage = File
  Messages = Standard
  Pool = File
  Priority = 10
  Write Bootstrap = /var/lib/bacula/%c.bsr
}
Schedule{
Name = programme2
Run = at 14:30
}

Job {
  Name = Backupftp
  Type = Backup
  Base = Backupftp, BackupBasse
  Accurate=yes
  Client = basseftp-fd
  FileSet = TOUT
  Schedule = programme
  Storage = File
  Messages = Standard
  Pool = File
  Priority = 10
  Write Bootstrap = /var/lib/bacula/%c.bsr
}

Schedule{
Name = programme
Run = Level=Full at 14:31
Run = Level=Incremental at 14:32

}

FileSet {
  Name = TOUT
  Include {
Options {
BaseJob = pmugcs5
Accurate = mcs
Verify = pin5
}
File = /
  }
}

Thank you very much for your help,

Juan David DIAZ

--
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems with Jumbo packets?

[Pablo Marques]

Hi,

You can use ping to check the end-to-end maximum MTU working size:

ping -Mdo -s 8000 192.168.X.X

increase or reduce the size (8000) until you get pings back from the FD client 
(mine goes up to 8972)

Pablo
- Original Message -
From: Frank Sweetser f...@wpi.edu
To: bacula-users@lists.sourceforge.net
Sent: Friday, January 6, 2012 11:34:44 AM
Subject: Re: [Bacula-users] Problems with Jumbo packets?

On 01/06/2012 11:30 AM, Wolfgang Denk wrote:
 Hello,
 
 I know that this is not exactly related to Bacula, but maybe some
 other user has seen similar behaviour.
 
 I have problems when trying to enable support for jumbo frames on  the
 network.  All NICs and switches are supposed to support that, however
 on some systems the communication of the FD to the SD stops as soon as
 I change on the FD the MTU from the default of 1500 to a higher value
 (9000).  The MTU on the DIR and SD can be set to 9000 without visible
 impact on bacula.

Are all of the machines on the same subnet?  If not, you'll also have to check
the MTU on all of your local router interfaces.

Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual 
desktops for less than the cost of PCs and save 60% on VDI infrastructure 
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] amazing backup size

[Pablo Marques]

From FAQ Bacula Wiki:

Why is my backup larger than my disk space usage?

The most common culprit of this is having one or more sparse files.

A sparse file is one with large blocks of nothing but zeroes that the operating 
system has optimized. Instead of actually storing disk blocks of nothing but 
zeroes, the filesystem simply contains a note that from point A to point B, the 
file is nothing but zeroes. Only blocks that contain non-zero data are 
allocated physical disk blocks.

The single biggest culprit seems to be the contents of /var/log/lastlog on 64 
bit systems. Since the lastlog file is extended to preallocate space for all 
UIDs, the switch from a 32 bit UID space to a 64 bit UID increases the full 
size to over 1TB.

Luckily the fix is simple - turn on sparse file support in the FileSet, will 
detect sparse files and not store the zerofill blocks.

Another possible cause is that your fileset accidentally includes some folders 
twice. Taken from the manual:

Take special care not to include a directory twice or Bacula will backup 
the same files two times wasting a lot of space on your archive device. 
Including a directory twice is very easy to do. For example:

Include {
  File = /
  File = /usr
  Options { compression=GZIP }
}

on a Unix system where /usr is a subdirectory (rather than a mounted 
filesystem) will cause /usr to be backed up twice.


- Original Message -
From: John Drescher dresche...@gmail.com
To: Tilman Schmidt t.schm...@phoenixsoftware.de
Cc: bacula-users bacula-users@lists.sourceforge.net
Sent: Sunday, December 18, 2011 10:21:58 AM
Subject: Re: [Bacula-users] amazing backup size

On Sun, Dec 18, 2011 at 9:09 AM, Tilman Schmidt
t.schm...@phoenixsoftware.de wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 A newly installed CentOS 6 / Bacula 5 backup server is reporting
 this when backing itself up:

  FD Bytes Written:       53,655,908,904 (53.65 GB)
  SD Bytes Written:       53,664,006,577 (53.66 GB)
  Last Volume Bytes:      53,705,852,928 (53.70 GB)

 Which is truly amazing because the actual amount of data stored
 on this system is far less than that:

 [r2d2@backup ~]$ LANG=C df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/vg_backup-lv_root
                       50G  2.2G   45G   5% /
 tmpfs                 1.9G     0  1.9G   0% /dev/shm
 /dev/sda1             485M   47M  413M  11% /boot
 /dev/mapper/vg_backup-lv_home
                      174G  2.4G  163G   2% /home

 Any explanations for that discrepancy?


No, I have never seen anything like this. And this is from a bacula
user for 8+ years who has run tens of thousands of backups for a
department with 50+ machines and 30 to 50TB on tape. I suggest you
examine what bacula has saved in the backup. The simplest way is to
use the bat version browser or the new restore viewer if you have
bacula-5.2.X.

John

--
Learn Windows Azure Live!  Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for 
developers. It will provide a great way to learn Windows Azure and what it 
provides. You can attend the event by watching it streamed LIVE online.  
Learn more at http://p.sf.net/sfu/ms-windowsazure
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Learn Windows Azure Live!  Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for 
developers. It will provide a great way to learn Windows Azure and what it 
provides. You can attend the event by watching it streamed LIVE online.  
Learn more at http://p.sf.net/sfu/ms-windowsazure
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Noob user impressions and why I chose not to use Bacula

[Pablo Marques]

Thanks you Jesse for the feedback.

Regarding the disaster recovery, I have a suggestion for the bacula team: 

Why not make the director write the bacula config files and any relevant bsr 
files at the beginning of each tape?
The space wasted on the tape to save these file would be very small.  

These files could be easily recovered with btape on a total disaster recovery 
situation when you only have tapes (and hopefully a tape drive) and nothing 
else.

How difficult could it be to modify bacula to make the above automated when a 
tape is labeled and/or a recycled tape is written on again for example?

Pablo


- Original Message -
From: Jesse Molina je...@opendreams.net
To: bacula-users@lists.sourceforge.net
Sent: Sunday, December 4, 2011 9:39:06 PM
Subject: [Bacula-users] Noob user impressions and why I chose not to use
Bacula


Hi

Recently I was looking for new backup app for my small network.  Here's 
my story and why I decided that Bacula was not a good choice for me.

I am not a long time user, so opinions and views may not be shared by 
others, but they are true none the less.  You can only be a noob once 
and I hope this criticism can he constructive and helpful.

I am not looking for any response from any users.  You don't need to 
defend Bacula from some noob with a questionable opinion.

Note that some of my notes below were jotted down in haste during 
testing Bacula and some of my comments might be rather harsh or vulgar. 
  I'm not trying to troll or bash, and I hope these comments can be used 
to improve Bacula, and maybe I will get to use it again some day and it 
will be a better product.

I tried to throw all of these notes into a coherent whole, but I'm sure 
some of it will come off as being out of order to not making any sense.

--

I've got two Linux servers, a Mac, Windows, and Linux desktop, and a 
number of remote hosts.  The main host fileset is about 750GB, and all 
of the other various clients roll into about 600GB.  I have a single DLT 
S4 (800GB native) drive direct attached to a primary server host via 
Ultra320 bus.

I am an experienced sysadmin and I've previously been the primary 
maintainer of one TSM system and assisted with multiple others.

In the end, I just wrote my own scripts using ssh, rsync and tar.  It's 
good enough.

In summary, I could say I was attracted to the idea that Bacula used 
it's own data archive format and had a database for it's catalog, but 
was really turned off when I figured out that configuration was not also 
stored in a database, and how complicated actual restoration was.

I find the modular nature of Bacula's components very attractive, as it 
allows for scaling across multiple hosts for various functions. 
However, I don't understand the historical need to call the File Daemon 
anything other than what it is and what everyone seems to want call it: 
a Client.  Rename it to the Client Daemon and get over it.

While I appreciate the SQL DB used for historical data (Catalog), I find 
that primary configuration and some temporary data is scattered across 
various files.  It makes things complicated and difficult.  It will 
always be necessary to have some small configuration to point towards 
the other daemons and provide passwords, but using config files just 
makes management difficult.  SQL is not hard and Bacula isn't a simple 
program.  I would refer to Nagios vs Icinga as a good example of 
complicated text config systems gone bad.  When you have so much 
re-usable configuration data and complicated relationships, that's what 
DBs were made for.  Add a separate config DB and then all configuration 
should be done via bconsole, and a webUI.  Configuration could be dumped 
and loaded via bconsole or maybe an import/export commands alla 
Juniper/Cisco.

As for user interfaces, bconsole is good and I never really bothered 
with anything else.  The one huge complaint I have is that eject and 
other basic loader controls are absent and should be added.  I got 
really tired of having to umount, ctrl-z, and then call mt just to eject 
my tape during testing.  I realize that this is more complicated for 
autoloader libs, but allow the user to configure a backend-script for 
the command and there you go.  This can be done.



Documentation sucks.  It's just not a priority for this project and it 
shows.  Tons of typos, the formatting and layout is horrible, and for 
the English language I get the impression there are a lot of 
translation-isms.  It was like reading a paper written by five different 
college students where each one wrote a different portion with a 
different writing style.

In a number of cases, two sentences having the same or very similar 
meaning will be in the same paragraph.  Effectively saying the same 
thing twice or more.

For example:
Bacula can automatically label Volumes if instructed to do so, but this 
feature is not yet fully implemented. 
Really, WTF?  If it's not implemented, don't document 

Re: [Bacula-users] Noob user impressions and why I chose not to use Bacula

[Pablo Marques]

 Thanks you Jesse for the feedback.
 
 Regarding the disaster recovery, I have a suggestion for the bacula
 team:
 
 Why not make the director write the bacula config files and any
 relevant bsr files at the beginning of each tape? The space wasted on
 the tape to save these file would be very small.

Well, the first problem here is that the Director would have to know how
much space it was going to need for BSR files.  Of course, it could
pre-allocate a fixed-size block of, say, 1MB for BSR files.

Agree, 1 MB is basically nothing on a tape and it can accommodate easily a huge 
amount of bsr files.
My /etc/bacula is 88k uncompressed.  

The second problem, it seems to me, is that this would break
compatibility with all older Bacula volumes and installations.

not necessarily if you make this information at the beginning of the tape look 
like a volume file.
It will be ignored by old directors because it will look the same as a failed 
job that took space on the tape.  


Pablo

--
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] VirtualFull and Base Jobs

[Pablo Marques]

question: 

I have a Base job for a server. 
Full backups based on the Base Job run successfully. 
Incrementals run also fine. 

When I run a VirtualFull job the end result is not a Full backup but an 
Incremental one, based (I guess) on the last Incrementals. 
If I try to restore from the VirtualFull I only see a few files. 

The next Incremental backup run after the VirtualFull will backup all files 
except the ones present on the latest Incrementals. 

Is VirtualFull not compatible with Base Jobs? 

Pablo 
--
The demand for IT networking professionals continues to grow, and the
demand for specialized networking skills is growing even more rapidly.
Take a complimentary Learning@Cisco Self-Assessment and learn 
about Cisco certifications, training, and career opportunities. 
http://p.sf.net/sfu/cisco-dev2dev___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Base jobs and incremental backups

[Pablo Marques]

You are right about the documentation. It should work the way you describe it.

When you do a FULL after the BASE, it should examine all the files and only 
backup those that have changed since the BASE (which they should be very few or 
none).

Can you show us your configuration files to verify you are not missing 
something?
As I understand it, the accurate backup requires the director to send all the 
file definitions of a previous backup to the client, for the client to decide 
what has changed.
That may take a while if the bandwidth is not there.

These are my job definitions config file for one of my servers:
Job {
Name = server-base-fd-data
JobDefs = jobbaculadefs
Client = server-fd
FileSet = server-set
Pool = Yearly
Level = Base
}
Job {
Name = server-fd-data
JobDefs = jobbaculadefs
Client = server-fd
Schedule = IncWeeklyCycle
FileSet = server-set
Client Run Before Job = C:/windows/system32/ntbackup.exe backup systemstate /J 
\vmbackup Backup\ /F \C:\\SystemStateBackup.bkf\ /R:yes /L:f /SNAP:on
Base = server-base-fd-data
Accurate = yes
}

What I do on my setup is I first run the BASE manually, then I run a manual 
FULL, then I just let the regular client schedule run the usual INCREMENTALS 
and DIFFERENTIALS (and FULLS).
I guess I am not using a generic BASE backup but a specific one per server.

Pablo

- Original Message -
From: TipMeAbout tipmeab...@gmail.com
To: bacula-users@lists.sourceforge.net
Cc: Pablo Marques pmarq...@miamilinux.net
Sent: Tuesday, May 24, 2011 2:41:07 AM
Subject: Re: [Bacula-users] Base jobs and incremental backups



Le mardi 24 mai 2011 j'ai reçu le message suivant:



 Maybe laptop-lan and laptop-wlan need to be the same client (same name). 



 Just change the ip address of the client configuration of laptop-lan to be

 the wlan ip and run a FULL backup on it.



 Let me know.



 Pablo



Hello,



That would be strange if the 2 needs to be the same name or IP address as in 
the doc it is said that it can use backups from other clients as template of 
backup:



-

extract of the doc:



A new Job directive Base=Jobx, Joby... permits to specify the list of files 
that will be used during Full backup as base. Job { Name = BackupLinux Level= 
Base ... } Job { Name = BackupZog4 Base = BackupZog4, BackupLinux Accurate = 
yes ... }

In this example, the job BackupZog4 will use the most recent version of all 
files contained in BackupZog4 and BackupLinux jobs. Base jobs should have run 
with level=Base to be used.

-



Changing IP as you request would make me change the IP of the server too (WiFi 
and LAN are not in the same network), so first I would like to do more 
investigations without modifying IPs.

I will make some quick tests with small directories.



Could you post here the example of you base job and related job please ?



Thanks !



JC



--

http://www.tipmeabout.org

http://www.tipmeabout.com --
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery, 
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now. 
http://p.sf.net/sfu/quest-d2dcopy1___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Base jobs and incremental backups

[Pablo Marques]

Maybe laptop-lan and laptop-wlan need to be the same client (same name).

Just change the ip address of the client configuration of laptop-lan to be the 
wlan ip and run a FULL backup on it.

Let me know.

Pablo
- Original Message -
From: TipMeAbout tipmeab...@gmail.com
To: bacula-users@lists.sourceforge.net
Cc: Pablo Marques pmarq...@miamilinux.net
Sent: Monday, May 23, 2011 5:40:17 PM
Subject: Re: [Bacula-users] Base jobs and incremental backups

Le mercredi 18 mai 2011 j'ai reçu le message suivant:

  Are there any informations in the log file that prove the first full
  after a base job is effectively using the base job ?
 [...]


Hello,

I'm back with news, but not good.

I have done a base job backup of the laptop system through LAN connection (job 
name: laptop-lan_system.basejob). Below the definition of this job:

Job {
Name= laptop-lan_system.basejob
Enabled= yes
Type= Backup
Level= Base # can be defined in the schedule resource
Client= laptop-lan
FileSet= system
Messages= Standard
Pool= full
Full Backup Pool= full
Differential Backup Pool= diff
Incremental Backup Pool= incr
Schedule= full_1st_of_month
Storage= VTD_08
Priority= 9

}


Then a full backup of this client through WIFI (job name:laptop-wlan_system) 
and mentionning that this job should help itself with the previous base job.  
Below the definition of this job:

Job {
Name= laptop-wlan_system
Enabled= yes
Type= backup
Base= laptop-lan_system.basejob
Level= incremental # can be defined in the schedule resource
Client= laptop-wlan
FileSet= system
Messages= Standard
Pool= incr
Full Backup Pool= full
Differential Backup Pool= diff
Incremental Backup Pool= incr
Schedule= incr_everyday_but_1st
Storage= VTD_04

}

As I have launched manually the full, I have change in command line the level 
and pool of the job laptop-wlan_system to set them to full.


But unfortunately, it take more than 5 hours to do the backup and from the 
report at the end of the backup, it did not take the base job into 
consideration as no line appears to indicate the percentage of use from the 
base job:

  Build OS:   x86_64-redhat-linux-gnu redhat 
  JobId:  1235
  Job:laptop-wlan_system.2011-05-23_08.46.24_03
  Backup Level:   Full
  Client: laptop-wlan 5.0.3 (04Aug10) x86_64-redhat-linux-
gnu,redhat,
  FileSet:system 2011-03-18 08:47:54
  Pool:   full (From Job FullPool override)
  Catalog:bacula (From Client resource)
  Storage:VTD_04 (From Job resource)
  Scheduled time: 23-mai-2011 08:46:06
  Start time: 23-mai-2011 08:46:26
  End time:   23-mai-2011 14:05:18
  Elapsed time:   5 hours 18 mins 52 secs
  Priority:   10
  FD Files Written:   131,130
  SD Files Written:   131,130
  FD Bytes Written:   7,233,904,662 (7.233 GB)
  SD Bytes Written:   7,252,157,429 (7.252 GB)
  Rate:   378.1 KB/s
  Software Compression:   None
  VSS:no
  Encryption: no
  Accurate:   no
  Volume name(s): full_vol-027|full_vol-028|full_vol-029
  Volume Session Id:  19
  Volume Session Time:1306014285
  Last Volume Bytes:  574,149,829 (574.1 MB)
  Non-fatal FD errors:0
  SD Errors:  0
  FD termination status:  OK
  SD termination status:  OK
  Termination:Backup OK



So what's wrong with my configuration ?

Thanks for your help !

JC


-- 
http://www.tipmeabout.org
http://www.tipmeabout.com

--
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery, 
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now. 
http://p.sf.net/sfu/quest-d2dcopy1
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Base jobs and incremental backups

[Pablo Marques]

 Are there any informations in the log file that prove the first full after a 
 base job is effectively using the base job ? 
The Bacula report after the FULL gives you very useful information, including 
the BASE backup files used. 

This is a sample report after a FULL backup, notice the line Base files/Used 
files: 

JobId: 371 
Job: server-fd-data.2011-05-09_16.58.33_17 
Backup Level: Full (upgraded from Incremental) 
Client: server-fd 5.0.3 (04Aug10) Linux,Cross-compile,Win32 
FileSet: server-set 2011-05-09 14:29:09 
Pool: Yearly (From User input) 
Catalog: MyCatalog (From Client resource) 
Storage: FileStorage (From Job resource) 
Scheduled time: 09-May-2011 16:58:24 
Start time: 09-May-2011 17:02:10 
End time: 09-May-2011 18:01:16 
Elapsed time: 59 mins 6 secs 
Priority: 10 
FD Files Written: 39,344 
SD Files Written: 39,344 
FD Bytes Written: 8,600,464,982 (8.600 GB) 
SD Bytes Written: 8,606,510,471 (8.606 GB) 
Rate: 2425.4 KB/s 
Software Compression: 39.7 % 
Base files/Used files: 39336/39114 (99.44%) 
VSS: yes 
Encryption: no 
Accurate: yes 
Volume name(s): Tape-Year-0001 
Volume Session Id: 358 
Volume Session Time: 1302812565 
Last Volume Bytes: 252,270,109,076 (252.2 GB) 
Non-fatal FD errors: 0 
SD Errors: 0 
FD termination status: OK 
SD termination status: OK 
Termination: Backup OK 

For your case, depending on how much data changes on the Laptop, it maybe OK 
just to leave it on the WiFi for all backups (except for the BASE). 

Pablo 
- Original Message -
From: - - tipmeab...@gmail.com 
To: Pablo Marques pmarq...@miamilinux.net 
Cc: bacula-users@lists.sourceforge.net 
Sent: Wednesday, May 18, 2011 9:10:55 AM 
Subject: Re: [Bacula-users] Base jobs and incremental backups 


2011/5/18 Pablo Marques  pmarq...@miamilinux.net  



When you do a Base backup, you need to do a FULL immediately after, 
because the BASE backup is a special backup and you cannot restore only from 
it. 

From then on you can do Incrementals, Fulls or Differentials. 

All the FULL backups that you will run after that fist FULL are going to be 
very small, 
so you should only need to have the laptop on Gig on your first BASE backup. 

Hope this helps. 

Pablo 








Hello ! 

That's an interesting point ! I did not understand when read the documentation 
that a full is necessary after a base job. I though a base replaces a full. 
For me the interesting point of a base job was that you can do a base job of a 
template server, and then backup n servers directly without having n full 
backups to do. Writing that, I now understand better the concept of base jobs, 
all the n full would contain pointers on the base jobs. Are there any 
informations in the log file that prove the first full after a base job is 
effectively using the base job ? 
Something like running 'client-bkp job' based on 'based-job'  ? 

So for my problem, I should try to do a floating IP between the 2 interface - 
GiB and wifi - and do the backup on this interface: if the LAN cable is 
connected, backup goes through the LAN, if there is only the wifi it would use 
it. I think I can do that with a piece of script. eth0 would be client-lan, 
wlan0 client-wlan and the backup interface would be client-bkp pointing on 
eth0:0 if cable is connected or wlan0:0. 

Thanks for your help ! 

JC 



blockquote


- Original Message - 
From: TipMeAbout  tipmeab...@gmail.com  
To: bacula-users@lists.sourceforge.net 
Sent: Tuesday, May 17, 2011 5:09:10 PM 
Subject: Re: [Bacula-users] Base jobs and incremental backups 

 On Mon, May 16, 2011 at 11:07 PM, TipMeAbout  tipmeab...@gmail.com  wrote: 
  Hello all, 
  
  I use Bacula for some times now and I experiment now the base job backup. 
  I have to solve a problem and I need help: I have to backup a laptop. 
  
  This laptop is most of the time connected by wifi. As it contains quite 
  large 
  data, I have decided to do a full the 1st of each month and an 
  incremental the 
  other days. 
  To let the full run quickly, I decided that the laptop will be connected 
  by its LAN 1Gb connection the 1st and by wifi the rest of the month. So 
  I have 2 
  client instances configured: client-lan and client-wlan, one for each 
  type 
  of backup, each with its own IP address. But when I start an incremental 
  for 
  client-wlan, Bacula tells me it does not find a valid full, so it 
  starts a 
  full through the wifi connection. Too long !! 
  I have decided to do a base job the 1st of the month by LAN for instance 
  client-lan and then each incremental for client-wlan would be based 
  on this base job. It does not work neither as it still starts a full 
  backup in wifi mode instead of incremental. I have read that a base job 
  is like a full 
  and let full backup base on it: so I change my incremental backup by a 
  full with hope it would saved only some datas, for instance 
  client-wlan. But after a while running, I have the impression Bacula 
  does a new total full without taking

Re: [Bacula-users] Base jobs and incremental backups

[Pablo Marques]

When you do a Base backup, you need to do a FULL immediately after, 
because the BASE backup is a special backup and you cannot restore only from 
it.

From then on you can do Incrementals, Fulls or Differentials.

All the FULL backups that you will run after that fist FULL are going to be 
very small,
so you should only need to have the laptop on Gig on your first BASE backup. 

Hope this helps.

Pablo

- Original Message -
From: TipMeAbout tipmeab...@gmail.com
To: bacula-users@lists.sourceforge.net
Sent: Tuesday, May 17, 2011 5:09:10 PM
Subject: Re: [Bacula-users] Base jobs and incremental backups

 On Mon, May 16, 2011 at 11:07 PM, TipMeAbout tipmeab...@gmail.com wrote:
  Hello all,
  
  I use Bacula for some times now and I experiment now the base job backup.
  I have to solve a problem and I need help: I have to backup a laptop.
  
  This laptop is most of the time connected by wifi. As it contains quite
  large
  data, I have decided to do a full the 1st of each month and an
  incremental the
  other days.
  To let the full run quickly, I decided that the laptop will be connected
  by its LAN 1Gb connection the 1st and by wifi the rest of the month. So
  I have 2
  client instances configured: client-lan and client-wlan, one for each
  type
  of backup, each with its own IP address. But when I start an incremental
  for
  client-wlan, Bacula tells me it does not find a valid full, so it
  starts a
  full through the wifi connection. Too long !!
  I have decided to do a base job the 1st of the month by LAN for instance
  client-lan and then each incremental for client-wlan would be based
  on this base job. It does not work neither as it still starts a full
  backup in wifi mode instead of incremental. I have read that a base job
  is like a full
  and let full backup base on it: so I change my incremental backup by a
  full with hope it would saved only some datas, for instance
  client-wlan. But after a while running, I have the impression Bacula
  does a new total full without taking into consideration the base job
  backup.
  
  So my questions are simple: does base job can be used with incremental
  backups
  to realise what I would like and how to do that ?
  
  Thanks in advance for your help !
  
  
  JC
  
  
  --
  http://www.tipmeabout.org
 
 Rather than try to backup the laptop with Bacula, I would use RSYNC to on
 the laptop and then backup the rsync mirror of the laptop, that way you
 don't rely on the laptop being connected and once you have done the initial
 rsync the  incrementals are much easier to manage over your WiFi bandwidth.
 You could also trigger your rsync job to run when the interface comes up,
 and if you did this over SSH via an Internet resolvable f.q.d.n then you
 can backup from anywhere
 
 Although Bacula is fantastic for backing up, sometimes other tools can
 make the overall process better


Hello,
 
Thanks for your answer !
Of course I could use other tools, but the challenge is to do that with bacula 
and to highlight how to use base job backup as a base for incremental backup.
  
By using RSYNC, I have to provide space left on the bacula server which would 
act as mirror, and then space to backup this mirror with bacula (I use backup 
on disk file of 4GiB). 
Moreover, the rsync would mirror data I don't care (iso, mpeg, jpg) which are 
space and bandwidth consuming through wifi; with bacula, it excluded from the 
beginning. And maintaining 2 lists of exclusion - one for each tool - is not 
the most recommended I think.
 
If someone else has a proposal, I'm still open for an answer.
I will myself think if a solution based on virtual IPs can be set up. If I 
found something, I would of course complete this thread.
 
 See you all !
 
 JC


-- 
http://www.tipmeabout.org

--
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its 
next-generation tools to help Windows* and Linux* C/C++ and Fortran 
developers boost performance applications - including clusters. 
http://p.sf.net/sfu/intel-dev2devmay
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its 
next-generation tools to help Windows* and Linux* C/C++ and Fortran 
developers boost performance applications - including clusters. 
http://p.sf.net/sfu/intel-dev2devmay
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pool per client

[Pablo Marques]

Thanks Randy, 

Unfortunately Maximum Concurrent Jobs won't cut it because I need a different 
tape per client. Still I set it to 1000. 
When spooling is enabled, bacula still wants a tape from the client pool on the 
drive before the spooling starts. 

I could create a virtual library with MHVTL and use several drives or use a 
disk-changer emulator. But I am not sure how scalable these solutions are. 

Pablo 
- Original Message -
From: Randy Katz rk...@simplicityhosting.com 
To: bacula-users@lists.sourceforge.net 
Sent: Wednesday, April 13, 2011 6:08:54 AM 
Subject: Re: [Bacula-users] Pool per client 

On 4/12/2011 4:04 PM, Pablo Marques wrote: 


I have a setup to backup lots of clients over slow links. 
I want to have each client (or group of clients) backed up to dedicated client 
pools, so client1 will go to pool client1, and so on. 
That way I have better control of the space used, if a client goes away I can 
simply delete the tapes (or files) an get the space back immediately. 
Also it gives me better control on the retention on a per client basis. 

The problem is that when I try to backup multiple clients at the same time, the 
storage process has to wait for each job to finish before it can move to the 
next because it needs to change the tape (different client -- different pool). 
Some clients may take many hours to finish, forcing everybody else to wait. 

I enabled spooling, but it seems like Bacula requires to mount a tape from the 
client pool on a drive before the client spooling can begin. 
Can this be avoided? 

A possible solution would be to do all backups on a special pool and after they 
are done migrate later each client job to each client pool. 
But I cannot find a way to modify the Next Pool dynamically. It is a fixed 
setting on the Pool definition. 

Does anybody have suggestions on how to accomplish this? 


Look into Maximum Concurrent Jobs in your Storage definition. 

Regards, 
Randy 

-- 
Forrester Wave Report - Recovery time is now measured in hours and minutes 
not days. Key insights are discussed in the 2010 Forrester Wave Report as 
part of an in-depth evaluation of disaster recovery service providers. 
Forrester found the best-in-class provider in terms of services and vision. 
Read this report now! http://p.sf.net/sfu/ibm-webcastpromo 
___ 
Bacula-users mailing list 
Bacula-users@lists.sourceforge.net 
https://lists.sourceforge.net/lists/listinfo/bacula-users 
--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pool per client

[Pablo Marques]

Thanks Greg,

But I would still have the problem that I need a device tied up backing up each 
client.
The problem I am facing is that I need to backup lots of slow clients, and I 
need to come up with something so I can back them up all at the _same_ time on 
one or maybe a few devices, and still have a Pool per client.

Pablo
- Original Message -
From: Greg Kurzawa gkurz...@pamida.com
To: Pablo Marques pmarq...@miamilinux.net
Cc: bacula-users@lists.sourceforge.net
Sent: Wednesday, April 13, 2011 9:29:31 AM
Subject: RE: [Bacula-users] Pool per client




Hi Pablo,



If you have enough disk space handy, you could send each Client’s data to its 
own disk Pool with its own Next Pool specification. Each Client’s data would be 
in its own Pool on disk, then move to its own Pool on tape. This is exactly 
what I’ve done at my site, except the disk Pools all point to the same tape 
Pool.



Greg





From: Pablo Marques [mailto:pmarq...@miamilinux.net]
Sent: Wednesday, April 13, 2011 7:28 AM
To: Randy Katz
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Pool per client




Thanks Randy,

Unfortunately Maximum Concurrent Jobs won't cut it because I need a different 
tape per client. Still I set it to 1000.
When spooling is enabled, bacula still wants a tape from the client pool on the 
drive before the spooling starts.

I could create a virtual library with MHVTL and use several drives or use a 
disk-changer emulator. But I am not sure how scalable these solutions are.

Pablo
- Original Message -


From: Randy Katz rk...@simplicityhosting.com
To: bacula-users@lists.sourceforge.net
Sent: Wednesday, April 13, 2011 6:08:54 AM
Subject: Re: [Bacula-users] Pool per client

On 4/12/2011 4:04 PM, Pablo Marques wrote:


I have a setup to backup lots of clients over slow links.
I want to have each client (or group of clients) backed up to dedicated client 
pools, so client1 will go to pool client1, and so on.
That way I have better control of the space used, if a client goes away I can 
simply delete the tapes (or files) an get the space back immediately.
Also it gives me better control on the retention on a per client basis.

The problem is that when I try to backup multiple clients at the same time, the 
storage process has to wait for each job to finish before it can move to the 
next because it needs to change the tape (different client -- different pool). 
Some clients may take many hours to finish, forcing everybody else to wait.

I enabled spooling, but it seems like Bacula requires to mount a tape from the 
client pool on a drive before the client spooling can begin.
Can this be avoided?

A possible solution would be to do all backups on a special pool and after they 
are done migrate later each client job to each client pool.
But I cannot find a way to modify the Next Pool dynamically. It is a fixed 
setting on the Pool definition.

Does anybody have suggestions on how to accomplish this?

Look into Maximum Concurrent Jobs in your Storage definition.

Regards,
Randy

--
Forrester Wave Report - Recovery time is now measured in hours and minutes 
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now! http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users --
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pool per client

[Pablo Marques]

 But I would still have the problem that I need a device tied up backing
 up each client.  The problem I am facing is that I need to backup lots of
 slow clients, and I need to come up with something so I can back them up
 all at the _same_ time on one or maybe a few devices, and still have a
 Pool per client.

I'm not clear if you're trying to avoid lots of physical devices or lots of
bacula storage device definitions.  You could create one Device {} entry
per client in the bacula-sd.conf.  These each correspond to a different
directory on some filesystem.   You then run each backup to its own file
Device -- these can all happen concurrently.

You should then be able to migrate each one in turn to tape.

Or maybe I've missed something?

I would try to avoid changing the device definitions in bacula-sd.conf every 
time I add or delete a client, as this could happen very often.

If I backup, let's say, 500 clients at night, the ideal would be to back them 
up all to the same device at the same time. If one client stalls or loses the 
connection, the others can continue without problems.
If I tie up a device per client, if a client has problems it could render the 
device unusable until the client finishes or the job times out.

I guess I could modify bacula-sd an add/remove a file device per client as 
needed. I am not sure if I can reload bacula-sd.conf without interrupting 
running backups.

When I add a client I have a template client definition with all per client 
definitions that I need:
I replace $CLIENT_NAME $IP_ADDRESS $PORT and generate a new file and then I do 
a reload on bconsole, and the client is ready to go. 
The clients, in my application, decide the backup schedule, and they run it 
from their bconsole client.
Each client can only run or restore its own backups. 

=
 # We need $CLIENT_NAME $IP_ADDRESS $PORT
Client {
  Name = $CLIENT_NAME-fd
  Address = $IP_ADDRESS
  FDPort = $PORT
  Catalog = MyCatalog
  Password = xccxcc  # password for FileDaemon
  File Retention = 5 year
  Job Retention = 15 years
  AutoPrune = yes # Prune expired Jobs/Files
}
Console {
  Name = $CLIENT_NAME
  Password = $CLIENT_NAMEpassword
  JobACL = $CLIENT_NAME-fd-data,$CLIENT_NAME-Restore
  ClientACL = $CLIENT_NAME-fd
  StorageACL = CHANGER
  ScheduleACL = *all*
  PoolACL = $CLIENT_NAME
  FileSetACL = $CLIENT_NAME-set
  CatalogACL = MyCatalog
  WhereACL = *all*
  CommandACL = run, restore
}

Job {
  Name = $CLIENT_NAME-base-fd-data
  JobDefs = jobbaculadefs
  Client = $CLIENT_NAME-fd
  FileSet = $CLIENT_NAME-set
  Pool = $CLIENT_NAME
  Level = Base
  SpoolData = yes
  Maximum Concurrent Jobs = 1000
  Max Run Sched Time = 86400
}

Job {
  Name = $CLIENT_NAME-fd-data
  JobDefs = jobbaculadefs
  Client = $CLIENT_NAME-fd
  FileSet = $CLIENT_NAME-set
  Pool = $CLIENT_NAME
  Base = $CLIENT_NAME-base-fd-data
  Accurate = yes
  SpoolData = yes
  Maximum Concurrent Jobs = 1000
  Max Run Sched Time = 86400
}

Job {
  Name = $CLIENT_NAME-restore
  Type = Restore
  Client = $CLIENT_NAME-fd
  FileSet=$CLIENT_NAME-set
  Storage = CHANGER
  Pool = $CLIENT_NAME
  Messages = Standard
  Where = /
}

Pool {
   Name = $CLIENT_NAME
   Pool Type = Backup
   Recycle = yes
   AutoPrune = yes
   Volume Retention = 15 years
   Recycle Oldest Volume = yes
   Recycle Pool = Scratch
}
# We need $CLIENT_NAME 
FileSet {
  Name = $CLIENT_NAME-set
  Include {
Options {
  signature = MD5
  compression = GZIP
  Sparse = yes
  }
  @/etc/bacula/clients-configs/$CLIENT_NAME-filelist
  }
}
===

Hope this clarifies my setup. 

Pablo
--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pool per client

[Pablo Marques]

Thanks Greg,

The key in my application is if I can modify bacula-sd.conf dynamically 
without a bacula-sd restart every time I add or remove a client.

bacula-sd restart would kill running jobs.

Pablo

Kurzawa, Greg wrote on Wednesday, April 13, 2011
 Pablo,
 
 
 
 If you’re backing up to disk, you can have as many Jobs running
 simultaneously as you want/need.  The trick here is to define not only a
 unique Pool resource for each Client, but unique Storage, Device and Media
 Type resources as well.  Each Client will have its own resources, so there
 will be no contention.  Following are the relevant pieces of my
 configuration, snipped for readability.
 
 
 
 I admit that when I was setting this up it seemed a bit bulky, but it
 works.  I back up many slow Clients to disk; their jobs all begin and run
 simultaneously.
 


 
 
 Of course, if you can’t back up to disk first, this is useless to you.
 
 
 
  CLIENTS 
 
 
 
 Client {
 
   Name = DORCAS
 
   Address = DORCAS.URTH.COM
 
 }
 
 
 
 Client {
 
   Name = SEVERIAN
 
   Address = SEVERIAN.URTH.COM
 
 }
 
 
 
  JOBS 
 
 
 
 Job {
 
   name = DAILY:severian
 
   client = SEVERIAN
 
   pool = SEVERIAN
 
 }
 
 
 
 Job {
 
   name = DAILY:dorcas
 
   client = DORCAS
 
   pool = DORCAS
 
 }
 
 
 
  POOLS 
 
 
 
 Pool {
 
   Name = SEVERIAN
 
   Pool Type = Backup
 
   Storage = SAN:severian
 
   next pool = LTO4
 
   LabelFormat = severian-
 
 }
 
 
 
 Pool {
 
   Name = DORCAS
 
   Pool Type = Backup
 
   Storage = SAN:dorcas
 
   next pool = LTO4
 
   LabelFormat = dorcas-
 
 }
 
 
 
  STORAGE 
 
 
 
 Storage {
 
   Name = SAN:severian
 
   Device = SAN:severian
 
   Media Type = FILE:severian
 
 }
 
 
 
 Storage {
 
   Name = SAN:dorcas
 
   Device = SAN:dorcas
 
   Media Type = FILE:dorcas
 
 }
 
 
 
 And in bacula-sd.conf:
 
 
 
 Device {
 
   Name = SAN:severian
 
   Media Type = FILE:severian
 
   Archive Device = /dp-SAN/severian
 
 }
 
 
 
 Device {
 
   Name = SAN:dorcas
 
   Media Type = FILE:dorcas
 
   Archive Device = /dp-SAN/dorcas
 
 }
 
 
 
 Greg
 
 
 
 
 
 From: Pablo Marques [mailto:pmarq...@miamilinux.net]
 Sent: Wednesday, April 13, 2011 11:01 AM
 To: Kurzawa, Greg
 Cc: bacula-users@lists.sourceforge.net
 Subject: Re: [Bacula-users] Pool per client
 
 
 
 Thanks Greg,
 
 But I would still have the problem that I need a device tied up backing up
 each client. The problem I am facing is that I need to backup lots of slow
 clients, and I need to come up with something so I can back them up all at
 the _same_ time on one or maybe a few devices, and still have a Pool per
 client.
 
 Pablo
 
 
 
 From: Greg Kurzawa gkurz...@pamida.com
 To: Pablo Marques pmarq...@miamilinux.net
 Cc: bacula-users@lists.sourceforge.net
 Sent: Wednesday, April 13, 2011 9:29:31 AM
 Subject: RE: [Bacula-users] Pool per client
 
 
 
 
 Hi Pablo,
 
 
 
 If you have enough disk space handy, you could send each Client’s data to
 its own disk Pool with its own Next Pool specification.  Each Client’s
 data would be in its own Pool on disk, then move to its own Pool on tape. 
 This is exactly what I’ve done at my site, except the disk Pools all point
 to the same tape Pool.
 
 
 
 Greg
 
 
 
 From: Pablo Marques [mailto:pmarq...@miamilinux.net]
 Sent: Wednesday, April 13, 2011 7:28 AM
 To: Randy Katz
 Cc: bacula-users@lists.sourceforge.net
 Subject: Re: [Bacula-users] Pool per client
 
 
 
 Thanks Randy,
 
 Unfortunately Maximum Concurrent Jobs won't cut it because I need a
 different tape per client.  Still I set it to 1000. When spooling is
 enabled, bacula still wants a tape from the client pool on the drive
 before the spooling starts.
 
 I could create a virtual library with MHVTL and use several drives or use a
 disk-changer emulator. But I am not sure how scalable these solutions are.
 
 Pablo
 
 
 
 From: Randy Katz rk...@simplicityhosting.com
 To: bacula-users@lists.sourceforge.net
 Sent: Wednesday, April 13, 2011 6:08:54 AM
 Subject: Re: [Bacula-users] Pool per client
 
 On 4/12/2011 4:04 PM, Pablo Marques wrote:
 
 I have a setup to backup lots of clients over slow links.
 I want to have each client (or group of clients) backed up to dedicated
 client pools, so client1 will go to pool client1, and so on. That way I
 have better control of the space used, if a client goes away I can simply
 delete the tapes (or files) an get the space back immediately. Also it
 gives me better control on the retention on a per client basis.
 
 The problem is that when I try to backup multiple clients at the same time,
 the storage process has to wait for each job to finish before it can move
 to the next because it needs to change the tape (different client --
 different pool). Some clients may take many hours to finish, forcing
 everybody else to wait.
 
 I enabled spooling, but it seems like Bacula requires to mount a tape from
 the client pool on a drive before the client

Re: [Bacula-users] Pool per client

[Pablo Marques]

Martin,

This hack looks very promissing.
I will test it and let you know.

Pablo

- Original Message -
From: Martin Simmons mar...@lispworks.com
To: bacula-users@lists.sourceforge.net
Sent: Wednesday, April 13, 2011 11:24:24 AM
Subject: Re: [Bacula-users] Pool per client

 On Tue, 12 Apr 2011 19:04:40 -0400 (EDT), Pablo Marques said:
 
 I enabled spooling, but it seems like Bacula requires to mount a tape from
 the client pool on a drive before the client spooling can begin. 
 Can this be avoided? 

AFAIK, no.


 A possible solution would be to do all backups on a special pool and after
 they are done migrate later each client job to each client pool. 
 But I cannot find a way to modify the Next Pool dynamically. It is a fixed
 setting on the Pool definition.
 
 Does anybody have suggestions on how to accomplish this? 

There is a hack:

http://thread.gmane.org/gmane.comp.sysutils.backup.bacula.devel/14084

__Martin

--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pool per client

[Pablo Marques]

This setup would work, but as it turns out, when you have Base jobs, you cannot 
do migrations.

Fatal error: sql_create.c:1147 Can't Copy/Migrate job using BaseJob

Pablo
- Original Message -
From: Pablo Marques pmarq...@miamilinux.net
To: Martin Simmons mar...@lispworks.com
Cc: bacula-users@lists.sourceforge.net
Sent: Wednesday, April 13, 2011 3:03:44 PM
Subject: Re: [Bacula-users] Pool per client

Martin,

This hack looks very promissing.
I will test it and let you know.

Pablo

- Original Message -
From: Martin Simmons mar...@lispworks.com
To: bacula-users@lists.sourceforge.net
Sent: Wednesday, April 13, 2011 11:24:24 AM
Subject: Re: [Bacula-users] Pool per client

 On Tue, 12 Apr 2011 19:04:40 -0400 (EDT), Pablo Marques said:
 
 I enabled spooling, but it seems like Bacula requires to mount a tape from
 the client pool on a drive before the client spooling can begin. 
 Can this be avoided? 

AFAIK, no.


 A possible solution would be to do all backups on a special pool and after
 they are done migrate later each client job to each client pool. 
 But I cannot find a way to modify the Next Pool dynamically. It is a fixed
 setting on the Pool definition.
 
 Does anybody have suggestions on how to accomplish this? 

There is a hack:

http://thread.gmane.org/gmane.comp.sysutils.backup.bacula.devel/14084

__Martin

--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Benefiting from Server Virtualization: Beyond Initial Workload 
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve 
application availability and disaster protection. Learn more about boosting 
the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Pool per client

[Pablo Marques]

Gavin:

I tried signalling bacula-sd, but it does not seem to work and still sees the 
same configuration until it is completely restarted.
I will try creating a bacula-sd instance per client and see...


Pablo
- Original Message -
From: Gavin McCullagh gavin.mccull...@gcd.ie
To: Pablo Marques pmarq...@miamilinux.net
Cc: bacula-users@lists.sourceforge.net
Sent: Wednesday, April 13, 2011 4:21:23 PM
Subject: Re: [Bacula-users] Pool per client

Hi,

On Wed, 13 Apr 2011, Pablo Marques wrote:

 I guess I could modify bacula-sd an add/remove a file device per client
 as needed. I am not sure if I can reload bacula-sd.conf without
 interrupting running backups.

My understanding is that you need a restart, which is likely to kill any
running backups.

 When I add a client I have a template client definition with all per client 
 definitions that I need:
 I replace $CLIENT_NAME $IP_ADDRESS $PORT and generate a new file and then I 
 do a reload on bconsole, and the client is ready to go. 
 The clients, in my application, decide the backup schedule, and they run it 
 from their bconsole client.
 Each client can only run or restore its own backups. 

I do something similar actually, but it's not quite as big or dynamic so I
manually modify the bacula-sd and don't generally have to worry about
backups that are running at the time I restart the bacula-sd.

This might seem (or be) crazy, but in principal you could run a bacula-sd
for each client with its own bacula-sd instance running on a particular
ip:port.  That would avoid the issue of restarting bacula-sd.

Of course it might be possible to signal bacula-sd to re-read its config,
which would be much simpler :-)

Gavin


-- 
Gavin McCullagh
Senior System Administrator
IT Services
Griffith College 
South Circular Road
Dublin 8
Ireland
Tel: +353 1 4163365
http://www.gcd.ie
http://www.gcd.ie/brochure.pdf
http://www.gcd.ie/opendays
http://www.gcd.ie/ebrochure

This E-mail is from Griffith College.
The E-mail and any files transmitted with it are confidential and may be
privileged and are intended solely for the use of the individual or entity
to whom they are addressed. If you are not the addressee you are prohibited
from disclosing its content, copying it or distributing it otherwise than to
the addressee. If you have received this e-mail in error, please immediately
notify the sender by replying to this e-mail and delete the e-mail from your
computer.

Bellerophon Ltd, trades as Griffith College (registered in Ireland No.
60469) with its registered address as Griffith College Campus, South
Circular Road, Dublin 8, Ireland.


--
Benefiting from Server Virtualization: Beyond Initial Workload 
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve 
application availability and disaster protection. Learn more about boosting 
the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Pool per client

[Pablo Marques]

I have a setup to backup lots of clients over slow links. 
I want to have each client (or group of clients) backed up to dedicated client 
pools, so client1 will go to pool client1, and so on. 
That way I have better control of the space used, if a client goes away I can 
simply delete the tapes (or files) an get the space back immediately. 
Also it gives me better control on the retention on a per client basis. 

The problem is that when I try to backup multiple clients at the same time, the 
storage process has to wait for each job to finish before it can move to the 
next because it needs to change the tape (different client -- different pool). 
Some clients may take many hours to finish, forcing everybody else to wait. 

I enabled spooling, but it seems like Bacula requires to mount a tape from the 
client pool on a drive before the client spooling can begin. 
Can this be avoided? 

A possible solution would be to do all backups on a special pool and after they 
are done migrate later each client job to each client pool. 
But I cannot find a way to modify the Next Pool dynamically. It is a fixed 
setting on the Pool definition. 

Does anybody have suggestions on how to accomplish this? 

Thanks 

Pablo 


--
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users