Re: [Bacula-users] can file retention be job rather than client based?

[mark . bergman]

In the message dated: Wed, 11 Apr 2018 06:25:43 -0700,
The pithy ruminations from Stephen Thompson on 
[Re: [Bacula-users] can file retention be job rather than client based?] were:
=> 
=> Thanks Kern.
=> 
=> I think given the limited nature of his need, I may use a postrun script 

+1 for this request.

I've got what may be a similar environment -- one major Bacula "client"
(ie., a floating VIP/hostname pointing to a fileserver) with ~150 jobs,
ranging from KB to multi-TB. Varying retention periods per-job would be
a tremendous enhancement.

=> to simply wipe database records out of band.
=> 
=> Also if I did use multi-client definitions, I would need to use the same 
=> pool as they all go to the same monthly tapes.

Same here.

Mark


=> 
=> Stephen
=> 

-- 
Mark Bergman   voice: 215-746-4061  
 
mark.berg...@uphs.upenn.edu  fax: 215-614-0266
http://www.med.upenn.edu/cbica/
IT Technical Director, Center for Biomedical Image Computing and Analytics
Department of Radiology University of Pennsylvania

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] can file retention be job rather than client based?

[Dimitri Maziuk]

On 2018-04-11 01:59, Kern Sibbald wrote:

I have never used two Client definitions to backup the same machine, but 
in principle it would work fine.


The closest I have is a "cluster ip" floating between two machines, with 
3 client/job definitions: one for each physical machine (backing up /etc 
and crontabs) and one for the cluster ip (backing up DRBD filesystem 
mounted on the host that has cluster ip).


The only thing non-obvious to me was that bacula-fd running as

FileDaemon {
  name = nodea-fd
...
}

will happily answer to (bacula-dir)

Client {
  name = nodea-fd
  address = 1.2.3.4
  ...
}
Client {
  name = nodeb-fd
  address = 1.2.3.5
  ...
}
Client {
  name = cluster-fd
  address = 1.2.3.1
  ...
}
Job {
  name = cluster-drbd
  client = cluster-fd
  fileset = drbd
  ...
}

(I.e there is no FileDaemon named "cluster-fd" running anywhere but the 
Job and Client definitions for it work just fine.)


Dima

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] can file retention be job rather than client based?

[Stephen Thompson]

Thanks Kern.

I think given the limited nature of his need, I may use a postrun script 
to simply wipe database records out of band.


Also if I did use multi-client definitions, I would need to use the same 
pool as they all go to the same monthly tapes.


Stephen


On 4/10/18 11:59 PM, Kern Sibbald wrote:

Hello Stephen,

What you are asking for, as you suspect, does not exist and implementing 
it would be a bit problematic because every Job would need to keep it's 
own retention period.  For one client, there can be any number of Jobs 
-- typically thousands.  Thus the catalog would grow faster (more data 
for the File table having the most records), and the complexity of 
pruning including the time to prune would probably explode -- probably 
thousands of times slower.


I have never used two Client definitions to backup the same machine, but 
in principle it would work fine.  If you name your Clients appropriately 
it might be easier to remember what was done.  E.g. 
Client1-Normal-Files, Client1-Archived-Files, ... Also, if you put clear 
comments on the resource definitions, it would help.  Note two things, 
if you go this route:


1. Be sure to define each of your two Client1-xxx with different Pools 
with different Volume retention periods
2. I would appreciate feedback on how this works -- especially 
operationally


Best regards,
Kern

PS: At the current time the Enterprise version of Bacula has a number of 
performance improvements that should significantly speed up the backups 
of 50+million files.  It does this at a small extra expense (size) of 
the catalog.


On 04/07/2018 06:21 AM, Stephen Thompson wrote:


I believe the answer is no, but as a happy bacula user for 10 years I 
am somewhat surprised at the lack of flexibility.


The scenarios is this:  A fileserver (1 client) with dozens of large 
(size-wise) filesystems (12 jobs), but a couple of those filesystems 
are large (filecount-wise).  We would really like to set different 
file retention periods on those high-filecount jobs (50+million), 
because they are forcing the Catalog to go beyond our size 
constraints. However, we also don't want to lose the file retention 
longevity of that client's other jobs (5 years).  The only hack I can 
think of is to define 2 clients for 1 actual host, but I'd rather not 
go down that route, because tracking jobs and associating them, 
especially over multiple years, will get that much more tricky.


Ideas?

thanks,
Stephen


--
Stephen Thompson   Berkeley Seismo Lab
step...@seismo.berkeley.edu215 McCone Hall
Office: 510.664.9177   University of California
Remote: 510.214.6506 (Tue) Berkeley, CA 94720-4760

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] can file retention be job rather than client based?

[Kern Sibbald]

Hello Stephen,

What you are asking for, as you suspect, does not exist and implementing 
it would be a bit problematic because every Job would need to keep it's 
own retention period.  For one client, there can be any number of Jobs 
-- typically thousands.  Thus the catalog would grow faster (more data 
for the File table having the most records), and the complexity of 
pruning including the time to prune would probably explode -- probably 
thousands of times slower.


I have never used two Client definitions to backup the same machine, but 
in principle it would work fine.  If you name your Clients appropriately 
it might be easier to remember what was done.  E.g. 
Client1-Normal-Files, Client1-Archived-Files, ... Also, if you put clear 
comments on the resource definitions, it would help.  Note two things, 
if you go this route:


1. Be sure to define each of your two Client1-xxx with different Pools 
with different Volume retention periods

2. I would appreciate feedback on how this works -- especially operationally

Best regards,
Kern

PS: At the current time the Enterprise version of Bacula has a number of 
performance improvements that should significantly speed up the backups 
of 50+million files.  It does this at a small extra expense (size) of 
the catalog.


On 04/07/2018 06:21 AM, Stephen Thompson wrote:


I believe the answer is no, but as a happy bacula user for 10 years I 
am somewhat surprised at the lack of flexibility.


The scenarios is this:  A fileserver (1 client) with dozens of large 
(size-wise) filesystems (12 jobs), but a couple of those filesystems 
are large (filecount-wise).  We would really like to set different 
file retention periods on those high-filecount jobs (50+million), 
because they are forcing the Catalog to go beyond our size 
constraints. However, we also don't want to lose the file retention 
longevity of that client's other jobs (5 years).  The only hack I can 
think of is to define 2 clients for 1 actual host, but I'd rather not 
go down that route, because tracking jobs and associating them, 
especially over multiple years, will get that much more tricky.


Ideas?

thanks,
Stephen



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] can file retention be job rather than client based?

[Stephen Thompson]

Thanks... Yeah, I'm leaning towards a post or pre job script that 
actually prunes (or more likely purges) the file records I need to jettison.


Stephen


On 4/7/18 3:38 AM, Heitor Faria wrote:

Hello Stefaphen,


I believe the answer is no, but as a happy bacula user for 10 years I am
somewhat surprised at the lack of flexibility.


Alternative solutions with proprietary catalog data are much more inflexible.


The scenarios is this:  A fileserver (1 client) with dozens of large
(size-wise) filesystems (12 jobs), but a couple of those filesystems are
large (filecount-wise).  We would really like to set different file
retention periods on those high-filecount jobs (50+million), because
they are forcing the Catalog to go beyond our size constraints.
However, we also don't want to lose the file retention longevity of that
client's other jobs (5 years).  The only hack I can think of is to
define 2 clients for 1 actual host, but I'd rather not go down that
route, because tracking jobs and associating them, especially over
multiple years, will get that much more tricky.


File & Job Retention can be set in Pool resource instead of Client one.
You can also try to modify the prior used manual Bacula pruning Perl script to only 
prune files you need and not delete the whole job 
.
Finally, you can even use dynamically generated Filesets with lower or greater file 
sizes to automate their backup distribution. 



Ideas?

thanks,
Stephen
--


Regards,



--
Stephen Thompson   Berkeley Seismo Lab
step...@seismo.berkeley.edu215 McCone Hall
Office: 510.664.9177   University of California
Remote: 510.214.6506 (Tue) Berkeley, CA 94720-4760

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] can file retention be job rather than client based?

[Heitor Faria]

s/Stefaphen/Stephen/
Sorry for the typo. When you type as Joe Satriani playing "Speed of Light" this 
might happens. lol

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] can file retention be job rather than client based?

[Heitor Faria]

Hello Stefaphen,

> I believe the answer is no, but as a happy bacula user for 10 years I am
> somewhat surprised at the lack of flexibility.

Alternative solutions with proprietary catalog data are much more inflexible.

> The scenarios is this:  A fileserver (1 client) with dozens of large
> (size-wise) filesystems (12 jobs), but a couple of those filesystems are
> large (filecount-wise).  We would really like to set different file
> retention periods on those high-filecount jobs (50+million), because
> they are forcing the Catalog to go beyond our size constraints.
> However, we also don't want to lose the file retention longevity of that
> client's other jobs (5 years).  The only hack I can think of is to
> define 2 clients for 1 actual host, but I'd rather not go down that
> route, because tracking jobs and associating them, especially over
> multiple years, will get that much more tricky.

File & Job Retention can be set in Pool resource instead of Client one.
You can also try to modify the prior used manual Bacula pruning Perl script to 
only prune files you need and not delete the whole job 
.
Finally, you can even use dynamically generated Filesets with lower or greater 
file sizes to automate their backup distribution. 


> Ideas?
> 
> thanks,
> Stephen
> --

Regards,
-- 
=== 
Heitor Medrado de Faria | CEO Bacula do Brasil & USA | Visto EB-1 | LPIC-III | 
EMC 05-001 | ITIL-F 
• Não seja tarifado pelo tamanho dos seus backups, conheça o Bacula Enterprise 
http://www.bacula.com.br/enterprise/ 
• Ministro treinamento e implementação in-company do Bacula Community 
http://www.bacula.com.br/in-company/ 
• Compre o novo livro do Bacula http://www.bacula.com.br/livro 
Brazil +55 (61) 98268-4220 | USA +1 (323) 300-5387 | www.bacula.com.br 
 
Indicamos as capacitações complementares: 
Shell básico e Programação em Shell com Julio Neves | Zabbix com Adail Host. 


-- 
=== 
Heitor Medrado de Faria | EB-1 Visa | LPIC-III | ITIL-F | EMC 05-001 
• Check my Bacula Book at Amazon.com http://bacula.us/book 
• Do you need Bacula training? http://bacula.us/video-classes/ 
• USA +1 (323) 300-53 87 | Brazil +55 (61) 98268-4220 | http://bacula.us 
===

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users