Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Radosław Korzeniewski]

Hello,

sob., 11 kwi 2020 o 15:55 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
yateen.shaligram_bha...@nokia.com> napisał(a):

> Hi Radoslaw, Josh,
>
>
>
> Thanks for your recommendations, much appreciated.
>
>
>
> The way I am currently trying out Bacula Solution is as explained below:
>
>
>
>1. Bacula Server which hosts bacula-dir, bacula-sd, PostgreSQL
>2. FreeBSD based Filer with ZFS disk storage to hold Bacula volume
>files. This disk storage is NFS exported and mounted on the Bacula Server.
>
> ( This Filer holds only the Bacula volume files, nothing else)
>
The best should be to run Bacula SD on this server. Your current
configuration is not optimal as you are cutting your bandwidth in half.

>
>1. A dedicated point to point Gigabit network between the  Bacula
>server and the Filer
>2. There is a separate LAN between Bacula clients and the Bacula server
>3. I will try  shifting  the bacula-sd from Bacula server to Filer as
>suggested by both of you.
>
> Yes. This is a good step.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[dmaziuk via Bacula-users]

On 4/11/2020 8:55 AM, Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) wrote:


   1.  Bacula Server which hosts bacula-dir, bacula-sd, PostgreSQL
   2.  FreeBSD based Filer with ZFS disk storage to hold Bacula volume files. 
This disk storage is NFS exported and mounted on the Bacula Server.

( This Filer holds only the Bacula volume files, nothing else)

   1.  A dedicated point to point Gigabit network between the  Bacula server 
and the Filer
   2.  There is a separate LAN between Bacula clients and the Bacula server
   3.  I will try  shifting  the bacula-sd from Bacula server to Filer as 
suggested by both of you.


We have a setup with ZFS on linux storage mounted directly on bacula 
server, that works pretty well without spooling, but in your setup I 
would put a large(-ish, dep. on your volume size and number of 
concurrent jobs) SSD in the bacula server and spool data to it.


That way bacula will stream complete volumes from spool to the filer at 
your full bandwidth and avoid (potentially) many small writes over NFS.


Dima


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)]

Hi Radoslaw, Josh,

Thanks for your recommendations, much appreciated.

The way I am currently trying out Bacula Solution is as explained below:


  1.  Bacula Server which hosts bacula-dir, bacula-sd, PostgreSQL
  2.  FreeBSD based Filer with ZFS disk storage to hold Bacula volume files. 
This disk storage is NFS exported and mounted on the Bacula Server.

( This Filer holds only the Bacula volume files, nothing else)

  1.  A dedicated point to point Gigabit network between the  Bacula server and 
the Filer
  2.  There is a separate LAN between Bacula clients and the Bacula server
  3.  I will try  shifting  the bacula-sd from Bacula server to Filer as 
suggested by both of you.

-Yateen Bhagat




From: Radosław Korzeniewski 
Sent: Thursday, April 9, 2020 5:57 PM
To: Josh Fisher 
Cc: Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) 
; bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Issue with concurrent jobs in disk based auto 
changer

Hello,

czw., 9 kwi 2020 o 13:20 Josh Fisher 
mailto:jfis...@pvct.com>> napisał(a):


On 4/9/2020 4:09 AM, Radosław Korzeniewski wrote:
Hello,

wt., 7 kwi 2020 o 14:40 Josh Fisher mailto:jfis...@pvct.com>> 
napisał(a):


On 4/7/2020 7:20 AM, Radosław Korzeniewski wrote:
Hello,

wt., 7 kwi 2020 o 09:38 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) 
mailto:yateen.shaligram_bha...@nokia.com>> 
napisał(a):
Hi,

The issue is resolved after I increased the number of devices under a 
filechanger.
Nevertheless, the suggestion to keep the file server and the bacula-sd on the 
same host is good one.

If you are using backup to tape then yes, running a dedicated bacula-sd on file 
server is a good recommendation.

Also true if the file server in question is only for the backup volumes.
I assume a "file server" mentioned above is not a server which only holds 
backup volume files but a common sense of this term like storing user profiles, 
documents, production files, photos, movies, etc.
If the "file server" holds only backup volumes, then I personally do not name 
it "file server" but a backup server. Exporting backup volume files used by SD 
without a proper operational synchronization is not a good idea. It does not 
harm your backups when exported as read-only, but full-access...



OK. Backup server, then.
Great!

The point was to move SD to the host where the backup volume files are stored 
to prevent doubling the network traffic required.
Yes, it is a very recommended way to optimize backup paths.

If data is stored on that host as well, then care must be taken to ensure that 
the storage that the volume files are written to is isolated, physically and 
logically, from the storage that data is written to.
Absolutely. Thanks for clarification then.

best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net<mailto:rados...@korzeniewski.net>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Radosław Korzeniewski]

Hello,

czw., 9 kwi 2020 o 13:20 Josh Fisher  napisał(a):

>
> On 4/9/2020 4:09 AM, Radosław Korzeniewski wrote:
>
> Hello,
>
> wt., 7 kwi 2020 o 14:40 Josh Fisher  napisał(a):
>
>>
>> On 4/7/2020 7:20 AM, Radosław Korzeniewski wrote:
>>
>> Hello,
>>
>> wt., 7 kwi 2020 o 09:38 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
>> yateen.shaligram_bha...@nokia.com> napisał(a):
>>
>>> Hi,
>>>
>>>
>>>
>>> The issue is resolved after I increased the number of devices under a
>>> filechanger.
>>>
>>> Nevertheless, the suggestion to keep the file server and the bacula-sd
>>> on the same host is good one.
>>>
>>
>> If you are using backup to tape then yes, running a dedicated bacula-sd
>> on file server is a good recommendation.
>>
>> Also true if the file server in question is only for the backup volumes.
>>
> I assume a "file server" mentioned above is not a server which only holds
> backup volume files but a common sense of this term like storing user
> profiles, documents, production files, photos, movies, etc.
> If the "file server" holds only backup volumes, then I personally do not
> name it "file server" but a backup server. Exporting backup volume files
> used by SD without a proper operational synchronization is not a good idea.
> It does not harm your backups when exported as read-only, but
> full-access...
>
>
> OK. Backup server, then.
>
Great!

> The point was to move SD to the host where the backup volume files are
> stored to prevent doubling the network traffic required.
>
Yes, it is a very recommended way to optimize backup paths.

> If data is stored on that host as well, then care must be taken to ensure
> that the storage that the volume files are written to is isolated,
> physically and logically, from the storage that data is written to.
>
Absolutely. Thanks for clarification then.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Josh Fisher]

On 4/9/2020 4:09 AM, Radosław Korzeniewski wrote:

Hello,

wt., 7 kwi 2020 o 14:40 Josh Fisher > napisał(a):



On 4/7/2020 7:20 AM, Radosław Korzeniewski wrote:

Hello,

wt., 7 kwi 2020 o 09:38 Shaligram Bhagat, Yateen (Nokia -
IN/Bangalore) mailto:yateen.shaligram_bha...@nokia.com>> napisał(a):

Hi,

The issue is resolved after I increased the number of devices
under a filechanger.

Nevertheless, the suggestion to keep the file server and the
bacula-sd on the same host is good one.


If you are using backup to tape then yes, running a dedicated
bacula-sd on file server is a good recommendation.


Also true if the file server in question is only for the backup
volumes.

I assume a "file server" mentioned above is not a server which only 
holds backup volume files but a common sense of this term like storing 
user profiles, documents, production files, photos, movies, etc.
If the "file server" holds only backup volumes, then I personally do 
not name it "file server" but a backup server. Exporting backup volume 
files used by SD without a proper operational synchronization is not a 
good idea. It does not harm your backups when exported as read-only, 
but full-access...



OK. Backup server, then. The point was to move SD to the host where the 
backup volume files are stored to prevent doubling the network traffic 
required. If data is stored on that host as well, then care must be 
taken to ensure that the storage that the volume files are written to is 
isolated, physically and logically, from the storage that data is 
written to.





But! If you backup to file volumes and disks mounted directly on
this file server then it is - generally - not recommended. In the
worst case scenario you can lose all your data including backups.
Be aware.


Yes. It would never be a good idea to have backup volume files and
data on the same set of physical disks.

NO, it doesn't matter if it is the same physical disk, any number of 
separate devices, lvm or raid groups or just a network filesystem 
mounted remotely. If it is available for writing with a standard file 
io api then it exposes a huge risk! You should never do that.

best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net 
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Radosław Korzeniewski]

Hello,

wt., 7 kwi 2020 o 14:40 Josh Fisher  napisał(a):

>
> On 4/7/2020 7:20 AM, Radosław Korzeniewski wrote:
>
> Hello,
>
> wt., 7 kwi 2020 o 09:38 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
> yateen.shaligram_bha...@nokia.com> napisał(a):
>
>> Hi,
>>
>>
>>
>> The issue is resolved after I increased the number of devices under a
>> filechanger.
>>
>> Nevertheless, the suggestion to keep the file server and the bacula-sd on
>> the same host is good one.
>>
>
> If you are using backup to tape then yes, running a dedicated bacula-sd on
> file server is a good recommendation.
>
> Also true if the file server in question is only for the backup volumes.
>
I assume a "file server" mentioned above is not a server which only holds
backup volume files but a common sense of this term like storing user
profiles, documents, production files, photos, movies, etc.
If the "file server" holds only backup volumes, then I personally do not
name it "file server" but a backup server. Exporting backup volume files
used by SD without a proper operational synchronization is not a good idea.
It does not harm your backups when exported as read-only, but
full-access...

>
> But! If you backup to file volumes and disks mounted directly on this file
> server then it is - generally - not recommended. In the worst case scenario
> you can lose all your data including backups. Be aware.
>
> Yes. It would never be a good idea to have backup volume files and data on
> the same set of physical disks.
>
NO, it doesn't matter if it is the same physical disk, any number of
separate devices, lvm or raid groups or just a network filesystem mounted
remotely. If it is available for writing with a standard file io api then
it exposes a huge risk! You should never do that.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Josh Fisher]

On 4/7/2020 7:20 AM, Radosław Korzeniewski wrote:

Hello,

wt., 7 kwi 2020 o 09:38 Shaligram Bhagat, Yateen (Nokia - 
IN/Bangalore) > napisał(a):


Hi,

The issue is resolved after I increased the number of devices
under a filechanger.

Nevertheless, the suggestion to keep the file server and the
bacula-sd on the same host is good one.


If you are using backup to tape then yes, running a dedicated 
bacula-sd on file server is a good recommendation.



Also true if the file server in question is only for the backup volumes.




But! If you backup to file volumes and disks mounted directly on this 
file server then it is - generally - not recommended. In the worst 
case scenario you can lose all your data including backups. Be aware.



Yes. It would never be a good idea to have backup volume files and data 
on the same set of physical disks.





best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net 


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Radosław Korzeniewski]

Hello,

wt., 7 kwi 2020 o 09:38 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
yateen.shaligram_bha...@nokia.com> napisał(a):

> Hi,
>
>
>
> The issue is resolved after I increased the number of devices under a
> filechanger.
>
> Nevertheless, the suggestion to keep the file server and the bacula-sd on
> the same host is good one.
>

If you are using backup to tape then yes, running a dedicated bacula-sd on
file server is a good recommendation.

But! If you backup to file volumes and disks mounted directly on this file
server then it is - generally - not recommended. In the worst case scenario
you can lose all your data including backups. Be aware.

best regards

> --
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)]

Hi,

Thanks,

Yes, I increased the number of devices and the issue is resolved.

-Yateen



From: Radosław Korzeniewski 
Sent: Sunday, April 5, 2020 3:02 AM
To: Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) 

Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Issue with concurrent jobs in disk based auto 
changer

Hello,

sob., 4 kwi 2020 o 23:27 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) 
mailto:yateen.shaligram_bha...@nokia.com>> 
napisał(a):

Issue:
With the above mentioned configs, When I start 200 virtual full jobs I expect 
all these jobs to run concurrently.


Every Virtual Full job requires at least 2 devices to operate. You need more 
devices or less jobs.

best regards
--
Radosław Korzeniewski
rados...@korzeniewski.net<mailto:rados...@korzeniewski.net>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)]

Hi,

The issue is resolved after I increased the number of devices under a 
filechanger.
Nevertheless, the suggestion to keep the file server and the bacula-sd on the 
same host is good one.

Thanks
Yateen

From: Josh Fisher 
Sent: Monday, April 6, 2020 6:22 PM
To: Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) 
; bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Issue with concurrent jobs in disk based auto 
changer



On 4/4/2020 1:50 PM, Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) wrote:
Hello,

...

All these jobs including the ones that are initially shown as  "created not yet 
running" eventually complete successfully, but after a long time (~36 Hours),
But the very purpose of concurrency is defeated.




You may also be looking at a network bottleneck. Is the file server on the same 
network as the clients? If so, then client data is traversing the same network 
twice and it would be better to run bacula-sd on the file server.

Also, if the db is being accessed across the network, then turning on attribute 
spooling may help.


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Josh Fisher]

On 4/4/2020 1:50 PM, Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) wrote:


Hello,

...

All these jobs including the ones that are initially shown as 
 “created not yet running” eventually complete successfully, but after 
a long time (~36 Hours),


But the very purpose of concurrency is defeated.



You may also be looking at a network bottleneck. Is the file server on 
the same network as the clients? If so, then client data is traversing 
the same network twice and it would be better to run bacula-sd on the 
file server.


Also, if the db is being accessed across the network, then turning on 
attribute spooling may help.



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Issue with concurrent jobs in disk based auto changer

[Radosław Korzeniewski]

Hello,

sob., 4 kwi 2020 o 23:27 Shaligram Bhagat, Yateen (Nokia - IN/Bangalore) <
yateen.shaligram_bha...@nokia.com> napisał(a):

>
>
> Issue:
>
> With the above mentioned configs, When I start 200 virtual full jobs I
> expect all these jobs to run concurrently.
>
>
>

Every Virtual Full job requires at least 2 devices to operate. You
need more devices or less jobs.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Issue with concurrent jobs in disk based auto changer

[Shaligram Bhagat, Yateen (Nokia - IN/Bangalore)]

Hello,

I am trying out Bacula Community version 9.4.4 on centos 6.4 with PostGreSQL
There are 200 bacula clients from where data of average size of 30GB each needs 
to be backed up

There will be incremental nightly backup every weekday and a virtual full on 
the weekend.
The storage is disk based, there are 10 NFS mounted disks on Bacula server (zfs 
exports from a remote filer host).

Each disk corresponds to one storage, namely,  StorageA, StorageB,..StorageJ
Each Storage has its own media type defined.
Each Storage has one Autochanger associated with it, and each Autochanger has 
20 devices,

StorageA-> AutochnagerA-> DeviceA1, DeviceA2...DeviceA20
StorageB-> AutochnagerB->DeviceB1, DeviceB2...Deviceb20
..
SorageJ-> AutochangerJ-> DeviceJ1, DeviceJ2...DeviceJ20

Each Device has Maximum Concurrent Job = 1

Hence as per my understanding the maximum Concurrent jobs that can be handled 
by this configuration
is 200 (10 Storages X 20 Devices per storage X 1 max concurent job per device )

I have defined the Maximum Concurrent Jobs in other places as under :
1. for each storage definition, namely StorageA, StorageB, ... 
StorageJ : Maximum Concurrent Jobs = 100
2. for the bacula storage daemon : Maximum Concurrent Jobs =500
3. for the bacula daemon : Maximum Concurrent Jobs =500
4. in PostGreSQL database : max_connections set to 500

Issue:
With the above mentioned configs, When I start 200 virtual full jobs I expect 
all these jobs to run concurrently.

However I find that although few jobs run concurrently, many jobs still show 
the state "created not yet running".
The bconsole status command DOES NOT show a single job in state "waiting to 
reserve a device", Also many devices are still
shown as "not open". Hence I assume that there are enough free devices 
available, to handle all the 200 concurrent jobs.

All these jobs including the ones that are initially shown as  "created not yet 
running" eventually complete successfully, but after a long time (~36 Hours),
But the very purpose of concurrency is defeated.

So my question is how to find the reason why a job is getting into "created not 
yet running" state?
( Note : all jobs have equal priority of 10)

Will setting the debug level dynamically (console command setdebug) help in 
getting more info about the jobs that are already in state "created not yet 
running" ?. I tired that but does not yiled any extra info in joblog.

Thanks,

Yateen

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users