Re: [Bacula-users] Slow spooling and hashing speed

[heitor]

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Chris Wilkinson]

FWIW, I also use SMB (CIFS v2) to backup to a cheapo DLINK DNS-325 NAS. I
typically get ~30MB/s on a Gb local LAN for long jobs, maybe half that for
shorter incremental. Compression is enabled.

Like yours, the NAS is locked down so no chance to put the SD there.

You might be lucky and find a toolchain to roll your own SD but only useful
if you can hack the NAS. Sounds like that's not an option for you.

-Chris-

On Sat, 23 Sept 2023, 18:36 Rob Gerber,  wrote:

> Heitor,
>
> Thank you for the reply.
>
> The backup source is a studio network solutions Evo NAS appliance.
> Unfortunately, I have no access to run bacula FD, nor most of other things
> suggested on this appliance as access is only via appliance developer
> designer means. I do not have any shell access to this appliance. Off the
> top of my head, I believe my only access options to this Evo NAS are smb (I
> believe 3.x), read only iSCSI, or afs (apple file sharing - obviously not
> something I want to try). This Evo NAS appliance is running Gentoo Linux.
>
> I am able to transfer data using the Linux cp command at link speeds. I am
> open to trying anything to see if I can improve these backup times since we
> have a lot of data to backup (total dataset about 194 TiB).
>
> Robert Gerber
> 402-237-8692
> r...@craeon.net
>
> On Sat, Sep 23, 2023, 1:24 PM  wrote:
>
>> Hello,
>>
>> On 9/14/23 15:35, Rob Gerber wrote:
>>>
>>> Bacula is transferring data at a fraction of the available link speed. I
>>> am backing up an SMB share hosted on a fast NAS appliance. The share is
>>> mounted on the bacula server in /mnt/NAS/sharename. I have dedicated 10gbe
>>> copper interfaces on the NAS and the bacula server.
>>>
>>>
>> Most likely your bottleneck is the SMB data source. Some documentations
>> say the protocol has a 30 MBs limitation. I would't use for backup
>> apllications.
>> Bacula Clients at the source machine, NFS, CIFS and NDMP are better
>> options.
>> I would also use the Linux TCP BBR protocol.
>>
>> Rgds.
>>
>>
>> *MSc Heitor Faria (Miami/USA)* Bacula LATAM CIO
>> mobile1: + 1 909 655-8971
>> mobile2: + 55 61 98268-4220
>>
>> [image: logo] 
>> bacula.lat | bacula.com.br 
>> 
>>
>>
>>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Phil Stracchino]

On 9/23/23 14:25, Rob Gerber wrote:
You know, asking them to add a Bacula FD plugin is a good idea! I'm sure 
it wouldn't be available for quite a while, but that is genuinely a good 
idea.


If they're building on Gentoo, they ought to be able to do it pretty 
quickly if you can persuade them to.  the Gentoo ebuild is mature and, 
particularly for client-only, is about as close to plug-and-play as you 
can get.



This is a commercial market NAS, aimed at videographers. The devs are 
actually pretty damn good, and I'm proud of them for choosing Gentoo. 
They have a bunch of software sitting on top of Gentoo aiming to provide 
fast access to stored media files for video editors.


Good on them, then.

The main data array is formatted XFS, by way of trivia for y'all. The 
core ethic is "get media go the end users as fast as possible so they 
can edit and render as fast as possible". Everything else is secondary.


XFS is probably the ideal choice for that application.  It was 
specifically designed to optimize streaming media performance.



--
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Rob Gerber]

You know, asking them to add a Bacula FD plugin is a good idea! I'm sure it
wouldn't be available for quite a while, but that is genuinely a good idea.

This is a commercial market NAS, aimed at videographers. The devs are
actually pretty damn good, and I'm proud of them for choosing Gentoo. They
have a bunch of software sitting on top of Gentoo aiming to provide fast
access to stored media files for video editors. It does its job well, but
backup is obviously our responsibility. They do integrate well with Storage
DNA, but given that some of our goals included archiving media (that would
not remain on the NAS), and given the boss's reaction to the storage DNA
price point, we went with Bacula instead. I'm more comfortable using an
open source highly configurable software solution. I'm not shit talking the
storage DNA folks at all - they seemed pretty good and provided some useful
information on tape hardware vendors (backupworks.com were very helpful as
we built our hardware stack).


Overall, I understand why they locked it down. Their target market uses
apple computers for virtually everything, and a modified appliance may well
break when they apply their updates. I would be pretty reluctant to modify
the appliance even if given access, unless I had no other choice. It's
designed and configured to do one thing and I don't know what they've done
with their changes. Breaking the really expensive appliance isn't really on
my to do list ;)

(I wasn't involved in selecting the Evo appliance, and have been tasked
with building infrastructure around it).

The main data array is formatted XFS, by way of trivia for y'all. The core
ethic is "get media go the end users as fast as possible so they can edit
and render as fast as possible". Everything else is secondary.



Robert Gerber
402-237-8692
r...@craeon.net

On Sat, Sep 23, 2023, 2:05 PM Phil Stracchino  wrote:

> On 9/23/23 13:34, Rob Gerber wrote:
> > Heitor,
> >
> > Thank you for the reply.
> >
> > The backup source is a studio network solutions Evo NAS appliance.
> > Unfortunately, I have no access to run bacula FD, nor most of other
> > things suggested on this appliance as access is only via appliance
> > developer designer means. I do not have any shell access to this
> > appliance. Off the top of my head, I believe my only access options to
> > this Evo NAS are smb (I believe 3.x), read only iSCSI, or afs (apple
> > file sharing - obviously not something I want to try). This Evo NAS
> > appliance is running Gentoo Linux.
>
>
> Wait, it runs Gentoo and they locked it down that hard somehow?
>
> Gut feeling says you should be able to install almost anything you want
> onto it, but it may be some work depending on how they locked it down.
> (This would of course probably void your warranty.)
>
> That said, if you can't put a fd on it directly, given those options the
> next best thing would probably be to mount it as an iSCSI target.
>
> You might also consider asking them if they would consider a Bacula-fd
> package available.  But my experience is that to NAS consumer-appliance
> manufacturers, backup is at best an afterthought.  I went through this
> with a NAS appliance which QNAP proudly claimed ran Linux and ZFS, only
> to find out that through idiotic choices and Byzantine design decisions
> they had crippled it so badly it might as well have been running Windows.
>
>
>
> --
>Phil Stracchino
>Babylon Communications
>ph...@caerllewys.net
>p...@co.ordinate.org
>Landline: +1.603.293.8485
>Mobile:   +1.603.998.6958
>
>
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Phil Stracchino]

On 9/23/23 13:34, Rob Gerber wrote:

Heitor,

Thank you for the reply.

The backup source is a studio network solutions Evo NAS appliance. 
Unfortunately, I have no access to run bacula FD, nor most of other 
things suggested on this appliance as access is only via appliance 
developer designer means. I do not have any shell access to this 
appliance. Off the top of my head, I believe my only access options to 
this Evo NAS are smb (I believe 3.x), read only iSCSI, or afs (apple 
file sharing - obviously not something I want to try). This Evo NAS 
appliance is running Gentoo Linux.



Wait, it runs Gentoo and they locked it down that hard somehow?

Gut feeling says you should be able to install almost anything you want 
onto it, but it may be some work depending on how they locked it down. 
(This would of course probably void your warranty.)


That said, if you can't put a fd on it directly, given those options the 
next best thing would probably be to mount it as an iSCSI target.


You might also consider asking them if they would consider a Bacula-fd 
package available.  But my experience is that to NAS consumer-appliance 
manufacturers, backup is at best an afterthought.  I went through this 
with a NAS appliance which QNAP proudly claimed ran Linux and ZFS, only 
to find out that through idiotic choices and Byzantine design decisions 
they had crippled it so badly it might as well have been running Windows.




--
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Rob Gerber]

Heitor,

Thank you for the reply.

The backup source is a studio network solutions Evo NAS appliance.
Unfortunately, I have no access to run bacula FD, nor most of other things
suggested on this appliance as access is only via appliance developer
designer means. I do not have any shell access to this appliance. Off the
top of my head, I believe my only access options to this Evo NAS are smb (I
believe 3.x), read only iSCSI, or afs (apple file sharing - obviously not
something I want to try). This Evo NAS appliance is running Gentoo Linux.

I am able to transfer data using the Linux cp command at link speeds. I am
open to trying anything to see if I can improve these backup times since we
have a lot of data to backup (total dataset about 194 TiB).

Robert Gerber
402-237-8692
r...@craeon.net

On Sat, Sep 23, 2023, 1:24 PM  wrote:

> Hello,
>
> On 9/14/23 15:35, Rob Gerber wrote:
>>
>> Bacula is transferring data at a fraction of the available link speed. I
>> am backing up an SMB share hosted on a fast NAS appliance. The share is
>> mounted on the bacula server in /mnt/NAS/sharename. I have dedicated 10gbe
>> copper interfaces on the NAS and the bacula server.
>>
>>
> Most likely your bottleneck is the SMB data source. Some documentations
> say the protocol has a 30 MBs limitation. I would't use for backup
> apllications.
> Bacula Clients at the source machine, NFS, CIFS and NDMP are better
> options.
> I would also use the Linux TCP BBR protocol.
>
> Rgds.
>
>
> *MSc Heitor Faria (Miami/USA)* Bacula LATAM CIO
> mobile1: + 1 909 655-8971
> mobile2: + 55 61 98268-4220
>
> [image: logo] 
> bacula.lat | bacula.com.br 
> 
>
>
>
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[heitor]

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Rob Gerber]

Thank you for your reply, josh. Sorry for delay in reply - have been on
vacation. See replies below.

Robert Gerber
402-237-8692
r...@craeon.net

On Fri, Sep 15, 2023, 9:52 AM Josh Fisher via Bacula-users <
bacula-users@lists.sourceforge.net> wrote:

>
> On 9/14/23 15:35, Rob Gerber wrote:
>
> Bacula is transferring data at a fraction of the available link speed. I
> am backing up an SMB share hosted on a fast NAS appliance. The share is
> mounted on the bacula server in /mnt/NAS/sharename. I have dedicated 10gbe
> copper interfaces on the NAS and the bacula server.
>
> When backing up the NAS, cifsiostat shows around 250MB/s during the
> spooling phase (and 0 kb/s during the despool phase). When using cp to copy
> files from the NAS to the Bacula server, I can easily saturate my 10gbe
> link (avg throughput around 1GB/s, or a little lower).
>
>
> So that tells you that there's nothing wrong with the underlying  SMB file
> system. The Bacula client just reads the files like any other directory
> it's backing up.
>
>
>
> I think the problem lies in Bacula because I can copy data much faster
> using cp instead of bacula. Obviously bacula is doing a lot more than cp,
> so there will be differences. However I would hope for transfer speeds
> closer to the available link speed.
>
> top shows that a couple cores are maxed out during the spooling process.
> Maybe hashing speed is the limitation here? If so, could multicore hashing
> support speed this up? I have two e5-2676 v3 processors in this server. I
> am using SHA512 right now, but I saw similar speeds from bacula when using
> MD5.
>
>
> The hashing speed doesn't account for a 4x slower transfer, and likely not
> for saturating 2 cores. Do you have compression enabled for the job? Or
> encryption? You definitely do not want compression, since the tape drive
> will handle compression itself. Also, the client and sd are the same
> machine in this case, but make sure it is not configured to use TLS
> connections.
>

I have not affirmatively enabled compression. Is there any sort of default
compression mode that I could disable?

I'll see what I can find regarding TLS connections. I haven't set up such a
thing, but maybe it works that way by default?

>
>
> Average write speed to LTO-8 media winds up being about 120-150MB/s once
> the times to spool and despool are considered.
>
> My spool is on a 76GB ramdisk (spool size is 75G in bacula dir conf), so I
> don't think spool disk access speed is a factor.
>
>
> Might be overkill. A NVMe SSD is plenty fast enough for both the 10G
> network and for despooling to the LTO8 drive. If the catalog DB is also on
> this server, then you might be better off with the spool on SSD and far
> more RAM dedicated to postgresql. If the DB is on another server, then the
> attributes are being despooled to the DB over the 1G network.
>

I was using a sata SSD for my spool, but switched to a ramdisk for
troubleshooting purposes.

I have about 50gb ram available (not counting ramdisk). Top shows about
1.6gb ram in use, though I don't know how accurate that is.

My postgres DB server is on the same machine. I haven't specifically
configured postgres to use a certain amount of ram. I don't know what
performance enhancements I might be able to use with postgres.

I'm tempted to try setting up a direct to disk backup and test with local
files, or files stored on my TrueNAS server (which has its own dedicated
optical 10gbe link, which I can also a saturate when using cp).

Is there a way I could troubleshoot and try to pinpoint the location of my
bottleneck?

I will investigate the questions you raised in your email and report back.
Some delay is to be expected because a 124tb backup is running and is
currently at 98tb complete, averaging about 125MiB/s according to bacularis.

>
> I have not tested to see if bacula could back up faster if it wasn't
> accessing a share via SMB. I don't think SMB should be an issue here but I
> have to consider every possibility. The SMB share I'm backing up is mounted
> on /mnt/NAS/sharename. Bacula is backing that mount folder up.
>
> Currently, my only access to the NAS appliance is via SMB. The appliance
> does support iscsi in read only mode but i'm not sure if there would be any
> performance improvements.
>
> I don't think the traffic could be going out through the wrong interface.
> The NAS is directly attached to my bacula server using a short cat6 cable.
> The NAS and my server each have 10gbe copper interfaces. The relevant
> interfaces have ip addresses statically assigned. These addresses are
> unique to the LAN configuration (local lan is 10.1.1.0/24, 10gbe
> interfaces assigned to 192.168.6.25 and 192.168.6.100). My bacula
> server's only other connection is to the gigabit LAN switch.
>
> Is there any information that I could provide to help the list help me, or
> does anyone have any thoughts for me?
>
> Regards,
> Robert Gerber
> 402-237-8692
> r...@craeon.net
>
>
>
>
>
> 

Re: [Bacula-users] Slow spooling and hashing speed

[Josh Fisher via Bacula-users]

On 9/14/23 15:35, Rob Gerber wrote:
Bacula is transferring data at a fraction of the available link speed. 
I am backing up an SMB share hosted on a fast NAS appliance. The share 
is mounted on the bacula server in /mnt/NAS/sharename. I have 
dedicated 10gbe copper interfaces on the NAS and the bacula server.


When backing up the NAS, cifsiostat shows around 250MB/s during the 
spooling phase (and 0 kb/s during the despool phase). When using cp to 
copy files from the NAS to the Bacula server, I can easily saturate my 
10gbe link (avg throughput around 1GB/s, or a little lower).



So that tells you that there's nothing wrong with the underlying SMB 
file system. The Bacula client just reads the files like any other 
directory it's backing up.





I think the problem lies in Bacula because I can copy data much faster 
using cp instead of bacula. Obviously bacula is doing a lot more than 
cp, so there will be differences. However I would hope for transfer 
speeds closer to the available link speed.


top shows that a couple cores are maxed out during the spooling 
process. Maybe hashing speed is the limitation here? If so, could 
multicore hashing support speed this up? I have two e5-2676 v3 
processors in this server. I am using SHA512 right now, but I saw 
similar speeds from bacula when using MD5.



The hashing speed doesn't account for a 4x slower transfer, and likely 
not for saturating 2 cores. Do you have compression enabled for the job? 
Or encryption? You definitely do not want compression, since the tape 
drive will handle compression itself. Also, the client and sd are the 
same machine in this case, but make sure it is not configured to use TLS 
connections.





Average write speed to LTO-8 media winds up being about 120-150MB/s 
once the times to spool and despool are considered.


My spool is on a 76GB ramdisk (spool size is 75G in bacula dir conf), 
so I don't think spool disk access speed is a factor.



Might be overkill. A NVMe SSD is plenty fast enough for both the 10G 
network and for despooling to the LTO8 drive. If the catalog DB is also 
on this server, then you might be better off with the spool on SSD and 
far more RAM dedicated to postgresql. If the DB is on another server, 
then the attributes are being despooled to the DB over the 1G network.





I have not tested to see if bacula could back up faster if it wasn't 
accessing a share via SMB. I don't think SMB should be an issue here 
but I have to consider every possibility. The SMB share I'm backing up 
is mounted on /mnt/NAS/sharename. Bacula is backing that mount folder up.


Currently, my only access to the NAS appliance is via SMB. The 
appliance does support iscsi in read only mode but i'm not sure if 
there would be any performance improvements.


I don't think the traffic could be going out through the wrong 
interface. The NAS is directly attached to my bacula server using a 
short cat6 cable. The NAS and my server each have 10gbe copper 
interfaces. The relevant interfaces have ip addresses statically 
assigned. These addresses are unique to the LAN configuration (local 
lan is 10.1.1.0/24 , 10gbe interfaces assigned to 
192.168.6.25 and 192.168.6.100). My bacula server's only other 
connection is to the gigabit LAN switch.


Is there any information that I could provide to help the list help 
me, or does anyone have any thoughts for me?


Regards,
Robert Gerber
402-237-8692
r...@craeon.net





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Slow spooling and hashing speed

[Rob Gerber]

Bacula is transferring data at a fraction of the available link speed. I am
backing up an SMB share hosted on a fast NAS appliance. The share is
mounted on the bacula server in /mnt/NAS/sharename. I have dedicated 10gbe
copper interfaces on the NAS and the bacula server.

When backing up the NAS, cifsiostat shows around 250MB/s during the
spooling phase (and 0 kb/s during the despool phase). When using cp to copy
files from the NAS to the Bacula server, I can easily saturate my 10gbe
link (avg throughput around 1GB/s, or a little lower).

I think the problem lies in Bacula because I can copy data much faster
using cp instead of bacula. Obviously bacula is doing a lot more than cp,
so there will be differences. However I would hope for transfer speeds
closer to the available link speed.

top shows that a couple cores are maxed out during the spooling process.
Maybe hashing speed is the limitation here? If so, could multicore hashing
support speed this up? I have two e5-2676 v3 processors in this server. I
am using SHA512 right now, but I saw similar speeds from bacula when using
MD5.

Average write speed to LTO-8 media winds up being about 120-150MB/s once
the times to spool and despool are considered.

My spool is on a 76GB ramdisk (spool size is 75G in bacula dir conf), so I
don't think spool disk access speed is a factor.

I have not tested to see if bacula could back up faster if it wasn't
accessing a share via SMB. I don't think SMB should be an issue here but I
have to consider every possibility. The SMB share I'm backing up is mounted
on /mnt/NAS/sharename. Bacula is backing that mount folder up.

Currently, my only access to the NAS appliance is via SMB. The appliance
does support iscsi in read only mode but i'm not sure if there would be any
performance improvements.

I don't think the traffic could be going out through the wrong interface.
The NAS is directly attached to my bacula server using a short cat6 cable.
The NAS and my server each have 10gbe copper interfaces. The relevant
interfaces have ip addresses statically assigned. These addresses are
unique to the LAN configuration (local lan is 10.1.1.0/24, 10gbe interfaces
assigned to 192.168.6.25 and 192.168.6.100). My bacula server's only other
connection is to the gigabit LAN switch.

Is there any information that I could provide to help the list help me, or
does anyone have any thoughts for me?

Regards,
Robert Gerber
402-237-8692
r...@craeon.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users