Re: [Bacula-users] LTO tape performances, again...
Mandi! Cejka Rudolf via Bacula-users In chel di` si favelave... > - Spool Data = yes, attribute spooling is automatically enabled too Done. > In device storage section: > - Maximum File Size = 32 GB, for LTO-6 and older you can think about 16 GB i have: Maximum File Size = 50G > - Sufficient Maximum Job Spool Size or Maximum Spool Size, so overall write > speed > could be effective, I use 200 GB and 1000 GB i have: Maximum Spool Size = 400G #Maximum Job Spool Size = 50G > - Maximum Block Size = 262144 or 131072, for LTO-6 and older 65536 (this is > tested > on my FreeBSD system, Linux or another systems may need higher values - but > do not > use too big, because probability of write errors increases) i have: Maximum blocksize = 512K > - SSD based disk spooling, M2 or SATA RAID I've just tried with an dedicated SSD disk: 31-Jan 18:07 svpve3-sd JobId 16548: Committing spooled data to Volume "SVPVE3_0003". Despooling 131,278,620,424 bytes ... 31-Jan 18:21 svpve3-sd JobId 16548: Despooling elapsed time = 00:14:22, Transfer rate = 152.2 M Bytes/second I've tried a quick fio test on SSD disk leading to: root@svpve3:~# fio --name TEST --eta-newline=5s --filename=/rpool-backup/bacula/spool/temp.file --rw=read --size=20g --io_size=50g --blocksize=1024k --ioengine=libaio --fsync=1 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process TEST: Laying out IO file (1 file / 20480MiB) Jobs: 1 (f=1): [R(1)][13.1%][r=521MiB/s][r=520 IOPS][eta 00m:53s] Jobs: 1 (f=1): [R(1)][23.0%][r=521MiB/s][r=520 IOPS][eta 00m:47s] Jobs: 1 (f=1): [R(1)][32.8%][r=520MiB/s][r=520 IOPS][eta 00m:41s] Jobs: 1 (f=1): [R(1)][42.6%][r=522MiB/s][r=521 IOPS][eta 00m:35s] Jobs: 1 (f=1): [R(1)][52.5%][r=520MiB/s][r=519 IOPS][eta 00m:29s] Jobs: 1 (f=1): [R(1)][62.3%][r=520MiB/s][r=520 IOPS][eta 00m:23s] Jobs: 1 (f=1): [R(1)][72.1%][r=521MiB/s][r=520 IOPS][eta 00m:17s] Jobs: 1 (f=1): [R(1)][82.0%][r=521MiB/s][r=520 IOPS][eta 00m:11s] Jobs: 1 (f=1): [R(1)][91.8%][r=521MiB/s][r=521 IOPS][eta 00m:05s] Jobs: 1 (f=1): [R(1)][100.0%][r=520MiB/s][r=520 IOPS][eta 00m:00s] TEST: (groupid=0, jobs=1): err= 0: pid=29942: Wed Jan 31 18:32:54 2024 read: IOPS=519, BW=520MiB/s (545MB/s)(30.5GiB/60061msec) slat (usec): min=72, max=563, avg=113.95, stdev=17.64 clat (msec): min=2, max=120, avg=61.42, stdev= 1.93 lat (msec): min=2, max=120, avg=61.53, stdev= 1.93 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 62], 95.00th=[ 62], | 99.00th=[ 62], 99.50th=[ 63], 99.90th=[ 90], 99.95th=[ 106], | 99.99th=[ 117] bw ( KiB/s): min=487424, max=534528, per=100.00%, avg=532325.72, stdev=4251.46, samples=120 iops: min= 476, max= 522, avg=519.83, stdev= 4.15, samples=120 lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.07%, 100=99.83% lat (msec) : 250=0.06% cpu : usr=0.86%, sys=7.44%, ctx=31284, majf=7, minf=8203 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=31224,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=30.5GiB (32.7GB), run=60061-60061msec Disk stats (read/write): sde: ios=31220/4, merge=0/1, ticks=1914913/325, in_queue=1872640, util=99.90% So disk seems to perform at least 500MB/s on reading, but bacula still write on tape at 150MB/s... -- Se vuoi navigare, comprati una barca. (sysmen dei.unipd.it) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO tape performances, again...
Mandi! Marco Gaiarin In chel di` si favelave... > I'll provide feedback. Thanks. Better, reached 180MB/s. Probably i need to move bacula spool on a dedicated SSD disk (currently, on an ZFS RAID pool, backed up with an SSD read cache) to reach better performaces. Still incremental are very slow... not writing to LTO, but caching data; true that currently read pool is the same pool where the cache reside, but i make a note... More to come... -- Bisogna saper scegliere il tempo non arrivarci per contrarieta`(F. Guccini) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO tape performances, again...
Hello, you need all of these: In job director section: - Spool Data = yes, attribute spooling is automatically enabled too In device storage section: - Maximum File Size = 32 GB, for LTO-6 and older you can think about 16 GB - Sufficient Maximum Job Spool Size or Maximum Spool Size, so overall write speed could be effective, I use 200 GB and 1000 GB - Maximum Block Size = 262144 or 131072, for LTO-6 and older 65536 (this is tested on my FreeBSD system, Linux or another systems may need higher values - but do not use too big, because probability of write errors increases) - SSD based disk spooling, M2 or SATA RAID I finished write speed tests with different file sizes, so here they are (+ is my recommended file size, - is alternative and I expect that LTO-9 is still similar to LTO-8 and LTO-7 in this view): LTO-5, block size 65536: File Simplified Simplified Bytes Files Rough time to sizefill timefill rate written written read one file (at 140 MB/s) 1 GB 6:44:1861.8 MB/s 1498688192512 149900:00:07 2 GB 4:51:3085.7 MB/s 1498737278976 75000:00:14 4 GB 3:54:57 106.3 MB/s 1498762641408 37500:00:29 8 GB 3:24:22 122.2 MB/s 1499224932352 18800:00:57 16 GB +3:11:09 130.7 MB/s 14994564055049400:01:54 32 GB -3:04:35 135.4 MB/s 14996903034884700:03:49 64 GB 3:01:17 137.8 MB/s 14995749601282400:07:37 128 GB 2:59:37 139.1 MB/s 14996036648961200:15:14 LTO-6, block size 65536: File Simplified Simplified Bytes Files Rough time to sizefill timefill rate written written read one file (at 160 MB/s) 1 GB 9:51:3870.4 MB/s 2500322394112 250100:00:06 2 GB 7:05:4597.9 MB/s 2500404314112 125100:00:13 4 GB 5:42:29 121.6 MB/s 2500445274112 62600:00:25 8 GB 4:56:39 140.5 MB/s 2501218009088 31300:00:50 16 GB +4:37:28 150.2 MB/s 2501603885056 15700:01:40 32 GB -4:27:55 155.6 MB/s 25016084725767900:03:20 64 GB 4:24:19 157.7 MB/s 25018018037764000:06:40 128 GB 4:20:58 159.7 MB/s 25019952660482000:13:20 256 GB 4:19:43 160.5 MB/s 25017795215361000:26:40 LTO-7, block size 262144 (65536 started to be limiting): File Simplified Simplified Bytes Files Rough time to sizefill timefill rate written written read one file (at 300 MB/s) 2 GB 9:17:20 179.8 MB/s 6013258301440 300700:00:07 4 GB 7:25:39 224.8 MB/s 6013256204288 150400:00:13 8 GB 6:30:19 256.9 MB/s 6017229258752 75300:00:27 16 GB 6:02:51 276.3 MB/s 6015444844544 37600:00:53 32 GB +5:48:56 287.4 MB/s 6017326514176 18900:01:47 64 GB -5:42:07 293.1 MB/s 60173259898889500:03:33 128 GB 5:38:45 296.0 MB/s 60171076239364800:07:07 256 GB 5:36:56 297.6 MB/s 60174560133122400:14:13 512 GB 5:36:12 298.3 MB/s 60174557511681200:28:27 LTO-8, block size 262144: File Simplified Simplified Bytes Files Rough time to sizefill timefill rate written written read one file (at 300 MB/s) 2 GB 17:59:35 185.3 MB/s 12005379407872 600400:00:07 4 GB 14:34:50 228.7 MB/s 12005942231040 300200:00:13 8 GB 12:52:26 259.2 MB/s 12013881786368 150200:00:27 16 GB 12:01:17 277.5 MB/s 12010304307200 75100:00:53 32 GB + 11:35:37 287.8 MB/s 12012189122560 37600:01:47 64 GB - 11:26:04 291.8 MB/s 12014075510784 18800:03:33 128 GB 11:16:23 296.0 MB/s 120145743708169400:07:07 256 GB 11:13:14 297.4 MB/s 120143392276484700:14:13 512 GB 11:11:39 298.1 MB/s 120131572203522400:28:27 Pierre Bernhardt wrote (2024/01/25): > Am 25.01.24 um 10:06 schrieb Marco Gaiarin: > > > > > 2) checked disk performance (data came only from local disk); i've > > > currently > > > 3 servers, some perform better, some worster, but the best one have a > > > read > > > disk performance pretty decent, at least 200MB/s on random access (1500 > > > MB/s > > > on sequential one). > > > > Jim Pollard on private email ask me about controllers: i've not specified, > > sorry, but LTO units are
Re: [Bacula-users] LTO tape performances, again...
Mandi! Josh Fisher via Bacula-users In chel di` si favelave... > Disk that is local to the server does not mean it is local to the > bacula-sd process or tape drive. If the connection is 1 gigabit > Ethernet, then max rate is going to be 125 MB/s. Nono, i meant literally 'local disk': i'm putting on tape with bacula the 'backup server', so SD and FD of this job reside on the same server. > That is probably not what you want to do. You want the the bacula-sd > process to spool data on its local disk so that when it is despooled to Anyway... noted that incremental jobs are far slower then full one, probably the FD lost too many time on seeks; so i've still enabled spooling, so i an write a big chunk of data sequentially. I'll provide feedback. Thanks. -- Qualche saggio scriveva che l'unico sistema economico funzionante gli pareva il libero mercato in cui il governo tiene la pistola puntata alla tempia delle corporations. Peccato che il governo sia più solito offrire modiche quantità di vaselina al consumatore, occasionalmente... (Emanuele Pucciarelli) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO tape performances, again...
On 1/24/24 12:48, Marco Gaiarin wrote: My new IBM LTO9 tape unit have a data sheet performace of: https://www.ibm.com/docs/it/ts4500-tape-library/1.10.0?topic=performance-lto-specifications so on worst case (compression disabled) seems to perform 400 MB/s on an LTO9 tape. Practically on Bacula i get 70-80 MB/s. I've just: 1) followed: https://www.bacula.org/9.6.x-manuals/en/problems/Testing_Your_Tape_Drive_Wit.html#SECTION00422000 getting 237.7 MB/s on random data (worst case). 2) checked disk performance (data came only from local disk); i've currently 3 servers, some perform better, some worster, but the best one have a read disk performance pretty decent, at least 200MB/s on random access (1500 MB/s on sequential one). Disk that is local to the server does not mean it is local to the bacula-sd process or tape drive. If the connection is 1 gigabit Ethernet, then max rate is going to be 125 MB/s. 3) disabled data spooling, of course; as just stated, data came only from local disks. Enabled attribute spooling. That is probably not what you want to do. You want the the bacula-sd process to spool data on its local disk so that when it is despooled to the tape drive it is reading only from local disk, not from a small RAM buffer that is being filled through a network socket. Even with a 10 G Ethernet network it is better to spool data for LTO tape drives, since the client itself might not be able to keep up with the tape drive, or is busy, or the network is congested, etc. Clearly i can expect some performance penalty on Bacula and mixed files, but really 70MB/s are slow... What else can i tackle with? Thanks. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO tape performances, again...
Am 25.01.24 um 10:06 schrieb Marco Gaiarin: 2) checked disk performance (data came only from local disk); i've currently 3 servers, some perform better, some worster, but the best one have a read disk performance pretty decent, at least 200MB/s on random access (1500 MB/s on sequential one). Jim Pollard on private email ask me about controllers: i've not specified, sorry, but LTO units are connected to a specific SAS controller, not the disks one. I'm registred also a lower than expected write performance. My LTO-6 drive should be handle 160 MB/s uncompressable random data. By the way mostly bacula said after writing a sequence that the tranfer speed is mostly round about 80 MB/s. I've not investigated yet but normally it should go faster. The job is spooled to /tmp and the swap is not in use. So the Transfer should be much more faster. My suggestion now is: Create a big data-random file as like a spool file in /tmp. Spool it with dd from tmp to /dev/null Spool from /dev/random to tape Spool from /tmp to tape Any suggestions about bs usage or something else? Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] LTO tape performances, again...
> 2) checked disk performance (data came only from local disk); i've currently > 3 servers, some perform better, some worster, but the best one have a read > disk performance pretty decent, at least 200MB/s on random access (1500 MB/s > on sequential one). Jim Pollard on private email ask me about controllers: i've not specified, sorry, but LTO units are connected to a specific SAS controller, not the disks one. -- ...mi dispiace solo un po' per gli svizzeri, ieri hanno giocato una partita di merda, oggi gli arriva bossi...(Piccia, il 27/6/2006) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] LTO tape performances, again...
My new IBM LTO9 tape unit have a data sheet performace of: https://www.ibm.com/docs/it/ts4500-tape-library/1.10.0?topic=performance-lto-specifications so on worst case (compression disabled) seems to perform 400 MB/s on an LTO9 tape. Practically on Bacula i get 70-80 MB/s. I've just: 1) followed: https://www.bacula.org/9.6.x-manuals/en/problems/Testing_Your_Tape_Drive_Wit.html#SECTION00422000 getting 237.7 MB/s on random data (worst case). 2) checked disk performance (data came only from local disk); i've currently 3 servers, some perform better, some worster, but the best one have a read disk performance pretty decent, at least 200MB/s on random access (1500 MB/s on sequential one). 3) disabled data spooling, of course; as just stated, data came only from local disks. Enabled attribute spooling. Clearly i can expect some performance penalty on Bacula and mixed files, but really 70MB/s are slow... What else can i tackle with? Thanks. -- If you hear something late at night some kind of trouble, some kind of fight just don't ask me what it was (S. Vega) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
