Some more output from a Full job that gets stuck (after many Incremental jobs
ran)
2017-11-27 23:30:11 bareos-sd JobId 714: Please mount append Volume "Full-0290"
or label a new one for:
Job: BackupCatalog.2017-11-27_23.10.00_39
Storage: "FileStorage-Data1" (/data1/bareos)
Pool: Full
Media type: File
2017-11-27 23:30:11 bareos-sd JobId 714: Warning: mount.c:250 Open device
"FileStorage-Data1" (/data1/bareos) Volume "Full-0290" failed: ERR=dev.c:664
Could not open: /data1/bareos/Full-0290, ERR=No space left on device
...
2017-11-27 23:30:10 bareos-sd JobId 714: Warning: label.c:376 Open device
"FileStorage-Data1" (/data1/bareos) Volume "Full-0290" failed: ERR=dev.c:664
Could not open: /data1/bareos/Full-0290, ERR=No space left on device
2017-11-27 23:30:10 bareos-dir JobId 714: Recycled volume "Full-0290"
2017-11-27 23:30:10 bareos-dir JobId 714: All records pruned from Volume
"Full-0290"; marking it "Purged"
2017-11-27 23:30:10 bareos-dir JobId 714: There are no more Jobs associated
with Volume "Full-0290". Marking it purged.
2017-11-27 23:30:10 bareos-sd JobId 714: End of medium on Volume "Full-0289"
Bytes=1,096,893 Blocks=17 at 27-Nov-2017 23:30.
2017-11-27 23:30:08 bareos-sd JobId 714: End of Volume "Full-0289" at 0:1096892
on device "FileStorage-Data1" (/data1/bareos). Write of 64512 bytes got 37699.
2017-11-27 23:30:08 bareos-sd JobId 714: New volume "Full-0289" mounted on
device "FileStorage-Data1" (/data1/bareos) at 27-Nov-2017 23:30.
2017-11-27 23:30:07 bareos-sd JobId 714: Wrote label to prelabeled Volume
"Full-0289" on device "FileStorage-Data1" (/data1/bareos)
2017-11-27 23:30:07 bareos-sd JobId 714: Labeled new Volume "Full-0289" on
device "FileStorage-Data1" (/data1/bareos).
2017-11-27 23:30:02 bareos-dir JobId 714: Recycled volume "Full-0289"
2017-11-27 23:30:02 bareos-dir JobId 714: All records pruned from Volume
"Full-0289"; marking it "Purged"
2017-11-27 23:30:01 bareos-dir JobId 714: There are no more Jobs associated
with Volume "Full-0289". Marking it purged.
2017-11-27 23:30:01 bareos-sd JobId 714: End of medium on Volume "Full-0288"
Bytes=285,336,758 Blocks=4,423 at 27-Nov-2017 23:30.
2017-11-27 23:30:01 bareos-sd JobId 714: End of Volume "Full-0288" at
0:285336757 on device "FileStorage-Data1" (/data1/bareos). Write of 64512 bytes
got 43850.
2017-11-27 23:30:00 bareos-sd JobId 714: Wrote label to prelabeled Volume
"Full-0288" on device "FileStorage-Data1" (/data1/bareos)
2017-11-27 23:30:00 bareos-sd JobId 714: Labeled new Volume "Full-0288" on
device "FileStorage-Data1" (/data1/bareos).
2017-11-27 23:29:59 bareos-dir JobId 714: Using Device "FileStorage-Data1" to
write.
2017-11-27 23:29:59 bareos-dir JobId 714: Start Backup JobId 714,
Job=BackupCatalog.2017-11-27_23.10.00_39
2017-11-27 23:29:36 bareos-dir JobId 714: shell command: run BeforeJob
"/usr/lib/bareos/scripts/make_catalog_backup.pl MyCatalog"
On Monday, November 27, 2017 at 2:32:49 PM UTC-8, [email protected]
wrote:
> Hello,
>
> I am currently having an issue when I run a Full job (50G vol size, 365d
> retention) the job does not switch to the next available device(s) or recycle
> an existing volume when the device(s) runs out of space, however it seems to
> work fine when I have a Incremental job (1G vol size, 30d retention).
>
> Below I have listed my setup and configuration information that I feel is
> relevant, along with output from three jobs (2 of the jobs are a small job
> that fails in Full pool but works when Incremental pool, and the 3rd is the
> output from a DB backup that spans multiple Full volumes and caused the last
> disk to fill up over night, which was hoping it would roll back to Data1 or
> at least recycle the oldest on Data4 but it did not
>
> Thank you for any assistance in this manner.
> -Chris
>
> ##
> ## Setup
> ##
> Bareos 17.2rc1/rc2
> CentOS 7 server with 4 4TB JBOD disks for backup storage
> Director and SD on same machine
> Multiple FD clients
>
> # Director storage/File.conf
> Storage {
> Name = File
> Address = backup
> Password = <password>
> Device = FileStorage-Data1
> Device = FileStorage-Data2
> Device = FileStorage-Data3
> Device = FileStorage-Data4
> Media Type = File
> Maximum Concurrent Jobs = 10
> }
>
> # Director pool/Full.conf
> Pool {
> Name = Full
> Pool Type = Backup
> Recycle = yes
> AutoPrune = yes
> Volume Retention = 365 days
> Maximum Volume Bytes = 50G
> Maximum Volumes = 200
> Label Format = "Full-"
> Recycle Oldest Volume = yes
> Action On Purge = Truncate
> Purge Oldest Volume = yes
> }
>
> # Director pool/Incremental.conf
> Pool {
> Name = Incremental
> Pool Type = Backup
> Recycle = yes
> AutoPrune = yes
> Volume Retention = 30 days
> Maximum Volume Bytes = 1G
> Maximum Volumes = 200
> Label Format = "Incremental-"
> Recycle Oldest Volume = yes
> Action On Purge = Truncate
> Purge Oldest Volume = yes
> }
>
> # Storage device/FileStorage-Data1.conf
> Device {
> Name = FileStorage-Data1
> Media Type = File
> Archive Device = /data1/bareos
> LabelMedia = yes;
> Random Access = yes;
> AutomaticMount = yes;
> RemovableMedia = no;
> AlwaysOpen = no;
> Description = "File device. A connecting Director must have the same Name
> and MediaType."
> Maximum Concurrent Jobs = 10
> }
>
> # Storage device/FileStorage-Data2.conf
> Device {
> Name = FileStorage-Data2
> Media Type = File
> Archive Device = /data2/bareos
> LabelMedia = yes;
> Random Access = yes;
> AutomaticMount = yes;
> RemovableMedia = no;
> AlwaysOpen = no;
> Description = "File device. A connecting Director must have the same Name
> and MediaType."
> Maximum Concurrent Jobs = 10
> }
>
> # Storage device/FileStorage-Data3, Data4
> Same as above just Name and Archive Device updated accordingly
>
>
> ##
> ## Expected actions
> ##
> # When all drives are not full
> When FileStorage-Data1 fills up, the volume is created on FileStorage-Data2..4
>
> # When all drives are full
> When all drives are full, recycle the oldest volume which should be on
> FileStorage-Data1
>
> ## Workaround so far
> When FileStorage-Data1 filled up, comment it out in Director
> storage/File.conf which allows it to be written to FileStorage-Data2, repeat
> till Data4.
>
> This allowed me to kick the can down the road until last night when Data4
> filled up and it did not recycle and got stuck.
>
> I should note, that when Data1 first filled up, again only Full backups would
> not jump to Data2..4 but Incremental and Differential backups would.
>
>
> ##
> ## Output
> ##
> # df
> /dev/mapper/clc-data3 3.7T 3.6T 62G 99% /data3
> /dev/mapper/cld-data4 3.7T 3.7T 747M 100% /data4
> /dev/mapper/clb-data2 3.7T 3.7T 40K 100% /data2
> /dev/mapper/cla-data1 3.0T 3.0T 1001M 100% /data1
>
> #
> # FAILS - Full job output (works as Incremental)
> #
> 2017-11-27 14:00:03 bareos-dir JobId 702: Bareos bareos-dir 17.2.4 (21Sep17):
> Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise
> Linux Server release 7.0 (Maipo)
> JobId: 702
> Job: Machine1-Wiki.2017-11-27_13.52.25_51
> Backup Level: Full
> Client: "Machine1-fd" 17.2.3 (14Aug17)
> x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0
> (Maipo),RHEL_7,x86_64
> FileSet: "Machine1-Wiki" 2017-10-03 15:58:33
> Pool: "Full" (From command line)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "File" (From Job resource)
> Scheduled time: 27-Nov-2017 13:52:25
> Start time: 27-Nov-2017 13:52:28
> End time: 27-Nov-2017 14:00:03
> Elapsed time: 7 mins 35 secs
> Priority: 10
> FD Files Written: 0
> SD Files Written: 0
> FD Bytes Written: 0 (0 B)
> SD Bytes Written: 0 (0 B)
> Rate: 0.0 KB/s
> Software Compression: None
> VSS: no
> Encryption: no
> Accurate: no
> Volume name(s):
> Volume Session Id: 2
> Volume Session Time: 1511818757
> Last Volume Bytes: 0 (0 B)
> Non-fatal FD errors: 1
> SD Errors: 0
> FD termination status: Error
> SD termination status: Canceled
> Termination: Backup Canceled
>
> 2017-11-27 14:00:03 bareos-dir JobId 702: Fatal error: No Job status returned
> from FD.
> 2017-11-27 14:00:03 bareos-sd JobId 702: Fatal error: Job 702 canceled.
> 2017-11-27 14:00:03 bareos-dir JobId 702: Fatal error: Network error with FD
> during Backup: ERR=Interrupted system call
> 2017-11-27 13:57:28 bareos-sd JobId 702: Warning: mount.c:250 Open device
> "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed: ERR=dev.c:664
> Could not open: /data1/bareos/Full-0288, ERR=No such file or directory
>
> 2017-11-27 13:52:28 bareos-sd JobId 702: Please mount append Volume
> "Full-0288" or label a new one for:
> Job: Machine1-Wiki.2017-11-27_13.52.25_51
> Storage: "FileStorage-Data1" (/data1/bareos)
> Pool: Full
> Media type: File
> 2017-11-27 13:52:28 bareos-sd JobId 702: Warning: mount.c:250 Open device
> "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed: ERR=dev.c:664
> Could not open: /data1/bareos/Full-0288, ERR=No space left on device
>
> ...
>
> 2017-11-27 13:52:28 bareos-sd JobId 702: Warning: label.c:376 Open device
> "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed: ERR=dev.c:664
> Could not open: /data1/bareos/Full-0288, ERR=No space left on device
>
> 2017-11-27 13:52:28 bareos-dir JobId 702: Using Device "FileStorage-Data1" to
> write.
> 2017-11-27 13:52:27 bareos-dir JobId 702: Start Backup JobId 702,
> Job=Machine1-Wiki.2017-11-27_13.52.25_51
>
>
>
> #
> # WORKS - Incremental job output (Same job)
> #
> 2017-11-27 14:00:08 bareos-dir JobId 703: Bareos bareos-dir 17.2.4 (21Sep17):
> Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise
> Linux Server release 7.0 (Maipo)
> JobId: 703
> Job: Machine1-Wiki.2017-11-27_13.59.58_02
> Backup Level: Incremental, since=2017-11-27 07:48:39
> Client: "Machine1-fd" 17.2.3 (14Aug17)
> x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0
> (Maipo),RHEL_7,x86_64
> FileSet: "Machine1-Wiki" 2017-10-03 15:58:33
> Pool: "Incremental" (From command line)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "File" (From command line)
> Scheduled time: 27-Nov-2017 13:59:58
> Start time: 27-Nov-2017 14:00:04
> End time: 27-Nov-2017 14:00:08
> Elapsed time: 4 secs
> Priority: 10
> FD Files Written: 139
> SD Files Written: 139
> FD Bytes Written: 24,751,160 (24.75 MB)
> SD Bytes Written: 24,770,990 (24.77 MB)
> Rate: 6187.8 KB/s
> Software Compression: 53.5 % (lzo)
> VSS: no
> Encryption: no
> Accurate: no
> Volume name(s): Incremental-0094
> Volume Session Id: 3
> Volume Session Time: 1511818757
> Last Volume Bytes: 24,800,601 (24.80 MB)
> Non-fatal FD errors: 0
> SD Errors: 0
> FD termination status: OK
> SD termination status: OK
> Termination: Backup OK
>
> 2017-11-27 14:00:08 bareos-dir JobId 703: sql_create.c:872 Insert of
> attributes batch table done
> 2017-11-27 14:00:07 bareos-sd JobId 703: Elapsed time=00:00:01, Transfer
> rate=24.77 M Bytes/second
> 2017-11-27 14:00:06 bareos-sd JobId 703: New volume "Incremental-0094"
> mounted on device "FileStorage-Data1" (/data1/bareos) at 27-Nov-2017 14:00.
> 2017-11-27 14:00:06 bareos-sd JobId 703: Recycled volume "Incremental-0094"
> on device "FileStorage-Data1" (/data1/bareos), all previous data lost.
> 2017-11-27 14:00:05 bareos-dir JobId 703: Recycled volume "Incremental-0094"
> 2017-11-27 14:00:05 bareos-dir JobId 703: All records pruned from Volume
> "Incremental-0094"; marking it "Purged"
> 2017-11-27 14:00:05 bareos-dir JobId 703: There are no more Jobs associated
> with Volume "Incremental-0094". Marking it purged.
> 2017-11-27 14:00:05 bareos-sd JobId 703: End of medium on Volume
> "Incremental-0093" Bytes=1,073,729,940 Blocks=2 at 27-Nov-2017 14:00.
> 2017-11-27 14:00:05 bareos-sd JobId 703: User defined maximum volume capacity
> 1,073,741,824 exceeded on device "FileStorage-Data1" (/data1/bareos).
> 2017-11-27 14:00:04 bareos-sd JobId 703: Ready to append to end of Volume
> "Incremental-0093" size=1073729940
> 2017-11-27 14:00:04 bareos-sd JobId 703: Volume "Incremental-0093" previously
> written, moving to end of data.
> 2017-11-27 14:00:04 bareos-dir JobId 703: Using Device "FileStorage-Data1" to
> write.
> 2017-11-27 14:00:04 bareos-dir JobId 703: Start Backup JobId 703,
> Job=Machine1-Wiki.2017-11-27_13.59.58_02
>
>
> #
> # FAILS Full job that filled Data4 device
> #
> 2017-11-27 07:43:33 bareos-dir JobId 687: Bareos bareos-dir 17.2.3 (14Aug17):
> Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise
> Linux Server release 7.0 (Maipo)
> JobId: 687
> Job: Machine2-MySQL.2017-11-26_21.05.00_43
> Backup Level: Full
> Client: "Machine2-fd" 17.2.3 (14Aug17)
> x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0
> (Maipo),RHEL_7,x86_64
> FileSet: "MySQL-Data" 2017-10-04 15:39:20
> Pool: "Full" (From Job FullPool override)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "File" (From Job resource)
> Scheduled time: 26-Nov-2017 21:05:00
> Start time: 26-Nov-2017 21:05:03
> End time: 27-Nov-2017 07:43:32
> Elapsed time: 10 hours 38 mins 29 secs
> Priority: 10
> FD Files Written: 0
> SD Files Written: 0
> FD Bytes Written: 0 (0 B)
> SD Bytes Written: 846,033 (846.0 KB)
> Rate: 0.0 KB/s
> Software Compression: None
> VSS: no
> Encryption: no
> Accurate: no
> Volume name(s): Full-0284|Full-0285|Full-0286
> Volume Session Id: 564
> Volume Session Time: 1507787926
> Last Volume Bytes: 1,354,941 (1.354 MB)
> Non-fatal FD errors: 1
> SD Errors: 0
> FD termination status: Error
> SD termination status: Canceled
> Termination: Backup Canceled
>
> 2017-11-27 04:43:14 bareos-sd JobId 687: Please mount append Volume
> "Full-0287" or label a new one for:
> Job: Machine2-MySQL.2017-11-26_21.05.00_43
> Storage: "FileStorage-Data4" (/data4/bareos)
> Pool: Full
> Media type: File
> 2017-11-27 04:38:14 bareos-sd JobId 687: Warning: mount.c:250 Open device
> "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663
> Could not open: /data4/bareos/Full-0287, ERR=No such file or directory
>
> ...
>
> 2017-11-27 00:48:14 bareos-sd JobId 687: Warning: mount.c:250 Open device
> "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663
> Could not open: /data4/bareos/Full-0287, ERR=No such file or directory
>
> 2017-11-27 00:43:14 bareos-sd JobId 687: Please mount append Volume
> "Full-0287" or label a new one for:
> Job: Machine2-MySQL.2017-11-26_21.05.00_43
> Storage: "FileStorage-Data4" (/data4/bareos)
> Pool: Full
> Media type: File
> 2017-11-27 00:38:14 bareos-sd JobId 687: Warning: mount.c:250 Open device
> "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663
> Could not open: /data4/bareos/Full-0287, ERR=No such file or directory
>
> ...
>
> 2017-11-26 21:43:13 bareos-sd JobId 687: Warning: label.c:376 Open device
> "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663
> Could not open: /data4/bareos/Full-0287, ERR=No space left on device
>
> 2017-11-26 21:43:13 bareos-dir JobId 687: All records pruned from Volume
> "Full-0287"; marking it "Purged"
> 2017-11-26 21:43:13 bareos-dir JobId 687: There are no more Jobs associated
> with Volume "Full-0287". Marking it purged.
> 2017-11-26 21:43:13 bareos-dir JobId 687: 1 File on Volume "Full-0287" purged
> from catalog.
> 2017-11-26 21:43:11 bareos-dir JobId 687: Purging oldest volume "Full-0287"
> 2017-11-26 21:43:11 bareos-sd JobId 687: End of medium on Volume "Full-0286"
> Bytes=1,354,941 Blocks=21 at 26-Nov-2017 21:43.
> 2017-11-26 21:43:11 bareos-sd JobId 687: End of Volume "Full-0286" at
> 0:1354940 on device "FileStorage-Data4" (/data4/bareos). Write of 64512 bytes
> got 21315.
> 2017-11-26 21:43:11 bareos-sd JobId 687: New volume "Full-0286" mounted on
> device "FileStorage-Data4" (/data4/bareos) at 26-Nov-2017 21:43.
> 2017-11-26 21:43:11 bareos-sd JobId 687: Wrote label to prelabeled Volume
> "Full-0286" on device "FileStorage-Data4" (/data4/bareos)
> 2017-11-26 21:43:10 bareos-sd JobId 687: Labeled new Volume "Full-0286" on
> device "FileStorage-Data4" (/data4/bareos).
> 2017-11-26 21:43:07 bareos-dir JobId 687: Recycled volume "Full-0286"
> 2017-11-26 21:43:07 bareos-dir JobId 687: All records pruned from Volume
> "Full-0286"; marking it "Purged"
> 2017-11-26 21:43:07 bareos-dir JobId 687: There are no more Jobs associated
> with Volume "Full-0286". Marking it purged.
> 2017-11-26 21:43:07 bareos-sd JobId 687: End of medium on Volume "Full-0285"
> Bytes=35,648,683,931 Blocks=552,590 at 26-Nov-2017 21:43.
> 2017-11-26 21:43:06 bareos-sd JobId 687: End of Volume "Full-0285" at
> 8:1288945562 on device "FileStorage-Data4" (/data4/bareos). Write of 64512
> bytes got 61541.
> 2017-11-26 21:26:55 bareos-sd JobId 687: New volume "Full-0285" mounted on
> device "FileStorage-Data4" (/data4/bareos) at 26-Nov-2017 21:26.
> 2017-11-26 21:26:55 bareos-sd JobId 687: Wrote label to prelabeled Volume
> "Full-0285" on device "FileStorage-Data4" (/data4/bareos)
> 2017-11-26 21:26:55 bareos-sd JobId 687: Labeled new Volume "Full-0285" on
> device "FileStorage-Data4" (/data4/bareos).
> 2017-11-26 21:26:55 bareos-dir JobId 687: Recycled volume "Full-0285"
> 2017-11-26 21:26:54 bareos-dir JobId 687: All records pruned from Volume
> "Full-0285"; marking it "Purged"
> 2017-11-26 21:26:54 bareos-dir JobId 687: There are no more Jobs associated
> with Volume "Full-0285". Marking it purged.
> 2017-11-26 21:26:53 bareos-sd JobId 687: End of medium on Volume "Full-0284"
> Bytes=53,687,081,703 Blocks=832,204 at 26-Nov-2017 21:26.
> 2017-11-26 21:26:53 bareos-sd JobId 687: User defined maximum volume capacity
> 53,687,091,200 exceeded on device "FileStorage-Data4" (/data4/bareos).
> 2017-11-26 21:05:03 bareos-sd JobId 687: Ready to append to end of Volume
> "Full-0284" size=4857758442
> 2017-11-26 21:05:03 bareos-sd JobId 687: Volume "Full-0284" previously
> written, moving to end of data.
> 2017-11-26 21:05:03 Machine2-fd JobId 687: shell command: run ClientBeforeJob
> "/usr/local/bin/mysqladmin flush-tables"
> 2017-11-26 21:05:03 Machine2-fd JobId 687: ClientBeforeJob: Slave stopped
> 2017-11-26 21:05:03 Machine2-fd JobId 687: shell command: run ClientBeforeJob
> "/usr/local/bin/mysqladmin stop-slave"
> 2017-11-26 21:05:03 bareos-dir JobId 687: Using Device "FileStorage-Data4" to
> write.
> 2017-11-26 21:05:02 bareos-dir JobId 687: Start Backup JobId 687,
> Job=Machine2-MySQL.2017-11-26_21.05.00_43
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.
