Hi Brock,
thank you. If that might help somebody, instead of 2xN, I managed N+1
emails with the following query (which is also better in regard to failed
backups, and making sure it copies the backups, not the copies, and
assuming the latest full will always have the highest jobid):
...
Selection Type = SQLQuery
Selection Pattern = "select max(jobid), name from job where type = 'B' and
level = 'F' and jobstatus IN ('T','W') and name =
any(array['server1-sys-file','server2-sys-file']) group by name;"
...
On Friday, April 29, 2022 at 3:49:09 PM UTC+2 [email protected]
wrote:
> That’s pretty much how it works, a copy jobs actually spawns other jobs
> that do the actual copy.
>
> One thing about copy jobs that prevented me form using them the way I
> wanted (this may not be an issue for you as you have to SD’s online) is
> that Consolidation jobs for always incremental jobs works flaky with it,
> and has no provisioned for an offline copy.
>
> I ended up using the archive job for my offline second copy.
>
>
> Brock Palen
> [email protected]
> www.mlds-networks.com
> Websites, Linux, Hosting, Joomla, Consulting
>
>
>
> > On Apr 29, 2022, at 8:51 AM, [email protected] <[email protected]>
> wrote:
> >
> > Hello,
> >
> > There doesn't seem to be a lot of documentation for copy jobs, although
> I think they are extremely practical when having 2 storage daemons, and
> backing up remote clients to only use bandwidth once.
> >
> > Problem: whenever I (manually, or scheduled) run a copy job, bareos
> actually runs 3 jobs:
> > 1) one "copy job" named "myserver-sys-copy-file-to-tape" which only
> selects the job to be copied, and runs for not even a second
> > 2) one "copy job" named "myserver-sys-copy-file-to-tape" as well, which
> seems to be actual copy job (it has the read, and write storage attributes)
> > 3) one "copy of a job" named "myserver-sys-file" which is initially
> empty, but then fills with all the log-data of the initial job which was
> copied (also having the jobid of the file-backup which is copied to the
> tape)
> >
> > Is this "as intended", or can I improve my config so I declutter my
> inbox/messages when doing a copy? I plan to have a couple of copy jobs
> which copies the last Full backup from file to tape. I could try one copy
> job for all remote clients (doing e.g. a SQL group by/distinct by
> client-name, and limiting the result to the amount of clients I want to
> backup), but I guess it would only save me one email/job per client (so for
> 5 clients, I would still have 11 emails).
> >
> > It would be great if somebody can have a look at my config and point out
> what is wrong, so I only get (ideally) 1 email per copy job?
> >
> > ==== Here are the logs of the 3 jobs ====
> > 1) copy job which seems to select the jobid to be copied:
> > *Director Running Jobs*
> > Running Jobs:
> > Console connected at 29-Apr-22 12:38
> > Console connected at 29-Apr-22 12:38
> > JobId Level Name Status
> > ======================================================================
> > 1084 Increme myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11 is
> waiting on max Job jobs
> > 1085 Full myserver-sys-file.2022-04-29_12.38.35_12 is waiting execution
> > ====
> > *Joblog*
> > Connecting to Director localhost:9101
> > Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
> > 1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
> > bareos.org build binary
> > bareos.org binaries are UNSUPPORTED by bareos.com.
> > Get official binaries and vendor support on https://www.bareos.com
> > You are connected using the default console
> >
> > Enter a period (.) to cancel a command.
> > list joblog jobid=1083
> > Automatically selected Catalog: MyCatalog
> > Using Catalog "MyCatalog"
> > 2022-04-29 12:38:35 bareos-dir JobId 1083: The following 1 JobId was
> chosen to be copied: 1069
> > 2022-04-29 12:38:35 bareos-dir JobId 1083: Automatically selected
> Catalog: MyCatalog
> > 2022-04-29 12:38:35 bareos-dir JobId 1083: Using Catalog "MyCatalog"
> > 2022-04-29 12:38:35 bareos-dir JobId 1083: Job queued. JobId=1084
> > 2022-04-29 12:38:35 bareos-dir JobId 1083: Copying JobId 1084 started.
> > 2022-04-29 12:38:35 bareos-dir JobId 1083: Bareos bareos-dir 21.0.0
> (21Dec21):
> > Build OS: Ubuntu 20.04.3 LTS
> > Current JobId: 1083
> > Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.33_06
> > Catalog: "MyCatalog" (From Default catalog)
> > Start time: 29-Apr-2022 12:38:35
> > End time: 29-Apr-2022 12:38:35
> > Elapsed time: 0 secs
> > Priority: 20
> > Bareos binary info: bareos.org build: Get official binaries and vendor
> support on bareos.com
> > Job triggered by: User
> > Termination: Copying OK
> >
> > You have messages.
> >
> > *Files backed up*
> >
> > You have messages.
> >
> > 2) the actual copy job between the 2 storage daemons
> >
> >
> > *Director Running Jobs*
> > Running Jobs:
> > Console connected at 29-Apr-22 12:38
> > Console connected at 29-Apr-22 12:39
> > JobId Level Name Status
> > ======================================================================
> > 1085 Full myserver-sys-file.2022-04-29_12.38.35_12 has terminated
> > ====
> > *Joblog*
> > Connecting to Director localhost:9101
> > Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
> > 1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
> > bareos.org build binary
> > bareos.org binaries are UNSUPPORTED by bareos.com.
> > Get official binaries and vendor support on https://www.bareos.com
> > You are connected using the default console
> >
> > Enter a period (.) to cancel a command.
> > list joblog jobid=1084
> > Automatically selected Catalog: MyCatalog
> > Using Catalog "MyCatalog"
> > 2022-04-29 12:38:35 bareos-dir JobId 1084: Copying using JobId=1069
> Job=myserver-sys-file.2022-04-29_12.01.16_47
> > 2022-04-29 12:38:35 bareos-dir JobId 1084: Bootstrap records written to
> /var/lib/bareos/bareos-dir.restore.9.bsr
> > 2022-04-29 12:38:37 bareos-dir JobId 1084: Start Copying JobId 1084,
> Job=myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11
> > 2022-04-29 12:38:37 bareos-dir JobId 1084: Connected Storage daemon at
> bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256
> TLSv1.3
> > 2022-04-29 12:38:37 bareos-dir JobId 1084: Using Device "bareos-file" to
> read.
> > 2022-04-29 12:38:37 bareos-dir JobId 1085: Connected Storage daemon at
> eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256
> TLSv1.3
> > 2022-04-29 12:38:37 bareos-dir JobId 1085: Using Device "Ultrium1760" to
> write.
> > 2022-04-29 12:38:37 bareos-sd JobId 1084: Connected Storage daemon at
> eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256
> TLSv1.3
> > 2022-04-29 12:38:37 bareos-sd JobId 1084: Ready to read from volume
> "Full-0154" on device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
> > 2022-04-29 12:38:37 eljadid-sd JobId 1085: Spooling data ...
> > 2022-04-29 12:38:37 bareos-sd JobId 1084: Forward spacing Volume
> "Full-0154" to file:block 3:1118992026.
> > 2022-04-29 12:38:57 bareos-sd JobId 1084: End of Volume at file 3 on
> device "bareos-file" (/mnt/leah/var_lib_bareos_storage), Volume "Full-0154"
> > 2022-04-29 12:38:57 bareos-sd JobId 1084: End of all volumes.
> > 2022-04-29 12:38:57 eljadid-sd JobId 1085: Committing spooled data to
> Volume "XM2724L3". Despooling 931,953,968 bytes ...
> > 2022-04-29 12:39:17 eljadid-sd JobId 1085: Despooling elapsed time =
> 00:00:20, Transfer rate = 46.59 M Bytes/second
> > 2022-04-29 12:39:17 eljadid-sd JobId 1085: Releasing device
> "Ultrium1760" (/dev/tape/by-id/scsi-HU19326DV6).
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: smartctl 7.1
> 2019-12-30 r5022 [x86_64-linux-5.4.0-109-generic] (local build)
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Copyright (C) 2002-19,
> Bruce Allen, Christian Franke, www.smartmontools.org
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert:
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: === START OF READ
> SMART DATA SECTION ===
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: TapeAlert Not Supported
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Error Counter logging
> not supported
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert:
> >
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Elapsed time=00:00:42,
> Transfer rate=22.07 M Bytes/second
> > 2022-04-29 12:39:19 eljadid-sd JobId 1085: Sending spooled attrs to the
> Director. Despooling 24,595,188 bytes ...
> > 2022-04-29 12:39:21 bareos-sd JobId 1084: Releasing device "bareos-file"
> (/mnt/leah/var_lib_bareos_storage).
> > 2022-04-29 12:39:21 bareos-dir JobId 1085: Insert of attributes batch
> table with 74025 entries start
> > 2022-04-29 12:39:23 bareos-dir JobId 1085: Insert of attributes batch
> table done
> > 2022-04-29 12:39:23 bareos-dir JobId 1084: Bareos bareos-dir 21.0.0
> (21Dec21):
> > Build OS: Ubuntu 20.04.3 LTS
> > Prev Backup JobId: 1069
> > Prev Backup Job: myserver-sys-file.2022-04-29_12.01.16_47
> > New Backup JobId: 1085
> > Current JobId: 1084
> > Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11
> > Backup Level: Incremental
> > Client: myserver-fd
> > FileSet: "myserver-sys-file"
> > Read Pool: "Full" (From Job resource)
> > Read Storage: "bareos-file" (From Pool resource)
> > Write Pool: "tape-automated" (From Job's NextPool resource)
> > Write Storage: "eljadid_HP_1_8_G2" (From Storage from Job's NextPool
> resource)
> > Next Pool: "tape-automated" (From Job's NextPool resource)
> > Catalog: "MyCatalog" (From Default catalog)
> > Start time: 29-Apr-2022 12:38:37
> > End time: 29-Apr-2022 12:39:23
> > Elapsed time: 46 secs
> > Priority: 20
> > SD Files Written: 74,025
> > SD Bytes Written: 927,271,942 (927.2 MB)
> > Rate: 20158.1 KB/s
> > Volume name(s): XM2724L3
> > Volume Session Id: 10
> > Volume Session Time: 1651214033
> > Last Volume Bytes: 19,536,942,080 (19.53 GB)
> > SD Errors: 0
> > SD termination status: OK
> > Bareos binary info: bareos.org build: Get official binaries and vendor
> support on bareos.com
> > Job triggered by: User
> > Termination: Copying OK
> >
> > You have messages.
> >
> > *Files backed up*
> >
> > You have messages.
> >
> >
> >
> >
> > 3) the "copy of a job" with a new jobid (1085 in this case), but the
> logs of the job which is copied
> > *Director Running Jobs*
> > Running Jobs:
> > Console connected at 29-Apr-22 13:01
> > No Jobs running.
> > ====
> > *Joblog*
> > Connecting to Director localhost:9101
> > Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
> > 1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
> > bareos.org build binary
> > bareos.org binaries are UNSUPPORTED by bareos.com.
> > Get official binaries and vendor support on https://www.bareos.com
> > You are connected using the default console
> >
> > Enter a period (.) to cancel a command.
> > list joblog jobid=1085
> > Automatically selected Catalog: MyCatalog
> > Using Catalog "MyCatalog"
> > 2022-04-29 12:01:18 bareos-dir JobId 1069: Start Backup JobId 1069,
> Job=myserver-sys-file.2022-04-29_12.01.16_47
> > 2022-04-29 12:01:18 bareos-dir JobId 1069: Connected Storage daemon at
> bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256
> TLSv1.3
> > 2022-04-29 12:01:18 bareos-dir JobId 1069: Using Device "bareos-file" to
> write.
> > 2022-04-29 12:01:18 bareos-dir JobId 1069: Probing client protocol...
> (result will be saved until config reload)
> > 2022-04-29 12:01:22 bareos-dir JobId 1069: Connected Client: myserver-fd
> at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
> > 2022-04-29 12:01:23 bareos-dir JobId 1069: Handshake: Immediate TLS
> 2022-04-29 12:01:23 bareos-dir JobId 1069: Encryption:
> TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
> > 2022-04-29 12:01:25 myserver-fd JobId 1069: shell command: run
> ClientBeforeJob "/usr/local/bin/bareos-runbefore.sh"
> > 2022-04-29 12:01:25 myserver-fd JobId 1069: Extended attribute support
> is enabled
> > 2022-04-29 12:01:25 myserver-fd JobId 1069: ACL support is enabled
> > 2022-04-29 12:01:23 bareos-sd JobId 1069: Connected File Daemon at
> sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
> > 2022-04-29 12:01:25 bareos-sd JobId 1069: Volume "Full-0154" previously
> written, moving to end of data.
> > 2022-04-29 12:01:25 bareos-sd JobId 1069: Ready to append to end of
> Volume "Full-0154" size=14003893914
> > 2022-04-29 12:25:39 bareos-sd JobId 1069: Releasing device "bareos-file"
> (/mnt/leah/var_lib_bareos_storage).
> > 2022-04-29 12:25:39 bareos-sd JobId 1069: Elapsed time=00:24:14,
> Transfer rate=637.7 K Bytes/second
> > 2022-04-29 12:25:39 bareos-dir JobId 1069: Insert of attributes batch
> table with 74025 entries start
> > 2022-04-29 12:25:41 bareos-dir JobId 1069: Insert of attributes batch
> table done
> > 2022-04-29 12:25:41 bareos-dir JobId 1069: Bareos bareos-dir 21.0.0
> (21Dec21):
> > Build OS: Ubuntu 20.04.3 LTS
> > JobId: 1069
> > Job: myserver-sys-file.2022-04-29_12.01.16_47
> > Backup Level: Full
> > Client: "myserver-fd" 21.0.0 (21Dec21) Debian GNU/Linux 11
> (bullseye),debian
> > FileSet: "myserver-sys-file" 2022-04-28 17:29:32
> > Pool: "Full" (From command line)
> > Catalog: "MyCatalog" (From Client resource)
> > Storage: "bareos-file" (From Pool resource)
> > Scheduled time: 29-Apr-2022 12:01:16
> > Start time: 29-Apr-2022 12:01:25
> > End time: 29-Apr-2022 12:25:41
> > Elapsed time: 24 mins 16 secs
> > Priority: 10
> > FD Files Written: 74,025
> > SD Files Written: 74,025
> > FD Bytes Written: 881,441,457 (881.4 MB)
> > SD Bytes Written: 927,271,942 (927.2 MB)
> > Rate: 605.4 KB/s
> > Software Compression: 61.5 % (lz4)
> > VSS: no
> > Encryption: yes
> > Accurate: no
> > Volume name(s): Full-0154
> > Volume Session Id: 4
> > Volume Session Time: 1651214033
> > Last Volume Bytes: 14,936,345,934 (14.93 GB)
> > Non-fatal FD errors: 0
> > SD Errors: 0
> > FD termination status: OK
> > SD termination status: OK
> > Bareos binary info: bareos.org build: Get official binaries and vendor
> support on bareos.com
> > Job triggered by: User
> > Termination: Backup OK
> >
> > 2022-04-29 12:25:41 bareos-dir JobId 1069: shell command: run AfterJob
> "/bin/bash -c '/bin/echo "run myserver-sys-file-v yes" | bconsole
> >/dev/null'"
> > You have messages.
> >
> > *Files backed up*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ==== Here is my config ====
> >
> > Job { #the job which backs up the remote client to file/disk
> > Name = "myserver-sys-file"
> > Type = Backup
> > Level = Incremental
> > Client = "myserver-fd"
> > FileSet = myserver-sys-file
> > Schedule = myserver-sys-file
> > Storage = bareos-file
> > Messages = Standard
> > Maximum Concurrent Jobs = 7
> > Pool = Incremental
> > Priority = 10
> > Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr"
> > Allow Mixed Priority = yes
> >
> > #prevent duplicate jobs
> > Allow Duplicate Jobs = no
> > Cancel Lower Level Duplicates = yes
> > Cancel Queued Duplicates = yes
> >
> > Full Backup Pool = Full # write Full Backups into "Full" Pool (#05)
> > Differential Backup Pool = Differential # write Diff Backups into
> "Differential" Pool (#08)
> > Incremental Backup Pool = Incremental # write Incr Backups into
> "Incremental" Pool (#11)
> >
> > RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-file-v yes\" |
> bconsole >/dev/null'"
> > RunScript {
> > Command = "/usr/local/bin/bareos-runbefore.sh"
> > RunsWhen = before
> > RunsOnClient = yes
> > }
> > }
> >
> > Job { #the job which copies to tape
> > Name = myserver-sys-copy-file-to-tape
> > Type = Copy
> > Level = Full
> > Pool = Full
> > Selection Type = SQLQuery
> > Selection Pattern = "select jobid from job where name =
> 'myserver-sys-file' and level ='F' order by endtime DESC limit 1;"
> > Next Pool = tape-automated
> > Schedule = myserver-sys-copy-file-to-tape
> > Spool Data = Yes
> > Messages = Standard
> > Priority = 20
> > Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr"
> >
> > Allow Duplicate Jobs = no
> > Cancel Lower Level Duplicates = yes
> > Cancel Queued Duplicates = yes
> >
> > RunAfterJob = "/bin/bash -c '/bin/echo \"run
> myserver-sys-copy-file-to-tape-v yes\" | bconsole >/dev/null'"
> > }
> >
> > Pool { #file/disk pool
> > Name = Full
> > Pool Type = Backup
> > Recycle = yes # Bareos can automatically recycle Volumes
> > AutoPrune = yes # Prune expired volumes
> > Volume Retention = 365 days # How long should the Full Backups be kept?
> (#06)
> > Job Retention = 365 days
> > File Retention = 365 days
> > Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
> > Maximum Volumes = 200 # Limit number of Volumes in Pool
> > Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"
> > Storage = bareos-file
> > Next Pool = tape-automated
> > }
> >
> >
> > Pool { #tape pool
> > Name = tape-automated
> > Pool Type = Backup
> > Recycle = yes # Bareos can automatically recycle Volumes
> > AutoPrune = yes # Prune expired volumes
> > #Recycle Oldest Volume = yes
> > RecyclePool = Scratch
> > Maximum Volume Bytes = 0
> > Job Retention = 365 days
> > Volume Retention = 4 weeks
> > Volume Use Duration = 12 days
> > Cleaning Prefix = "CLN"
> > Catalog Files = yes #needed, this is default
> > Storage = tapeSD_HP_1_8_G2
> > }
> >
> > Storage { #this is the file/disk storage daemon
> > Name = bareos-file
> > Address = "bareos.mydomain.org" # N.B. Use a fully qualified name here
> (do not use "localhost" here).
> > Password = "XXXX"
> > Device = bareos-file
> > Media Type = File
> > Maximum Concurrent Jobs = 7
> >
> > #dashboard progress / statistics collection
> > Collect Statistics = yes
> >
> > TLS Enable = yes
> > TLS Require = yes
> > TLS DH File = /etc/bareos/ssl/dh4096.pem
> > }
> >
> >
> > Storage { #this is the tape storage daemon on another server
> > Name = "tapeSD_HP_1_8_G2"
> > Address = "tapeSD.mydomain.org"
> > Lan Address = "tapeSD.directlink.mydomain.org"
> > Password = "XXXX"
> > Device = HP_1_8_G2
> > Autochanger = yes
> > MediaType = "LTO"
> > #dashboard progress / statistics collection
> > Collect Statistics = yes
> >
> > TLS Enable = yes
> > TLS Require = yes
> > TLS DH File = /etc/bareos/ssl/dh4096.pem
> > }
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "bareos-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected].
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/bareos-users/508d21ad-1d5e-449f-9753-9735dd97c03cn%40googlegroups.com
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/3f73f0a7-5916-4538-96a7-a4fc66598e19n%40googlegroups.com.
