Re: [Bacula-users] Building the S3/Amazon options for Bacula 15

[Dan Langille]

On Fri, Apr 12, 2024, at 8:27 AM, Martin Simmons wrote:
>>>>>> On Thu, 11 Apr 2024 16:34:48 -0400, Dan Langille said:
>> 
>> A problem with building the S3 options on Bacula 15.0.2 has been reported[1] 
>> and I'm trying to figure it out. I'm not an s3 user myself, and I am the 
>> maintainer of the FreeBSD port/package.
>> 
>> When building 13.0.4, see:
>> 
>> src/stored/.libs/bacula-sd-cloud-s3-driver.so
>> src/stored/.libs/bacula-sd-cloud-s3-driver-13.0.4.so
>> 
>> When building 15.0.2, I see
>> 
>> src/stored/.libs/bacula-sd-cloud-driver.so
>> src/stored/.libs/bacula-sd-cloud-driver-15.0.2.so
>> 
>> Of note, the '-s3' part of the name has been removed. That doesn't seem to 
>> be my problem though.
>
> These are different libraries and you need both of them
> (bacula-sd-cloud-driver loads bacula-sd-cloud-s3-driver if required).  I think
> this is the same in 13.0.4.
>
>> The problem is the files do not get installed, and I'm not sure why.
>
> The build log contains:
>
> checking libs3.h usability... no
> checking libs3.h presence... no
> checking for libs3.h... no
>
> so that looke like the problem.  Where is your libs3.h?

First goal, who supplies that?

Searching at https://www.freshports.org by pkg-plist, I find 
sysutils/bacula-libs3

How is that used:

[12:30 pkg01 dvl ~/ports/head/sysutils] % grep -r bacula-libs3 bacula*  

bacula11-server/Makefile:LIB_DEPENDS+=  libs3.so:sysutils/bacula-libs3
bacula13-server/Makefile:LIB_DEPENDS+=  libs3.so:sysutils/bacula-libs3
bacula15-server/Makefile:MS3_LIB_DEPENDS=   libs3.so:sysutils/bacula-libs3

Interesting how it's MS3 for 15, and S3 for 11 and 13. I have only myself to 
blame and I have no idea why it's like that.

It appears to be cruft leftover from my mangled transformation from "if 
${PORT_OPTIONS:MS3}" to modern methods.

compare old and new:

https://cgit.freebsd.org/ports/tree/sysutils/bacula13-server/Makefile#n135
https://cgit.freebsd.org/ports/tree/sysutils/bacula15-server/Makefile#n136

Doing this change allowed the build:

[14:41 pkg01 dvl ~/ports/head/sysutils/bacula15-server] % git diff .
diff --git a/sysutils/bacula15-server/Makefile 
b/sysutils/bacula15-server/Makefile
index 6e7aec421cd7..a9d84c4720a0 100644
--- a/sysutils/bacula15-server/Makefile
+++ b/sysutils/bacula15-server/Makefile
@@ -133,10 +133,10 @@ MTX_RUN_DEPENDS=  ${LOCALBASE}/sbin/mtx:misc/mtx
 
 .endif
 
-.include 
+S3_CONFIGURE_ARGS= --enable-s3
+S3_LIB_DEPENDS=libs3.so:sysutils/bacula-libs3
 
-MS3_CONFIGURE_ARGS=--enable-s3
-MS3_LIB_DEPENDS=   libs3.so:sysutils/bacula-libs3
+.include 
 
 .if defined(WITH_CLIENT_ONLY)
 MP8+=  bacula.8 bacula-fd.8 bconsole.8

Now I find:

root@140amd64-dvl-primary:/wrkdirs/usr/ports/sysutils/bacula15-server/work # 
find . -name bacula-sd-cloud-s3-driver.so
./stage/usr/local/lib/bacula-sd-cloud-s3-driver.so
./bacula-15.0.2/src/stored/.libs/bacula-sd-cloud-s3-driver.so

Thanks for help me find my own error. I'll pass this onto the user for testing.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Building the S3/Amazon options for Bacula 15

[Dan Langille]

A problem with building the S3 options on Bacula 15.0.2 has been reported[1] 
and I'm trying to figure it out. I'm not an s3 user myself, and I am the 
maintainer of the FreeBSD port/package.

When building 13.0.4, see:

src/stored/.libs/bacula-sd-cloud-s3-driver.so
src/stored/.libs/bacula-sd-cloud-s3-driver-13.0.4.so

When building 15.0.2, I see

src/stored/.libs/bacula-sd-cloud-driver.so
src/stored/.libs/bacula-sd-cloud-driver-15.0.2.so

Of note, the '-s3' part of the name has been removed. That doesn't seem to be 
my problem though.

The problem is the files do not get installed, and I'm not sure why.

Is the correct option still --enable-s3

Checking the config options[2] I see no mention of it, nor mention of s3 or 
cloud.

I did find it in the configure file though:

work/bacula-15.0.2/configure:# Check whether --enable-s3 was given.

Is it me? What am I doing wrong? The 15 package is using the same build process 
as 13.

Thank you


[1] - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278312

[2] - 
https://www.bacula.org/15.0.x-manuals/en/main/Installing_Bacula.html#SECTION0019120
-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] FreeBSD: bacula9 and bacula11 are now deprecated

[Dan Langille]

Hello,

HEADS UP for all FreeBSD users: sysutils/bacula9-server and 
sysutils/bacula-11-server are deprecated and will be removed from the ports 
tree at the end of September 2024 (they won't be in the 2024Q4 ports tree 
branch).

Those releases have not updated in over 3 and 2 years respectively.  Bacula 13 
and 15 available.

Thank you.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula: Backup OK -- with warnings - what warning?

[Dan Langille]

On Tue, Jan 23, 2024, at 12:05 PM, Dan Langille wrote:
> On Tue, Jan 23, 2024, at 11:16 AM, Martin Simmons wrote:
>>>>>>> On Wed, 17 Jan 2024 09:54:23 -0500, Dan Langille said:
>>> 
>>> On Fri, Dec 29, 2023, at 6:26 PM, Martin Simmons wrote:
>>> >>>>>> On Fri, 29 Dec 2023 12:35:59 -0500, Dan Langille said:
>>> >> 
>>> >> On Fri, Dec 29, 2023, at 12:10 PM, Martin Simmons wrote:
>>> >> > 9.6.6 certainly displayed them for me, so I suspect a config issue.
>>> >> >
>>> >> > The messages would be omitted if !notsaved is in the Messages resource 
>>> >> > (but
>>> >> > they would still be counted as "Non-fatal FD errors" which makes it 
>>> >> > add "with
>>> >> > warnings" to the status).
>>> >> >
>>> >> > Maybe that changed in the client's bacula-fd.conf when you upgraded it?
>>> >> 
>>> >> That's a good idea.
>>> >> 
>>> >> [17:29 r730-01 dvl ~] % sudo ls -l /usr/local/etc/bacula/bacula-fd.conf
>>> >> -rw-r-  1 root bacula 1497 Feb 25  2023 
>>> >> /usr/local/etc/bacula/bacula-fd.conf
>>> >> 
>>> >> [17:31 r730-01 dvl ~] % sudo md5 /usr/local/etc/bacula/bacula-fd.conf
>>> >> MD5 (/usr/local/etc/bacula/bacula-fd.conf) = 
>>> >> e41a7d835766f563253c0a93418a1c61
>>> >> 
>>> >> 
>>> >> No change since February.
>>> >> 
>>> >> Let's look at snapshots taken before Dec 25, the date of the job in 
>>> >> question.
>>> >> 
>>> >> [17:32 r730-01 dvl /.zfs/snapshot] % cd 
>>> >> autosnap_2023-12-20_00:00:09_daily
>>> >> [17:32 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % 
>>> >> sudo md5 usr/local/etc/bacula/bacula-fd.conf 
>>> >> MD5 (usr/local/etc/bacula/bacula-fd.conf) = 
>>> >> e41a7d835766f563253c0a93418a1c61
>>> >> 
>>> >> 
>>> >> I'm confident this file has not changed.
>>> >
>>> > Hmm, looking at src/lib/message.h, I suspect this change broke version
>>> > compatibility in the message filtering infrastructure:
>>> >
>>> > commit fd926fc4671b054234fd3d5957bc05d303d87763
>>> > Author: Eric Bollengier 
>>> > Date:   Fri Nov 6 21:27:05 2020 +0100
>>> >
>>> > Fix unexpected connection event sent by the FD when the Message 
>>> > resource is not configured
>>> >
>>> > The problem is that the message types have been renumbered by moving 
>>> > M_EVENTS
>>> > higher up, but messages sent to the Director from other daemons use the
>>> > numeric value of the type so this is an incompatible change in the wire
>>> > protocol.
>>> >
>>> > Dispite the date of this change, it looks like it first appeared in 
>>> > Bacula 13,
>>> > so will cause problems if a Client < 13 sends a message to a Director >= 
>>> > 13 as
>>> > in your case.
>>> 
>>> That is concerning. It means backups may be incomplete and you don't know 
>>> it.
>>> 
>>> This has happened at home, and today I noticed it at $WORK.
>>> 
>>> It is no longer the case that versions can follow the rule:
>>> 
>>>   bacula-dir=bacula-sd>bacula-fd
>>> 
>>> That rule has been in effect as long as I've been associated with the 
>>> project (about 20 years). Is there any possibility of this being fixed? Or 
>>> is this change irrevocable?
>>
>> The change to the numbering is probably irrevocable (changing it back would
>> create a new set of incompatibilities), but some hack might be possible using
>> the jcr->FDVersion.
>>
>>
>>> If irrevocable, the users really need to be notified via an announcement. 
>>> Clients must be upgraded or the risk of data loss is present. In my case, 
>>> things I expected to be backed up were not being backed up. I could not 
>>> tell because the warnings were not presented to me.
>>
>> I suggest you create a bug report about it at
>> https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues
>> so it can tracked.
>
> And there we go: 
> https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues/2704

A fix has been committed. Expect a new release for 13.x and the fix will be in 
15.x

I have patched the FreeBSD port: 
https://cgit.freebsd.org/ports/commit/?id=3c57e9fa97a073c2911a6b3c76161376d3d185f6

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula: Backup OK -- with warnings - what warning?

[Dan Langille]

On Tue, Jan 23, 2024, at 11:16 AM, Martin Simmons wrote:
>>>>>> On Wed, 17 Jan 2024 09:54:23 -0500, Dan Langille said:
>> 
>> On Fri, Dec 29, 2023, at 6:26 PM, Martin Simmons wrote:
>> >>>>>> On Fri, 29 Dec 2023 12:35:59 -0500, Dan Langille said:
>> >> 
>> >> On Fri, Dec 29, 2023, at 12:10 PM, Martin Simmons wrote:
>> >> > 9.6.6 certainly displayed them for me, so I suspect a config issue.
>> >> >
>> >> > The messages would be omitted if !notsaved is in the Messages resource 
>> >> > (but
>> >> > they would still be counted as "Non-fatal FD errors" which makes it add 
>> >> > "with
>> >> > warnings" to the status).
>> >> >
>> >> > Maybe that changed in the client's bacula-fd.conf when you upgraded it?
>> >> 
>> >> That's a good idea.
>> >> 
>> >> [17:29 r730-01 dvl ~] % sudo ls -l /usr/local/etc/bacula/bacula-fd.conf
>> >> -rw-r-  1 root bacula 1497 Feb 25  2023 
>> >> /usr/local/etc/bacula/bacula-fd.conf
>> >> 
>> >> [17:31 r730-01 dvl ~] % sudo md5 /usr/local/etc/bacula/bacula-fd.conf
>> >> MD5 (/usr/local/etc/bacula/bacula-fd.conf) = 
>> >> e41a7d835766f563253c0a93418a1c61
>> >> 
>> >> 
>> >> No change since February.
>> >> 
>> >> Let's look at snapshots taken before Dec 25, the date of the job in 
>> >> question.
>> >> 
>> >> [17:32 r730-01 dvl /.zfs/snapshot] % cd autosnap_2023-12-20_00:00:09_daily
>> >> [17:32 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % 
>> >> sudo md5 usr/local/etc/bacula/bacula-fd.conf 
>> >> MD5 (usr/local/etc/bacula/bacula-fd.conf) = 
>> >> e41a7d835766f563253c0a93418a1c61
>> >> 
>> >> 
>> >> I'm confident this file has not changed.
>> >
>> > Hmm, looking at src/lib/message.h, I suspect this change broke version
>> > compatibility in the message filtering infrastructure:
>> >
>> > commit fd926fc4671b054234fd3d5957bc05d303d87763
>> > Author: Eric Bollengier 
>> > Date:   Fri Nov 6 21:27:05 2020 +0100
>> >
>> > Fix unexpected connection event sent by the FD when the Message 
>> > resource is not configured
>> >
>> > The problem is that the message types have been renumbered by moving 
>> > M_EVENTS
>> > higher up, but messages sent to the Director from other daemons use the
>> > numeric value of the type so this is an incompatible change in the wire
>> > protocol.
>> >
>> > Dispite the date of this change, it looks like it first appeared in Bacula 
>> > 13,
>> > so will cause problems if a Client < 13 sends a message to a Director >= 
>> > 13 as
>> > in your case.
>> 
>> That is concerning. It means backups may be incomplete and you don't know it.
>> 
>> This has happened at home, and today I noticed it at $WORK.
>> 
>> It is no longer the case that versions can follow the rule:
>> 
>>   bacula-dir=bacula-sd>bacula-fd
>> 
>> That rule has been in effect as long as I've been associated with the 
>> project (about 20 years). Is there any possibility of this being fixed? Or 
>> is this change irrevocable?
>
> The change to the numbering is probably irrevocable (changing it back would
> create a new set of incompatibilities), but some hack might be possible using
> the jcr->FDVersion.
>
>
>> If irrevocable, the users really need to be notified via an announcement. 
>> Clients must be upgraded or the risk of data loss is present. In my case, 
>> things I expected to be backed up were not being backed up. I could not tell 
>> because the warnings were not presented to me.
>
> I suggest you create a bug report about it at
> https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues
> so it can tracked.

And there we go: 
https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues/2704
-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula: Backup OK -- with warnings - what warning?

[Dan Langille]

On Fri, Dec 29, 2023, at 6:26 PM, Martin Simmons wrote:
>>>>>> On Fri, 29 Dec 2023 12:35:59 -0500, Dan Langille said:
>> 
>> On Fri, Dec 29, 2023, at 12:10 PM, Martin Simmons wrote:
>> > 9.6.6 certainly displayed them for me, so I suspect a config issue.
>> >
>> > The messages would be omitted if !notsaved is in the Messages resource (but
>> > they would still be counted as "Non-fatal FD errors" which makes it add 
>> > "with
>> > warnings" to the status).
>> >
>> > Maybe that changed in the client's bacula-fd.conf when you upgraded it?
>> 
>> That's a good idea.
>> 
>> [17:29 r730-01 dvl ~] % sudo ls -l /usr/local/etc/bacula/bacula-fd.conf
>> -rw-r-  1 root bacula 1497 Feb 25  2023 
>> /usr/local/etc/bacula/bacula-fd.conf
>> 
>> [17:31 r730-01 dvl ~] % sudo md5 /usr/local/etc/bacula/bacula-fd.conf
>> MD5 (/usr/local/etc/bacula/bacula-fd.conf) = e41a7d835766f563253c0a93418a1c61
>> 
>> 
>> No change since February.
>> 
>> Let's look at snapshots taken before Dec 25, the date of the job in question.
>> 
>> [17:32 r730-01 dvl /.zfs/snapshot] % cd autosnap_2023-12-20_00:00:09_daily
>> [17:32 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % sudo 
>> md5 usr/local/etc/bacula/bacula-fd.conf 
>> MD5 (usr/local/etc/bacula/bacula-fd.conf) = e41a7d835766f563253c0a93418a1c61
>> 
>> 
>> I'm confident this file has not changed.
>
> Hmm, looking at src/lib/message.h, I suspect this change broke version
> compatibility in the message filtering infrastructure:
>
> commit fd926fc4671b054234fd3d5957bc05d303d87763
> Author: Eric Bollengier 
> Date:   Fri Nov 6 21:27:05 2020 +0100
>
> Fix unexpected connection event sent by the FD when the Message 
> resource is not configured
>
> The problem is that the message types have been renumbered by moving M_EVENTS
> higher up, but messages sent to the Director from other daemons use the
> numeric value of the type so this is an incompatible change in the wire
> protocol.
>
> Dispite the date of this change, it looks like it first appeared in Bacula 13,
> so will cause problems if a Client < 13 sends a message to a Director >= 13 as
> in your case.

That is concerning. It means backups may be incomplete and you don't know it.

This has happened at home, and today I noticed it at $WORK.

It is no longer the case that versions can follow the rule:

  bacula-dir=bacula-sd>bacula-fd

That rule has been in effect as long as I've been associated with the project 
(about 20 years). Is there any possibility of this being fixed? Or is this 
change irrevocable?

If irrevocable, the users really need to be notified via an announcement. 
Clients must be upgraded or the risk of data loss is present. In my case, 
things I expected to be backed up were not being backed up. I could not tell 
because the warnings were not presented to me.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula: Backup OK -- with warnings - what warning?

[Dan Langille]

On Fri, Dec 29, 2023, at 12:10 PM, Martin Simmons wrote:
> 9.6.6 certainly displayed them for me, so I suspect a config issue.
>
> The messages would be omitted if !notsaved is in the Messages resource (but
> they would still be counted as "Non-fatal FD errors" which makes it add "with
> warnings" to the status).
>
> Maybe that changed in the client's bacula-fd.conf when you upgraded it?

That's a good idea.

[17:29 r730-01 dvl ~] % sudo ls -l /usr/local/etc/bacula/bacula-fd.conf
-rw-r-  1 root bacula 1497 Feb 25  2023 /usr/local/etc/bacula/bacula-fd.conf

[17:31 r730-01 dvl ~] % sudo md5 /usr/local/etc/bacula/bacula-fd.conf
MD5 (/usr/local/etc/bacula/bacula-fd.conf) = e41a7d835766f563253c0a93418a1c61


No change since February.

Let's look at snapshots taken before Dec 25, the date of the job in question.

[17:32 r730-01 dvl /.zfs/snapshot] % cd autosnap_2023-12-20_00:00:09_daily
[17:32 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % sudo 
md5 usr/local/etc/bacula/bacula-fd.conf 
MD5 (usr/local/etc/bacula/bacula-fd.conf) = e41a7d835766f563253c0a93418a1c61


I'm confident this file has not changed.

Here is it, with some redactions:

[17:33 r730-01 dvl /.zfs/snapshot/autosnap_2023-12-20_00:00:09_daily] % sudo 
cat usr/local/etc/bacula/bacula-fd.conf
Director {
  Name = bacula-dir
  Password = "[redacted]"

  TLS Enable  = yes
  TLS Require = yes
  TLS Verify Peer = yes

  # Allow only the Director to connect
  TLS Allowed CN  = "bacula.example.org"
  TLS CA Certificate File = /usr/local/etc/ssl/MyBigCA.crt

  TLS Certificate = /usr/local/etc/ssl/r730-01.int.unixathome.org.crt
  TLS Key = /usr/local/etc/ssl/r730-01.int.unixathome.org.nopassword.key
}

Director {
  Name = nagios-mon
  Password = "[redacted]"
  Monitor = yes
}

FileDaemon {
  Name= r730-01-fd
  FDAddress   = 10.55.0.141
  FDport  = 9102
  WorkingDirectory= /var/db/bacula
  Pid Directory   = /var/run
  Maximum Concurrent Jobs = 20

  TLS Enable  = yes
  TLS Require = yes

  TLS CA Certificate File = /usr/local/etc/ssl/MyBigCA.crt

  TLS Certificate = /usr/local/etc/ssl/r730-01.int.unixathome.org.crt
  TLS Key = /usr/local/etc/ssl/r730-01.int.unixathome.org.nopassword.key
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = bacula-dir = all, !skipped, !restored
}


>
> __Martin
>
>
>>>>>> On Mon, 25 Dec 2023 07:06:58 -0500, Dan Langille said:
>> 
>> Hello,
>> 
>> This is more for advising others than looking for a fix. 
>> 
>> bacula9 client mentions warnings and does not list them.
>> bacula13 client mentions the warnings.
>> 
>> It turns out, the missing warnings are rather important to know. The ZFS 
>> datasets in question are jailed, and need a different path to the mount 
>> point. Checking the log file, the warnings do not appear there either.
>> 
>> Here are examples of the problem.
>> 
>> The following email is from bacula9-client: 9.6.7_3 on FreeBSD 14. The 
>> subject of this message from Bacula mentions warnings. 
>> 
>> No warnings are supplied.
>> 
>> From: (Bacula) d...@langille.org
>> Subject: Bacula: Backup OK -- with warnings of r730-01-fd Incremental
>> Sender: bac...@bacula.int.unixathome.org
>> To: [redacted]
>> Date: Mon, 25 Dec 2023 03:09:16 + (UTC)
>> 
>> 25-Dec 03:09 bacula-dir JobId 362083: Start Backup JobId 362083, 
>> Job=r730-01_snapshots.2023-12-25_03.05.00_24
>> 25-Dec 03:09 bacula-dir JobId 362083: Connected to Storage 
>> "bacula-sd-04-IncrFile" at bacula-sd-04.int.unixathome.org:9103 with TLS
>> 25-Dec 03:09 bacula-dir JobId 362083: There are no more Jobs associated with 
>> Volume "IncrAuto-04-14761". Marking it purged.
>> 25-Dec 03:09 bacula-dir JobId 362083: All records pruned from Volume 
>> "IncrAuto-04-14761"; marking it "Purged"
>> 25-Dec 03:09 bacula-dir JobId 362083: Recycled volume "IncrAuto-04-14761"
>> 25-Dec 03:09 bacula-dir JobId 362083: Using Device "vDrive-IncrFile-5" to 
>> write.
>> 25-Dec 03:09 bacula-dir JobId 362083: Connected to Client "r730-01-fd" at 
>> r730-01.int.unixathome.org:9102 with TLS
>> 25-Dec 03:09 r730-01-fd JobId 362083: shell command: run ClientBeforeJob 
>> "/usr/local/sbin/snapshots-for-backup.sh create"
>> 25-Dec 03:09 bacula-sd-04 JobId 362083: Recycled volume "IncrAuto-04-14761" 
>> on File device "vDrive-IncrFile-5" (/usr/local/bacula/volumes/IncrFile), all 
>> previous data lost.
>> 25-Dec 03:09 bacula-dir JobId 362083: Max Vo

[Bacula-users] Bacula: Backup OK -- with warnings - what warning?

[Dan Langille]

 volume "IncrAuto-04-14785"
25-Dec 11:53 bacula-dir JobId 362099: Using Device "vDrive-IncrFile-5" to write.
25-Dec 11:53 bacula-dir JobId 362099: Connected to Client "r730-01-fd" at 
r730-01.int.unixathome.org:9102 with TLS
25-Dec 11:53 r730-01-fd JobId 362099: Connected to Storage at 
bacula-sd-04.int.unixathome.org:9103 with TLS
25-Dec 11:53 r730-01-fd JobId 362099: shell command: run ClientBeforeJob 
"/usr/local/sbin/snapshots-for-backup.sh create"
25-Dec 11:53 bacula-sd-04 JobId 362099: Recycled volume "IncrAuto-04-14785" on 
File device "vDrive-IncrFile-5" (/usr/local/bacula/volumes/IncrFile), all 
previous data lost.
25-Dec 11:53 bacula-dir JobId 362099: Max Volume jobs=1 exceeded. Marking 
Volume "IncrAuto-04-14785" as Used.
25-Dec 11:53 r730-01-fd JobId 362099:  Could not stat 
"/var/db/ingress/latest_commits/.zfs/snapshot/snapshot-for-backup": ERR=No such 
file or directory
25-Dec 11:53 r730-01-fd JobId 362099:  Could not stat 
"/var/db/ingress/latest_commits/.zfs/snapshot/snapshot-for-backup": ERR=No such 
file or directory
25-Dec 11:53 r730-01-fd JobId 362099:  Could not stat 
"/var/db/ingress/latest_commits/.zfs/snapshot/snapshot-for-backup": ERR=No such 
file or directory
25-Dec 11:53 r730-01-fd JobId 362099: shell command: run ClientAfterJob 
"/usr/local/sbin/snapshots-for-backup.sh destroy"
25-Dec 11:53 bacula-sd-04 JobId 362099: Elapsed time=00:00:11, Transfer 
rate=446.2 K Bytes/second
25-Dec 11:53 bacula-sd-04 JobId 362099: Sending spooled attrs to the Director. 
Despooling 77,691 bytes ...
25-Dec 11:53 bacula-dir JobId 362099: Bacula bacula-dir 13.0.3 (02May23):
  Build OS:   amd64-portbld-freebsd14.0 freebsd 14.0-RELEASE-p1
  JobId:  362099
  Job:r730-01_snapshots.2023-12-25_11.52.58_26
  Backup Level:   Incremental, since=2023-12-25 03:09:01
  Client: "r730-01-fd" 13.0.3 (02May23) 
amd64-portbld-freebsd14.0,freebsd,14.0-RELEASE-p1
  FileSet:"r730-01 snapshots" 2023-12-25 01:30:49
  Pool:   "IncrFile-04" (From Job IncPool override)
  Catalog:"MyCatalog" (From Client resource)
  Storage:"bacula-sd-04-IncrFile" (From Pool resource)
  Scheduled time: 25-Dec-2023 11:52:57
  Start time: 25-Dec-2023 11:53:01
  End time:   25-Dec-2023 11:53:15
  Elapsed time:   14 secs
  Priority:   10
  FD Files Written:   194
  SD Files Written:   194
  FD Bytes Written:   4,854,123 (4.854 MB)
  SD Bytes Written:   4,909,040 (4.909 MB)
  Rate:   346.7 KB/s
  Software Compression:   None
  Comm Line Compression:  73.1% 3.7:1
  Snapshot/VSS:   no
  Encryption: no
  Accurate:   no
  Volume name(s): IncrAuto-04-14785
  Volume Session Id:  55
  Volume Session Time:1703358455
  Last Volume Bytes:  4,919,905 (4.919 MB)
  Non-fatal FD errors:3
  SD Errors:  0
  FD termination status:  OK
  SD termination status:  OK
  Termination:Backup OK -- with warnings

25-Dec 11:53 bacula-dir JobId 362099: Begin pruning Jobs older than 3 years .
25-Dec 11:53 bacula-dir JobId 362099: No Jobs found to prune.
25-Dec 11:53 bacula-dir JobId 362099: Begin pruning Files.
25-Dec 11:53 bacula-dir JobId 362099: No Files found to prune.
25-Dec 11:53 bacula-dir JobId 362099: End auto prune.

FYI, the Messages directive (with email addresses modified) appears below.

Messages {
  Name = Standard

  mailcommand = "/usr/local/sbin/bsmtp -h cliff.example.org -f \"\(Bacula\) 
%r\" -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/usr/local/sbin/bsmtp -h cliff.example.org -f \"\(Bacula\) 
%r\" -s \"Bacula: Intervention needed for %j\" %r"
  operator = d...@example.org = mount
  mail = d...@example.org = all
  console  = all

  append = "/var/log/bacula/bacula.log" = all, !skipped, !restored

  catalog = all, !skipped, !saved
}

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Beta Release 15.0.0 - packages

[Dan Langille]

On Fri, Nov 24, 2023, at 9:22 AM, Dan Langille wrote:
> On Fri, Nov 3, 2023, at 1:07 PM, Eric Bollengier via Bacula-users wrote:
>> Hello,
>>
>> I have pushed a couple of binaries on bacula.org (ubuntu jammy, debian 
>> bullseye
>> and almalinux 8), you might find them under:
>
>
> Is there any advice, alerts, etc to package builders? I usually base 
> the new package building on the old package building.  A HEADS UP for 
> any differences which might affect packaging is appreciated.

I've created a Bacula 15.0.0 port for FreeBSD and run a simple test job.

If any FreeBSD users want to test it themselves, please see:

https://git.langille.org/dvl/ports/src/branch/master/sysutils

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Beta Release 15.0.0 - packages

[Dan Langille]

On Fri, Nov 3, 2023, at 1:07 PM, Eric Bollengier via Bacula-users wrote:
> Hello,
>
> I have pushed a couple of binaries on bacula.org (ubuntu jammy, debian 
> bullseye
> and almalinux 8), you might find them under:


Is there any advice, alerts, etc to package builders? I usually base the new 
package building on the old package building.  A HEADS UP for any differences 
which might affect packaging is appreciated.

Thank you.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] PostgreSQL update script fails

[Dan Langille]

On Thu, Nov 23, 2023, at 8:14 PM, Dan Langille wrote:

> I hope to do a blog post with more details about the upgrade process 
> including some suggestions:
>
> * copy the scripts to the database server
> * run it as the postgresql user, whatever that happens to be on the OS 
> in question

The blog post:

https://dan.langille.org/2023/11/23/bacula-moving-from-9-x-to-13-x-and-upgrading-the-postgresql-database/


-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] PostgreSQL update script fails

[Dan Langille]

On Thu, Nov 23, 2023, at 8:14 PM, Dan Langille wrote:
> I started an upgrade from Bacula 9.6.7 to Bacula 13.0.1 today and I 
> have two suggestions for fixing bugs. If a patch with a merge request 
> is preferred, I can do that next.


Patches here:

https://github.com/dlangille/bacula-community/commit/0200b1461310b4f5d4cdb89aedaa2ca9c1e1ad77
https://github.com/dlangille/bacula-community/commit/716952b15b71be44f152bfb522666514422acc95

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] PostgreSQL update script fails

[Dan Langille]

I started an upgrade from Bacula 9.6.7 to Bacula 13.0.1 today and I have two 
suggestions for fixing bugs. If a patch with a merge request is preferred, I 
can do that next.

Action item 1:

There is an error on this line. There is more than one such line.

echo "Inserting File data from file.$$.data failed

That should be file1016.data (in this example), not file.$$.data - when an 
error occurs, it misleads the user.


The error I hit is:

+ cat file1017.data
+ /usr/bin/gzip -d
+ psql --set 'ON_ERROR_STOP=1' -d bacula -c $'BEGIN; TRUNCATE File; COPY File 
FROM STDIN; set maintenance_work_mem=\'2000MB\'; CREATE INDEX file_jpfid_idx on 
File (JobId, PathId, Filename text_pattern_ops); ALTER TABLE ONLY File ADD 
CONSTRAINT file_pkey PRIMARY KEY (FileId); COMMIT;'
BEGIN
TRUNCATE TABLE
ERROR:  invalid input syntax for type bigint: "SET"
CONTEXT:  COPY file, line 1, column fileid: "SET"
+ [ 1 -ne 0 ]
+ echo 'Inserting File data from file.26318.data failed.'
Inserting File data from file.26318.data failed.
+ exit 1

Looking at the file in question, I see some interesting stuff:

$ cat file1017.data | /usr/bin/gzip -d | head -20
SET
SET
SET
290658834   47767   129209  442701  next_vol.patch  0   0   Bs jlhS 
IGk B A A COQKg ro EAA I BRYSdD BLhUbc BLk9ZE A A C fSmIlVrj9x4ETUxuH9PAIQ
290658835   50573   129209  237850  part_preface.xml.svn-base   0   
0   Bs n0Gc IEk B Pp Pp CfDWu i EAA E BRYSdQ BHf/U1 BHf/U2 A A C
JebC91WLdIQADU0JDepbkg
290658836   44185   129209  227351  019.jpg 0   0   Bs P0IB IGk B 
Pp Pp /XtI xL1 EAA HA BRYScm BFx9n8 BFx9n8 A A C  G4caSihayh/RmpHvhl+wew
290658837   54991   129209  438580  T137.ithmb  0   0   Bs 34gB 
IHk B Pp Pp Dfgkg Cj9w EAA VA BRYSeJ BJwsMf BLUldS A A C
CpSByTuNV1ZXrVej/29vfg
290658838   44189   129209  227351  023.jpg 0   0   Bs P0IF IGk B 
Pp Pp /X0o y9Q EAA HA BRYScm BFx9n8 BFx9n8 A A C  V0wtD4fr4t4GGYpcZNrkYA
290658839   44092   129209  227351  019--thumb.jpg  0   0   Bs P0Gk 
IGk B Pp Pp /Wl4 EAZ EAA k BRYScm BFx9n4 BFx9n4 A A C   ufFcqL3JsasSXNL8dz2OBg

I have an idea why SET SET SET occurs there.

I believe it relates to this line in the script:

psql --set ON_ERROR_STOP=1 -d ${db_name} $* -c "set work_mem='$WORKMEM';"'set 
enable_mergejoin to off ; set enable_hashjoin to off;

Action item 2:

That SET SET SET is the output of the three SET commands above.  It should be 
suppressed via an option on the psql invocation.  Sorry, I don't know that 
option off-hand.

To move onward with the upgrade, I modified the script.

I added the following to ignore the first three lines of the script.

# we do everything in the same commit to avoid creating WALs on this 
operation
cat file1016.data | $COMP -d | tail -n +4 | psql...

Side note: "cat filename  |"   can be replaced by "< filename"

I restarted the script after removing code related to work already done.

It succeeded.

First job failed though:

24-Nov 01:02 bacula-dir JobId 361289: Fatal error: sql_create.c:894 Fill File 
table Query failed: INSERT INTO File (FileIndex, JobId, PathId, Filename, 
LStat, MD5, DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId, 
batch.Name, batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN Path ON 
(batch.Path = Path.Path) : ERR=ERROR:  permission denied for table file

There seem to be a few tables with incorrect permissions.

Running grant_bacula_privileges fixed that.

I hope to do a blog post with more details about the upgrade process including 
some suggestions:

* copy the scripts to the database server
* run it as the postgresql user, whatever that happens to be on the OS in 
question


Full output at 
https://gist.github.com/dlangille/d959a48b26843e97e023c9ed5389b7a2


-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] HOME for client scripts

[Dan Langille]

On Thu, Nov 23, 2023, at 10:57 AM, Martin Simmons wrote:
> I've just tested it with the Bacula 15 Beta on FreeBSD 12.4 and found that the
> value of HOME in the ClientRunBeforeJob script comes from when bacula-fd was
> started and Bacula doesn't change it.
>
> How are you starting Bacula 9.6.7?

Lack of upgrade. :)

>  Note that FreeBSD's system startup scripts
> such as /etc/rc and /usr/sbin/service explicitly set HOME=/ so that might be
> the cause of the difference and will also happen if Bacula 13.x is started
> that way.

I didn't know that. There it is right there:

[16:27 empty dan ~] % grep -i home /usr/sbin/service
exec env -i -L -/daemon HOME=/ 
PATH=/sbin:/bin:/usr/sbin:/usr/bin "$dir/$script" "$@"

> FWIW, the make_catalog_backup.pl sets HOME=$wd while running pg_dump.  Maybe
> you need to do something similar?

I have tried that while debugging an issue.

The problem came to light when I upgraded a FreeBSD host from a postgresql 12 
to a postgresql 16 client. The pg_dump stopped working because of a 
password-related issue.  A postgresql 12-14 client worked. If installing 
postgresql15 or 16-client, pg_dump was failing.

Eventually, it was tracked down to a PostgreSQL change in 15.x with respect to 
determination of HOME.  Details at 
https://dan.langille.org/2023/11/22/i-figured-out-why-pg_dump-was-failing-with-postgresql-15-16/

In short, PostgreSQL once checked /etc/passwd for HOME, now it checks and uses 
$HOME if set. Hence the failure on FreeBSD with a HOME of / (/root/.pgpass) was 
not being picked up.

Both setting HOME and PGPASSFILE were valid solution for the pg_dump problem.

Thank you.

>>>>>> On Thu, 23 Nov 2023 09:02:36 -0500, Dan Langille said:
>> 
>> Hello,
>> 
>> One of the features of a is a script. I frequently use ClientRunBeforeJob to 
>> invoke pg_dump - I'm sure others may do similar.
>> 
>> For Bacula 9.6.7, when in that script, the value for $HOME is /
>> 
>> The UID is 0, i.e. root.  On FreeBSD, root's home directory is /root.
>> 
>> I'm not sure why that differs.  I've been told that Bacula 13.x does the 
>> right thing and HOME is /root - I was looking through the commits trying to 
>> find something which fixed this. I failed. I also checked the release notes 
>> and GitLab issues; no mention.
>> 
>> Does anyone recall this change?
>> 
>> To aid in tracking down this issue, could you add "echo $HOME" to your 
>> script. Is it / ?  Regardless, what version are you running and what HOME 
>> directory is reported?
>> 
>> NOTE:
>> 
>> - I'm not asking for bug fix
>> - I'm looking for a commit which made a deliberate change to the behavior
>> - Knowing the history as to why it changed might be useful
>> 
>> Thank you.
>> 
>> -- 
>>   Dan Langille
>>   d...@langille.org
>> 
>> 
>> ___
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] HOME for client scripts

[Dan Langille]

Hello,

One of the features of a is a script. I frequently use ClientRunBeforeJob to 
invoke pg_dump - I'm sure others may do similar.

For Bacula 9.6.7, when in that script, the value for $HOME is /

The UID is 0, i.e. root.  On FreeBSD, root's home directory is /root.

I'm not sure why that differs.  I've been told that Bacula 13.x does the right 
thing and HOME is /root - I was looking through the commits trying to find 
something which fixed this. I failed. I also checked the release notes and 
GitLab issues; no mention.

Does anyone recall this change?

To aid in tracking down this issue, could you add "echo $HOME" to your script. 
Is it / ?  Regardless, what version are you running and what HOME directory is 
reported?

NOTE:

- I'm not asking for bug fix
- I'm looking for a commit which made a deliberate change to the behavior
- Knowing the history as to why it changed might be useful

Thank you.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Running backup job without file records in catalog

[Dan Langille]

On Thu, Sep 28, 2023, at 6:17 AM, Yateen Shaligram Bhagat (Nokia) wrote:
> Hi All,
>  
> I would like to know if we can define a backup job (real or virtual) that 
> DOES NOT put the file records in the catalog at all.
> 

Why?

Whenever an unusual question arises, it helps to explain the background, the 
goal, the objective, etc.

Thanks.

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Adding missing foreign keys to PostgreSQL - avoiding dbcheck

[Dan Langille]

On Tue, Oct 3, 2023, at 1:45 PM, Eric Bollengier via Bacula-users wrote:
> Hello Dan,
>
> On 9/30/23 15:54, Dan Langille wrote:
>> Hello,
>> 
>> The Bacula PostgreSQL schema is missing several foreign keys (FK). Foreign 
>> keys are not a new database concept; they've been around for decades. They 
>> are reliable and robust.
>
> Yes, it's robust, but Bacula is doing mass insertion, and here, the cost is
> bigger than the benefits IMHO. Of course, if someone can run tests on a 
> multi-billion record database and show us that the cost is around few %, we 
> can
> consider them.

Here's hoping someone with that database can step up.

> Basically, each time you insert something, you need to query the other tables 
> to
> check the existence or not of records. The checks for the existence of the
> records is already done at the bacula level one time per job, no real need to 
> do
> it one time per insertion/update/delete.

Is that one-time-per-job check done via a SELECT on the database? I should 
explain analyze that one too and see if we can optimize that.

>> Wednesday, I started a dbcheck on a Bacula database. Granted, that database 
>> is 19 years old and this is the first time I've run dbcheck (as far as I 
>> know). That dbcheck is still going. FYI, the dump to disk is about 140GB; 
>> lots of cruft removal.
>> 
>> When PostgreSQL was first added to Bacula, there was resistance to FK, and I 
>> did not pursue the issue. Thus, it persists to this day. I hope to change 
>> that.
>
> I don't think they are necessary, we use transaction in the cleanup now days, 
> so
> orphan records should not exist anymore.
>
>> I would like to take that development work back up (pun intended), and start 
>> adding foreign keys back into Bacula, at least for PostgresQL. That might 
>> remove the need for dbcheck (again, at least for Bacula on PostgreSQL).
>
> dbcheck is needed from time to time to purge the Path table mostly, other 
> tables
> should be ok with a recent version. Not sure the algorithms in dbcheck are
> always the best ones to find a remove records.
>
>> For example, one index I have been using this index for years. I find it 
>> referenced[1] in the the 5.x documentation, but it is not part of the 
>> catalog creation.
>> 
>>  "file_jobid_idx" btree (jobid)
>> 
>> This index vastly improves the construction of the files, often going from 
>> hours to seconds. I don't recall when that index was added here, but 
>> building trees has never been an issue here.
>
> This index is great, but it uses GB of space and it has a cost during the 
> insertion of the attributes. PostgreSQL can use the composed index on JobId,
> PathId instead to locate records, so you may add it back if you want, but it's
> not mandatory.

How many GB? Do we know? We can easily find out too. 

bacula=# select pg_table_size('file_jobid_idx');
 pg_table_size 
---
   37937438720
(1 row)

That's about 40GB.  I'll check the explain analyse and see how things go and 
come up with cost. Right now, we have no data.

Some might say: if the cost > 0 it's not worth it. However, it is worth it, 
IMO. Databases are much better at enforcing relational integrity than code is. 
It's what they do.

Thank you

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Adding missing foreign keys to PostgreSQL - avoiding dbcheck

[Dan Langille]

On Tue, Oct 3, 2023, at 1:31 PM, Martin Simmons wrote:
>>>>>> On Tue, 03 Oct 2023 09:35:27 -0400, Dan Langille said:
>> 
>> On Mon, Oct 2, 2023, at 11:17 AM, Martin Simmons wrote:
>> >>>>>> On Sat, 30 Sep 2023 09:54:51 -0400, Dan Langille said:
>> >> 
>> >> Hello,
>> >> 
>> >> The Bacula PostgreSQL schema is missing several foreign keys (FK). 
>> >> Foreign keys are not a new database concept; they've been around for 
>> >> decades. They are reliable and robust.
>> >> 
>> >> Wednesday, I started a dbcheck on a Bacula database. Granted, that 
>> >> database is 19 years old and this is the first time I've run dbcheck (as 
>> >> far as I know). That dbcheck is still going. FYI, the dump to disk is 
>> >> about 140GB; lots of cruft removal. 
>> >> 
>> >> When PostgreSQL was first added to Bacula, there was resistance to FK, 
>> >> and I did not pursue the issue. Thus, it persists to this day. I hope to 
>> >> change that.
>> >> 
>> >> I would like to take that development work back up (pun intended), and 
>> >> start adding foreign keys back into Bacula, at least for PostgresQL. That 
>> >> might remove the need for dbcheck (again, at least for Bacula on 
>> >> PostgreSQL).
>> >
>> > What is the performance cost of foreign keys?
>> 
>> I'm replying so it does not appear as if I am ignoring you. Short answer: I 
>> don't know. Yet. That is the purpose of my project.
>
> OK, fair enough.
>
>> 
>> I can't answer that in a way which would sound satisfying. I have not 
>> started the work. I have only my personal experience - My backups seem fast 
>> enough to me.
>> 
>> It is easy enough to test. There are several ways to optimize foreign keys 
>> usage.
>> 
>> >> For example, one index I have been using this index for years. I find it 
>> >> referenced[1] in the the 5.x documentation, but it is not part of the 
>> >> catalog creation.
>> >> 
>> >> "file_jobid_idx" btree (jobid)
>> >> 
>> >> This index vastly improves the construction of the files, often going 
>> >> from hours to seconds. I don't recall when that index was added here, but 
>> >> building trees has never been an issue here.
>> >
>> > It was removed in this change:
>> >
>> > commit 740704c9c66d0b049a7cd548ac1204ef1aaf7356
>> > Author: Eric Bollengier 
>> > Date:   Mon May 11 17:11:40 2020 +0200
>> >
>> > BEE Backport bacula/src/cats/make_postgresql_tables.in
>> > 
>> > Does PostgreSQL use file_jpfid_idx for the query if you don't have
>> > file_jobid_idx?
>> 
>> Testing will show that. I am not at that stage yet. I will be examining the 
>> queries used and running them through the PostgreSQL 'EXPLAIN ANALYZE' 
>> process. I'll post results at https://explain.depesz.com so progress can be 
>> seen and compared. Others will be able to run the same non-destructive 
>> commands on their own databases for comparison.
>> 
>> I just check my database and it has these row counts:
>> 
>> filename:22,232,549
>> file: 1,208,708,804
>> path: 8,340,411
>> job: 97,139
>> jobmedia:   331,379
>> media:   12,848
>
> Is this an old version?  The filename table shouldn't exist now.  The new
> catalog format could make a big difference to the queries (and foreign key
> performance).

Yes, it's Bacula 9... for shame...
-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Adding missing foreign keys to PostgreSQL - avoiding dbcheck

[Dan Langille]

On Mon, Oct 2, 2023, at 11:17 AM, Martin Simmons wrote:
>>>>>> On Sat, 30 Sep 2023 09:54:51 -0400, Dan Langille said:
>> 
>> Hello,
>> 
>> The Bacula PostgreSQL schema is missing several foreign keys (FK). Foreign 
>> keys are not a new database concept; they've been around for decades. They 
>> are reliable and robust.
>> 
>> Wednesday, I started a dbcheck on a Bacula database. Granted, that database 
>> is 19 years old and this is the first time I've run dbcheck (as far as I 
>> know). That dbcheck is still going. FYI, the dump to disk is about 140GB; 
>> lots of cruft removal. 
>> 
>> When PostgreSQL was first added to Bacula, there was resistance to FK, and I 
>> did not pursue the issue. Thus, it persists to this day. I hope to change 
>> that.
>> 
>> I would like to take that development work back up (pun intended), and start 
>> adding foreign keys back into Bacula, at least for PostgresQL. That might 
>> remove the need for dbcheck (again, at least for Bacula on PostgreSQL).
>
> What is the performance cost of foreign keys?

I'm replying so it does not appear as if I am ignoring you. Short answer: I 
don't know. Yet. That is the purpose of my project.

I can't answer that in a way which would sound satisfying. I have not started 
the work. I have only my personal experience - My backups seem fast enough to 
me.

It is easy enough to test. There are several ways to optimize foreign keys 
usage.

>> For example, one index I have been using this index for years. I find it 
>> referenced[1] in the the 5.x documentation, but it is not part of the 
>> catalog creation.
>> 
>> "file_jobid_idx" btree (jobid)
>> 
>> This index vastly improves the construction of the files, often going from 
>> hours to seconds. I don't recall when that index was added here, but 
>> building trees has never been an issue here.
>
> It was removed in this change:
>
> commit 740704c9c66d0b049a7cd548ac1204ef1aaf7356
> Author: Eric Bollengier 
> Date:   Mon May 11 17:11:40 2020 +0200
>
> BEE Backport bacula/src/cats/make_postgresql_tables.in
> 
> Does PostgreSQL use file_jpfid_idx for the query if you don't have
> file_jobid_idx?

Testing will show that. I am not at that stage yet. I will be examining the 
queries used and running them through the PostgreSQL 'EXPLAIN ANALYZE' process. 
I'll post results at https://explain.depesz.com so progress can be seen and 
compared. Others will be able to run the same non-destructive commands on their 
own databases for comparison.

I just check my database and it has these row counts:

filename:22,232,549
file: 1,208,708,804
path: 8,340,411
job: 97,139
jobmedia:   331,379
media:   12,848

If someone feels this database is not representative of their use case, as 
mentioned above, they will be able to run the query tests on their own 
databases and provide feedback.

Hope this helps.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Adding missing foreign keys to PostgreSQL - avoiding dbcheck

[Dan Langille]

Hello,

The Bacula PostgreSQL schema is missing several foreign keys (FK). Foreign keys 
are not a new database concept; they've been around for decades. They are 
reliable and robust.

Wednesday, I started a dbcheck on a Bacula database. Granted, that database is 
19 years old and this is the first time I've run dbcheck (as far as I know). 
That dbcheck is still going. FYI, the dump to disk is about 140GB; lots of 
cruft removal. 

When PostgreSQL was first added to Bacula, there was resistance to FK, and I 
did not pursue the issue. Thus, it persists to this day. I hope to change that.

I would like to take that development work back up (pun intended), and start 
adding foreign keys back into Bacula, at least for PostgresQL. That might 
remove the need for dbcheck (again, at least for Bacula on PostgreSQL).

For example, one index I have been using this index for years. I find it 
referenced[1] in the the 5.x documentation, but it is not part of the catalog 
creation.

"file_jobid_idx" btree (jobid)

This index vastly improves the construction of the files, often going from 
hours to seconds. I don't recall when that index was added here, but building 
trees has never been an issue here.

I have no timeline for this work, but just posting the intent often gives me 
the incentive to get started.

1 - https://www.bacula.org/5.2.x-manuals/en/main/main/Catalog_Maintenance.html

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] TLS using certs with X509v3 extensions

[Dan Langille]

If anyone is using X509v3 extensions with copy jobs, I'm keenly interested in 
the certs you are using. See below.

On Thu, Sep 14, 2023, at 2:39 PM, Dan Langille wrote:
> On Thu, Sep 14, 2023, at 2:33 PM, Martin Simmons wrote:
>>>>>>> On Tue, 12 Sep 2023 08:41:42 -0400, Dan Langille said:
>>> 
>>> >  
>>> >> 
>>> >> I ask because yesterday I started running some copy jobs. The cert used 
>>> >> by bacula-sd was acceptable for receiving backups. It was not acceptable 
>>> >> for copy jobs.
>>> >> 
>>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Error: openssl.c:68 Connect 
>>> >> failure: ERR=error:1417C086:SSL 
>>> >> routines:tls_process_client_certificate:certificate verify failed
>>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: bnet.c:75 TLS 
>>> >> Negotiation failed.
>>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: TLS negotiation 
>>> >> failed with FD at "10.55.0.7:27230"
>>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: Incorrect 
>>> >> authorization key from File daemon at client rejected.
>>> >> For help, please see: 
>>> >> http://www.bacula.org/rel-manual/en/problems/Bacula_Frequently_Asked_Que.html
>>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Security Alert: Unable to 
>>> >> authenticate File daemon
>>> > 
>>> > I wonder if your SD connects to itself here, and fails to validate 
>>> > itself? The log above does mention an FD at 10.55.0.7. Does that FD 
>>> > component have a certificate? maybe there's mis-match with the CN of that 
>>> > certificate and the FDAddress directive in the bacula-fd.conf file?
>>> 
>>> There is no bacula-fd at 10.55.0.7 - it is not running and not configured. 
>>> It is bacula-sd only at that IP address.
>>> 
>>> Yes, bacula-sd-04 is at  10.55.0.7 - I don't know why FD is mentioned in 
>>> the error.
>>> 
>>> From the docs 
>>> (https://bacula.org/13.0.x-manuals/en/main/Migration_Copy.html): 
>>> 
>>> The Copy and the Migration jobs run without using the File daemon by 
>>> copying the data from the old backup Volume to a different Volume in a 
>>> different Pool
>>> 
>>> My reading of that: an FD should not be involved here.
>>
>> My guess is that Copy and Migration jobs work with the reading SD pretending
>> to be an FD to send data to the writing SD.
>>
>> __Martin
>
> Tests this afternoon have confirmed that. I’m still figuring this out. 
> I might resume testing in the next few days. 

It seems the only problem is copy/migrations jobs. In this case, bacula-sd is 
sending to bacula-sd and I have been unable to configuration a cert with X509v3 
which is accepted for this task.  The errors I get are below.

These certs are good for backups, not good for copy/migration (I have tested 
only copy, but I'm sure migration will have the same problem).

If I change the certificate, and *only* the certificate, to not include X509v3 
extensions, this error does not occur.

18-Sep 21:08 bacula-dir JobId 359528: Warning: FileSet MD5 digest not found.
18-Sep 21:08 bacula-dir JobId 359528: The following 1 JobId was chosen to be 
copied: 359391
18-Sep 21:08 bacula-dir JobId 359528: Copying using JobId=359391 
Job=r730-03_basic_testing.2023-09-15_12.57.14_14
18-Sep 21:08 bacula-dir JobId 359528: Start Copying JobId 359528, 
Job=CopyToSD04-testing-deleteme.2023-09-18_21.08.04_42
18-Sep 21:08 bacula-dir JobId 359528: Using Device "vDrive-FullFile-0" to read.
18-Sep 21:08 bacula-dir JobId 359529: Using Device "vDrive-FullFile-0" to write.
18-Sep 21:08 bacula-sd-01-sd JobId 359528: Error: openssl.c:68 Connect failure: 
ERR=error:1417C086:SSL routines:tls_process_client_certificate:certificate 
verify failed
18-Sep 21:08 bacula-sd-01-sd JobId 359528: Fatal error: bnet.c:75 TLS 
Negotiation failed.
18-Sep 21:08 bacula-sd-01-sd JobId 359528: Fatal error: TLS negotiation failed 
with FD at "10.55.0.7:61827"
18-Sep 21:08 bacula-sd-01-sd JobId 359528: Fatal error: Incorrect authorization 
key from File daemon at client rejected.
For help, please see: 
http://www.bacula.org/rel-manual/en/problems/Bacula_Frequently_Asked_Que.html
18-Sep 21:08 bacula-sd-01-sd JobId 359528: Security Alert: Unable to 
authenticate File daemon
18-Sep 21:08 bacula-dir JobId 359529: Fatal error: Bad response to Storage 
command: wanted 2000 OK storage
, got 2902 Bad storage

18-Sep 21:08 bacula-dir JobId 359529: Fatal error: mac.c:301 Response failure: 
storeddr=bacula-sd-01.int.unixathome.org:

Re: [Bacula-users] TLS using certs with X509v3 extensions

[Dan Langille]

On Thu, Sep 14, 2023, at 2:33 PM, Martin Simmons wrote:
>>>>>> On Tue, 12 Sep 2023 08:41:42 -0400, Dan Langille said:
>> 
>> >  
>> >> 
>> >> I ask because yesterday I started running some copy jobs. The cert used 
>> >> by bacula-sd was acceptable for receiving backups. It was not acceptable 
>> >> for copy jobs.
>> >> 
>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Error: openssl.c:68 Connect 
>> >> failure: ERR=error:1417C086:SSL 
>> >> routines:tls_process_client_certificate:certificate verify failed
>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: bnet.c:75 TLS 
>> >> Negotiation failed.
>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: TLS negotiation 
>> >> failed with FD at "10.55.0.7:27230"
>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: Incorrect 
>> >> authorization key from File daemon at client rejected.
>> >> For help, please see: 
>> >> http://www.bacula.org/rel-manual/en/problems/Bacula_Frequently_Asked_Que.html
>> >> 09-Sep 10:19 bacula-sd-04 JobId 358322: Security Alert: Unable to 
>> >> authenticate File daemon
>> > 
>> > I wonder if your SD connects to itself here, and fails to validate itself? 
>> > The log above does mention an FD at 10.55.0.7. Does that FD component have 
>> > a certificate? maybe there's mis-match with the CN of that certificate and 
>> > the FDAddress directive in the bacula-fd.conf file?
>> 
>> There is no bacula-fd at 10.55.0.7 - it is not running and not configured. 
>> It is bacula-sd only at that IP address.
>> 
>> Yes, bacula-sd-04 is at  10.55.0.7 - I don't know why FD is mentioned in the 
>> error.
>> 
>> From the docs 
>> (https://bacula.org/13.0.x-manuals/en/main/Migration_Copy.html): 
>> 
>> The Copy and the Migration jobs run without using the File daemon by copying 
>> the data from the old backup Volume to a different Volume in a different Pool
>> 
>> My reading of that: an FD should not be involved here.
>
> My guess is that Copy and Migration jobs work with the reading SD pretending
> to be an FD to send data to the writing SD.
>
> __Martin

Tests this afternoon have confirmed that. I’m still figuring this out. I might 
resume testing in the next few days. 

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] TLS using certs with X509v3 extensions

[Dan Langille]

On Tue, Sep 12, 2023, at 6:23 AM, Vanush "Misha" Paturyan wrote:
> On Mon, 11 Sept 2023 at 20:19, Dan Langille  wrote:
>> 
>> Yes, I think it's SSL erroring out, I agree with your theory.
>> 
>> Which means: what Key Usage needs to be included for each of:
>> 
>> * bacula-fd
>> * bacula-sd
>> * bacula-dir
>> 
>> Thank you for sharing your details.  Is this cert used with bacula-sd or 
>> bacula-fd?
> 
> That was a certificate from bacula-fd. bacula-sd certificate has the same 
> extensions (Key Usage: Digital Signature, Non Repudiation, Key Encipherment, 
> Data Encipherment). Its CN matches the value of SDAddress in the `Storage` 
> section of bacula-sd.conf file. For completeness, the TLS related entries in 
> that file are:
> TLS Enable = yes
> TLS Require = no
> TLS Verify Peer = yes
> TLS CA Certificate = 
> TLS Certificate = 
> TLS Key =  

We differ only in TLS Require

One thing I just realized: There are two clauses in configuration files, each 
of which can take a certificate. Until now, I never considered that each one 
could be a different cert; one client, one server.

Let me us my bacula-sd as an example:

Storage {
Name = "bacula-sd-04"

TLS Certificate = server key goes here
}

Director {
Name = "bacula-dir"
TLS Certificate = client cert goes here
}

Perhaps that is what I need to investigate.  Also, I could look into a a 
dual-use certificate: it's
possible for the EKU to assert both "Web Client" and "Web Server"

>  
>> 
>> I ask because yesterday I started running some copy jobs. The cert used by 
>> bacula-sd was acceptable for receiving backups. It was not acceptable for 
>> copy jobs.
>> 
>> 09-Sep 10:19 bacula-sd-04 JobId 358322: Error: openssl.c:68 Connect failure: 
>> ERR=error:1417C086:SSL routines:tls_process_client_certificate:certificate 
>> verify failed
>> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: bnet.c:75 TLS 
>> Negotiation failed.
>> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: TLS negotiation failed 
>> with FD at "10.55.0.7:27230"
>> 09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: Incorrect authorization 
>> key from File daemon at client rejected.
>> For help, please see: 
>> http://www.bacula.org/rel-manual/en/problems/Bacula_Frequently_Asked_Que.html
>> 09-Sep 10:19 bacula-sd-04 JobId 358322: Security Alert: Unable to 
>> authenticate File daemon
> 
> I wonder if your SD connects to itself here, and fails to validate itself? 
> The log above does mention an FD at 10.55.0.7. Does that FD component have a 
> certificate? maybe there's mis-match with the CN of that certificate and the 
> FDAddress directive in the bacula-fd.conf file?

There is no bacula-fd at 10.55.0.7 - it is not running and not configured. It 
is bacula-sd only at that IP address.

Yes, bacula-sd-04 is at  10.55.0.7 - I don't know why FD is mentioned in the 
error.

>From the docs (https://bacula.org/13.0.x-manuals/en/main/Migration_Copy.html): 

The Copy and the Migration jobs run without using the File daemon by copying 
the data from the old backup Volume to a different Volume in a different Pool

My reading of that: an FD should not be involved here.

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] TLS using certs with X509v3 extensions

[Dan Langille]

> On Sep 11, 2023, at 12:14 PM, Vanush Misha Paturyan  wrote:
> 
> Hello Dan,
> 
> On Sat, 9 Sept 2023 at 12:39, Dan Langille  <mailto:d...@langille.org>> wrote:
>> Hello,
>> 
>> Is anyone using self-signed certificates using X509v3 extensions?
>> 
>> To be clear: I am not trying to make use of X509v3 extensions for any 
>> particular purpose - A recent upgrade to the tool I am using recently 
>> started X509v3 extensions
>> 
> 
> Our system works with sellf-signed certificates with X509v3 extensions. 
> here's what the extensions look like on our setup:
> 
> X509v3 extensions:
> X509v3 Subject Key Identifier: 
> 5E:67:4E:42:8B:F3:3B:8E:F4:C4:BE:B9:29:B3:5E:41:DC:DE:12:81
> X509v3 Authority Key Identifier: 
> 
> keyid:88:38:87:5E:B1:E0:FF:59:98:BB:0F:2F:8B:55:F5:E0:85:E1:82:9D
> DirName:/C=IE/ST=Co Kildare/L=Maynooth/O=Maynooth 
> University/OU=Computer Science Department/CN=CS Dept Internal 
> CA/emailAddress=supp...@cs.nuim.ie <mailto:supp...@cs.nuim.ie>
> serial:CC:A9:72:5F:96:CF:3B:53
> 
> X509v3 Basic Constraints: 
> CA:FALSE
> X509v3 Key Usage: 
> Digital Signature, Non Repudiation, Key Encipherment, Data 
> Encipherment
> X509v3 CRL Distribution Points: 
> 
> Full Name:
>   URI:http://www.cs.nuim.ie/nuimcs.crl
> 
> Comparing to your example, I don't have the "Extended Key Usage" part, and I 
> don't remember why is there Subject Key Identifier and Authority Key 
> Identifier extensions: something wasn't working without them, but I can't 
> find my notes from when I was setting up our internal "CA", so have no idea 
> if it was related to Bacula or not.
> 
> But I have a feeling it is not bacula that is failing: this 
> "ERR=error:1416F086:SSL routines:tls_process_server_certificate:certificate 
> verify failed" feels like it is coming from the SSL library?
> 

Yes, I think it's SSL erroring out, I agree with your theory.

Which means: what Key Usage needs to be included for each of:

* bacula-fd
* bacula-sd
* bacula-dir

Thank you for sharing your details.  Is this cert used with bacula-sd or 
bacula-fd?

I ask because yesterday I started running some copy jobs. The cert used by 
bacula-sd was acceptable for receiving backups. It was not acceptable for copy 
jobs.

09-Sep 10:19 bacula-sd-04 JobId 358322: Error: openssl.c:68 Connect failure: 
ERR=error:1417C086:SSL routines:tls_process_client_certificate:certificate 
verify failed
09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: bnet.c:75 TLS Negotiation 
failed.
09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: TLS negotiation failed 
with FD at "10.55.0.7:27230"
09-Sep 10:19 bacula-sd-04 JobId 358322: Fatal error: Incorrect authorization 
key from File daemon at client rejected.
For help, please see: 
http://www.bacula.org/rel-manual/en/problems/Bacula_Frequently_Asked_Que.html
09-Sep 10:19 bacula-sd-04 JobId 358322: Security Alert: Unable to authenticate 
File daemon

I've been using 10.55.0.7 (bacula-sd-04.int.unixathome.org) – for backups for 
some time. This was the first copy job.

* it is not the password - I changed it, got a different error
* I change the cert to the type used on a bacula-sd (ie. client cert), that 
worked fine

I'm sure I need to change the extensions I am using.

— 
Dan Langille
http://langille <http://langille/>.org/





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Using LibreSSL?

[Dan Langille]

Coming back to this years later.

It seems that Bacula 9, 11, and 13 all build with LibreSSL now.

see https://marc.info/?l=bacula-users=152256798530318=2

On Sun, Apr 1, 2018, at 3:31 AM, Kern Sibbald wrote:
> Hello Dan,
>
> Is it possible to implement Martin's suggestion?
>
> Best regards,
>
> Kern
>
>
> On 03/09/2018 06:05 PM, Dan Langille wrote:
>>> On Feb 9, 2018, at 1:34 PM, Martin Simmons  wrote:
>>>
>>>>>>>> On Wed, 7 Feb 2018 20:33:49 -0500, Dan Langille said:
>>>>> On Nov 30, 2017, at 11:41 AM, Martin Simmons  wrote:
>>>>>
>>>>> It looks like LibreSSL defines OPENSSL_VERSION_NUMBER as 0x2000L, 
>>>>> i.e. as
>>>>> if it was OpenSSL 2.0.  Bacula uses this variable to detect OpenSSL >= 
>>>>> 1.1,
>>>>> which causes it to compile the 1.1 code when using LibreSSL, even though
>>>>> LibreSSL only claims to provide the API from OpenSSL 1.0.
>>>>>
>>>>> Bacula probably needs to detect LibreSSL (e.g. look for
>>>>> LIBRESSL_VERSION_NUMBER) and treat that as OpenSSL 1.0 regardless of
>>>>> OPENSSL_VERSION_NUMBER.
>>>> This seems to fit closely with a patch provided to FreeBSD: 
>>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223994
>>>>
>>>> I could file a bug and pass this directly to the Bacula project, and in 
>>>> the meantime, patch the FreeBSD port.
>>>>
>>>> Both will take time and I'm preoccupied with conferences just now.
>>> That patch looks incomplete though (there are a few other uses of
>>> OPENSSL_VERSION_NUMBER).
>> New patches have been uploaded to 
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223994
>>
>> Does that look better?
>>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] TLS using certs with X509v3 extensions

[Dan Langille]

Hello,

Is anyone using self-signed certificates using X509v3 extensions?

To be clear: I am not trying to make use of X509v3 extensions for any 
particular purpose - A recent upgrade to the tool I am using recently started 
X509v3 extensions

I ask because so far I have been unable to get TLS working when using X509v3 
extensions on a certificate used by bacula-fd

If I use a certificate with X509v3 extensions bacula-fd, I get these types of 
messages:

08-Sep 12:47 bacula-dir JobId 358290: Error: tls.c:96 Error with certificate at 
depth: 0, issuer = /C=US/ST=PA/L=Media/O=BSD Cabal Headquarters/CN=BSD Cabal 
Headquarters/emailAddress=d...@langille.org 
<mailto:Headquarters/emailAddress=d...@langille.org>, subject = 
/C=US/ST=PA/O=BSD Cabal 
Headquarters/CN=r730-03.int.unixathome.org/emailAddress=d...@langille.org 
<mailto:Headquarters/CN=r730-03.int.unixathome.org/emailAddress=d...@langille.org>,
 ERR=26:unsupported certificate purpose
08-Sep 12:47 bacula-dir JobId 358290: Error: openssl.c:68 Connect failure: 
ERR=error:1416F086:SSL routines:tls_process_server_certificate:certificate 
verify failed
08-Sep 12:47 bacula-dir JobId 358290: Fatal error: TLS negotiation failed with 
FD at "r730-03.int.unixathome.org:9102 
<http://r730-03.int.unixathome.org:9102/>".
08-Sep 12:47 bacula-dir JobId 358290: Fatal error: bsock.c:520 Packet 
size=386073346 too big from "Client: r730-03-fd:r730-03.int.unixathome.org:9102 
<http://r730-03.int.unixathome.org:9102/>". Maximum permitted 100. 
Terminating connection.

If I move back to certificate without X509v3 extensions, the backups succeed.

At first, I thought "unsupported certificate purpose" meant client versus 
server type certs, but no that was not it. That brought in a new type of error. 
 See https://dan.langille.org/2023/09/09/getting-the-right-type-of-certificate/ 
<https://dan.langille.org/2023/09/09/getting-the-right-type-of-certificate/>

What X509v3 extensions you might ask? These.

X509v3 extensions:
X509v3 Basic Constraints: 
CA:FALSE
X509v3 Key Usage: 
Digital Signature, Non Repudiation, Key Encipherment, Key 
Agreement
X509v3 Extended Key Usage: 
TLS Web Client Authentication
X509v3 CRL Distribution Points: 

Full Name:
      URI:http://CRL_URI <http://crl_uri/>
Ideas welcome.

-- 
Dan Langille
d...@langille.org <mailto:d...@langille.org>


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Copying jobs from one SD to another

[Dan Langille]

Hello,

I've started work on migrating backups from one SD with 45TB of backups (of 
which 18TB are just Catalog).

My goals:

* copy over the latest backup for each job - some may be old for jobs no longer 
run
* copy over the past 12 months of full backups.

I'm doing this via SQL query. There will be overlap between the above two 
goals, however, the query caters for that.

Details at: 
https://dan.langille.org/2023/08/20/bacula-copying-the-latest-jobs-over-from-one-sd-to-another/

I've been estimating space required, and have some ideas to reduce the number 
of Catalog backups I keep on hand.

At 160GB for each catalog backup, it may be time for me to run dbcheck in batch 
mode.

— 
Dan Langille
http://langille.org/







___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] .bsr file not updated after backup

[Dan Langille]

If files were backed up, should the .bsr file be updated?

[bacula dan /usr/local/bacula/bsr] % ls -l *mydev*
-rw-r-  1 bacula  bacula  5948 2023.03.03 03:05 mydev-fd_mydev_basic.bsr
-rw-r-  1 bacula  bacula  3844 2023.03.02 03:05 mydev-fd_mydev_home_dir.bsr

The second file was not updated today.

Yet, a job ran today and backed up a file:

*llist files jobid=352170
+---+
| filename  |
+---+
| /usr/home/dan/.touch_a_file_to_force_backupes |
+---+
   jobid: 352,170
 job: mydev_home_dir.2023-03-03_03.05.00_33
name: mydev home dir
 purgedfiles: 0
type: B
   level: I
clientid: 52
  clientname: mydev-fd
   jobstatus: T
   schedtime: 2023-03-03 03:05:00
   starttime: 2023-03-03 03:05:08
 endtime: 2023-03-03 03:05:09
 realendtime: 2023-03-03 03:05:09
jobtdate: 1,677,812,709
volsessionid: 460
  volsessiontime: 1,676,583,692
jobfiles: 1
jobbytes: 0
   readbytes: 0
   joberrors: 0
 jobmissingfiles: 0
  poolid: 24
poolname: IncrFile
  priorjobid: 0
   filesetid: 294
 fileset: mydev home dir
 hasbase: 0
hascache: 0
 comment: 

The job:

Job {
  Name= "mydev home dir"
  JobDefs = "DefaultJob"
  Client  = mydev-fd 
  FileSet = "mydev home dir"
}

The file set for that job:


FileSet {
  Name = "mydev home dir"
  Include { 
Options {
  signature=MD5
verify=pnugsmcs5
} 
Exclude Dir Containing = .NOBACKUP

File = /usr/home
  }
  Exclude {
File = *~
  }
}

Job defs:

JobDefs {
  Name= "DefaultJob"
  Type= Backup
  Level   = Incremental
  Schedule= "WeeklyCycle"
  Storage = CreyFile
  Messages= Standard

  Write Bootstrap = "/usr/local/bacula/bsr/%c_%n.bsr"

  Pool= FullFile  # required parameter for all Jobs

  Full Backup Pool = FullFile
  Differential Backup Pool = DiffFile
  Incremental  Backup Pool = IncrFile

  Priority= 10

  # don't spool date when backing up to disk
  Spool Data  = no
  Spool Attributes = yes

  PreferMountedVolumes = no
}






-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula 13 & PostgreSQL 15 ?

[Dan Langille]

On Wed, Feb 1, 2023, at 10:25 AM, Frédéric F. wrote:
> Hello,
> 
> Is Bacula 13 compatible with PostgreSQL v15 & 15.1 or should I install the 14 
> version ?
> 
> I saw Bacularis is PostgreSQL v15 compatible but what about bacula ?
> 
> https://bacularis.app/news/44/36/New-release-Bacularis-1.3.0/d,Bacularis%20news%20detail
>  
> <https://bacularis.app/news/44/36/New-release-Bacularis-1.3.0/d,Bacularis%20news%20details>

This is where regression testing would be useful.

However, regress.bacula.org died and has not been replaced. The hostname has 
been removed so I doubt it will be coming back.

In December I asked if anyone wants to host cdash.  Nothing came of it so I 
will be shutting down my regression farm soon. I had not added PostgreSQL 15 to 
it because regression tests were not being collected.

https://sourceforge.net/p/bacula/mailman/message/37752245/

I'm sorry to see it go. Regression testing is the best way to ensure your 
preferred OS and database works as expected with Bacula. I am disappointed that 
this is now dead.

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] contribute to Bacula via cdash

[Dan Langille]

On Thu, Dec 22, 2022, at 2:58 PM, Dan Langille wrote:
> If you've been looking for a way to contribute to the Bacula projecet, this 
> is one.
> 
> Install cdash and have it publicly available for use by Bacula regression 
> testers (like me).
> 
> The cdash install of the regression testing hosts allowed users to run 
> regression tests using their combination of OS and database. I do this for 
> entirely selfish reasons. I want problems created by recent changes to my 
> choice of OS and database to be caught and fixed early. I don't want a 
> release to come out which is broken for me.  Regression testing is very much 
> a set-it-up-and-it-runs project. Is it not a time-sink.
> 
> Historically, we have used cdash. That host has died and not been replaced. 
> Previous instances of cdash were run by a community member. If someone wants 
> to set up a cdash instance, I can easily point my regression  testing hosts 
> at it and start uploading my test results there.  6 months (they disappeared 
> in early June 2022) is a long time to go without adequate regression testing 
> coverage.
> 
> Please consider it. I know I'm asking others to do something I could do 
> myself. I am already over-committed to existing projects.
> 
> Also, consider setting up a regression testing instance. Fire up a small VM 
> somewhere.
> 
> see 
> https://www.bacula.org/13.0.x-manuals/en/developers/Bacula_Regression_Testing.html
> 
> re:
> 
> Eric Bollengier via Bacula-users wrote on 8/30/22 2:50 AM:
>> Unfortunately, CDASH and the Wiki are still not available today - we 
>> anticipate these will be replaced by Gitlab. Indeed, the wiki has already 
>> been ported into 
>> GitLab.

In December I asked if anyone wants to host cdash.  Nothing came of it so I 
will be shutting down my regression farm soon. I had not added PostgreSQL 15 to 
it because regression tests were not being collected.

I'm sorry to see it go. Regression testing is the best way to ensure your 
preferred OS and database works as expected with Bacula. I am disappointed that 
this is now dead.

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] contribute to Bacula via cdash

[Dan Langille]

If you've been looking for a way to contribute to the Bacula projecet, 
this is one.


Install cdash and have it publicly available for use by Bacula 
regression testers (like me).


The cdash install of the regression testing hosts allowed users to run 
regression tests using their combination of OS and database. I do this 
for entirely selfish reasons. I want problems created by recent changes 
to my choice of OS and database to be caught and fixed early. I don't 
want a release to come out which is broken for me.  Regression testing 
is very much a set-it-up-and-it-runs project. Is it not a time-sink.


Historically, we have used cdash. That host has died and not been 
replaced. Previous instances of cdash were run by a community member. If 
someone wants to set up a cdash instance, I can easily point my 
regression  testing hosts at it and start uploading my test results 
there.  6 months (they disappeared in early June 2022) is a long time to 
go without adequate regression testing coverage.


Please consider it. I know I'm asking others to do something I could do 
myself. I am already over-committed to existing projects.


Also, consider setting up a regression testing instance. Fire up a small 
VM somewhere.


see 
https://www.bacula.org/13.0.x-manuals/en/developers/Bacula_Regression_Testing.html


re:

Eric Bollengier via Bacula-users wrote on 8/30/22 2:50 AM:
Unfortunately, CDASH and the Wiki are still not available today - we 
anticipate these will be replaced by Gitlab. Indeed, the wiki has 
already been ported into
GitLab. 

--
Dan Langille
d...@langille.org : https://langille.org/
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Setting the to address for Bacula GDB traceback

[Dan Langille]

On Mon, Nov 7, 2022, at 10:59 AM, Martin Simmons wrote:
>>>>>> On Sun, 6 Nov 2022 20:00:55 -0500, Dan Langille said:
>> 
>> I'm getting some traceback emails like this:
>> 
>> From: root@localhost
>> Subject: Bacula GDB traceback of bacula-dir on bacula.int.example.org
>> Sender: bac...@bacula.int.example.org
>> To: root@localhost
>> 
>> There is no 'root@localhost' defined with my bacula-dir configuation.
>> 
>> [bacula dan /usr/local/etc/bacula] % sudo grep root@localhost * 0:56:01
>> bacula-dir.conf:#  mail = root@localhost = all, !skipped, saved
>> bacula-dir.conf.sample:  mail = root@localhost = all, !skipped
>> bacula-dir.conf.sample:  operator = root@localhost = mount
>> bacula-dir.conf.sample:  mail = root@localhost = all, !skipped
>> 
>> The above are either comments or .sample - none should be active.
>> 
>> Where is this address coming from?
>
> It is the default for dump_email in configure (used in scripts/btraceback.in).

Confirmed, I found it in there. Thank you.

I will probably modify the FreeBSD port to accomomdate local changes to this 
script.

While looking around, I found these:

[pkg01 dan ~/ports/head/sysutils/bacula9-server] % grep -r root *
Makefile:   --with-dump-email=root@localhost \
Makefile:   --with-job-email=root@localhost \

Those are configuration arguments for building DIR, SD, and FD. If those are 
specified at build time, it might be difficult for users to modify them. I'm 
wondering if they are still used or if they are deprecated. Sorry, I can't 
search the code just now.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Setting the to address for Bacula GDB traceback

[Dan Langille]

Hello,

I'm getting some traceback emails like this:

From: root@localhost
Subject: Bacula GDB traceback of bacula-dir on bacula.int.example.org
Sender: bac...@bacula.int.example.org
To: root@localhost

There is no 'root@localhost' defined with my bacula-dir configuation.

[bacula dan /usr/local/etc/bacula] % sudo grep root@localhost * 0:56:01
bacula-dir.conf:#  mail = root@localhost = all, !skipped, saved
bacula-dir.conf.sample:  mail = root@localhost = all, !skipped
bacula-dir.conf.sample:  operator = root@localhost = mount
bacula-dir.conf.sample:  mail = root@localhost = all, !skipped

The above are either comments or .sample - none should be active.

Where is this address coming from?

Thanks.

--
Dan Langille - d...@langille.org
https://langille.org/


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Bacula 13.0.0 now in the FreeBSD ports tree

[Dan Langille]

Hello,

Bacula 13 is now in the FreeBSD ports tree. Please proceed with caution. 
I have not tested it to see how stable it is.


It built cleanly based on the Bacula 11 port files, but I have not 
tested it.  Please open a PR at https://bugs.freebsd.org/bugzilla/ is 
you see a problem.


sysutils/bacula13-client-static did not build cleanly. That may come later.

===> Checking for items in STAGEDIR missing from pkg-plist
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: sbin/bacula-fd-static
Error: Missing: sbin/bconsole-static

Details at 
https://services.unixathome.org/poudriere/data/13amd64-dvl-testing/2022-07-10_00h33m03s/logs/errors/bacula13-client-static-13.0.0.log


--
Dan Langille - d...@langille.org
https://langille.org/


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [Bacula-announce] Beta Release 11.3.2

[Dan Langille]

Dan Langille wrote on 3/25/22 9:16 PM:

Eric Bollengier via Bacula-announce wrote on 3/24/22 1:50 PM:

Hello,

We are pleased to announce the first BETA release of the next major 
Bacula
version 11.3.2 to both the Bacula website (www.bacula.org) and to 
SourceForge.




Now might be the time to add 11.3 to 
https://regress.bacula.org/viewProjects.php


I've just updated my test hosts to run 11.3 regression testing.


Oh, wait, they are showing up:

https://regress.bacula.org/index.php?project=Bacula-11.0=2022-03-25

Perhaps adding 11.3 is not needed. I hope those failure tests are helpful.

--
Dan Langille - d...@langille.org
https://langille.org/


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [Bacula-announce] Beta Release 11.3.2

[Dan Langille]

Eric Bollengier via Bacula-announce wrote on 3/24/22 1:50 PM:

Hello,

We are pleased to announce the first BETA release of the next major 
Bacula
version 11.3.2 to both the Bacula website (www.bacula.org) and to 
SourceForge.




Now might be the time to add 11.3 to 
https://regress.bacula.org/viewProjects.php


I've just updated my test hosts to run 11.3 regression testing.

--
Dan Langille - d...@langille.org
https://langille.org/


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] S3 cloud plugin does not load on FreeBSD 13.0

[Dan Langille]

On Sun, Jul 18, 2021, at 6:32 AM, Ronny Forberger wrote:
> Hi,
> 
> I have a problem that I cannot load the s3 cloud plugin on FreeBSD 
> 13.0-RELEASE-p3 amd64.
> 
> The error in bacula-sd I am getting is the following:
> 
> > my-sd: init_dev.c:488-0 Driver=cloud entry point=800d4fe60
> > my-sd: cloud_dev.c:876-0 Open SD driver at 
> > /usr/local/lib/bacula-sd-cloud-s3-driver-11.0.5.so
> > my-sd: cloud_dev.c:879-0 Driver=s3 handle=800d19408
> > my-sd: cloud_dev.c:881-0 Lookup "BaculaCloudDriver" in driver=s3
> > my-sd: cloud_dev.c:883-0 Driver=s3 entry point=0
> > my-sd: cloud_dev.c:889-0 Lookup of symbol "BaculaCloudDriver" 
> > driver=/usr/local/lib/bacula-sd-cloud-s3-driver-11.0.5.so failed: 
> > ERR=Undefined symbol "BaculaCloudDriver"

This line above ...

> > my-sd: message.c:1831-0 cloud_dev.c:1001 Could not open Cloud driver 
> > type=1 for Device=CloudStorage.
> > my-sd: message.c:1831-0 cloud_dev.c:1032 Cloud driver initialization 
> > error Cloud driver not properly loaded
> > my-sd: htable.c:67-0 malloc buf=801c71ae8 size=100 rem=76
> > my-sd: init_dev.c:235-0 init_dev allocated: 801c1cc28
> > my-sd: init_dev.c:437-0 init_dev: tape=0 
> > dev_name=/backup/backupdisk1/bacula-storage
> > my-sd: dev.c:1143-0 DEVICE::register_metrics called. 0x801c1cc28 
> > collector=0x80101c938
> > my-sd: stored.c:706-0 SD init done CloudStorage (0x801c1cc28)
> I comppiled the cloud-driver and the cloud-s3-driver with the following 
> commands:
> > $ /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/src/stored 
> > >   make install-cloud
> >
> > LT Compiling cloud_parts.c
> > LT Compiling cloud_transfer_mgr.c
> > LT Compiling cloud_dev.c
> > cloud_dev.c:1818:17: warning: address of array 'tpkt->m_hash64' will 
> > always evaluate to 'true' [-Wpointer-bool-conversion]
> >   if (tpkt->m_hash64) {
> >   ~~  ~~^~~~
> > 1 warning generated.
> > LT Compiling file_driver.c
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/libtool 
> > --silent --tag=CXX --mode=link /usr/bin/c++ -L/usr/local/lib 
> > -L/usr/local/lib  -L/usr/local/lib -fstack-protector-strong -shared 
> > cloud_parts.lo cloud_transfer_mgr.lo cloud_dev.lo file_driver.lo -o 
> > bacula-sd-cloud-driver.la -R /usr/local/lib -rpath /usr/local/lib 
> > -module -export-dynamic -release 11.0.5
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/autoconf/mkinstalldirs
> >  
> > /usr/local/lib
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/libtool 
> > --silent --tag=CXX --mode=install /usr/bin/install -c -m 755 
> > bacula-sd-cloud-driver.la /usr/local/lib
> > /bin/rm -f /usr/local/lib/bacula-sd-cloud-driver.la
> >
> > $ /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/src/stored 
> > >   make install-s3-cloud
> >
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/autoconf/mkinstalldirs
> >  
> > /usr/local/lib
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/libtool 
> > --silent --tag=CXX --mode=install /usr/bin/install -c -m 755 
> > bacula-sd-cloud-driver.la /usr/local/lib
> > /bin/rm -f /usr/local/lib/bacula-sd-cloud-driver.la
> > LT Compiling s3_driver.c
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/libtool 
> > --silent --tag=CXX --mode=link /usr/bin/c++ -L/usr/local/lib 
> > -L/usr/local/lib  -L/usr/local/lib -fstack-protector-strong -shared 
> > cloud_parts.lo cloud_transfer_mgr.lo s3_driver.lo -o 
> > bacula-sd-cloud-s3-driver.la  -R /usr/local/lib -rpath /usr/local/lib 
> > -module -export-dynamic -release 11.0.5
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/autoconf/mkinstalldirs
> >  
> > /usr/local/lib
> > /usr/ports/sysutils/bacula11-server/work/bacula-11.0.5/libtool 
> > --silent --tag=CXX --mode=install /usr/bin/install -c -m 755 
> > bacula-sd-cloud-s3-driver.la /usr/local/lib
> > /bin/rm -f /usr/local/lib/bacula-sd-cloud-s3-driver.la
> >
> But when I start bacula-sd with bacula-sd -d 255, I am getting the above
> 
> > ERR=Undefined symbol "BaculaCloudDriver"
> error message.

Matches this line.  My thoughts: don't try to start bacula-sd until that 
compile time error is fixed.

> 
> Any ideas what cloud be wrong?

Let's see what the list comes up with over the next few days. If nothing comes 
of it, try the devel mailing list.

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [Bacula-devel] Release 11.0.5

[Dan Langille]

On Fri, Jun 4, 2021, at 8:19 AM, Sven Hartge wrote:
> On 04.06.21 08:46, Eric Bollengier via Bacula-users wrote:
> 
> > It seems that the release process is very attractive these days,
> > the tags are usually pushed at the end of the procedure (after
> > few days or when preparing the next release. It's better to
> > be sure that everything is ok because we cannot overwrite tags
> > once they are cloned on other git repositories.
> > 
> > Of course, I can push them now, it seems that the release
> > is correct this time.
> 
> I am sorry to say, but that is really strange. Of all the projects I 
> follow, Bacula is the only with that interpretation of tag usage.
> 
> Everyone else tags a release as soon as it is released and then the 
> release it not changed afterwards.
> 
> If something is wrong, a new release is made and tagged as well.
> 
> Same goes for the provided archives. Please don't remove existing ones 
> and replace them with other archives with the same version.

A project should never do this. 

As a packager, you have now made my job much bigger than it needs to be.
This is a non-trivial situation. If there is a problem with the release,
create another release. Your downstream will have much more respect for you
than if you do otherwise.

Please, never do this.

Archives are archives, never to be modified. By definition.

> This makes life really difficult for distributions when you add a new 
> version and then suddenly it is revoked and replaces by a new release 
> with the same version number.
> 
> If you for example release 11.0.5 and we upload this to Debian and two 
> days later you remove the release and replace it, we would then need to 
> "invent" a version number like 11.0.5.1 to be able to upload the new 
> version.
> 
> Yes, it looks bad when you have to release several minor version over a 
> few days, but please think of the people downstream of you.

It should not be about the project. It should be about your consumers.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] postgres createdb for bacula fails

[Dan Langille]

On Mon, May 3, 2021, at 6:16 PM, kjohn...@eclypse.org wrote:

> The next step in my testing is to create an empty bacula database on the
> test system and restore it from the pg_dump file created on the conversion
> system.  I have learned that the bacula postgres database needs to have
> encoding SQL_ASCII.  

For what it's worth, SQL_ASCII was decided upon many years ago.

> Here's the difficulty.  As user bacula on the test system:
> 
> $ export LC_ALL=C
> $ createdb -E SQL_ASCII -T template0 bacula
> createdb: database creation failed: ERROR:  encoding "SQL_ASCII" does not
> match locale "en_US.UTF-8"
> DETAIL:  The chosen LC_CTYPE setting requires encoding "UTF8".
> 
> Probably, fixing this is something that everyone else knows.  Unfortunately,
> my Google searches find too much or almost nothing.  A specific search of
> the mailing list archive found a discussion of why SQL_ASCII was so
> important (circa 2009),

The decision was made earlier I think.

> but not an answer to this problem.  At least, not
> one that I recognized.  (same failure without the export, by the way).  The
> documentation for Bacula 7.4
> (https://www.bacula.org/7.4.x-manuals/en/main/Installing_Configuring_Post.ht
> ml) also states that SQL_ASCII is essential, but I did not recognize a
> solution to this problem there.

I think the problem is more PostgreSQL & environment specific. Here is my test
on FreeBSD 12.2 with PostgreSQL 12.6 - these commands are run as the postgres
user:

$ createdb -E SQL_ASCII -T template0 bacula
$ dropdb bacula
$ export LC_ALL=C
$ createdb -E SQL_ASCII -T template0 bacula
$ 

Works there. I suspect it is something specific to the environment.

What do you have for this value?

$ echo $locale

$ 


-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Restore not working on compressed backup

[Dan Langille]

On Wed, Apr 21, 2021, at 4:09 AM, jerome raoul wrote:
> Hello everyone,

> 

> I'm testing Bacula, I have compiled it from source with gzip and lzo to 
> compress my backup. This is working without any issue. But when I try to 
> restore one file on the same client using the default job “RestoreFiles”, the 
> file is restored but compressed. If I decompress it using a trick (printf 
> "\x1f\x8b\x08\x00\x00\x00\x00\x00" |cat - /tmp/restore/myfile | gzip -dc 
> >/tmp/restore/un_myfile).

> I can read it. Of course, in log I got an error message: Error: attribs.c:621 
> File size of restored file /tmp/restore/myfile not correct. Original 239, 
> restored 173.

> Does someone have an idea of what is causing the issue? Bad configuration 
> when compiling, wrong option on restore?


Perhaps if you shared the backup job so we can see the settings you used...

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] regression testing - more volunteers wanted

[Dan Langille]

> On Apr 17, 2021, at 14:22, Andrea Venturoli  wrote:
> 
> On 4/17/21 2:14 PM, Dan Langille wrote:
> 
>> Looking at 
>> https://regress.bacula.org/index.php?project=Bacula-9.6=2021-04-07
>> I consistently get errors on langille_mysql57_bacula but not the others.
>> By all means, post to the developers channel if you get consistent failures. 
>> It is vital that they at least know the tests are failing.
>> I have noticed for a while that I'm regression testing Bacula-9.6 - 
>> eventually I should move to or add Bacula-11.0.
>> Thank you.
> 
> Thanks for your answer.
> 
> See https://regress.bacula.org/index.php?project=Bacula-11.0
> 
> (Notice I moved from FreeBSD 11 to 12).

There’s a good point. My regression testing is done in FreeBSD jails. I should 
install FreeBSD 11.4 jails as well. I hope to update the host to FreeBSD 13 
soon. Then I’ll copy the existing regression jails to new jails & update them 
to FreeBSD 13. Then we’ll have good coverage on all current versions of 
FreeBSD. 

> Shouldn't the developer see all from here?

Perhaps. Posting a regression failure over on the devs list is a good idea 
though. It is the dev mailing list & a regression test failure is a dev issue. 

Between us both, FreeBSD should be well covered. 


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] regression testing - more volunteers wanted

[Dan Langille]

On Thu, Apr 8, 2021, at 6:29 AM, Andrea Venturoli wrote:
> On 3/21/21 2:48 PM, Dan Langille wrote:
> 
> Hello.
> 
> > More diversity in OS versions is a goal. At present, only FreeBSD is 
> > consistently represented.
> 
> While we are at it, I'm one of those representing FreeBSD :)
> Since I moved my tests to version 11, I'm getting a few persistent failures:
> > The following tests FAILED:
> >  69 - disk:file-span-vol-test (Failed)
> >  85 - disk:backup-bacula-test (BAD_COMMAND)
> >  88 - disk:bconsole-test (Failed)
> > 172 - disk:truncate-test (Failed)
> 
> Should I investigate better? Report something in some way? Or is the 
> automatic posting of results enough?

Looking at 
https://regress.bacula.org/index.php?project=Bacula-9.6=2021-04-07

I consistently get errors on langille_mysql57_bacula but not the others.

By all means, post to the developers channel if you get consistent failures. It 
is vital that they at least know the tests are failing.

I have noticed for a while that I'm regression testing Bacula-9.6 - eventually 
I should move to or add Bacula-11.0.

Thank you.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Backing up Bacula itself

[Dan Langille]

On Mon, Mar 29, 2021, at 9:46 AM, Doruk Fisek via Bacula-users wrote:
> Hi,
> 
>  What do you recommend for backing up the Bacula server itself?
> 
>  We're backing up the Bacula VM at the virtualization level, I'd like
> to hear other (perhaps better) practices.

That's a good start.  It's similar to how I back up my FreeBSD jail which
runs Bacula.

The catalog is dumped to disk on a daily basis. That is backed up every day.

The dump is in plain text which makes it easier/faster to rsync somewhere else. 
That rsync is to multiple locations.

I rsync the configuration files away on a daily basis, to a few other locations.

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Restoring encrypted files to a different host

[Dan Langille]

On Wed, Mar 24, 2021, at 3:37 PM, Shawn Rappaport wrote:
> What do I need to do in order to be able to restore from one server to the 
> other? Do I need to copy the private key from portal02-px to portal01-px and 
> update bacula-fd.conf on them as well? 

Yes, but note, I have never tried this.

> If so, what would I put in bacula-fd.conf?

Basically, the same as what you had in the other client for PKI Keypair

see

https://www.bacula.org/11.0.x-manuals/en/main/Data_Encryption.html

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] regression testing - more volunteers wanted

[Dan Langille]

On Mon, Mar 22, 2021, at 8:23 AM, Adolf Belka (gmail) wrote:
> Hi All,
> 
> I have run the regression tests and had a couple of failures. One of 
> those disappeared after re-running the test. The other I have re-run 
> with debug set and saved the output to a file. I also had a syntax error 
> from the regression script itself right at the end for the total time 
> printing.

What was the error?


-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] regression testing - more volunteers wanted

[Dan Langille]

On Tue, Mar 23, 2021, at 8:26 AM, Adolf Belka (gmail) wrote:
> Hi All,
> 
> Found the README.ctest file and the CDashboard url. Have successfully 
> run the experimental-disk script and the results have got onto the 
> CDashboard.
> 
> So everything seems to be working well except that the bacula configure 
> script does not recognise Arch Linux.

Looking at 
http://regress.bacula.org/index.php?project=Bacula-9.6=2021-03-24 I see

9.6.7-10Dec20-mysql-unknown-unknown

Are you referring to 'unknown-unknown' where as other results have 
'freebsd-12.2-RELEASE-p4'?

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] regression testing - more volunteers wanted

[Dan Langille]

Hello,

Members of project communities often wonder how they can contribute without 
coding.

I have one suggestion: regression testing.

More diversity in OS versions is a goal. At present, only FreeBSD is 
consistently represented. That's not enough. Your OS should be there too.With 
daily regression testing on a wide variety of operating systems and versions, 
changes to code which introduce regressions in behavior can be quickly caught. 
Catching them early is key to getting them fixed.

It is not difficult to set up and it run automatically. No special skills 
required. If you are running Bacula
now, this is an easy way for you to contribute to the community.

The testing is run as non-root and requires no special privileges. I run my 
testing under my own login.

In my setup, I create a dedicated host (in a FreeBSD jail for this) but that is 
not required. The resources
required are not great.

Details at 
https://www.bacula.org/11.0.x-manuals/en/developers/Bacula_Regression_Testing.html

-- 
Dan Langille
d...@langille.org




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] ERR=20:"unable to get local issuer certificate"

[Dan Langille]

On Tue, Nov 10, 2020, at 2:11 PM, David Newman wrote:
> Director: FreeBSD 12.2, bacula-server-9.6.6 from pkgs
> Client: OpenBSD 6.8, bacula-client-9.6.5 from pkgs
> 
> After upgrading a bacula client's OS from OpenBSD 6.7 to 6.8, nightly
> backups run successfully but throw this warning:
> 
> ERR=20:"unable to get local issuer certificate"
> 
> This setup uses self-signed certificates and worked without errors or
> warnings before this OS upgrade.
> 
> There has been no bacula configuration change on either the client or
> director . A diff of the client bacula-fd.conf file (excerpted below)
> before and after the upgrade shows no change.
> 
> I tried revoking the old client cert and generating a new one, but this
> had no effect on the warning message.
> 
> I also tried command-line "openssl s_client -connect" commands both
> ways. Both connections worked on the respective ports 9101 and 9102.
> 
> Besides the bacula client configuration -- which hasn't changed, aside
> from pointing to new certs with the same filenames -- is there something
> else that needs tweaking on the client?
> 
> Many thanks.
> 
> dn
> 
> -
> 
> client bacula-fd.conf
> 
> Director {
>   Name = nye-dir
>  ..
> 
>   TLS Require = yes
>   TLS Enable = yes
>   TLS Verify Peer = yes
> 
>  # Allow only the Director to connect
>   TLS Allowed CN = "backups.example.com"
>   TLS CA Certificate File = /etc/bacula/cacert.pem
>   TLS Certificate = /etc/bacula/client.pem
>   TLS Key = /etc/bacula/client.key
> 
> }
> 
> ..
> 
> FileDaemon {
>   Name = client-fd
>   FDport = 9102  # where we listen for the director
>   WorkingDirectory = /var/db/bacula
>   Pid Directory = /var/run
>   Maximum Concurrent Jobs = 20
> 
>   TLS Require = yes
>   TLS Enable = yes
> 
>   TLS CA Certificate File = /etc/bacula/cacert.pem
>   TLS Certificate = /etc/bacula/client.pem
>   TLS Key = /etc/bacula/client.key
> 
> }

Did you solve this one?

-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Discrepancy between backup/restore size and files

[Dan Langille]

On Sat, Sep 12, 2020, at 6:23 AM, Ben Laurie wrote:
> I restored a Windows incremenental backup (to a FreeBSD machine).
> 
> The restore job looks like:
> 
>   Build OS:   amd64-portbld-freebsd12.1 freebsd 12.1-SYNTH
>   JobId:  92
>   Job:RestoreFiles.2020-09-12_10.36.22_08
>   Restore Client: b2-fd
>   Where:  /tmp/bacula-restores
>   Replace:Always
>   Start time: 12-Sep-2020 10:36:24
>   End time:   12-Sep-2020 10:45:48
>   Elapsed time:   9 mins 24 secs
>   Files Expected: 3,858
>   Files Restored: 3,858
>   Bytes Restored: 82,256,949,051 (82.25 GB)
>   Rate:   145845.7 KB/s
>   FD Errors:  0
>   FD termination status:  OK
>   SD termination status:  OK
>   Termination:Restore OK
> 
> But...
> 
> b2 /tmp/bacula-restores# du -sh *   
> 804Mc:
>  22Kj:
>  25Gk:
> 
> Why are the restored files less than a third of the size of the restore job?

Is zfs compression involved?

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] TLS issues

[Dan Langille]

On Mon, Aug 3, 2020, at 12:30 PM, Shawn Rappaport wrote:
> 
> Any idea why it would not be encrypted even though I am requiring TLS?

TLS is just transit.  Transport Layer Security.

https://www.bacula.org/9.6.x-manuals/en/main/Bacula_TLS_Communications_E.html

You want Data Encryption:

https://www.bacula.org/9.6.x-manuals/en/main/Data_Encryption.html

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] TLS issues

[Dan Langille]

run up to 10 jobs a the same time
>   Autochanger = File1 # point to ourself
>   TLS Enable = yes
>   TLS Require = yes
>   TLS CA Certificate File = /etc/ssl/bacula_ca.crt
>   TLS Certificate = /etc/ssl/bacula_server.crt
>   TLS Key = /etc/ssl/bacula_server.key
> }
> 
> I also added the TLS directives to *bacula-sd.conf*:
> 
> Storage { # definition of myself 
>   Name = xbacdirector01-lv.internal.shutterfly.com-sd
>   SDPort = 9103  # Director's port
>   WorkingDirectory = "/var/bacula"
>   Pid Directory = "/var/run"
>   Plugin Directory = "/usr/lib64"
>   Maximum Concurrent Jobs = 20
>   TLS Enable = yes
>   TLS Require = yes
>   TLS CA Certificate File = /etc/ssl/bacula_ca.crt
>   TLS Certificate = /etc/ssl/bacula_server.crt
>   TLS Key = /etc/ssl/bacula_server.key
> }
> 
> Director { 
>   Name = xbacdirector01-lv.internal.shutterfly.com-dir
>   Password = "password"
>   TLS Enable = yes
>   TLS Require = yes
>   TLS CA Certificate File = /etc/ssl/bacula_ca.crt
>   TLS Certificate = /etc/ssl/bacula_server.crt
>   TLS Key = /etc/ssl/bacula_server.key
> }
> 
> After those changes, I bounced Bacula and tried running bconsole. Here is the 
> error I'm receiving:
> 
> Connecting to Director xbacdirector01-lv.internal.shutterfly.com:9101 
> bconsole: tls.c:87-0 Error with certificate at depth: 0, issuer = 
> /C=US/ST=Arizona/L=Tempe/O=Shutterfly/OU=ops-syseng/CN=xbacdirector01-lv.internal.shutterfly.com/emailAddress=t...@shutterfly.com,
>  subject = 
> /C=US/ST=Arizona/L=Tempe/O=Shutterfly/OU=ops-syseng/CN=xbacdirector01-lv.internal.shutterfly.com/emailAddress=t...@shutterfly.com,
>  ERR=18:self signed certificate
> 31-Jul 14:41 bconsole JobId 0: Error: tls.c:87 Error with certificate at 
> depth: 0, issuer = 
> /C=US/ST=Arizona/L=Tempe/O=Shutterfly/OU=ops-syseng/CN=xbacdirector01-lv.internal.shutterfly.com/emailAddress=t...@shutterfly.com,
>  subject = 
> /C=US/ST=Arizona/L=Tempe/O=Shutterfly/OU=ops-syseng/CN=xbacdirector01-lv.internal.shutterfly.com/emailAddress=t...@shutterfly.com,
>  ERR=18:self signed certificate
> TLS negotiation failed
> Director authorization problem.
> Most likely the passwords do not agree.
> If you are using TLS, there may have been a certificate validation error 
> during the TLS handshake.
> For help, please see 
> http://www.bacula.org/rel-manual/en/problems/Bacula_Frequently_Asked_Que.html
> 
> 
> I'm guessing I didn't configure the certs properly but I don't know why. I 
> also tried creating things with the CN not being fully-qualified, i.e.: 
> CN=xbacdirector01-lv. That didn't make a difference. I'm pretty green when it 
> comes to configuring certs so I apologize if I'm making rookie mistakes.
> 
> Does anyone know what I'm doing wrong? Please let me know if you need more 
> details of my setup and configuration.
> 
> Thanks!
> 
> --Shawn
> 
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> 

--
  Dan Langille
  d...@langille.org

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] BSR file not updated with every job

[Dan Langille]

On Tue, Jul 28, 2020, at 12:52 PM, Martin Simmons wrote:
> >>>>> On Tue, 28 Jul 2020 07:55:21 -0400, Dan Langille said:
> > 
> > On Mon, Jul 27, 2020, at 11:55 AM, Dan Langille wrote:
> > > On Sun, Jul 26, 2020, at 7:33 AM, Dan Langille wrote:
> > > > > On Jul 24, 2020, at 6:00 PM, Martin Simmons  
> > > > > wrote:
> > > > > 
> > > > >>>>>> On Fri, 24 Jul 2020 09:09:12 -0400, Dan Langille said:
> > > > >> 
> > > > >> I'm using Bacula 9.6.5 on FreeBSD 12.1
> > > > >> 
> > > > >> I have noticed a BSR file which is not getting updated after every 
> > > > >> job run.
> > > > >> ...
> > > > >> This is the most recent job, from today:
> > > > >> 
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: Start Backup JobId 314258, 
> > > > >> Job=tallboy_home.2020-07-24_03.05.00_08
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: There are no more Jobs 
> > > > >> associated with Volume "IncrAuto-7393". Marking it purged.
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: All records pruned from Volume 
> > > > >> "IncrAuto-7393"; marking it "Purged"
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: Recycled volume "IncrAuto-7393"
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: Using Device 
> > > > >> "vDrive-IncrFile-6" to write.
> > > > >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Recycled volume 
> > > > >> "IncrAuto-7393" on File device "vDrive-IncrFile-6" 
> > > > >> (/usr/local/bacula/volumes/IncrFile), all previous data lost.
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: Max Volume jobs=1 exceeded. 
> > > > >> Marking Volume "IncrAuto-7393" as Used.
> > > > >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Elapsed time=00:00:03, 
> > > > >> Transfer rate=0  Bytes/second
> > > > >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Sending spooled attrs to 
> > > > >> the Director. Despooling 0 bytes ...
> > > > >> 24-Jul 03:05 bacula-dir JobId 314258: Bacula bacula-dir 9.6.5 
> > > > >> (11Jun20):
> > > > >>  Build OS:   amd64-portbld-freebsd12.1 freebsd 
> > > > >> 12.1-RELEASE-p7
> > > > >>  JobId:  314258
> > > > >>  Job:tallboy_home.2020-07-24_03.05.00_08
> > > > >>  Backup Level:   Incremental, since=2020-07-23 03:05:17
> > > > >>  Client: "tallboy-fd" 9.6.5 (11Jun20) 
> > > > >> amd64-portbld-freebsd12.1,freebsd,12.1-RELEASE-p7
> > > > >>  FileSet:"tallboy home" 2012-12-20 14:19:02
> > > > >>  Pool:   "IncrFile" (From Job IncPool override)
> > > > >>  Catalog:"MyCatalog" (From Client resource)
> > > > >>  Storage:"bacula-sd-01-IncrFile" (From Pool resource)
> > > > >>  Scheduled time: 24-Jul-2020 03:05:00
> > > > >>  Start time: 24-Jul-2020 03:05:17
> > > > >>  End time:   24-Jul-2020 03:05:21
> > > > >>  Elapsed time:   4 secs
> > > > >>  Priority:   10
> > > > >>  FD Files Written:   0
> > > > >>  SD Files Written:   0
> > > > >>  FD Bytes Written:   0 (0 B)
> > > > >>  SD Bytes Written:   0 (0 B)
> > > > > 
> > > > > This job wrote nothing to the volume, so there is nothing to add to 
> > > > > the BSR
> > > > > file.
> > > > 
> > > > When I look at this post now, that's easy to see. I find myself 
> > > > smirking to myself when it see it now. Thanks.
> > > > 
> > > > Why was this a problem to me? I'm backing up my BSR files with an rsync 
> > > > to other hosts.
> > > > 
> > > > There is a monitoring script to make sure the rcync'd files are not 
> > > > more than 48 hours old.
> > > > 
> > > > I have 'fixed' with with a crontab on the above referenced host:
> > > > 
> > > > 0   0   *   *  

Re: [Bacula-users] BSR file not updated with every job

[Dan Langille]

On Mon, Jul 27, 2020, at 11:55 AM, Dan Langille wrote:
> On Sun, Jul 26, 2020, at 7:33 AM, Dan Langille wrote:
> > > On Jul 24, 2020, at 6:00 PM, Martin Simmons  wrote:
> > > 
> > >>>>>> On Fri, 24 Jul 2020 09:09:12 -0400, Dan Langille said:
> > >> 
> > >> I'm using Bacula 9.6.5 on FreeBSD 12.1
> > >> 
> > >> I have noticed a BSR file which is not getting updated after every job 
> > >> run.
> > >> ...
> > >> This is the most recent job, from today:
> > >> 
> > >> 24-Jul 03:05 bacula-dir JobId 314258: Start Backup JobId 314258, 
> > >> Job=tallboy_home.2020-07-24_03.05.00_08
> > >> 24-Jul 03:05 bacula-dir JobId 314258: There are no more Jobs associated 
> > >> with Volume "IncrAuto-7393". Marking it purged.
> > >> 24-Jul 03:05 bacula-dir JobId 314258: All records pruned from Volume 
> > >> "IncrAuto-7393"; marking it "Purged"
> > >> 24-Jul 03:05 bacula-dir JobId 314258: Recycled volume "IncrAuto-7393"
> > >> 24-Jul 03:05 bacula-dir JobId 314258: Using Device "vDrive-IncrFile-6" 
> > >> to write.
> > >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Recycled volume 
> > >> "IncrAuto-7393" on File device "vDrive-IncrFile-6" 
> > >> (/usr/local/bacula/volumes/IncrFile), all previous data lost.
> > >> 24-Jul 03:05 bacula-dir JobId 314258: Max Volume jobs=1 exceeded. 
> > >> Marking Volume "IncrAuto-7393" as Used.
> > >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Elapsed time=00:00:03, 
> > >> Transfer rate=0  Bytes/second
> > >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Sending spooled attrs to the 
> > >> Director. Despooling 0 bytes ...
> > >> 24-Jul 03:05 bacula-dir JobId 314258: Bacula bacula-dir 9.6.5 (11Jun20):
> > >>  Build OS:   amd64-portbld-freebsd12.1 freebsd 
> > >> 12.1-RELEASE-p7
> > >>  JobId:  314258
> > >>  Job:tallboy_home.2020-07-24_03.05.00_08
> > >>  Backup Level:   Incremental, since=2020-07-23 03:05:17
> > >>  Client: "tallboy-fd" 9.6.5 (11Jun20) 
> > >> amd64-portbld-freebsd12.1,freebsd,12.1-RELEASE-p7
> > >>  FileSet:"tallboy home" 2012-12-20 14:19:02
> > >>  Pool:   "IncrFile" (From Job IncPool override)
> > >>  Catalog:"MyCatalog" (From Client resource)
> > >>  Storage:"bacula-sd-01-IncrFile" (From Pool resource)
> > >>  Scheduled time: 24-Jul-2020 03:05:00
> > >>  Start time: 24-Jul-2020 03:05:17
> > >>  End time:   24-Jul-2020 03:05:21
> > >>  Elapsed time:   4 secs
> > >>  Priority:   10
> > >>  FD Files Written:   0
> > >>  SD Files Written:   0
> > >>  FD Bytes Written:   0 (0 B)
> > >>  SD Bytes Written:   0 (0 B)
> > > 
> > > This job wrote nothing to the volume, so there is nothing to add to the 
> > > BSR
> > > file.
> > 
> > When I look at this post now, that's easy to see. I find myself 
> > smirking to myself when it see it now. Thanks.
> > 
> > Why was this a problem to me? I'm backing up my BSR files with an rsync 
> > to other hosts.
> > 
> > There is a monitoring script to make sure the rcync'd files are not 
> > more than 48 hours old.
> > 
> > I have 'fixed' with with a crontab on the above referenced host:
> > 
> > 0   0   *   *   *   touch ~/.backmeup
> > 
> > Cheers
> 
> Today we have:
> 
> -rw-r-  1 bacula  bacula   1035 Jul 26 03:05 tallboy-fd_tallboy_home.bsr
> 
> The file updated by cron is:
> 
> [dan@tallboy:~] $ ls -l .backmeup 
> -rw-r--r--  1 dan  dan  0 Jul 27 00:00 .backmeup
> 
> 
> The BSR file contains:
> 
> [dan@bacula:/usr/local/bacula/bsr] $ cat tallboy-fd_tallboy_home.bsr
> # 16-Jul-2020 16:19:59 - tallboy_home.2020-07-05_03.05.02_01 - Full, 
> since=2020-07-16 03:05:07
> Volume="HQ0024L4"
> MediaType="LTO4"
> Slot=3
> VolSessionId=32
> VolSessionTime=1594911018
> VolAddr=1791001367956-1791001368387
> FileIndex=1-91
> Volume="HQ0024L4"
> MediaType="LTO4"
> Slot=3
> VolSessionId=32
> VolSessionTime=1594911018
> VolAddr=1833951043364-1833951043364
> 

Re: [Bacula-users] BSR file not updated with every job

[Dan Langille]

On Mon, Jul 27, 2020, at 11:55 AM, Dan Langille wrote:

> There are tree other jobs on this host and their respective BSR files 
> are dated today.

*There are three other jobs ...



-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] BSR file not updated with every job

[Dan Langille]

On Sun, Jul 26, 2020, at 7:33 AM, Dan Langille wrote:
> > On Jul 24, 2020, at 6:00 PM, Martin Simmons  wrote:
> > 
> >>>>>> On Fri, 24 Jul 2020 09:09:12 -0400, Dan Langille said:
> >> 
> >> I'm using Bacula 9.6.5 on FreeBSD 12.1
> >> 
> >> I have noticed a BSR file which is not getting updated after every job run.
> >> ...
> >> This is the most recent job, from today:
> >> 
> >> 24-Jul 03:05 bacula-dir JobId 314258: Start Backup JobId 314258, 
> >> Job=tallboy_home.2020-07-24_03.05.00_08
> >> 24-Jul 03:05 bacula-dir JobId 314258: There are no more Jobs associated 
> >> with Volume "IncrAuto-7393". Marking it purged.
> >> 24-Jul 03:05 bacula-dir JobId 314258: All records pruned from Volume 
> >> "IncrAuto-7393"; marking it "Purged"
> >> 24-Jul 03:05 bacula-dir JobId 314258: Recycled volume "IncrAuto-7393"
> >> 24-Jul 03:05 bacula-dir JobId 314258: Using Device "vDrive-IncrFile-6" to 
> >> write.
> >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Recycled volume "IncrAuto-7393" 
> >> on File device "vDrive-IncrFile-6" (/usr/local/bacula/volumes/IncrFile), 
> >> all previous data lost.
> >> 24-Jul 03:05 bacula-dir JobId 314258: Max Volume jobs=1 exceeded. Marking 
> >> Volume "IncrAuto-7393" as Used.
> >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Elapsed time=00:00:03, Transfer 
> >> rate=0  Bytes/second
> >> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Sending spooled attrs to the 
> >> Director. Despooling 0 bytes ...
> >> 24-Jul 03:05 bacula-dir JobId 314258: Bacula bacula-dir 9.6.5 (11Jun20):
> >>  Build OS:   amd64-portbld-freebsd12.1 freebsd 12.1-RELEASE-p7
> >>  JobId:  314258
> >>  Job:tallboy_home.2020-07-24_03.05.00_08
> >>  Backup Level:   Incremental, since=2020-07-23 03:05:17
> >>  Client: "tallboy-fd" 9.6.5 (11Jun20) 
> >> amd64-portbld-freebsd12.1,freebsd,12.1-RELEASE-p7
> >>  FileSet:"tallboy home" 2012-12-20 14:19:02
> >>  Pool:   "IncrFile" (From Job IncPool override)
> >>  Catalog:"MyCatalog" (From Client resource)
> >>  Storage:"bacula-sd-01-IncrFile" (From Pool resource)
> >>  Scheduled time: 24-Jul-2020 03:05:00
> >>  Start time: 24-Jul-2020 03:05:17
> >>  End time:   24-Jul-2020 03:05:21
> >>  Elapsed time:   4 secs
> >>  Priority:   10
> >>  FD Files Written:   0
> >>  SD Files Written:   0
> >>  FD Bytes Written:   0 (0 B)
> >>  SD Bytes Written:   0 (0 B)
> > 
> > This job wrote nothing to the volume, so there is nothing to add to the BSR
> > file.
> 
> When I look at this post now, that's easy to see. I find myself 
> smirking to myself when it see it now. Thanks.
> 
> Why was this a problem to me? I'm backing up my BSR files with an rsync 
> to other hosts.
> 
> There is a monitoring script to make sure the rcync'd files are not 
> more than 48 hours old.
> 
> I have 'fixed' with with a crontab on the above referenced host:
> 
> 0   0   *   *   *   touch ~/.backmeup
> 
> Cheers

Today we have:

-rw-r-  1 bacula  bacula   1035 Jul 26 03:05 tallboy-fd_tallboy_home.bsr

The file updated by cron is:

[dan@tallboy:~] $ ls -l .backmeup 
-rw-r--r--  1 dan  dan  0 Jul 27 00:00 .backmeup


The BSR file contains:

[dan@bacula:/usr/local/bacula/bsr] $ cat tallboy-fd_tallboy_home.bsr
# 16-Jul-2020 16:19:59 - tallboy_home.2020-07-05_03.05.02_01 - Full, 
since=2020-07-16 03:05:07
Volume="HQ0024L4"
MediaType="LTO4"
Slot=3
VolSessionId=32
VolSessionTime=1594911018
VolAddr=1791001367956-1791001368387
FileIndex=1-91
Volume="HQ0024L4"
MediaType="LTO4"
Slot=3
VolSessionId=32
VolSessionTime=1594911018
VolAddr=1833951043364-1833951043364
FileIndex=91-130
# 18-Jul-2020 03:05:24 - tallboy_home.2020-07-18_03.05.01_50 - Incremental, 
since=2020-07-17 03:05:08
Volume="IncrAuto-11646"
MediaType="IncrFile"
VolSessionId=349
VolSessionTime=1594656307
VolAddr=257-14794
FileIndex=1-3
# 19-Jul-2020 03:05:14 - tallboy_home.2020-07-19_03.05.01_23 - Differential, 
since=2020-07-05 03:07:39
Volume="DiffAuto-1473"
MediaType="DiffFile"
VolSessionId=525
VolSessionTime=1594656307
VolAddr=235-19992
FileIndex=1-5
# 26-Jul-2020 03:05:12 - tallboy_home.2020-07-26_03.05.01_14 - Differential, 
since=2020-

Re: [Bacula-users] BSR file not updated with every job

[Dan Langille]

> On Jul 24, 2020, at 6:00 PM, Martin Simmons  wrote:
> 
>>>>>> On Fri, 24 Jul 2020 09:09:12 -0400, Dan Langille said:
>> 
>> I'm using Bacula 9.6.5 on FreeBSD 12.1
>> 
>> I have noticed a BSR file which is not getting updated after every job run.
>> ...
>> This is the most recent job, from today:
>> 
>> 24-Jul 03:05 bacula-dir JobId 314258: Start Backup JobId 314258, 
>> Job=tallboy_home.2020-07-24_03.05.00_08
>> 24-Jul 03:05 bacula-dir JobId 314258: There are no more Jobs associated with 
>> Volume "IncrAuto-7393". Marking it purged.
>> 24-Jul 03:05 bacula-dir JobId 314258: All records pruned from Volume 
>> "IncrAuto-7393"; marking it "Purged"
>> 24-Jul 03:05 bacula-dir JobId 314258: Recycled volume "IncrAuto-7393"
>> 24-Jul 03:05 bacula-dir JobId 314258: Using Device "vDrive-IncrFile-6" to 
>> write.
>> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Recycled volume "IncrAuto-7393" 
>> on File device "vDrive-IncrFile-6" (/usr/local/bacula/volumes/IncrFile), all 
>> previous data lost.
>> 24-Jul 03:05 bacula-dir JobId 314258: Max Volume jobs=1 exceeded. Marking 
>> Volume "IncrAuto-7393" as Used.
>> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Elapsed time=00:00:03, Transfer 
>> rate=0  Bytes/second
>> 24-Jul 03:05 bacula-sd-01-sd JobId 314258: Sending spooled attrs to the 
>> Director. Despooling 0 bytes ...
>> 24-Jul 03:05 bacula-dir JobId 314258: Bacula bacula-dir 9.6.5 (11Jun20):
>>  Build OS:   amd64-portbld-freebsd12.1 freebsd 12.1-RELEASE-p7
>>  JobId:  314258
>>  Job:tallboy_home.2020-07-24_03.05.00_08
>>  Backup Level:   Incremental, since=2020-07-23 03:05:17
>>  Client: "tallboy-fd" 9.6.5 (11Jun20) 
>> amd64-portbld-freebsd12.1,freebsd,12.1-RELEASE-p7
>>  FileSet:"tallboy home" 2012-12-20 14:19:02
>>  Pool:   "IncrFile" (From Job IncPool override)
>>  Catalog:"MyCatalog" (From Client resource)
>>  Storage:"bacula-sd-01-IncrFile" (From Pool resource)
>>  Scheduled time: 24-Jul-2020 03:05:00
>>  Start time: 24-Jul-2020 03:05:17
>>  End time:   24-Jul-2020 03:05:21
>>  Elapsed time:   4 secs
>>  Priority:   10
>>  FD Files Written:   0
>>  SD Files Written:   0
>>  FD Bytes Written:   0 (0 B)
>>  SD Bytes Written:   0 (0 B)
> 
> This job wrote nothing to the volume, so there is nothing to add to the BSR
> file.

When I look at this post now, that's easy to see. I find myself smirking to 
myself when it see it now. Thanks.

Why was this a problem to me? I'm backing up my BSR files with an rsync to 
other hosts.

There is a monitoring script to make sure the rcync'd files are not more than 
48 hours old.

I have 'fixed' with with a crontab on the above referenced host:

0   0   *   *   *   touch ~/.backmeup

Cheers

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] BSR file not updated with every job

[Dan Langille]

dex=91-130
# 18-Jul-2020 03:05:24 - tallboy_home.2020-07-18_03.05.01_50 - Incremental, 
since=2020-07-17 03:05:08
Volume="IncrAuto-11646"
MediaType="IncrFile"
VolSessionId=349
VolSessionTime=1594656307
VolAddr=257-14794
FileIndex=1-3
# 19-Jul-2020 03:05:14 - tallboy_home.2020-07-19_03.05.01_23 - Differential, 
since=2020-07-05 03:07:39
Volume="DiffAuto-1473"
MediaType="DiffFile"
VolSessionId=525
VolSessionTime=1594656307
VolAddr=235-19992
FileIndex=1-5
[dan@bacula:/usr/local/bacula/bsr] $ 

Ideas?

Thank you


-- 
  Dan Langille
  d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] SD fails to create new volumes: trace included

[Dan Langille]

Resending. Sorry, the previous email was incomplete.


On Tue, Jan 9, 2018, at 3:29 PM, Dan Langille wrote:
> > On Jan 9, 2018, at 3:07 PM, Dan Langille  wrote:
> > 
> >> On Jan 9, 2018, at 3:01 PM, Dan Langille  wrote:
> >> 
> >> I wonder if this line is related:
> >> 
> >> 
> >> bacula-sd-01-sd: mount.c:663-269534 No autolabel because polling.
> >> 
> >> If no labeling, then no volume is created.
> >> 
> >> See 
> >> https://gist.github.com/dlangille/513cb50351f7bc6ab6d33a6859b20326#file-bacula-trace-L84
> >> 
> >> 
> >> 
> >> bacula-sd-01-sd: mount.c:182-269534 want vol=FullAuto-8867 devvol= 
> >> dev="vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile)
> >> bacula-sd-01-sd: mount.c:190-269534 Try open Vol=FullAuto-8867
> >> bacula-sd-01-sd: file_dev.c:138-269534 Enter: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:113-269534 Enter: virtual bool 
> >> DEVICE::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:325-269534 Enter: virtual bool DEVICE::mount(int)
> >> bacula-sd-01-sd: file_dev.c:184-269534 open disk: mode=OPEN_READ_WRITE 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867, 0x2, 0640)
> >> bacula-sd-01-sd: file_dev.c:191-269534 open failed: file_dev.c:190 Could 
> >> not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> bacula-sd-01-sd: file_dev.c:211-269534 open dev: disk fd=-1 opened
> >> bacula-sd-01-sd: file_dev.c:214-269534 Leave: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: mount.c:192-269534 Try autolabel Vol=FullAuto-8867
> >> bacula-sd-01-sd: file_dev.c:138-269534 Enter: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:113-269534 Enter: virtual bool 
> >> DEVICE::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:325-269534 Enter: virtual bool DEVICE::mount(int)
> >> bacula-sd-01-sd: file_dev.c:184-269534 open disk: mode=OPEN_READ_WRITE 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867, 0x2, 0640)
> >> bacula-sd-01-sd: file_dev.c:191-269534 open failed: file_dev.c:190 Could 
> >> not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> bacula-sd-01-sd: file_dev.c:211-269534 open dev: disk fd=-1 opened
> >> bacula-sd-01-sd: file_dev.c:214-269534 Leave: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: mount.c:198-269534 open_device failed: ERR=file_dev.c:190 
> >> Could not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> bacula-sd-01-sd: mount.c:663-269534 No autolabel because polling.
> >> bacula-sd-01-sd: mount.c:215-269534 Open of File device 
> >> "vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile) Volume 
> >> "FullAuto-8867" failed: ERR=file_dev.c:190 Could not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> 
> >> bacula-sd-01-sd: mount.c:220-269534 Volume "FullAuto-8867" not loaded on 
> >> File device "vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile).
> > 
> > This is why it is not labelling, from bacula/src/stored/mount.c
> > 
> > int DCR::try_autolabel(bool opened)
> > {
> >   DCR *dcr = this;
> > 
> >   if (dev->poll && !dev->is_tape()) {
> >  Dmsg0(100, "No autolabel because polling.\n");
> >  return try_default;   /* if polling, don't try to create new 
> > labels */
> >   }
> > 
> 
> To get this backup running again:
> 
> I manually deleted the last volume which failed to create (it was in 
> the Catalog but not in the filesystem).
> 
> I manually labeled that Volume (which creates a Catalog entry and a 
> file on disk).
> 
> https://gist.github.com/dlangille/513cb50351f7bc6ab6d33a6859b20326#file-1319

This is still an issue.  This time it's on a new bacula-sd I have recently 
created. The bacula-sd configuration is based upon other SD instances I have.

*m
02-Dec 13:06 bacula-dir JobId 302468: Created new Volume="FullAuto-11080", 
Pool="FullFile-03", MediaType="FullFile03" in catalog.
02-Dec 13:06 bacula-sd-03 JobId 302468: Warning: mount.c:215 Open of File 
device &qu

Re: [Bacula-users] SD fails to create new volumes: trace included

[Dan Langille]

On Tue, Jan 9, 2018, at 3:29 PM, Dan Langille wrote:
> > On Jan 9, 2018, at 3:07 PM, Dan Langille  wrote:
> > 
> >> On Jan 9, 2018, at 3:01 PM, Dan Langille  wrote:
> >> 
> >> I wonder if this line is related:
> >> 
> >> 
> >> bacula-sd-01-sd: mount.c:663-269534 No autolabel because polling.
> >> 
> >> If no labeling, then no volume is created.
> >> 
> >> See 
> >> https://gist.github.com/dlangille/513cb50351f7bc6ab6d33a6859b20326#file-bacula-trace-L84
> >> 
> >> 
> >> 
> >> bacula-sd-01-sd: mount.c:182-269534 want vol=FullAuto-8867 devvol= 
> >> dev="vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile)
> >> bacula-sd-01-sd: mount.c:190-269534 Try open Vol=FullAuto-8867
> >> bacula-sd-01-sd: file_dev.c:138-269534 Enter: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:113-269534 Enter: virtual bool 
> >> DEVICE::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:325-269534 Enter: virtual bool DEVICE::mount(int)
> >> bacula-sd-01-sd: file_dev.c:184-269534 open disk: mode=OPEN_READ_WRITE 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867, 0x2, 0640)
> >> bacula-sd-01-sd: file_dev.c:191-269534 open failed: file_dev.c:190 Could 
> >> not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> bacula-sd-01-sd: file_dev.c:211-269534 open dev: disk fd=-1 opened
> >> bacula-sd-01-sd: file_dev.c:214-269534 Leave: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: mount.c:192-269534 Try autolabel Vol=FullAuto-8867
> >> bacula-sd-01-sd: file_dev.c:138-269534 Enter: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:113-269534 Enter: virtual bool 
> >> DEVICE::open_device(DCR *, int)
> >> bacula-sd-01-sd: dev.c:325-269534 Enter: virtual bool DEVICE::mount(int)
> >> bacula-sd-01-sd: file_dev.c:184-269534 open disk: mode=OPEN_READ_WRITE 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867, 0x2, 0640)
> >> bacula-sd-01-sd: file_dev.c:191-269534 open failed: file_dev.c:190 Could 
> >> not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> bacula-sd-01-sd: file_dev.c:211-269534 open dev: disk fd=-1 opened
> >> bacula-sd-01-sd: file_dev.c:214-269534 Leave: virtual bool 
> >> file_dev::open_device(DCR *, int)
> >> bacula-sd-01-sd: mount.c:198-269534 open_device failed: ERR=file_dev.c:190 
> >> Could not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> bacula-sd-01-sd: mount.c:663-269534 No autolabel because polling.
> >> bacula-sd-01-sd: mount.c:215-269534 Open of File device 
> >> "vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile) Volume 
> >> "FullAuto-8867" failed: ERR=file_dev.c:190 Could not 
> >> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640):
> >>  ERR=No such file or directory
> >> 
> >> bacula-sd-01-sd: mount.c:220-269534 Volume "FullAuto-8867" not loaded on 
> >> File device "vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile).
> > 
> > This is why it is not labelling, from bacula/src/stored/mount.c
> > 
> > int DCR::try_autolabel(bool opened)
> > {
> >   DCR *dcr = this;
> > 
> >   if (dev->poll && !dev->is_tape()) {
> >  Dmsg0(100, "No autolabel because polling.\n");
> >  return try_default;   /* if polling, don't try to create new 
> > labels */
> >   }
> > 
> 
> To get this backup running again:
> 
> I manually deleted the last volume which failed to create (it was in 
> the Catalog but not in the filesystem).
> 
> I manually labeled that Volume (which creates a Catalog entry and a 
> file on disk).
> 
> https://gist.github.com/dlangille/513cb50351f7bc6ab6d33a6859b20326#file-1319

This is still an issue.  This time it's on a new bacula-sd I have recently 
created. The bacula-sd configuration is based upon other SD instances I have.

*m
02-Dec 13:06 bacula-dir JobId 302468: Created new Volume="FullAuto-11080", 
Pool="FullFile-03", MediaType="FullFile03" in catalog.
02-Dec 13:06 bacula-sd-03 JobId 302468: Warning: mount.c:215 Open of File 
device "vDrive-FullFile-1" (/usr/local/bacula/volumes/

Re: [Bacula-users] Is there a test for LTO-4 tapes or just pull them from service?

[Dan Langille]

On Mon, May 6, 2019, at 10:46 PM, John R Zoch wrote:
> I'm running Bacula 9.4.2 on an FreeNAS 11.2-U3 in an iocage jail. Attached is 
> an image from bat indicating a bunch of failed tapes. They passed the rewind 
> - write EOF test. 
> 

The rewind-write EOF test is more a test of Bacula and the tape drive, not of 
the tape itself.

That image from BAT shows the status of that tape in the Catalog. The Error 
could be for reasons other than a bad tape.

To know for sure, you need to look at the job email, or the bacula log on the 
system itself (provided there is one).

> 
> Is there something I should try to make them usable again (a test/format of 
> sorts) or is the Bacula error indication sufficient that I should just pull 
> them from service?
> 
> I've used "btape qfill" in the past but it only tests the first tape in the 
> library. I'm not certain qfill verifies the data it wrote to the tape to 
> provide some assurance that the tape is good.

Show us some of these errors and we'll know for sure if it's a tape error. It 
could be drive loading error. It wanted tape X but couldn't load it, for 
whatever reason.___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] MySQL 8.0 mysql.c:201:4: error: unknown type name 'my_bool'

[Dan Langille]

> On Apr 27, 2019, at 4:03 PM, Carsten Leonhardt  wrote:
> 
> Hi Dan,
> 
>> I am setting up regression testing for MySQL 8.0 and getting this error on 
>> the build:
> 
>> /usr/home/dan/src/bacula/regress/build/libtool --silent --tag=CXX 
>> --mode=link /usr/bin/c++-o libbaccats.la cats_null.lo -export-dynamic 
>> -rpath /usr/home/dan/src/bacula/regress/bin -release 9.4.3
>> Compiling mysql.c
>> mysql.c:201:4: error: unknown type name 'my_bool'
>>   my_bool reconnect = 1;
>>   ^
>> 1 error generated.
>> *** Error code 1
> 
> MySQL removed "my_bool" in version 8, a patch is needed. For example:
> 
> see https://patches.ubuntu.com/b/bacula/bacula_9.4.2-2ubuntu1.patch
> 
> There might be cleverer approaches.
> 
> Regards,
> 
> Carsten

That got me farther into the compile.  Now it's:

cats_test.c:533:75: error: too many arguments provided to function-like macro 
invocation
   ok(db_get_file_list(jcr, jcr->db_batch, buf, false, false, list_files, ),
  ^
../cats/protos.h:224:9: note: macro 'db_get_file_list' defined here
#define db_get_file_list(jcr, mdb, jobids, opts, result_handler, ctx) \
^
1 error generated.
*** Error code 1

— 
Dan Langille
http://langille <http://langille/>.org/





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] bacula and SQLite

[Dan Langille]

On Wed, May 1, 2019, at 4:59 AM, Kern Sibbald wrote:
> Hello,
> 
> Just so everyone knows how I feel about SQLite -- it is a great program,
> written by a very good programmer.  It is probably closer to Postgres
> than to MySQL in its syntax. 

An aside...

There is a reason for that similarity.  When SQLite was being written,
PostgreSQL was the reference for what to do.

D. Richard Hipp gave the keynote talk at PGCon 2014.  I was present.

The slides are at: https://www.pgcon.org/2014/schedule/events/736.en.html

> However, it is really not an appropriate
> database for many reasons for Bacula, and maintaining the update tables
> scripts is very time consuming.  I will continue to keep the source code
> as long as Sven and others contribute what is necessary to keep it
> working.

In that talk, you will find these quotes:

“SQLite is not a replacement for PostgreSQL.
SQLite is a replacement for fopen().”

"Use PostgreSQL as an enterprise data depot, and use SQLite for application 
files."

The Catalog is definitely not an application file.

Please move from SQLite to PostgreSQL. You will be better off.

 


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] MySQL 8.0 mysql.c:201:4: error: unknown type name 'my_bool'

[Dan Langille]

Thank you. 

With that patch, we have MySQL 8 covered via regression testing. 

http://regress.bacula.org/index.php?project=Bacula-9.4=2019-04-27

FYI I apply that patch during testing. It needs to get into the code. 

--
 Dan Langille
 d...@langille.org



On Sat, Apr 27, 2019, at 4:20 PM, Carsten Leonhardt wrote:
> Hi Dan,
> 
> > I am setting up regression testing for MySQL 8.0 and getting this error on 
> > the build:
> 
> > /usr/home/dan/src/bacula/regress/build/libtool --silent --tag=CXX 
> > --mode=link /usr/bin/c++ -o libbaccats.la cats_null.lo -export-dynamic 
> > -rpath /usr/home/dan/src/bacula/regress/bin -release 9.4.3
> > Compiling mysql.c
> > mysql.c:201:4: error: unknown type name 'my_bool'
> > my_bool reconnect = 1;
> > ^
> > 1 error generated.
> > *** Error code 1
> 
> MySQL removed "my_bool" in version 8, a patch is needed. For example:
> 
> see https://patches.ubuntu.com/b/bacula/bacula_9.4.2-2ubuntu1.patch
> 
> There might be cleverer approaches.
> 
> Regards,
> 
> Carsten
> 
> 
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> 
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] MySQL 8.0 mysql.c:201:4: error: unknown type name 'my_bool'

[Dan Langille]

I am setting up regression testing for MySQL 8.0 and getting this error on the 
build:

Making libbacsql.la ...
/usr/home/dan/src/bacula/regress/build/libtool --silent --tag=CXX --mode=link 
/usr/bin/c++-o libbacsql.la bvfs.lo cats.lo sql.lo sql_cmds.lo 
sql_create.lo sql_delete.lo sql_find.lo sql_get.lo sql_list.lo sql_update.lo 
-export-dynamic -rpath /usr/home/dan/src/bacula/regress/bin -release 9.4.3 
Compiling cats_null.c
Making libbaccats.la ...
/usr/home/dan/src/bacula/regress/build/libtool --silent --tag=CXX --mode=link 
/usr/bin/c++-o libbaccats.la cats_null.lo -export-dynamic -rpath 
/usr/home/dan/src/bacula/regress/bin -release 9.4.3
Compiling mysql.c
mysql.c:201:4: error: unknown type name 'my_bool'
   my_bool reconnect = 1;
   ^
1 error generated.
*** Error code 1

mysql80-client-8.0.15
mysql80-server-8.0.15

   C++ Compiler: /usr/bin/c++ version
   Compiler flags:-g -O2 -Wall -x c++ -fno-strict-aliasing 
-fno-exceptions -fno-rtti
   Linker flags:  
   Libraries:-lpthread 
   Statically Linked Tools:  no
   Statically Linked FD: no
   Statically Linked SD: no
   Statically Linked DIR:no
   Statically Linked CONS:   no
   Database backends:MySQL
   Database port: 
   Database name:regress
   Database user:regress
   Database SSL options: 

   SMTP Host Address:localhost

   Director Port:8101
   File daemon Port: 8102
   Storage daemon Port:  8103

   Director User:
   Director Group:   
   Storage Daemon User:  
   Storage DaemonGroup:  
   File Daemon User: 
   File Daemon Group:

   Large file support:   yes
   Bacula conio support: yes -ltinfo
   readline support: no 
   TCP Wrappers support: yes -lwrap
   TLS support:  yes
   Encryption support:   yes
   ZLIB support: yes
   LZO support:  no
   enable-smartalloc:yes
   enable-lockmgr:   yes
   bat support:  no
   client-only:  no
   build-dird:   yes
   build-stored: yes
   Plugin support:   yes
   AFS support:  no
   ACL support:  yes
   XATTR support:yes
   systemd support:  no 
   Batch insert enabled: MySQL

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [External] Re: [Bacula-devel] IBAdmin

[Dan Langille]

> On Jan 28, 2019, at 1:19 PM, Radosław Korzeniewski 
>  wrote:
> 
> Hello Kern,
> 
> pt., 25 sty 2019 o 16:47 Kern Sibbald  <mailto:k...@sibbald.com>> napisał(a):
> Hello guys,
> 
> Interesting conversation.  I thought I would throw in some general comments 
> of my own.
> 
> - I really like seeing another GUI for Bacula, because it is something we 
> really need.
> 
> Thank you Kern, I really appreciate your support. You are the only one who 
> sees it as an opportunity and not a threat.

I think that's unfair.

I hope you have not taken the suggestions provided here are the result of 
viewing IBAdmin as a threat. They are not.

I hope you have not taken the suggestions for improvement as an attack on 
IBAdmin. They are not.

It is relatively straight forward, I should hope, to identify the parts of the 
database you need to read and the parts you need to update.

I recently did similar for another application:

It looks something like this:

create role freshsource_ro;
GRANT SELECT ON TABLE public.commit_log TO freshsource_ro;
GRANT SELECT ON TABLE public.commit_log_elements TO freshsource_ro;
GRANT SELECT ON TABLE public.element TO freshsource_ro;
GRANT SELECT ON TABLE public.latest_commits TO freshsource_ro;
GRANT SELECT ON TABLE public.repo TO freshsource_ro;
GRANT SELECT ON TABLE public.security_notice TO freshsource_ro;
GRANT SELECT ON TABLE public.system TO freshsource_ro;
GRANT SELECT ON TABLE public.users TO freshsource_ro;
GRANT UPDATE(cookie) ON TABLE public.users TO freshsource_ro;
GRANT UPDATE(lastlogin) ON TABLE public.users TO freshsource_ro;
GRANT SELECT ON TABLE public.watch_list TO freshsource_ro;
GRANT SELECT ON TABLE public.watch_list_element TO freshsource_ro;
GRANT SELECT ON TABLE public.watch_notice TO freshsource_ro;

Then a user is created and added to that freshsource_ro role:

create user freshsource_dev with password '[redacted]' IN ROLE freshsource_ro;

Nobody sees the application itself as threat.

These are straight forward security practices which are carried out in many 
organizations, both small and large.

Best wishes.

--
Dan Langille - BSDCan / PGCon
d...@langille.org



signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [Bacula-devel] IBAdmin

[Dan Langille]

> On Jan 24, 2019, at 5:30 AM, Radosław Korzeniewski 
>  wrote:
> 
> Hello Dan,
> 
> śr., 23 sty 2019 o 23:59 Dan Langille  <mailto:d...@langille.org>> napisał(a):
>> On Jan 23, 2019, at 4:56 PM, Radosław Korzeniewski 
>> mailto:rados...@korzeniewski.net>> wrote:
>> 
>> 
>> Does IBAdmin work with a readonly connection to the Catalog database?
>> 
>> 
>> No.
> 
> What features will be available with a read-only connection?
> 
> 
> Why do you need a read-only database?

Isolation.

Bacula manipulates the Catalog. IBAdmin reports upon it.  Why would it need to 
modify my Catalog?

> All management is based on database objects. When you want to create or 
> change any configuration it will change/update database objects for you. When 
> you log in your session is stored in a database. These are the most basic 
> operations which require read-write access to database objects.

OK, but all of this must be in the Bacula Catalog? If so, why? Why not put 
IBAdmin-specific data in a separate database?

> I assume you want to test it, right?

Yes, I do, I am interested, but not if I must give it write access to my Bacula 
database.

--
Dan Langille - BSDCan / PGCon
d...@langille.org





signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [Bacula-devel] IBAdmin

[Dan Langille]

> On Jan 23, 2019, at 4:56 PM, Radosław Korzeniewski 
>  wrote:
> 
> 
> Does IBAdmin work with a readonly connection to the Catalog database?
> 
> 
> No.

What features will be available with a read-only connection?



signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [Bacula-devel] IBAdmin

[Dan Langille]

> On Jan 22, 2019, at 6:19 AM, Radosław Korzeniewski 
>  wrote:
> 
> Hello Bacula Users!
> 
> There are a lot of different tools for managing a Bacula system, some GUI, 
> some for reports only and every new week we have a new one in the wild (just 
> kidding)! So I thought I'll introduce you another one. :) More GUI for 
> Bacula, more fun.
> 
> The GUI is called IBAdmin (Inteos Backup Administration GUI) and is developed 
> by Inteos for its BEE customers. The IBAdmin application is available for 
> community (IBAdmin Community version) for a few months now with AGPL v3 
> license. IBAdmin Community support Bacula Enterprise Edition 8.x-10.x and 
> Bacula Community 9.2.x. You can check the code at 
> https://github.com/inteos/IBAdmin <https://github.com/inteos/IBAdmin>, its 
> Wiki at https://github.com/inteos/IBAdmin/wiki 
> <https://github.com/inteos/IBAdmin/wiki> and especially the installation 
> (https://github.com/inteos/IBAdmin/wiki/Installation-procedure 
> <https://github.com/inteos/IBAdmin/wiki/Installation-procedure>) or upgrade 
> (https://github.com/inteos/IBAdmin/wiki/Upgrade-procedure 
> <https://github.com/inteos/IBAdmin/wiki/Upgrade-procedure>) procedures if you 
> run a previous version. 
> 
> What is IBAdmin, someone may ask? IBAdmin is a new Bacula Enterprise or 
> Community administration GUI dedicated to Inteos Backup users. But currently 
> as a general purpose tool it can be used by any BEE or BC enthusiasts. The 
> main goal for IBAdmin is simplicity, so you don't need to know much about 
> Bacula or Bacula Enterprise to start using it, just an overall backup 
> architecture. Thanks to this any newbie learning curve should be extremely 
> short, i.e. all the configuration and management is hidden as IBAdmin. 
> IBAdmin allows you to prepare and manage your backup solution as a whole 
> appliance, something you can find at other products (sort of, some 
> functionality is still missing). The IBAdmin Community support file level 
> backup only for Bacula Community and BEE which is included in the latest 
> release. The latest release brings a lot of new features including an 
> advanced users management feature with roles based access control and 
> departments support as a multitenant solution.
> I hope you will find it useful.

Can you elaborate upon the systemd requirement please?

I see: "Without systemd you can't manage system daemons and its logs directly 
from application and this part of ibadmin won't work"

By manage, what do you mean? Stop, start, etc? What does IBAdmin do with the 
logs?

Does IBAdmin work with a readonly connection to the Catalog database?

When a new client is created, does IBAdmin create the bacula-fd.conf file?

Thank you.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Backing up to NetApp

[Dan Langille]

Have you backed up to NetApp?

At $WORK we are likely to soon have heaps of NetApp storage at our disposal.

Of course my thoughts turned to backups.

Do you use a NetApp appliance with Bacula as a destination for backups?

I know of the NetApp plugin for Bacula, but that is the wrong direction: that 
is for backing up the NetApp device.

I've never mounted remote storage for bacula-sd over any of NFS, CIF, Samba, 
etc.

I can't imagine NFS would be useful given the throughput.  Mind you, I don't 
yet know how much we'll be backing up, but it'll be more than 1

Have you?

--
Dan Langille - BSDCan / PGCon
d...@langille.org




signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] check_bacula Segmentation fault if host is down

[Dan Langille]

> On Jan 18, 2019, at 11:24 AM, Josh Fisher  wrote:
> 
> On 1/17/2019 12:17 PM, Dan Langille wrote:
> 
>> I was noticing this in my /var/log/messages:
>> 
>> Jan 17 17:04:00 slocum kernel: pid 52623 (check_bacula), uid 181: exited on 
>> signal 11
>> Jan 17 17:04:21 slocum kernel: pid 53805 (check_bacula), uid 181: exited on 
>> signal 11
>> 
>> I tracked it down to a host which was not up:
>> 
>> $ time ./check_bacula -H tape02 -D fd -M nagios-mon -K '[redacted]'
>> Segmentation fault
>> 
>> real 1m15.101s
>> user 0m0.006s
>> sys  0m0.000s
>> 
>> 
>> Could someone else please try to replicate this situation for me please?
>> 
>> This check is being run on FreeBSD 12.0-RELEASE-p2 with check_bacula from 
>> Bacula 9.2.2
>> 
>> Thank you.
>> 
> Have you tried with debug level increased, say using -d7 in the command line? 
> Maybe that would give a clue as to what it does prior to the segfault.
> 
This was interesting:


$ time ./check_bacula -H tape02 -D fd -M nagios-mon -K '[redacted]' -d7
Segmentation fault

real1m15.137s
user0m0.000s
sys 0m0.004s

Note the time, similar to the first test.

Let's up the debug level

$ time ./check_bacula -H tape02 -D fd -M nagios-mon -K '[redacted]' -d77
check_bacula: bsockcore.c:384-0 Could not connect to server File daemon 
tape02:9102. ERR=Operation timed out
check_bacula: bsockcore.c:197-0 Unable to connect to File daemon on 
tape02:9102. ERR=Operation timed out
Segmentation fault

real1m15.024s
user0m0.000s
sys 0m0.004s


Ok debugging info.

Let's bump up:

$ time ./check_bacula -H tape02 -D fd -M nagios-mon -K '[redacted]' -d777
check_bacula: bsockcore.c:299-0 Current 10.55.0.110:9102 All 10.55.0.110:9102
check_bacula: bsockcore.c:384-0 Could not connect to server File daemon 
tape02:9102. ERR=Host is down
check_bacula: bsockcore.c:197-0 Unable to connect to File daemon on 
tape02:9102. ERR=Host is down
Segmentation fault

real0m0.013s
user0m0.000s
sys 0m0.004s


What: zero time?

Further tests with -d777, -d77, and -d7 all finished in near-zero time.  I 
tried again without -d, and it failed in 75 seconds, like the others.

Try again with -d (a larger value than previously tried), near-zero time 
again.

Then I waited and tried again.

[dan@webserver:/usr/local/libexec/nagios] $ time ./check_bacula -H tape02 -D fd 
-M nagios-mon -K '[redacted' -d
check_bacula: bsockcore.c:299-0 Current 10.55.0.110:9102 All 10.55.0.110:9102
check_bacula: bsockcore.c:384-0 Could not connect to server File daemon 
tape02:9102. ERR=Operation timed out
check_bacula: bsockcore.c:197-0 Unable to connect to File daemon on 
tape02:9102. ERR=Operation timed out
check_bacula: watchdog.c:82-0 Initialising NicB-hacked watchdog thread
check_bacula: watchdog.c:197-0 Registered watchdog 800c38098, interval 300 one 
shot
check_bacula: watchdog.c:254-0 NicB-reworked watchdog thread entered
check_bacula: watchdog.c:296-0 pthread_cond_timedwait 60
check_bacula: btimers.c:177-0 Start bsock timer 800c135e8 tid=800c15000 for 300 
secs at 1547849285
check_bacula: btimers.c:212-0 Stop bsock timer 800c135e8 tid=800c15000 at 
1547849285.
check_bacula: watchdog.c:217-0 Unregistered watchdog 800c38098
check_bacula: watchdog.c:296-0 pthread_cond_timedwait 60
Segmentation fault

real1m15.035s
user0m0.000s
sys 0m0.004s


--
Dan Langille - BSDCan / PGCon
d...@langille.org





signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] check_bacula Segmentation fault if host is down

[Dan Langille]

I was noticing this in my /var/log/messages:

Jan 17 17:04:00 slocum kernel: pid 52623 (check_bacula), uid 181: exited on 
signal 11
Jan 17 17:04:21 slocum kernel: pid 53805 (check_bacula), uid 181: exited on 
signal 11

I tracked it down to a host which was not up:

$ time ./check_bacula -H tape02 -D fd -M nagios-mon -K '[redacted]'
Segmentation fault

real1m15.101s
user0m0.006s
sys 0m0.000s


Could someone else please try to replicate this situation for me please?

This check is being run on FreeBSD 12.0-RELEASE-p2 with check_bacula from 
Bacula 9.2.2

Thank you.

--
Dan Langille - BSDCan / PGCon
d...@langille.org




signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] FreeBSD: 7.x ports deprecated - expire 2019-03-01

[Dan Langille]

Hello,

I write with my plans for the Bacula 7.x packages on FreeBSD.

Bacula 7.x was last updated 20 months ago. Bacula 9.x came out shortly 
thereafter.

(re https://sourceforge.net/projects/bacula/files/bacula/)

On all the 7.x ports, I plan to set:

DEPRECATED=Unsupported, please install Bacula 9.x

and

EXPIRATION_DATE=2019-03-01

I chose that date because it is more than 3 months from now, and still gives 
users the FreeBSD quarterly ports branch until 2019Q2.

Those changes should appear later this week.

Thank you.

--
Dan Langille - BSDCan / PGCon
d...@langille.org




signature.asc
Description: Message signed with OpenPGP
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] FileSet estimate listing sorting

[Dan Langille]

> On Sep 23, 2018, at 5:27 AM, George Anchev via Bacula-users 
>  wrote:
> 
> On Sun, 23 Sep 2018 10:42:20 +0200 Kern Sibbald wrote:
> 
>> Direct the output the estimate listing to a file,
>> then sort it any way you want.
> 
> I actually do this but I am afraid it is not that
> simple because sorting it e.g. by filename (including
> path) or by date would require additional scripting.
> Hence the whole question.
> 
> I was hoping that the listing could be sorted by
> Bacula itself.

What use cases exist for sorting the estimate?




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Retaining files indefinitely ?

[Dan Langille]

> On Aug 28, 2018, at 2:54 AM, Chris Wilkinson  wrote:
> 
> One of my requirements for backups is that the last version of every file 
> should remain in the backup storage indefinitely or at least for a long time.
> 
> Could I achieve this by writing "File Retention = 10 years" in the pool or 
> client definition? 

Yes. But you'll also need Volume and Job retention also set to 10 years if you 
want to easily restore that data.

This will keep all records, not just the latest.

Yes, this will use more database records, but my view is that this is a cheap 
price to pay.

Also, as mentioned by others, does this requirement mean that a file deleted 6 
years ago needs to be kept for another 4 years?

You'll need to clarify that 10 years meets the "indefinitely" requirement.

Given the originally stated requirements, it seems your only solution is not to 
ever recycle volumes or allow the Catalog to be pruned.

> My concern is that, if this is possible, then it would stop automatic volume 
> truncation and recycling and eventually result in out of space storage errors.

Yes, you need to have enough storage.


-- 
Dan Langille - BSDCan / PGCon
d...@langille.org



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] any way to "collapse" back to a single director?

[Dan Langille]

> On Aug 11, 2018, at 10:47 AM, Phil Stracchino  wrote:
> 
> On 08/09/18 11:47, Matthew Arguin wrote:
>> But then you lose the historical stuff?  Was hoping for a way to sort of
>> migrate everything from one to another.  I will say that I don’t expect
>> that this is doable with out more work than it is worth.
> 
> 
> Moving the clients is easy.  Moving the historical data is hard.  As
> others have said, you must do it manually.  You would have to start out
> by changing the numeric IDs of every Job, every Volume, every Pool,
> every Client, every Fileset, every Schedule, every type of resource, in
> the catalog of the Director(s) you are planning to shut down to values
> not used by the one you're trying to keep, and you need to do it
> CONSISTENTLY, making sure every single record in the database refers to
> every other record it should by the correct *NEW* row IDs.  Then you
> need to import that data into the Catalog you're keeping without
> overwriting any data you already have.
> 
> None of this is anything you should be attempting unless you have strong
> SQL database skills and understand how the Bacula catalog tables relate
> to each other.  And of course, in all cases you should back up all of
> your catalog databases first.
> 
> 
> By comparison, you will probably find it a lot simpler to just add your
> Clients and their Filesets and Schedules to the Director you're keeping
> and restart it, then keep the other Directors around as backups in case
> you need them for restores until their historical data becomes outdated.

So far, thought has been given to merging the database.

Consider not merging them, but keeping them.

My idea:

* You can shutdown the other directors.  The historical data is in a database.

* You can keep the databases.

* A given director can using multiple Catalogs.

This might be a line of investigation.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy Job to Another Catalog

[Dan Langille]

> On Jul 20, 2018, at 6:26 PM, Dan Langille  wrote:
> 
> Signed PGP part
>> On Jul 19, 2018, at 1:41 PM, Jari Fredriksson  wrote:
>> 
>> Signed PGP part
>> 
>> 
>>> Heitor Faria  kirjoitti 18.7.2018 kello 20.42:
>>> 
>>> Dear Users,
>>> 
>>> I'm planning to deploy a Copy Job for Geographical Redundancy Disaster 
>>> Recovery (Site A, Site B).
>>> Failover site (B) has a secondary Bacula Director, Catalog and Storage 
>>> Daemon.
>>> Do you think it is possible to perform Copy jobs from Site A to Site B, 
>>> using the failover Catalog as the metadata repository?
>>> 
>> 
>> Hello
>> 
>> As far as I know a Copy job is not possible to be running on two storage 
>> daemons. That is what I learned when I had a need for it.
> 
> That was once the case, it is no longer.  I copy jobs between two bacula-sd 
> now.
> 

It changed in 7.0.0: 
http://www.bacula.org/9.0.x-manuals/en/main/New_Features_in_7_0_0.html#SECTION00611000

The documentation is incorrect: 
http://www.bacula.org/9.0.x-manuals/en/main/Migration_Copy.html#SECTION00283

"Migration is only implemented for a single Storage daemon. You cannot read on 
one Storage daemon and write on another."

--
Dan Langille - BSDCan / PGCon
d...@langille.org





signature.asc
Description: Message signed with OpenPGP
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy Job to Another Catalog

[Dan Langille]

> On Jul 19, 2018, at 1:41 PM, Jari Fredriksson  wrote:
> 
> Signed PGP part
> 
> 
>> Heitor Faria  kirjoitti 18.7.2018 kello 20.42:
>> 
>> Dear Users,
>> 
>> I'm planning to deploy a Copy Job for Geographical Redundancy Disaster 
>> Recovery (Site A, Site B).
>> Failover site (B) has a secondary Bacula Director, Catalog and Storage 
>> Daemon.
>> Do you think it is possible to perform Copy jobs from Site A to Site B, 
>> using the failover Catalog as the metadata repository?
>> 
> 
> Hello
> 
> As far as I know a Copy job is not possible to be running on two storage 
> daemons. That is what I learned when I had a need for it.

That was once the case, it is no longer.  I copy jobs between two bacula-sd now.



signature.asc
Description: Message signed with OpenPGP
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] disaster recovery - full OS?

[Dan Langille]

> On Jun 18, 2018, at 10:59 AM, czezz  wrote:
> 
> Hi,
> Does bacula offer full disaster recovery?

http://www.bacula.org/9.0.x-manuals/en/main/Disaster_Recovery_Using_Bac.html 
<http://www.bacula.org/9.0.x-manuals/en/main/Disaster_Recovery_Using_Bac.html>

That said, I always prefer to backup data and let me take care of setting up 
the system.  The backup may have occurred on one type of filesystem, parition 
layout, OS, and I want to restore to another, for example.

> What I have read up until now it seems that Bacula backups only files and 
> directories.
> It seems that in case full system restoration, fresh installation of OS is 
> required.
> 
> Is this correct?
> Is ther an option to completely image OS like eg. Clonezilla offers (I know 
> it is offline backup though)

Everything specializes in something.  Create your priorities, make your 
choices.  if getting a new system back up and running ASAP is your priority, 
restoring from backup is your biggest problem. Deployment is.

If you like what Clonezilla does, make sure it does everything you want.  For 
example, do you want, and does it do, incrementals, differentials, can it be 
used to restore the 400GB file which was just deleted.

Can it gives you a list of files in the backup, and their checksums, and 
timestamps so you can decide which one to restore?

This is just a small list of things to consider.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula h/w write fails, but tar writes w/out error?

[Dan Langille]

> On Jun 7, 2018, at 11:38 AM, Stieneke, Dan  wrote:
> 
> The job ate through 4 tapes, with only 2 – 60GB on each tape. Then it hit 
> recycle limits and was asking for more media.
>  
> These are used tapes, but I can’t see 4 consecutive tapes going bad at the 
> same time.
>  
> Incidentally, this is the same behavior I saw 4 months ago, and at that time 
> I did test bacula to a brand-new tape, which also failed quickly.


If it is all tapes, is the issue with the tape drive?
-- 
Dan Langille - BSDCan / PGCon
d...@langille.org

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] RunAfterJob

[Dan Langille]

> On Jun 6, 2018, at 10:48 AM, Donna Hofmeister  wrote:
> 
> In a job{} definition, is it possible/advisable/wise to have multiple 
> RunAfterJob statements?
> 
> If so, does N+1 wait on N to complete?  - d

I understand it is possible.

My preference would be just one. Make is a /bin/sh script and do everything in 
there.

Then you know.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Accidentally deleted a running jobid during a backup - Any hint on how to address?

[Dan Langille]

> On Jun 3, 2018, at 3:42 PM, Pedro Oliveira  <mailto:oliveira...@gmail.com>> wrote:
> 
> No dia 03/06/2018, às 16:13, Dan Langille  <mailto:d...@langille.org>> escreveu:
> 
>>> On Jun 3, 2018, at 1:23 AM, Ben Wong >> <mailto:benjamin.w...@blur.compbio.ucsf.edu>> wrote:
>>> 
>>> Hello, 
>>> 
>>> I was trying to clean up my failed jobs from my catalog but I happened to 
>>> accidentally delete the jobid of a running job.  Stupid, I know.  From what 
>>> I can see, the backup looks like it succeeded, it only appears failed due 
>>> to bacula being unable to find original jobid because I deleted it.  
>>> 
>>> Is there a simple way to retrieve that catalog data for the jobid I 
>>> deleted?  The backup is written across four LTO7 tape volumes.  I was 
>>> thinking of redoing the backup and not deleting the jobid but maybe there's 
>>> an option  where I don't have to rewrite all that data?  
>>> 
>>> Here's the error I get:  JobID 245 is the job I accidentally deleted.  
>>> 02-Jun 20:38 bacula-dir JobId 245: Warning: Error updating job record. 
>>> sql_update.c:202 Update failed: affected_rows=0 for UPDATE Job SET 
>>> JobStatus='T',EndTime='2018-06-02 
>>> 20:38:50',ClientId=8,JobBytes=40063503226963,ReadBytes=40063503226963,JobFiles=2684608,JobErrors=28,VolSessionId=8,VolSessionTime=1527615994,PoolId=4,FileSetId=8,JobTDate=1527997130,RealEndTime='2018-06-02
>>>  20:38:50',PriorJobId=0,HasBase=0,PurgedFiles=0 WHERE JobId=245
>>> 02-Jun 20:38 bacula-dir JobId 245: Warning: Error getting Job record for 
>>> Job report: ERR=sql_get.c:318 No Job found for JobId 245 
>> 
>> If you really need it, perhaps bextract will help.
>> 
>> Keep a fresh backup of your Catalog.

> Use Bscan to scan the volume with bsr file 
> 

Yes,  bscan, not bextract.  Sorry.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org <mailto:d...@langille.org>


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Accidentally deleted a running jobid during a backup - Any hint on how to address?

[Dan Langille]

> On Jun 3, 2018, at 1:23 AM, Ben Wong  
> wrote:
> 
> Hello, 
> 
> I was trying to clean up my failed jobs from my catalog but I happened to 
> accidentally delete the jobid of a running job.  Stupid, I know.  From what I 
> can see, the backup looks like it succeeded, it only appears failed due to 
> bacula being unable to find original jobid because I deleted it.  
> 
> Is there a simple way to retrieve that catalog data for the jobid I deleted?  
> The backup is written across four LTO7 tape volumes.  I was thinking of 
> redoing the backup and not deleting the jobid but maybe there's an option  
> where I don't have to rewrite all that data?  
> 
> Here's the error I get:  JobID 245 is the job I accidentally deleted.  
> 02-Jun 20:38 bacula-dir JobId 245: Warning: Error updating job record. 
> sql_update.c:202 Update failed: affected_rows=0 for UPDATE Job SET 
> JobStatus='T',EndTime='2018-06-02 
> 20:38:50',ClientId=8,JobBytes=40063503226963,ReadBytes=40063503226963,JobFiles=2684608,JobErrors=28,VolSessionId=8,VolSessionTime=1527615994,PoolId=4,FileSetId=8,JobTDate=1527997130,RealEndTime='2018-06-02
>  20:38:50',PriorJobId=0,HasBase=0,PurgedFiles=0 WHERE JobId=245
> 02-Jun 20:38 bacula-dir JobId 245: Warning: Error getting Job record for Job 
> report: ERR=sql_get.c:318 No Job found for JobId 245 

If you really need it, perhaps bextract will help.

Keep a fresh backup of your Catalog.


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Intervention needed for backup

[Dan Langille]

> On May 26, 2018, at 5:39 PM, Heitor Faria <hei...@bacula.com.br> wrote:
> 
> Hello Dan and Donna,
> On May 25, 2018, at 1:31 PM, Donna Hofmeister <do...@allegro.com 
> <mailto:do...@allegro.com>> wrote:
> 
> 
> On Thu, May 24, 2018 at 9:26 AM, Donna Hofmeister <do...@allegro.com 
> <mailto:do...@allegro.com>> wrote:
> 
> Second interesting question -- given my "Label Format" statement, I would 
> expect my files to be named something like this: Vol-minerva.allegro.com 
> <http://vol-minerva.allegro.com/>-fd-File-20180517-2305-73-19. So why are my 
> files named: MinervaBackup.2018-05-23_23.05.00_53?
> 
> To answer my own question
> 
> I think the underlying issue was I had made several changes to the "label 
> format" statement while I've been testing Bacula...and had inadvertantly 
> created this situation.
> 
> To solve this, I wiped out my bacula storage environment and all seems to be 
> working well.
> 
> So -- reasking my second question -- if you change the "label format" 
> statement, is there additional steps after that?- donna 
> 
> 
> 
> Issue an update in bconsole.
> I think the update is not necessary when changing the Volume Format since 
> prior created volumes are already created and Bacula won't rename them.

How will bacula-dir know about the changed label format without issuing the 
update command?

> I also think Bacula could criticize erratic volume names when reloading. E.g. 
> a misspelled variable ${Poll} is accepted, and the user will only find the 
> error when a Job gets stalled unable to create new volumes.

Sanity testing? Sounds good.

> Finally, I think the Bacula configuration default Label Format should be 
> ${Pool}-${NumVols}. Vol- works, but it becomes ugly when you use multiple 
> Pools such as Daily, Weekly etc. since Volume numeration matches the MediaID. 
> Having the ${Pool} name on the Volume might also help to distinguish them in 
> a Bacula disaster recovery scenario.


I have never subscribed to the LOTS OF INFO IN THE VOLUME LABEL school of 
thought.  I use Pool names:

$ grep 'LabelFormat' *
pools.conf:  LabelFormat = "FullAuto-"
pools.conf:  LabelFormat = "DiffAuto-"
pools.conf:  LabelFormat = "IncrAuto-"
pools.conf:  LabelFormat = "IncrAutoNoNextPool-"
pools.conf:  LabelFormat = "FullAutoNoNextPool-"
pools.conf:  LabelFormat = "TwoHourly-"
pools.conf:  LabelFormat = "Daily-"
pools.conf:  LabelFormat = "Weekly-"
pools.conf:  LabelFormat = "Monthly-"

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Intervention needed for backup

[Dan Langille]

> On May 25, 2018, at 1:31 PM, Donna Hofmeister <do...@allegro.com> wrote:
> 
> 
> On Thu, May 24, 2018 at 9:26 AM, Donna Hofmeister <do...@allegro.com 
> <mailto:do...@allegro.com>> wrote:
> 
> Second interesting question -- given my "Label Format" statement, I would 
> expect my files to be named something like this: 
> Vol-minerva.allegro.com-fd-File-20180517-2305-73-19. So why are my files 
> named: MinervaBackup.2018-05-23_23.05.00_53?
> 
> To answer my own question
> 
> I think the underlying issue was I had made several changes to the "label 
> format" statement while I've been testing Bacula...and had inadvertantly 
> created this situation.
> 
> To solve this, I wiped out my bacula storage environment and all seems to be 
> working well.
> 
> So -- reasking my second question -- if you change the "label format" 
> statement, is there additional steps after that?- donna 
> 


Issue an update in bconsole.

The change will affect only future Volumes. Existing Volumes are never affected.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula 7+mysql

[Dan Langille]

> On May 23, 2018, at 1:02 PM, Chris Wilkinson <winstonia...@gmail.com> wrote:
> 
> This will seem a dumb question to the experienced Bacula user. I would prefer 
> to use mysql rather than the default sqlite. During installation (Debian 
> Stretch repo, Bacula v7.4.4) I said no to configuring sqlite expecting to be 
> offered the opportunity to configure mysql instead. That didn't happen. I 
> have created an empty bacula database with the appropriate user/permissions 
> but can't figure out how to make Bacula populate it or connect.

That sounds like a Debian packaging problem.

> 
> At present the director fails to start of course.
> 
> An extract from journalctl -xe follows indicating it is looking for sqlite.
> 
> -- Unit bacula-director.service has begun starting up.
> May 23 17:40:41 HOMESERVER bacula-dir[16139]: bacula-dir: dird.c:969-0 Could 
> not open Catalog "MyCatalog", databasMay 23 17:40:41 HOMESERVER 
> bacula-dir[16139]: bacula-dir: dird.c:974-0 sqlite.c:199 Database 
> /var/lib/bacula/XXX_DMay 23 17:40:41 HOMESERVER bacula-dir[16139]: 23-May 
> 17:40 bacula-dir ERROR TERMINATION
> May 23 17:40:41 HOMESERVER bacula-dir[16139]: Please correct configuration 
> file: /etc/bacula/bacula-dir.conf
> May 23 17:40:41 HOMESERVER systemd[1]: bacula-director.service: Control 
> process exited, code=exited status=1
> May 23 17:40:41 HOMESERVER systemd[1]: Failed to start Bacula Director Daemon 
> service.
> -- Subject: Unit bacula-director.service has failed
> -- Defined-By: systemd
> -- Support: https://www.debian.org/support <https://www.debian.org/support>
> --
> -- Unit bacula-director.service has failed.
> --
> -- The result is failed.
> May 23 17:40:41 HOMESERVER systemd[1]: bacula-director.service: Unit entered 
> failed state.
> May 23 17:40:41 HOMESERVER systemd[1]: bacula-director.service: Failed with 
> result 'exit-code'.
> May 23 17:47:01 HOMESERVER minissdpd[716]: 4 new devices added
> lines 2591-2644/2644 (END)
> 
> 
> This seems to indicate that bacula-dir is wrong but I haven't found an 
> example of what it should be.
> 
> Any help with that would be great.

Does this help? 
http://www.bacula.org/7.4.x-manuals/en/main/Installing_Configuring_MySQ.html 
<http://www.bacula.org/7.4.x-manuals/en/main/Installing_Configuring_MySQ.html>
Bacula will not create the required tables etc. You must do that.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] ERR=Operation timed out

[Dan Langille]

> On May 16, 2018, at 2:44 PM, Elam <elm.c...@gmail.com> wrote:
> 
> Hello everyone,
> 
> I have recently received the following error for a bacula job:
> 
> Fatal error: Network error with FD during Backup: ERR=Operation timed out
> Fatal error: No Job status returned from FD.
> 
> bacula starts the backup job, it runs for a few mins, ( I could see the 
> backup progress) but it terminate all of a sudden while reporting the above 
> error.
> 

This sounds like network issues.  What does 'status client' show for this FD?

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] BSCAN - what to expect?

[Dan Langille]

> On Apr 7, 2018, at 5:30 PM, Thomas plancon <tplan...@bkaarchs.com> wrote:
> 
> Hi Dan,
> 
> Thanks for your interest!
> 
> So I let bscan run  for almost 48 hours and then thought I should try and see 
> if any records were added to the database: nothing was added except a job 
> number and the volume name! I cancelled bscan and went into bconsole to 
> verify and yes, nothing except as mentioned above.

One thing to consider: automated pruning.  I recall previous posts saying 
records would get added, but later get pruned because they are too old.

> I tried bls, and that was actually pulling file names off of the tape and I 
> redirected it to a text file. From that info I ran bextract with the name of 
> a project retrieved with bls. Bextract successfully restored the project to a 
> local directory!!!
> 
> So now I'm letting bls run on the tapes and collecting the list in text 
> files. Hoping I'll see the project I need to recover. Unfortunately, bls is 
> also a very slow process; 48 hours and still not through listing a LTO-2, 400 
> Gb, tape; I've got 3 tapes to go through!

I think that is because, like bscan, it reads the whole tape.

> It is frustrating because I'm not even sure the needed project is even on 
> these tapes. The backup was full, so I'm just deducing, and hoping, (that 
> could be on a t-shirt: deducing and hoping), that it is there somewhere.
> 
> I'll post again in a day or so with the status.

Good.


-- 
Dan Langille - BSDCan / PGCon
d...@langille.org



> 
> Tom Plancon
> 
> 
> 
> On 4/7/2018 3:57 PM, Dan Langille wrote:
>>> On Apr 3, 2018, at 11:56 AM, Tom Plancon <tplan...@bkaarchs.com> wrote:
>>> 
>>> Hi folks,
>>> 
>>> I'm running BSCAN to recover old data from LTO-2 tapes. The backup job 
>>> spanned 3 tapes/volumes. I ran the BSCAN command listing the volumes as 
>>> required with the first tape in the drive - this is NOT an autochanger. 
>>> BSCAN seemed to start OK, found the tape, dbase etc., and has been running 
>>> now for about 24hrs with the first tape still in the drive!
>> bscan does take a very long time. I have no personal experience with it 
>> though.
>> 
>>> So, my questions, will BSCAN ask for the next tape when the first is done? 
>>> Or, how will I know how to change it? Should it be taking this long for a 
>>> 400Gb tape? The tape drive moves for a few seconds, pauses a few seconds 
>>> and moves again, now for a full day!
>>> 
>>> I'm trying to recover a specific project in this old data, there is 
>>> probably over 100 projects saved on these tapes. Is there a better way to 
>>> do this?
>>> 
>>> Any help appreciated! Thanks much!
>> It's been a few days. Any progress to report?
>> 
>> 
> 


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Autochanger not changing

[Dan Langille]

> On Apr 1, 2018, at 11:12 AM, Sebastian Suchanek <bac...@suchanek.de> wrote:
> 
> Hi everyone,
> 
> while still setting up Bacula (v5.2.6), I've come accross a rather
> strange issue with my tape library (an Overland NEO2000) with a
> LTO4-FC-Drive. When I do "fill" test with btape over multiple tapes,
> everything works fine. Also, for example, when I do a "label barcodes"
> command in btape over multiple slots, the individual tapes are moved
> into the drive, get written and are put back into their original slot.
> 
> But when I run a "real" backup job which spans more than one tape, the
> job gets stuck as soon as the end of the first tape is reached. The last
> messages from the job are:
> 
> | [...]
> | 2018-04-01 14:16:42 tigersclaw-sd JobId 111: Volume "SU4103L4"
> previously written, moving to end of data.
> | 2018-04-01 15:55:32 tigersclaw-sd JobId 111: End of Volume 
> "SU4103L4"
> at 21:2548 on device "LTO4-Drive-1" (/dev/nst0). Write of 200 bytes
> got -1.
> 
> And after that, nothing happens anymore. Not even if I wait for half an
> hour.
> 
> Here's the (hopefully) relevant part of my bacula-sd.conf:
> 

[snip]

> 
> Any idea what exactly is going wrong here? What can I do to track down
> the reason for this error? What can I do to fix it?

It is usually permissions. It it nearly always permission.

I have documented my experiences here: https://dan.langille.org/

Search for mtx and mtx-loader.  There are several configuration step and low 
level testing at the command line.


This might be your best bet: 
https://dan.langille.org/2016/02/04/dell-tl4000-tape-library-running-btape-and-configuring-baculas-mtx-changer/

start slow.  Make sure mtx-changer works. If you need to customize it, move it 
away from /etc/bacula/scripts/mtx-changer ... I put mine at /usr/local/sbin 
well away from the installed one.

Make sure all the steps with mtx-changer work: load, unload, loaded, list

Then make sure they work as the bacula user (your SD should be running as 
bacula, not root; there is no need for bacula-sd to run as root).

To get mtx-changer working, first make sure mtx works.

Try what I tried and make sure it all works.

HTH


-- 
Dan Langille - BSDCan / PGCon
d...@langille.org



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] BSCAN - what to expect?

[Dan Langille]

> On Apr 3, 2018, at 11:56 AM, Tom Plancon  wrote:
> 
> Hi folks,
> 
> I'm running BSCAN to recover old data from LTO-2 tapes. The backup job 
> spanned 3 tapes/volumes. I ran the BSCAN command listing the volumes as 
> required with the first tape in the drive - this is NOT an autochanger. BSCAN 
> seemed to start OK, found the tape, dbase etc., and has been running now for 
> about 24hrs with the first tape still in the drive!

bscan does take a very long time. I have no personal experience with it though.

> So, my questions, will BSCAN ask for the next tape when the first is done? 
> Or, how will I know how to change it? Should it be taking this long for a 
> 400Gb tape? The tape drive moves for a few seconds, pauses a few seconds and 
> moves again, now for a full day! 
> 
> I'm trying to recover a specific project in this old data, there is probably 
> over 100 projects saved on these tapes. Is there a better way to do this?
> 
> Any help appreciated! Thanks much!

It's been a few days. Any progress to report?



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Using LibreSSL?

[Dan Langille]

> On Feb 9, 2018, at 1:34 PM, Martin Simmons <mar...@lispworks.com> wrote:
> 
>>>>>> On Wed, 7 Feb 2018 20:33:49 -0500, Dan Langille said:
>> 
>>> On Nov 30, 2017, at 11:41 AM, Martin Simmons <mar...@lispworks.com> wrote:
>>> 
>>> It looks like LibreSSL defines OPENSSL_VERSION_NUMBER as 0x2000L, i.e. 
>>> as
>>> if it was OpenSSL 2.0.  Bacula uses this variable to detect OpenSSL >= 1.1,
>>> which causes it to compile the 1.1 code when using LibreSSL, even though
>>> LibreSSL only claims to provide the API from OpenSSL 1.0.
>>> 
>>> Bacula probably needs to detect LibreSSL (e.g. look for
>>> LIBRESSL_VERSION_NUMBER) and treat that as OpenSSL 1.0 regardless of
>>> OPENSSL_VERSION_NUMBER.
>> 
>> This seems to fit closely with a patch provided to FreeBSD: 
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223994
>> 
>> I could file a bug and pass this directly to the Bacula project, and in the 
>> meantime, patch the FreeBSD port.
>> 
>> Both will take time and I'm preoccupied with conferences just now.
> 
> That patch looks incomplete though (there are a few other uses of
> OPENSSL_VERSION_NUMBER).

New patches have been uploaded to 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223994

Does that look better?

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Using LibreSSL?

[Dan Langille]

> On Nov 30, 2017, at 11:41 AM, Martin Simmons <mar...@lispworks.com> wrote:
> 
> It looks like LibreSSL defines OPENSSL_VERSION_NUMBER as 0x2000L, i.e. as
> if it was OpenSSL 2.0.  Bacula uses this variable to detect OpenSSL >= 1.1,
> which causes it to compile the 1.1 code when using LibreSSL, even though
> LibreSSL only claims to provide the API from OpenSSL 1.0.
> 
> Bacula probably needs to detect LibreSSL (e.g. look for
> LIBRESSL_VERSION_NUMBER) and treat that as OpenSSL 1.0 regardless of
> OPENSSL_VERSION_NUMBER.

This seems to fit closely with a patch provided to FreeBSD: 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223994

I could file a bug and pass this directly to the Bacula project, and in the 
meantime, patch the FreeBSD port.

Both will take time and I'm preoccupied with conferences just now.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




> 
> 
> 
>>>>>> On Thu, 30 Nov 2017 09:53:33 -0500, Dan Langille said:
>> 
>> Are you using LibreSSL with Bacula 9.0.6?
>> 
>> I ask because of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223994
>> 
>> re:
>> 
>> 
>> --- openssl.lo ---
>> Compiling openssl.c
>> --- crypto.lo ---
>> crypto.c:199:1: error: unknown type name 'DEFINE_STACK_OF'
>> DEFINE_STACK_OF(SignerInfo);
>> ^
>> crypto.c:200:1: error: unknown type name 'DEFINE_STACK_OF'
>> DEFINE_STACK_OF(RecipientInfo);
>> ^
>> crypto.c:334:21: error: use of undeclared identifier 'ASN1_STRING_get0_data';
>> did you mean 'ASN1_STRING_data'?
>>  ext_value_data = ASN1_STRING_get0_data(asn1_ext_val);
>>   ^
>>   ASN1_STRING_data
>> /usr/local/include/openssl/asn1.h:787:17: note: 'ASN1_STRING_data' declared
>> here
>> unsigned char * ASN1_STRING_data(ASN1_STRING *x);
>>   ^
>> crypto.c:334:43: error: cannot initialize a parameter of type 'ASN1_STRING *'
>> (aka 'asn1_string_st *') with an lvalue of type 'const ASN1_STRING *' (aka
>> 'const asn1_string_st *')
>>  ext_value_data = ASN1_STRING_get0_data(asn1_ext_val);
>> 
>> 
>> -- 
>> Dan Langille - BSDCan / PGCon
>> d...@langille.org
>> 
>> 
>> 
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> ___
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>> 
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

> On Feb 7, 2018, at 1:04 PM, Dimitri Maziuk <dmaz...@bmrb.wisc.edu> wrote:
> 
> On 02/07/2018 11:57 AM, Dan Langille wrote:
>> Either Bacula is in charge of the changer or it is not.
> 
> Only when it's running a job. If access is not concurrent, then it's
> very likely the tool not cleaning up after itself properly or something.

What I mean: either use Bacula or not use Bacula.

The other tool is taking hold of the device and not releasing.  That is 
interfering with backups.

So.. make a decision: Bacula or not.

Then go from there.

--
Dan Langille - BSDCan / PGCon
d...@langille.org





signature.asc
Description: Message signed with OpenPGP
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

Either Bacula is in charge of the changer or it is not.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org


> On Feb 7, 2018, at 12:48 PM, Kenneth Garges <gar...@ucsc.edu> wrote:
> 
> That might be the issue. Somebody does something in the web interface that 
> takes the changer offline, clicks OK at the warning message. Then later when 
> it’s back online FreeBSD doesn’t figure that out and needs a reboot. 
> 
> Probably an issue that should be addressed with FreeBSD rather than Bacula.
> 
> 
> 
> 
>> On 6Feb 2018, at 6:50 PM, Alan Brown <a.br...@ucl.ac.uk> wrote:
>> 
>> On 07/02/18 02:01, Kenneth Garges wrote:
>>> Permissions and ownership are ok. I think I figured out the problem.
>>> 
>>> Both mtx and chio would work for a while, then stop working returning only 
>>> “Inappropriate ioctl for device” or other errors. The culprit I think is a 
>>> library management tool by Quantum. The library provides a web interface 
>>> that allows you to check status, move tapes around, and various utilities. 
>>> 
>>> Problem is if you use that web interface it gets the server (both Bacula SD 
>>> and Director on the same FreeBSD box) wedged such that mtx and chio always 
>>> fail.
>> 
>> Yes - and if it's about to do anything which takes the changer offline
>> the web ui gives a warning message and asked if you want to continue.
>> 
>>> Unknown to me, the operations staff had been using that web interface while 
>>> I was testing. After I asked them to stop my system seems to work reliably. 
>> 
>> I hope you changed the password and put the changer in its own IP subnet
>> after that.
>> 
>> I wrote some kludgy shell scripts (originally for Neo4000, then Neo 8000
>> and now Quantum i500 - portable as far as I know) which automate loading
>> and unloading of the changer (imported tapes are scattered randomly
>> around available slots to ensure even wear and tear) - these mean that
>> all operations staff have to do is open the magazine to load/unload
>> tapes when emailed to do so by Bacula.
>> 
>> Dan can probably improve them considerably.
>> 
>> As an adjunct to that another script works out which tapes are oldest
>> (if in the scratch pool) or about to expire (if none are in the scratch
>> pool) and asks staff to pull them out of the safe.
>> 
>> I also submitted some chages to the mtx-changer script a while ago which
>> check that the changer's actually ready before attempting to send it
>> commands. Kern's been sitting on them for a couple of years.
>> 
>> 
>> There are some pretty whizzy things you can do to monitor tape health
>> when a tape is in the changer using the sg_attr and sg_logs commands
>> that go _far _ beyond Tapealart or smartmontools capabilities.
>> 
>> I've submitted a number of updates to sg_utils and am in the process of
>> (slowly) reverse engineering what I can from the MAMs that's not well
>> documented using text output provided by veritape (proprietary, windows
>> tool from mptapes.com which reads MAMs).
>> 
>> Selfish motivation is (of course) to reduce the amount of time I have to
>> spend manually scanning tapes with the windows program/standalone
>> scanner then interpreting output. All of that information is available
>> when the tape is in the drive and with appropriate MAM reads at
>> load/unload it's possible to have the bacula server tell me when a tape
>> is approaching end of life (this is far more accurate than the metrics
>> bacula uses of simple load cycles.) - and more importantly, when tape
>> drives are going bad.
>> 
>> Having spent several man-years dealing with the fallout of bad LTO tapes
>> damaging drives and then those drives damaging tapes, I want to minimize
>> the pain if it ever happens again.
>> 
>> (LTO drives even return detailed information about the condition of each
>> of the heads if queried the right way. This data is interpreted by IBM
>> or HP's proprietary tools to give reports but they use the same queries
>> as sg_logs does to get it)
>> 
> 


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

> On Feb 6, 2018, at 9:01 PM, Kenneth Garges <gar...@ucsc.edu> wrote:
> 
> Permissions and ownership are ok. I think I figured out the problem.
> 
> Both mtx and chio would work for a while, then stop working returning only 
> “Inappropriate ioctl for device” or other errors. The culprit I think is a 
> library management tool by Quantum. The library provides a web interface that 
> allows you to check status, move tapes around, and various utilities. 
> 
> Problem is if you use that web interface it gets the server (both Bacula SD 
> and Director on the same FreeBSD box) wedged such that mtx and chio always 
> fail.
> 
> Unknown to me, the operations staff had been using that web interface while I 
> was testing. After I asked them to stop my system seems to work reliably. 
> 
> Thanks to all that chimed in on this thread. It was very helpful.

Wow, that would have been hard to track down.  How did you stumble across that?


-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org


> On Feb 6, 2018, at 7:07 AM, Alan Brown <a.br...@ucl.ac.uk> wrote:
> 
> On 03/02/18 13:45, Dan Langille wrote:
>>> *ahem*
>>> 
>>> MTX is _only_ designed to interface to scsi generic devices. Unless 
>>> /dev/pass42 is another name for the generic device you'll have problems.
>> Can confirm /dev/pass* devices on FreeBSD are the right thing to use. From 
>> https://dan.langille.org/2016/02/04/dell-tl4000-tape-library-running-btape-and-configuring-baculas-mtx-changer/
> 
> What about permissions and ownership vs what bacula-sd is running as?

Good point.

I see that's not covered in the above, but is here: 
https://dan.langille.org/2016/02/07/using-a-dell-tl4000-with-baculas-bacula-sd-on-freebsd-with-zfs/

Look for /etc/group
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

> On Feb 2, 2018, at 7:07 AM, Alan Brown  wrote:
> 
> On 31/01/18 03:23, Kenneth Garges wrote:
>> I’m having trouble getting a tape library to work. Individual tape drive 
>> seems ok but the library sometimes makes btape hang requiring a reboot. Or 
>> it fails.
> 
> 
> >  # Define a Virtual autochanger
> 
> Remove anything you're not actually using.  They're just going to add 
> confusion.
> 
>> # Quantum Scalar i80 tape library at UCSC
>> # Hostname of library: core-prod-jb-2
>> #
>> Autochanger {
>>   Name = "core-prod-jb-2"
>>   Device = "Drive1"
>> # Device = "Drive2"
>>   Changer Command = "/usr/local/etc/bacula/mtx-changer %c %o %S %a %d"
>> #  Changer Device = /dev/ch0
>>   Changer Device = /dev/pass42
>> }
> 
> *ahem*
> 
> MTX is _only_ designed to interface to scsi generic devices. Unless 
> /dev/pass42 is another name for the generic device you'll have problems.

Can confirm /dev/pass* devices on FreeBSD are the right thing to use. From 
https://dan.langille.org/2016/02/04/dell-tl4000-tape-library-running-btape-and-configuring-baculas-mtx-changer/

$ sudo camcontrol devlist
 at scbus0 target 0 lun 0 (pass0,sa0)
 at scbus0 target 0 lun 1 (pass1,ch0)
 at scbus0 target 1 lun 0 (pass2,sa1)
 at scbus0 target 1 lun 1 (pass3,ch1)
at scbus1 target 0 lun 0 (ada0,pass4)
at scbus3 target 0 lun 0 (ada1,pass5)

CAM is the Common Access Method for the SCSI/ATA subsystem: 
https://www.freebsd.org/cgi/man.cgi?query=cam=4=FreeBSD+11.0-RELEASE


$ sudo mtx -f /dev/pass0 inquiry
Product Type: Tape Drive
Vendor ID: 'IBM '
Product ID: 'ULT3580-HH4 '
Revision: 'C7Q1'
Attached Changer API: No
 
$ sudo tapeinfo -f /dev/pass0
Product Type: Tape Drive
Vendor ID: 'IBM '
Product ID: 'ULT3580-HH4 '
Revision: 'C7Q1'
Attached Changer API: No
SerialNumber: '1K10012996'
MinBlock: 1
MaxBlock: 16777215
Ready: yes
BufferedMode: yes
Medium Type: 0x48
Density Code: 0x46
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: -1
Partition 0 Size in Kbytes: -1
ActivePartition: 0
EarlyWarningSize: 0
 
$ sudo tapeinfo -f /dev/pass1
Product Type: Medium Changer
Vendor ID: 'IBM '
Product ID: '3573-TL '
Revision: 'B.60'
Attached Changer API: No
SerialNumber: '00X4U78G4272_LL0'
Ready: yes


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

> On Feb 1, 2018, at 6:42 PM, Kenneth Garges <gar...@ucsc.edu> wrote:
> 
> This is really weird. 
> 
> First mtx works:
> 
> # mtx -f /dev/pass42 status
>   Storage Changer /dev/pass42:2 Drives, 54 Slots ( 5 Import/Export )
> Data Transfer Element 0:Empty
> Data Transfer Element 1:Full (Storage Element 1 Loaded):VolumeTag = 000600
>   
>   Storage Element 1:Empty:VolumeTag=
>   Storage Element 2:Empty:VolumeTag=
>   Storage Element 3:Empty:VolumeTag=
>   Storage Element 4:Empty:VolumeTag=
>   Storage Element 5:Empty:VolumeTag=
>   Storage Element 6:Empty:VolumeTag=
>   Storage Element 7:Empty:VolumeTag=
>   Storage Element 8:Empty:VolumeTag=
>   Storage Element 9:Empty:VolumeTag=
>   Storage Element 10:Empty:VolumeTag=
>   Storage Element 11:Empty:VolumeTag=
>   Storage Element 12:Empty:VolumeTag=
>   Storage Element 13:Empty:VolumeTag=
>   Storage Element 14:Empty:VolumeTag=
>   Storage Element 15:Full :VolumeTag=B1  
>   Storage Element 16:Full :VolumeTag=B2  
>   Storage Element 17:Full :VolumeTag=B3  
>   Storage Element 18:Full :VolumeTag=B4  
>   Storage Element 19:Empty:VolumeTag=
>   Storage Element 20:Empty:VolumeTag=
>   Storage Element 21:Empty:VolumeTag=
>   Storage Element 22:Empty:VolumeTag=
>   Storage Element 23:Empty:VolumeTag=
>   Storage Element 24:Full :VolumeTag=B5  
>   Storage Element 25:Empty:VolumeTag=
>   Storage Element 26:Empty:VolumeTag=
>   Storage Element 27:Empty:VolumeTag=
>   Storage Element 28:Empty:VolumeTag=
>   Storage Element 29:Empty:VolumeTag=
>   Storage Element 30:Empty:VolumeTag=
>   Storage Element 31:Empty:VolumeTag=
>   Storage Element 32:Empty:VolumeTag=
>   Storage Element 33:Empty:VolumeTag=
>   Storage Element 34:Empty:VolumeTag=
>   Storage Element 35:Empty:VolumeTag=
>   Storage Element 36:Empty:VolumeTag=
>   Storage Element 37:Empty:VolumeTag=
>   Storage Element 38:Empty:VolumeTag=
>   Storage Element 39:Empty:VolumeTag=
>   Storage Element 40:Empty:VolumeTag=
>   Storage Element 41:Empty:VolumeTag=
>   Storage Element 42:Empty:VolumeTag=
>   Storage Element 43:Empty:VolumeTag=
>   Storage Element 44:Empty:VolumeTag=
>   Storage Element 45:Empty:VolumeTag=
>   Storage Element 46:Empty:VolumeTag=
>   Storage Element 47:Empty:VolumeTag=
>   Storage Element 48:Empty:VolumeTag=
>   Storage Element 49:Empty:VolumeTag=
>   Storage Element 50 IMPORT/EXPORT:Empty:VolumeTag=   
>  
>   Storage Element 51 IMPORT/EXPORT:Empty:VolumeTag=   
>  
>   Storage Element 52 IMPORT/EXPORT:Empty:VolumeTag=   
>  
>   Storage Element 53 IMPORT/EXPORT:Empty:VolumeTag=   
>  
>   Storage Element 54 IMPORT/EXPORT:Empty:VolumeTag=   
>      
> 
> Try it again immediately afterwards and it doesn’t:

I suspect timing.  Set the sleep option in mtx-changer's config

It can take a while for a tape to settle and for the tape drive to be ready.
-- 
Dan Langille - BSDCan 

Re: [Bacula-users] Problems getting a tape library to work under FreeBSD

[Dan Langille]

ote 1 EOF to "Drive1" (/dev/nsa0)
> btape: btape.c:582-0 Rewound "Drive1" (/dev/nsa0)
> btape: btape.c:1644-0 Now forward spacing 1 file.
> We should be in file 1. I am at file 1. This is correct!
> btape: btape.c:1656-0 Now forward spacing 2 files.
> We should be in file 3. I am at file 3. This is correct!
> btape: btape.c:582-0 Rewound "Drive1" (/dev/nsa0)
> btape: btape.c:1669-0 Now forward spacing 4 files.
> We should be in file 4. I am at file 4. This is correct!
> 
> btape: btape.c:1687-0 Now forward spacing 1 more file.
> We should be in file 5. I am at file 5. This is correct!
> 
> === End Forward space files test ===
> 
> 
> Ah, I see you have an autochanger configured.
> To test the autochanger you must have a blank tape
>  that I can write on in Slot 1.
> 
> Do you wish to continue with the Autochanger test? (y/n): y
> 
> 
> === Autochanger test ===
> 
> 3301 Issuing autochanger "loaded" command.
> 3991 Bad autochanger command: /usr/local/etc/bacula/mtx-changer /dev/pass42 
> loaded 1 /dev/nsa0 0
> 3991 result="mtx: Request Sense: Long Report=yes
> ": ERR=Child exited with code 1
> You must correct this error or the Autochanger will not work.
> *quit
> btape: dev.c:271-0 close_dev vol= fd=3 dev=802c6bc28 adata=0 dev="Drive1" 
> (/dev/nsa0)
> Pool   Maxsize  Maxused  Inuse
> NoPool  2568  0
> NAME1300  0
> FNAME   256   11  0
> MSG   645124  0
> EMSG   10243  0
> BSOCK  40960  0
> 
> # /usr/local/etc/bacula/mtx-changer /dev/pass42 loaded 1 /dev/nsa0 0
> mtx: Request Sense: Long Report=yes
> mtx: Request Sense: Valid Residual=no
> mtx: Request Sense: Error Code=0 (Unknown?!)
> mtx: Request Sense: Sense Key=No Sense
> mtx: Request Sense: FileMark=no
> mtx: Request Sense: EOM=no
> mtx: Request Sense: ILI=no
> mtx: Request Sense: Additional Sense Code = 00
> mtx: Request Sense: Additional Sense Qualifier = 00
> mtx: Request Sense: BPV=no
> mtx: Request Sense: Error in CDB=no
> mtx: Request Sense: SKSV=no
> INQUIRY command Failed
> 
> The FreeBSD version of mtx also fails. 
> 
> # /usr/local/sbin/mtx -f /dev/pass42 inquiry
> mtx: Request Sense: Long Report=yes
> mtx: Request Sense: Valid Residual=no
> mtx: Request Sense: Error Code=0 (Unknown?!)
> mtx: Request Sense: Sense Key=No Sense
> mtx: Request Sense: FileMark=no
> mtx: Request Sense: EOM=no
> mtx: Request Sense: ILI=no
> mtx: Request Sense: Additional Sense Code = 00
> mtx: Request Sense: Additional Sense Qualifier = 00
> mtx: Request Sense: BPV=no
> mtx: Request Sense: Error in CDB=no
> mtx: Request Sense: SKSV=no
> INQUIRY Command Failed
> # /usr/local/sbin/mtx -f /dev/pass42 status
> mtx: Request Sense: Long Report=yes
> mtx: Request Sense: Valid Residual=no
> mtx: Request Sense: Error Code=0 (Unknown?!)
> mtx: Request Sense: Sense Key=No Sense
> mtx: Request Sense: FileMark=no
> mtx: Request Sense: EOM=no
> mtx: Request Sense: ILI=no
> mtx: Request Sense: Additional Sense Code = 00
> mtx: Request Sense: Additional Sense Qualifier = 00
> mtx: Request Sense: BPV=no
> mtx: Request Sense: Error in CDB=no
> mtx: Request Sense: SKSV=no
> INQUIRY command Failed
> 
> 
> And using /dev/ch0 instead of /dev/pass42 doesn’t seem to help.
> 
> # /usr/local/sbin/mtx -f /dev/ch0 status
> cannot open SCSI device '/dev/ch0' - Input/output error
> 
> # /usr/local/etc/bacula/mtx-changer /dev/ch0 loaded 1 /dev/nsa0 0
> cannot open SCSI device '/dev/ch0' - Input/output error

I have documented most of my setup procedure on FreeBSD.

Getting mtx working first.  Then mtx-changers.  Verify permissions on devices 
for the bacula user.

These might help, based on https://dan.langille.org/?s=mtx-changer=

https://dan.langille.org/2016/02/04/dell-tl4000-tape-library-running-btape-and-configuring-baculas-mtx-changer/

This example has more on permissions:

https://www.freebsddiary.org/tape-library-integration.php

Hope that gets you started.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] problems with storage daemon

[Dan Langille]

> On Jan 11, 2018, at 11:56 AM, Jerry Lowry <michaiah2...@gmail.com> wrote:
> 
> Hi,  Last weekend I ran through a bunch of updates for my backup server.  Of 
> the many updates it updated the kernel and MariaDB ( 10.2.8 -> 10.2.12 ). I 
> have recently upgraded bacula to 9.0.6.  The problem is that the following 
> error:
>  10-Jan 20:36 distress-dir JobId 34724: Fatal error: Bad response to Storage 
> command: wanted 2000 OK storage
> , got 2902 Bad storage
> 
> I also get errors saying that the client can not connect to the storage 
> server:
> 
> 11-Jan 01:37 Denial-fd: BackupDenial.2018-01-11_00.05.01_44 Fatal error: 
> ../../lib/bnet.c:870 Unable to connect to Storage daemon on 
> distress.ACCOUNTING.EDT.LOCAL:9103. ERR=No error
> 11-Jan 01:38 distress-dir JobId 34726: Fatal error: Socket error on Storage 
> command: ERR=No data available
> 11-Jan 01:38 distress-dir JobId 34726: Fatal error: No Job status returned 
> from FD.
> 
> nothing was changed in the config files, just the updates were installed. I 
> can ping the storage server from all of the clients.
> 
> The jobs get to the point where they display "waiting for client"  but the 
> jobs never complete.
> 

When you did those update, did you keep all bacula-dir and all bacula-sd on the 
same exact version?

The general rule for software versions is:

bacula-dir = bacula-sd >= bacula-fd

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] SD fails to create new volumes: trace included

[Dan Langille]

> On Jan 9, 2018, at 3:07 PM, Dan Langille <d...@langille.org> wrote:
> 
>> On Jan 9, 2018, at 3:01 PM, Dan Langille <d...@langille.org> wrote:
>> 
>> I wonder if this line is related:
>> 
>> 
>> bacula-sd-01-sd: mount.c:663-269534 No autolabel because polling.
>> 
>> If no labeling, then no volume is created.
>> 
>> See 
>> https://gist.github.com/dlangille/513cb50351f7bc6ab6d33a6859b20326#file-bacula-trace-L84
>> 
>> 
>> 
>> bacula-sd-01-sd: mount.c:182-269534 want vol=FullAuto-8867 devvol= 
>> dev="vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile)
>> bacula-sd-01-sd: mount.c:190-269534 Try open Vol=FullAuto-8867
>> bacula-sd-01-sd: file_dev.c:138-269534 Enter: virtual bool 
>> file_dev::open_device(DCR *, int)
>> bacula-sd-01-sd: dev.c:113-269534 Enter: virtual bool 
>> DEVICE::open_device(DCR *, int)
>> bacula-sd-01-sd: dev.c:325-269534 Enter: virtual bool DEVICE::mount(int)
>> bacula-sd-01-sd: file_dev.c:184-269534 open disk: mode=OPEN_READ_WRITE 
>> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867, 0x2, 0640)
>> bacula-sd-01-sd: file_dev.c:191-269534 open failed: file_dev.c:190 Could not 
>> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640): 
>> ERR=No such file or directory
>> bacula-sd-01-sd: file_dev.c:211-269534 open dev: disk fd=-1 opened
>> bacula-sd-01-sd: file_dev.c:214-269534 Leave: virtual bool 
>> file_dev::open_device(DCR *, int)
>> bacula-sd-01-sd: mount.c:192-269534 Try autolabel Vol=FullAuto-8867
>> bacula-sd-01-sd: file_dev.c:138-269534 Enter: virtual bool 
>> file_dev::open_device(DCR *, int)
>> bacula-sd-01-sd: dev.c:113-269534 Enter: virtual bool 
>> DEVICE::open_device(DCR *, int)
>> bacula-sd-01-sd: dev.c:325-269534 Enter: virtual bool DEVICE::mount(int)
>> bacula-sd-01-sd: file_dev.c:184-269534 open disk: mode=OPEN_READ_WRITE 
>> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867, 0x2, 0640)
>> bacula-sd-01-sd: file_dev.c:191-269534 open failed: file_dev.c:190 Could not 
>> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640): 
>> ERR=No such file or directory
>> bacula-sd-01-sd: file_dev.c:211-269534 open dev: disk fd=-1 opened
>> bacula-sd-01-sd: file_dev.c:214-269534 Leave: virtual bool 
>> file_dev::open_device(DCR *, int)
>> bacula-sd-01-sd: mount.c:198-269534 open_device failed: ERR=file_dev.c:190 
>> Could not 
>> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640): 
>> ERR=No such file or directory
>> bacula-sd-01-sd: mount.c:663-269534 No autolabel because polling.
>> bacula-sd-01-sd: mount.c:215-269534 Open of File device "vDrive-FullFile-8" 
>> (/usr/local/bacula/volumes/FullFile) Volume "FullAuto-8867" failed: 
>> ERR=file_dev.c:190 Could not 
>> open(/usr/local/bacula/volumes/FullFile/FullAuto-8867,OPEN_READ_WRITE,0640): 
>> ERR=No such file or directory
>> 
>> bacula-sd-01-sd: mount.c:220-269534 Volume "FullAuto-8867" not loaded on 
>> File device "vDrive-FullFile-8" (/usr/local/bacula/volumes/FullFile).
> 
> This is why it is not labelling, from bacula/src/stored/mount.c
> 
> int DCR::try_autolabel(bool opened)
> {
>   DCR *dcr = this;
> 
>   if (dev->poll && !dev->is_tape()) {
>  Dmsg0(100, "No autolabel because polling.\n");
>  return try_default;   /* if polling, don't try to create new labels 
> */
>   }
> 

To get this backup running again:

I manually deleted the last volume which failed to create (it was in the 
Catalog but not in the filesystem).

I manually labeled that Volume (which creates a Catalog entry and a file on 
disk).

https://gist.github.com/dlangille/513cb50351f7bc6ab6d33a6859b20326#file-1319

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users