Re: [Bacula-users] Bacula Verification Procedures

2017-08-31 Thread Bill Arlofski 
On 08/30/2017 09:31 PM, Jim Richardson wrote:
> So I am looking to the community to help with the following questions to save
> me from trial and error:

Hi Jim!


>  1. Is there a verification job or a sequence of jobs “built in” to Bacula
> that can accomplish an automated restore with an automated verification of
> Volume to Restore on Disk?

No, but some of this can be scripted.

I have attached a script that I have been using here for some months, both on
my Bacula Community environment and in my Bacula Enterprise test environments.

The idea is that in a Runscript (RunsWhen = after), this script is called
which then does an automated restore of the current backup job, then runs
VolumeToCatalog, DiskToCatalog, and Data verification jobs. I am sure it is
not perfect yet, but I bet this can provide some ideas and a jumping point for
you.

Of course, since it accepts a jobid, client, fileset, and jobname on the
command line, you can run the restore, followed by the verifies on any job you
like and it does not have to be called from a Job's RunScript stanza. (Random
Job restore and verification tests, anyone?)


>  2. Will the VolumeToCatalog job verify Catalog files to the Tape without
> first hashing and when hardware compression is in place?

This one is "above my pay grade"™   :)

Actually, with hardware compression, that is completely transparent to Bacula.
The hashing is computed, the data is written to the drive, the drive does its
hardware compression, and writes the compressed data to the tape.   On a
read/verify, the opposite takes place, and Bacula sees the actual
(uncompressed) data, and the hashes will match.


>  3. Is there a verification job or a sequence of jobs “built in” to Bacula
> that can accomplish an automated Restore with an automated Verification of
> Tape to Restore on Disk?

This is the same as #1, just with "tape" swapped for "volume". In my mind,
file volumes and tape volumes can be considered and treated the same in most
scenarios.


>  4. Are there other options or best practices to accomplish both a post job
> verification of Catalog to Volume and a periodic verification of Volume
> (Tape or Disk) to Restore on Disk?

Nothing that I know of. I suspect that since (most of) the pieces are there:
RunScript{} stanzas, restore and verify jobs, custom scripts, etc. I think
that the only "best practice" would be what you or your company requires. eg:
How often should restores take place, which jobs/volumes should be tested,
should random jobs be tested, or should only some critical ones be
automatically tested, etc...


>  5. Is there a Bacula Systems whitepaper that covers best practices for
> Verification and Testing of backups? 

For this, I can answer a definite "no". I think it is mainly because it kind
of falls under #4, where a company's policy would probably dictate such
specific things.


> I have looked into DiskToCatalog this will not work.  (someone argue with me).

Nope... :)  I will only agree, and then add to your comment. :)

The issue you describe below is exactly the issue I plan on reporting
internally as a feature request to Bacula Systems.  My idea is exactly the one
you describe below, and I cannot see a way that this can be accomplished as
Bacula is currently written.

>  It will compare files on the client disk to the catalog.  After 15 hours +
> verification run time many files have changes - archived, reached retention
> period and deleted, etc.  Ideally, I want a complete restore to different
> location and a verification that the files in the Catalog and the Volume are
> actually the files on disk after the Restore.

I think (and I will let Kern answer for himself of course :), but I think Kern
will tell you that if the backup Job has a "Signature = md5" (or sha1), and it
ran with no errors, and then the restore job worked and reported no errors,
then for all intents and purposes, the restore is 100% fine, your data that
was restored is exactly what was backed up on the volumes and no further tests
are necessary.

Kern??  :)

While I completely agree with that logic, I do also like the idea of being
able to ask Bacula to do a verify job against data that was restored to a
different directory after the restore has finished.  As a matter of fact, when
running tests last year to certify Bacula Enterprise Edition with HPE's
"StoreOnce"™ storage devices, they required "diff" tests to verify that the
data that was restored was the same as the static data in the test directories
we were backing up. They would not accept my sound reasoning above, so I think
there can be some argument for such a feature to prove beyond "trust us that
we verify the data on the way in and on the way out..."

Please see my attached script (AutoRestoreAndVerify.sh), and let me know if
this is useful. I plan to post it to my website
(http://www.revpol.com/bacula/) at some point when I have a bit of time. And,
if a feature is adopted and implemented

[Bacula-users] Bacula 9.0.3 bat qwt requirements

2017-08-31 Thread Hicks, Daniel CTR OSD DMEA 
Hello,

I am trying to test the 9.0.3 build but cannot get past the make process. I am 
configuring with the following but when I try to run make I get an error:

CFLAGS="-g -O2 -Wall" \
  ./configure \
--sysconfdir=/etc/bacula \
--sbindir=/sbin \
--oldincludedir=/usr/include/mysql \
--enable-smartalloc \
--enable-bat \
--disable-nls \
--with-pid-dir=/var/run/bacula \
--with-subsys-dir=/var/run/subsys \
--with-systemd=/usr/lib/systemd/system \
--with-mysql \
--with-working-dir=/etc/bacula/working \
 --with-plugindir=/etc/bacula/plugins \
--with-logdir=/var/log/bacula \
--with-dump-email=admin@localhost \
--with-job-email=admin@localhost \
--with-smtp-host=localhost


Make[1]: *** [bat] Error 1
Make[1]: Leaving directory '/usr/local/src/Bacula-9.0.3/src/qt-console'

= Error in / usr/local/src/Bacula-9.0.3/src/qt-console 


This error is on a test system that already has Bacula 7.4.7 running with bat 
enabled.

Thank you

Daniel Hicks
Senior Systems Analyst
DMEA IT Support

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] bat install questions

2017-08-31 Thread Jerry Lowry 
That is correct it is not BAT

to solve this check out this:
https://github.com/marazmista/radeon-profile/issues/8

thanks,

jerry

On Thu, Aug 31, 2017 at 8:17 AM, Phil Stracchino 
wrote:

> On 08/31/17 10:37, Jerry Lowry wrote:
> > List,
> > So I installed qt as Heitor suggested.  compiles just fine.  But when I
> > run bat I get a blank screen with the following output on the terminal
> > window:
> > [root@tech bin]# ./bat
> > X Error: BadAccess (attempt to access private resource denied) 10
> >   Extension:131 (MIT-SHM)
> >   Minor opcode: 1 (X_ShmAttach)
> >   Resource id:  0x15a
> > X Error: BadShmSeg (invalid shared segment parameter) 128
> >   Extension:131 (MIT-SHM)
> >   Minor opcode: 5 (X_ShmCreatePixmap)
> >   Resource id:  0x269
>
>
> You have an X11 permissions/forwarding/authorization problem of some
> kind.  This isn't BAT's fault.  You need to make sure X11 is working
> correctly before you try to run BAT.
>
>
>
> --
>   Phil Stracchino
>   Babylon Communications
>   ph...@caerllewys.net
>   p...@co.ordinate.org
>   Landline: +1.603.293.8485
>   Mobile:   +1.603.998.6958
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] bat install questions

2017-08-31 Thread Phil Stracchino 
On 08/31/17 10:37, Jerry Lowry wrote:
> List,
> So I installed qt as Heitor suggested.  compiles just fine.  But when I
> run bat I get a blank screen with the following output on the terminal
> window:
> [root@tech bin]# ./bat
> X Error: BadAccess (attempt to access private resource denied) 10
>   Extension:    131 (MIT-SHM)
>   Minor opcode: 1 (X_ShmAttach)
>   Resource id:  0x15a
> X Error: BadShmSeg (invalid shared segment parameter) 128
>   Extension:    131 (MIT-SHM)
>   Minor opcode: 5 (X_ShmCreatePixmap)
>   Resource id:  0x269


You have an X11 permissions/forwarding/authorization problem of some
kind.  This isn't BAT's fault.  You need to make sure X11 is working
correctly before you try to run BAT.



-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] bat install questions

2017-08-31 Thread Jerry Lowry 
List,
So I installed qt as Heitor suggested.  compiles just fine.  But when I run
bat I get a blank screen with the following output on the terminal window:
[root@tech bin]# ./bat
X Error: BadAccess (attempt to access private resource denied) 10
  Extension:131 (MIT-SHM)
  Minor opcode: 1 (X_ShmAttach)
  Resource id:  0x15a
X Error: BadShmSeg (invalid shared segment parameter) 128
  Extension:131 (MIT-SHM)
  Minor opcode: 5 (X_ShmCreatePixmap)
  Resource id:  0x269
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x2a00012

 and it goes on for pages

thanks,

jerry


On Wed, Aug 30, 2017 at 12:30 AM, Simone Caronni 
wrote:

> On Wed, Aug 30, 2017 at 9:28 AM, Simone Caronni 
> wrote:
>
>> Hi,
>>
>> you might as well use pre-built binaries:
>>
>
> Sorry, keyboard error :/
> Damn gmail.
>
> https://copr.fedorainfracloud.org/coprs/slaanesh/Bacula/
>
> Those are exactly the same builds in Fedora that will eventually go into
> RHEL. I make sure that every release builds on all supported Fedora and
> CentOS/RHEL releases.
> Upgrades in place from bundled packages in the distribution is supported,
> just make sure to update the database.
>
> Regards,
> --Simone
>
>
>
> --
> You cannot discover new oceans unless you have the courage to lose sight
> of the shore (R. W. Emerson).
>
> http://xkcd.com/229/
> http://negativo17.org/
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Backup Storage Block Level Deduplication with Ddumbfs and Bacula

2017-08-31 Thread Heitor Faria 
Hello, Bacula Users, 

Just FYI, a paper published today about Aligned Volumes 
.
 

Regards, 
-- 
=== 
Heitor Medrado de Faria | EB-1 Visa | LPIC-III | ITIL-F | EMC 05-001| Bacula 
Systems Certified Administrator II 
• Bacula Enterprise does not charge per backup size and has plugins for many 
applications. In the USA: http://bacula.us/ 
• Do you need Bacula training? http://bacula.us/video-classes/ 
+55 61 98268-4220 | http://bacula.us 
=== 
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Windows FileSet

2017-08-31 Thread Martin Simmons 
Hi Kern,

I've just tested this in the latest sources and can confirm that changes to
Options { wild = ... } clauses do *not* trigger an upgrade to a Full backup.

Changes to File = lines do trigger it.

Maybe that was not the intent, but it has been like that forever AFAICS.

__Martin


> On Thu, 24 Aug 2017 18:06:12 +0100, Martin Simmons said:
> 
> Can you point me to the code that implements this "active" part check please?
> I can only find store_fname() and store_plugin_name() in dird/inc_conf.c that
> update the md5c while reading the config.
> 
> __Martin
> 
> 
> On Thu, 24 Aug 2017 16:33:41 +0200, Kern Sibbald said:
> > 
> > Bacula does an MD5 sum on the "active" part of a fileset, which means 
> > that even an immaterial change such as a space is important.  If the MD5 
> > sum changes, Bacula will do a Full backup unless the Ignore Fileset 
> > Changes is set.
> > 
> > Kern
> > 
> > On 24/08/2017 14:49, Martin Simmons wrote:
> > >> On Thu, 24 Aug 2017 06:33:30 +0100, Ben Laurie said:
> > >> May not be relevant that this is Windows, but ... in the docs it says
> > >> that if a change is made to a FileSet, the next backup will be a full
> > >> backup.
> > >>
> > >> However, I am trying to tune a windows fileset, and two things are 
> > >> happening:
> > >>
> > >> 1. If I change the FileSet then run a backup, it is an incremental 
> > >> backup,
> > > Are you just changing things inside Options clauses?  I think it only 
> > > notices
> > > changes to the "File = ..." lines.
> > >
> > >
> > >> 2. The incremental backup includes files that are excluded by the
> > >> FileSet, even though an estimate command does _not_ show those files.
> > > That's unexpected.  Is there a large difference between the number of 
> > > files?
> > >
> > > __Martin
> > >
> > > --
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > > ___
> > > Bacula-users mailing list
> > > Bacula-users@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/bacula-users
> > >
> > 
> 

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Storage Daemon crashes on Debian 9 with Bacula release 7.4.4+dfsg-6

2017-08-31 Thread christian . garling 

Am 2017-08-30 17:33, schrieb Sven Hartge:

On 30.08.2017 15:19, christian.garl...@cg-networks.de wrote:

We never had such a problem before the upgrade, so we are wondering 
what

could be the reason for that.



Could you please add

deb http://deb.debian.org/debian-debug/ stretch-debug main contrib 
non-free


to your source.list and install the packages bacula-sd-dbgsym and
bacula-common-dbgsym. This should make the backtrace a bit more 
readable.




Yes, here you can find a btraceback from last nigh with debug symbols:

[New LWP 24920]
[New LWP 20893]
[New LWP 20894]
[New LWP 20895]
[New LWP 20896]
[New LWP 20950]
[New LWP 21026]
[New LWP 21027]
[New LWP 21031]
[New LWP 21038]
[New LWP 21064]
[New LWP 21091]
[New LWP 21168]
[New LWP 21171]
[New LWP 21186]
[New LWP 21190]
[New LWP 21276]
[New LWP 21610]
[New LWP 21895]
[New LWP 21937]
[New LWP 21945]
[New LWP 22176]
[Thread debugging using libthread_db enabled] Using host libthread_db 
library "/lib/x86_64-linux-gnu/libthread_db.so.1".

0x7f9cdfd993f3 in select () at ../sysdeps/unix/syscall-template.S:84
84  ../sysdeps/unix/syscall-template.S: No such file or directory.
$1 = "31-Aug-2017 01:06:10\000\000\000\000\000\000\000\000\000"
$2 = 0x5644c1acb8c0  "s110180-sd"
$3 = 0x5644c2a61c68 "bacula-sd"
$4 = 0x5644c2a61ca8 "/usr/sbin/bacula-sd"
$5 = 0x0
$6 = '\000' 
$7 = 0x7f9ce183eac5 "7.4.4 (20 September 2016)"
$8 = 0x7f9ce183eaa6 "x86_64-pc-linux-gnu"
$9 = 0x7f9ce183ea9f "debian"
$10 = 0x7f9ce183eac1 "9.0"
$11 = "s110180", '\000' 
$12 = 0x7f9ce183eaba "debian 9.0"
Environment variable "TestName" not defined.
#0  0x7f9cdfd993f3 in select () at 
../sysdeps/unix/syscall-template.S:84
#1  0x7f9ce180b1ca in bnet_thread_server (addrs=, 
max_clients=201, client_wq=0x5644c1acba60 , 
handle_client_request=0x5644c188b5e0 ) 
at bnet_server.c:166
#2  0x5644c186aa84 in main (argc=, argv=out>) at stored.c:294


Thread 23 (Thread 0x7f9c86ffd700 (LWP 22176)):
#0  0x7f9ce15e5b3a in __waitpid (pid=pid@entry=22184, 
stat_loc=stat_loc@entry=0x7f9c86ffc0ac, options=options@entry=0) at 
../sysdeps/unix/sysv/linux/waitpid.c:29

#1  0x7f9ce1829f7e in signal_handler (sig=11) at signal.c:237
#2  
#3  new_tls_connection (ctx=0x0, fd=45) at tls.c:459
#4  0x7f9ce1809e4d in bnet_tls_server (ctx=, 
bsock=bsock@entry=0x5644c2adcc68, verify_list=verify_list@entry=0x0) at 
bnet.c:215
#5  0x5644c1878ae9 in authenticate_filed 
(jcr=jcr@entry=0x7f9cbc0008e8, fd=fd@entry=0x5644c2adcc68, FDVersion=0) 
at authenticate.c:197
#6  0x5644c188e2df in handle_client_connection 
(fd=fd@entry=0x5644c2adcc68) at hello.c:176
#7  0x5644c188b890 in handle_connection_request (arg=0x5644c2adcc68) 
at dircmd.c:170
#8  0x7f9ce1834d02 in workq_server (arg=0x5644c1acba60 ) 
at workq.c:326
#9  0x7f9ce15dc494 in start_thread (arg=0x7f9c86ffd700) at 
pthread_create.c:333
#10 0x7f9cdfda0aff in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 22 (Thread 0x7f9ca7fff700 (LWP 21945)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x7f9ce1828436 in rwl_writelock_p (rwl=rwl@entry=0x5644c1acbf20 
, file=file@entry=0x5644c18bb047 "reserve.c", 
line=line@entry=287) at rwlock.c:231
#2  0x5644c189e75e in _lock_reservations 
(file=file@entry=0x5644c18bb047 "reserve.c", line=line@entry=287) at 
reserve.c:80
#3  0x5644c18a1656 in use_device_cmd (jcr=) at 
reserve.c:287

#4  use_cmd (jcr=) at reserve.c:165
#5  0x5644c188ba4f in handle_connection_request (arg=0x5644c2ade638) 
at dircmd.c:226
#6  0x7f9ce1834d02 in workq_server (arg=0x5644c1acba60 ) 
at workq.c:326
#7  0x7f9ce15dc494 in start_thread (arg=0x7f9ca7fff700) at 
pthread_create.c:333
#8  0x7f9cdfda0aff in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 21 (Thread 0x7f9cc77fe700 (LWP 21937)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x7f9ce1828436 in rwl_writelock_p (rwl=rwl@entry=0x5644c1acbf20 
, file=file@entry=0x5644c18bb047 "reserve.c", 
line=line@entry=287) at rwlock.c:231
#2  0x5644c189e75e in _lock_reservations 
(file=file@entry=0x5644c18bb047 "reserve.c", line=line@entry=287) at 
reserve.c:80
#3  0x5644c18a1656 in use_device_cmd (jcr=) at 
reserve.c:287

#4  use_cmd (jcr=) at reserve.c:165
#5  0x5644c188ba4f in handle_connection_request (arg=0x5644c2ad6638) 
at dircmd.c:226
#6  0x7f9ce1834d02 in workq_server (arg=0x5644c1acba60 ) 
at workq.c:326
#7  0x7f9ce15dc494 in start_thread (arg=0x7f9cc77fe700) at 
pthread_create.c:333
#8  0x7f9cdfda0aff in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Thread 20 (Thread 0x7f9ca67fc700 (LWP 21895)):
#0  0x7f9ce15e57dd in nanosleep () at 
../sysdeps/unix/syscall-template.S:84
#1  0x7f9ce1807597 in bmicrosleep (sec=sec@entry=5, 
usec=usec@entry=0) at bsys.c:178
#2  0x5644c188b874 in