Nice that you ruled out problems with 4GB
In looking at your plugin command line, I recommend:
try turning off gzip on the plugin command line -- I don't think it will work
correctly with the plugin (we have never used that option). Then make sure
there is no compression in the Bacula FileSet. If that works, try turning on
compression in the Bacula FileSet. I think it will work, but it needs
testing. It is better to try things one step at a time when you have such
problems.
In any case, you do not want to compress on the plugin command line AND in the
Bacula FileSet.
Regards,
Kern
On Wednesday 27 May 2009 16:12:25 Ulrich Leodolter wrote:
> Hello,
>
> On Wed, 2009-05-27 at 14:28 +0200, Kern Sibbald wrote:
> > It is quite possible that mysqldump creates a much larger dump file
> > particularly if you have compression turned on.
> >
> > Perhaps you filled up your output filesystem.
>
> definitely NO
>
> > We have tested this many times and never had such problems. You might
> > try testing with a much smaller database to ensure that you are using the
> > proper syntax, ... before working with such a large database.
> >
> > One possibility is that some value in the plugin is limited to 32 bits
> > (i.e. a bit less than 4GB).
>
> I tried to dump single bacula catalog table,
> it worked without problems, FD Bytes Written are than 4 GB,
>
> Plugin = "bpipe:/MYSQL/bacula.sql:mysqldump -f --opt bacula File:gzip >
> /var/tmp/bacula-File.sql.gz"
>
>
> I feel helpless, dumping large File table (more than 10GB uncompressed)
> works in bpipe mode. But dumping the whole bacula database using bpipe
> results in incomplete dump.
>
>
> Here is the job log using the above Plugin line.
>
>
> troll-dir JobId 25217: Start Backup JobId 25217,
> Job=Backup-CatalogPipe.2009-05-27_15.08.03_11 troll-dir JobId 25217: Using
> Device "FileStorage"
> troll-sd JobId 25217: Volume "Backup-0615" previously written, moving to
> end of data. troll-sd JobId 25217: Ready to append to end of Volume
> "Backup-0615" size=30604524317 troll-sd JobId 25217: Job write elapsed time
> = 00:25:26, Transfer rate = 2.696 M bytes/second troll-dir JobId 25217:
> Bacula troll-dir 3.0.1 (30Apr09): 27-May-2009 15:33:31 Build OS:
> i686-pc-linux-gnu redhat Enterprise release JobId:
> 25217
> Job: Backup-CatalogPipe.2009-05-27_15.08.03_11
> Backup Level: Full
> Client: "troll-fd" 3.0.1 (30Apr09)
> i686-pc-linux-gnu,redhat,Enterprise release FileSet:
> "CatalogPipe" 2009-05-27 11:30:04
> Pool: "DiskBackup" (From Job resource)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "File" (From Job resource)
> Scheduled time: 27-May-2009 15:07:59
> Start time: 27-May-2009 15:08:05
> End time: 27-May-2009 15:33:31
> Elapsed time: 25 mins 26 secs
> Priority: 11
> FD Files Written: 1
> SD Files Written: 1
> FD Bytes Written: 4,114,114,420 (4.114 GB)
> SD Bytes Written: 4,114,114,606 (4.114 GB)
> Rate: 2696.0 KB/s
> Software Compression: 70.1 %
> VSS: no
> Encryption: no
> Accurate: no
> Volume name(s): Backup-0615
> Volume Session Id: 924
> Volume Session Time: 1242371530
> Last Volume Bytes: 34,721,566,627 (34.72 GB)
> Non-fatal FD errors: 0
> SD Errors: 0
> FD termination status: OK
> SD termination status: OK
> Termination: Backup OK
>
> troll-dir JobId 25217: Begin pruning Jobs.
> troll-dir JobId 25217: No Jobs found to prune.
> troll-dir JobId 25217: Begin pruning Files.
> troll-dir JobId 25217: No Files found to prune.
> troll-dir JobId 25217: End auto prune.
>
>
> Regards
> Ulrich
>
> > Regards,
> >
> > Kern
> >
> > On Wednesday 27 May 2009 13:35:53 Ulrich Leodolter wrote:
> > > Hello,
> > >
> > > I tried to setup bpipe plugin for Catalog backup.
> > > Below you can see FileSet/Job config, Backup/Restore logs.
> > > Everything looks ok, but mysqldump is incomplete!
> > >
> > > Running "mysqldump -f --opt bacula > bacula.sql" from shell
> > > results in 14G bacula.sql file.
> > >
> > > /MYSQL/bacula.sql is only 3,325,644,283 bytes as u can see in
> > > restore log.
> > >
> > > Here are the last lines of restored file /var/tmp/bacula.sql.gz,
> > > last lines look ok, but dump file is definitely incomplete.
> > >
> > > A A ou BAA I BKEH+e BEE6La BJWk5Q A A
> > > E','dBR5u4Hk9YoQc9qFPSQEfvDv4Rg'); /*!40103 SET
> > > time_zo...@old_time_zone */;
> > >
> > > /*!40101 SET sql_mo...@old_sql_mode */;
> > > /*!40014 SET foreign_key_chec...@old_foreign_key_checks */;
> > > /*!40014 SET unique_chec...@old_unique_checks */;
> > > /*!40101 SET character_set_clie...@old_character_set_client */;
> > > /*!40101 SET character_set_resul...@old_character_set_results */;
> > > /*!40101 SET collation_connecti...@old_collation_connection */;
> > > /*!40111 SET sql_not...@old_sql_notes */;
> > >
> > > -- Dump completed on 2009-05-27 10:56:00
> > >
> > >
> > >
> > > It looks like mysqldump called from bpipe terminates/stops
> > > while dumping File.
> > >
> > > [r...@troll tmp]# zgrep "CREATE TABLE" /var/tmp/bacula.sql.gz
> > > CREATE TABLE `BaseFiles` (
> > > CREATE TABLE `CDImages` (
> > > CREATE TABLE `Client` (
> > > CREATE TABLE `Counters` (
> > > CREATE TABLE `Device` (
> > > CREATE TABLE `File` (
> > > [r...@troll tmp]#
> > >
> > >
> > > Regards
> > > Ulrich
> > >
> > > -------
> > >
> > >
> > > FileSet {
> > > Name = CatalogPipe
> > > Ignore FileSet Changes = yes
> > > Include {
> > > Options {
> > > signature = MD5
> > > compression = GZIP
> > > }
> > > Plugin = "bpipe:/MYSQL/bacula.sql:mysqldump -f --opt bacula:gzip >
> > > /var/tmp/bacula.sql.gz" }
> > > }
> > >
> > > Job {
> > > Name = Backup-CatalogPipe
> > > Type = Backup
> > > Level = Full
> > > Client = troll-fd
> > > FileSet = "CatalogPipe"
> > > Storage = File
> > > Messages = StandardMailOnError
> > > Pool = DiskBackup
> > > Write Bootstrap = "/opt/bacula/working/%n.bsr"
> > > Priority = 11
> > > SpoolData = no
> > > Allow Duplicate Jobs = no
> > > }
> > >
> > >
> > >
> > >
> > > troll-dir JobId 25211: Start Backup JobId 25211,
> > > Job=Backup-CatalogPipe.2009-05-27_12.48.41_59
> > > troll-dir JobId 25211: Using Device "FileStorage"
> > > troll-sd JobId 25211: Volume "Backup-0615" previously written, moving
> > > to end of data. troll-sd JobId 25211: Ready to append to end of Volume
> > > "Backup-0615" size=29564841571 troll-sd JobId 25211: Job write elapsed
> > > time = 00:07:17, Transfer rate = 2.290 M bytes/second troll-dir JobId
> > > 25211: Bacula troll-dir 3.0.1 (30Apr09): 27-May-2009 12:56:00 Build OS:
> > > i686-pc-linux-gnu redhat Enterprise release JobId:
> > > 25211
> > > Job: Backup-CatalogPipe.2009-05-27_12.48.41_59
> > > Backup Level: Full
> > > Client: "troll-fd" 3.0.1 (30Apr09)
> > > i686-pc-linux-gnu,redhat,Enterprise release FileSet:
> > > "CatalogPipe" 2009-05-27 11:30:04
> > > Pool: "DiskBackup" (From Job resource)
> > > Catalog: "MyCatalog" (From Client resource)
> > > Storage: "File" (From Job resource)
> > > Scheduled time: 27-May-2009 12:48:34
> > > Start time: 27-May-2009 12:48:43
> > > End time: 27-May-2009 12:56:00
> > > Elapsed time: 7 mins 17 secs
> > > Priority: 11
> > > FD Files Written: 1
> > > SD Files Written: 1
> > > FD Bytes Written: 1,001,064,037 (1.001 GB)
> > > SD Bytes Written: 1,001,064,218 (1.001 GB)
> > > Rate: 2290.8 KB/s
> > > Software Compression: 69.9 %
> > > VSS: no
> > > Encryption: no
> > > Accurate: no
> > > Volume name(s): Backup-0615
> > > Volume Session Id: 918
> > > Volume Session Time: 1242371530
> > > Last Volume Bytes: 30,566,617,501 (30.56 GB)
> > > Non-fatal FD errors: 0
> > > SD Errors: 0
> > > FD termination status: OK
> > > SD termination status: OK
> > > Termination: Backup OK
> > >
> > > troll-dir JobId 25211: Begin pruning Jobs.
> > > troll-dir JobId 25211: No Jobs found to prune.
> > > troll-dir JobId 25211: Begin pruning Files.
> > > troll-dir JobId 25211: No Files found to prune.
> > > troll-dir JobId 25211: End auto prune.
> > >
> > >
> > >
> > > troll-dir JobId 25212: Start Restore Job
> > > RestoreFiles.2009-05-27_12.57.43_02
> > > troll-dir JobId 25212: Using Device "FileStorage"
> > > troll-sd JobId 25212: Ready to read from volume "Backup-0615" on device
> > > "FileStorage" (/disk0/bacula/files). troll-sd JobId 25212: Forward
> > > spacing Volume "Backup-0615" to file:block 6:3795037795. troll-sd JobId
> > > 25212: End of Volume at file 7 on device "FileStorage"
> > > (/disk0/bacula/files), Volume "Backup-0615" troll-sd JobId 25212: End
> > > of all volumes.
> > > troll-dir JobId 25212: Bacula troll-dir 3.0.1 (30Apr09): 27-May-2009
> > > 13:00:19 Build OS: i686-pc-linux-gnu redhat Enterprise
> > > release JobId: 25212
> > > Job: RestoreFiles.2009-05-27_12.57.43_02
> > > Restore Client: troll-fd
> > > Start time: 27-May-2009 12:57:45
> > > End time: 27-May-2009 13:00:19
> > > Files Expected: 1
> > > Files Restored: 1
> > > Bytes Restored: 3,325,644,283
> > > Rate: 21595.1 KB/s
> > > FD Errors: 0
> > > FD termination status: OK
> > > SD termination status: OK
> > > Termination: Restore OK
> > >
> > > troll-dir JobId 25212: Begin pruning Jobs.
> > > troll-dir JobId 25212: No Jobs found to prune.
> > > troll-dir JobId 25212: Begin pruning Files.
> > > troll-dir JobId 25212: No Files found to prune.
> > > troll-dir JobId 25212: End auto prune.
------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
