Your message dated Sun, 22 May 2016 22:21:16 +0000 with message-id <[email protected]> and subject line Bug#737821: fixed in bacula 7.4.0+dfsg-1 has caused the Debian Bug report #737821, regarding bacula-console-qt: .bvfs_update is carried out VERY long on jobs with several million files and a deep enclosure to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact [email protected] immediately.) -- 737821: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=737821 Debian Bug Tracking System Contact [email protected] with problems
--- Begin Message ---Package: bacula-console-qt Version: 5.2.6+dfsg-9 Severity: important Dear Maintainer, For recovery of files by means of bRestore, it is necessary to execute at first .bvfs_update. When trying to build the PathHierarchy with .bvfs_update, the update for big jobs take forever... Steps to Reproduce: - have a big dataset (> 1 million files AND a deep directory hierarchy) - .bvfs_update job=N - mysql used 100% CPU very mach time - at this time no run jobs on the schedule Additional Information: I updated bacula from version 5.0 to 5.2 (debian 6 -> 7). In the new bat version the "Browse Cataloged Files" function became deprecated. It is offered to use bRestore. My bat hanged at bRestore choice. At this time mysql uses 100% of Cpu I enable slow query log on mysql. The analysis of slow query is lower TOP 1 from "#mysqldumpslow mysql-slow.log" Count: 249 Time=243.29s (60579s) Lock=0.00s (0s) Rows=0.0 (0), bacula[bacula]@localhost INSERT INTO PathVisibility (PathId, JobId) SELECT a.PathId,N FROM ( SELECT DISTINCT h.PPathId AS PathId FROM PathHierarchy AS h JOIN PathVisibility AS p ON (h.PathId=p.PathId) WHERE p.JobId=N) AS a LEFT JOIN (SELECT PathId FROM PathVisibility WHERE JobId=N) AS b ON (a.PathId = b.PathId) WHERE b.PathId IS NULL I carried out the analysis and optimization of this query. Original query: INSERT INTO PathVisibility (PathId, JobId) SELECT a.PathId,N FROM ( SELECT DISTINCT h.PPathId AS PathId FROM PathHierarchy AS h JOIN PathVisibility AS p ON (h.PathId=p.PathId) WHERE p.JobId=N) AS a LEFT JOIN ( SELECT PathId FROM PathVisibility WHERE JobId=N) AS b ON (a.PathId = b.PathId) WHERE b.PathId IS NULL Optimized query: INSERT INTO PathVisibility (PathId, JobId) SELECT a.PathId,N FROM ( SELECT DISTINCT h.PPathId AS PathId FROM PathHierarchy AS h JOIN PathVisibility AS p ON (h.PathId=p.PathId) WHERE p.JobId=N) AS a LEFT JOIN PathVisibility AS b ON (b.JobId=N and a.PathId = b.PathId) WHERE b.PathId IS NULL Results of query are identical. In original query in join on the right there is a temporary table (subquery "SELECT PathId FROM PathVisibility WHERE JobId=N"). The temporary table has no index across the field of PathId. For each record of the left table there is a full search of the right table. The table PathVisibility has a compound index on the fields JobId and PathId. In the optimized query join is carried out with use of this index. After introduction of editings and recompilation bacula of .bvfs_update it was executed in 17 minutes. With old inquiry in 16-17 hours it didn't end and was still very far from end. Count: 19052 Time=0.01s (152s) Lock=0.00s (1s) Rows=0.0 (0), bacula[bacula]@localhost INSERT INTO PathVisibility (PathId, JobId) SELECT a.PathId,N FROM ( SELECT DISTINCT h.PPathId AS PathId FROM PathHierarchy AS h JOIN PathVisibility AS p ON (h.PathId=p.PathId) WHERE p.JobId=N) AS a LEFT JOIN PathVisibility AS b ON (JobId=N and a.PathId = b.PathId) WHERE b.PathId IS NULL The similar situation was stated in http://sourceforge.net/mailarchive/forum.php?thread_name=3ccaa3a6fb746ee27273ef0b1db9045b%40bugs.bacula.org&forum_name=bacula-bugs I have a base in mysql. So detailed check is necessary for other bases. I asked the friend to check on postgresql for a lot of work. After 1,5 hours of processing of one big job were compelled to stop postgresql. Patch: diff -Naur bacula-5.2.6+dfsg/src/cats/bvfs.c_orig bacula-5.2.6+dfsg/src/cats/bvfs.c --- bacula-5.2.6+dfsg/src/cats/bvfs.c_orig 2012-06-02 13:26:27.000000000 +0400 +++ bacula-5.2.6+dfsg/src/cats/bvfs.c 2014-02-06 10:42:34.798440879 +0400 @@ -404,10 +404,9 @@ "SELECT DISTINCT h.PPathId AS PathId " "FROM PathHierarchy AS h " "JOIN PathVisibility AS p ON (h.PathId=p.PathId) " - "WHERE p.JobId=%s) AS a LEFT JOIN " - "(SELECT PathId " - "FROM PathVisibility " - "WHERE JobId=%s) AS b ON (a.PathId = b.PathId) " + "WHERE p.JobId=%s) AS a " + "LEFT JOIN PathVisibility AS b " + "ON (b.JobId=%s and a.PathId = b.PathId) " "WHERE b.PathId IS NULL", jobid, jobid, jobid); } -- System Information: Debian Release: 7.3 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 3.2.0-4-amd64 (SMP w/4 CPU cores) Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/dash Versions of packages bacula-console-qt depends on: ii bacula-common 5.2.6+dfsg-9 ii libc6 2.13-38 ii libcap2 1:2.22-1.2 ii libgcc1 1:4.7.2-5 ii libqtcore4 4:4.8.2+dfsg-11 ii libqtgui4 4:4.8.2+dfsg-11 ii libssl1.0.0 1.0.1e-2+deb7u3 ii libstdc++6 4.7.2-5 ii libwrap0 7.6.q-24 ii zlib1g 1:1.2.7.dfsg-13 bacula-console-qt recommends no packages. bacula-console-qt suggests no packages. -- no debconf information
--- End Message ---
--- Begin Message ---Source: bacula Source-Version: 7.4.0+dfsg-1 We believe that the bug you reported is fixed in the latest version of bacula, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to [email protected], and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Carsten Leonhardt <[email protected]> (supplier of updated bacula package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing [email protected]) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 1.8 Date: Sun, 22 May 2016 23:41:08 +0200 Source: bacula Binary: bacula bacula-common bacula-common-sqlite3 bacula-common-pgsql bacula-common-mysql bacula-director-common bacula-director-sqlite3 bacula-director-mysql bacula-director-pgsql bacula-client bacula-fd bacula-server bacula-sd bacula-sd-sqlite3 bacula-sd-mysql bacula-sd-pgsql bacula-console bacula-console-qt Architecture: source Version: 7.4.0+dfsg-1 Distribution: unstable Urgency: low Maintainer: Debian Bacula Team <[email protected]> Changed-By: Carsten Leonhardt <[email protected]> Description: bacula - network backup service - metapackage bacula-client - network backup service - client metapackage bacula-common - network backup service - common support files bacula-common-mysql - network backup service - MySQL common files bacula-common-pgsql - network backup service - PostgreSQL common files bacula-common-sqlite3 - network backup service - SQLite v3 common files bacula-console - network backup service - text console bacula-console-qt - network backup service - Bacula Administration Tool bacula-director-common - network backup service - Director common files bacula-director-mysql - network backup service - MySQL storage for Director bacula-director-pgsql - network backup service - PostgreSQL storage for Director bacula-director-sqlite3 - network backup service - SQLite 3 storage for Director bacula-fd - network backup service - file daemon bacula-sd - network backup service - storage daemon bacula-sd-mysql - network backup service - MySQL SD tools bacula-sd-pgsql - network backup service - PostgreSQL SD tools bacula-sd-sqlite3 - network backup service - SQLite 3 SD tools bacula-server - network backup service - server metapackage Closes: 737821 Changes: bacula (7.4.0+dfsg-1) unstable; urgency=low . * New upstream release - Integrates fix for .bvfs_update (Closes: #737821) - patches removed because they are fixed upstream: fix-manpage-bconsole8.patch, ppc64el-libtoolize.patch, fix-src-tools-make-clean.patch, fix-update-sqlite3-12to14.patch, add-basic-kfreebsd-support.patch * now compiles on hurd * updated debian/copyright * refreshed debian/patches/fix-default-config, switch-nonfree-sha1-to-openssl.patch * *-dbg packages no longer provided, please use the automatic dbgsym packages if needed * Update Standards-Version to 3.9.8. No changes required. * bacula-director-pgsql no longer suggests postgresql-doc * created new scripts to cope with new database update scripts * added patch to fix update script for postgresql databases Checksums-Sha1: 4a17b5ffb62d75aaf3bdd7665b5f8d2842ed82e2 3347 bacula_7.4.0+dfsg-1.dsc 4872a48a38aab1a5204d77575f4a5c84a37c90f7 2191248 bacula_7.4.0+dfsg.orig.tar.xz e3ea8e0a3474b9253306aa4fb072cc7bc244d3c2 67156 bacula_7.4.0+dfsg-1.debian.tar.xz Checksums-Sha256: c0003ba25a30be7505d5e0621020aa72101d42280f85ef5bdd5003eaba225275 3347 bacula_7.4.0+dfsg-1.dsc 96327fd3d526c0839ddcf608e8da3b79c300d48e0df50290009dca889fbe45f9 2191248 bacula_7.4.0+dfsg.orig.tar.xz 8fb62345ac7143109a1cef20080c87ffa2b1aa8ffeb6b158c1fa6805353946dc 67156 bacula_7.4.0+dfsg-1.debian.tar.xz Files: e5a4e9d4c0cd0511f51bb98fdfc32976 3347 admin optional bacula_7.4.0+dfsg-1.dsc 5dd2973e2a25721c5bb9485fdef3c416 2191248 admin optional bacula_7.4.0+dfsg.orig.tar.xz 803b40f47ffb4aaf53940e4e363063af 67156 admin optional bacula_7.4.0+dfsg-1.debian.tar.xz -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJXQimpAAoJEFbggnbEQhZ0ITwP/1fqrmEjFGDT47CEKFlbx0+y yOhUsO+PHWI55nA2b+ZK6VDWtlqfjcfXcYx78Ys0FtqnIqnWJbvUGTWfpBCyeWWa j/KcoeJn/pzpujtJrljKZokCgrhcJjPAt16MuDRv45eYdLUbA2NKTZeoQ0OA/B1J 1XxSLuzQ/UIyk++dv0vdJhMFIFY0rGMKavfaNWSrrmXLP2oyorAS+2CHwYojrRB9 OeJ4t8Q/BUkwNCR6LOXHLXaOsEq3EGKKjTqV5RUxteGxjByIEq+dOPBUz0rjaO2z kdfKmDRkknecQJglUg4apW6xTUESzMIDyhdcE8cjNERbuYCg2Fx6gc5dsdhFioQf 60BVttbeRvf05rBqOlC340SYA5Ijeskl2fTKIqPxLHsqdAbSHph6G9rW0MNPpXjl dBiPK4tU5ceHmzGyt0ChnX3pgDOHibp4ARG95OCkt+nQiljHqpyjO1sAfyHXv7+y bNEcVpOlz9TA/1vePf4nT3yxdRkySRsNSMeDPqi1wRsxAS+l3ty7dyZ6wHoze9C0 xO+/XFKXPcEyWvV8Fznok5+s9XBpIb92O7BtHMju4M7fu/PVHiGqNbk5HcTsGngU wQHfvBkEqXpaQKHm/6PdJji/5Sfd69fKhiJ+Zg+QuamUn/i3UtEaZb2hiJHpyB/E hcnR3pZg0q9QV+pRRvgd =Azdz -----END PGP SIGNATURE-----
--- End Message ---
