Hello Eric,
I've been a bit busy, but just the same worried about what Bruno is describing
here, and interestingly enough, on my machine, I am also finding that for
MySQL batch insert it turned off, yet for PostgreSQL it is turned on.
After a bit of experimenting, I think the problem is that there are several
problems with the ./configure script.
First, for MySQL, the library that is found is /usr/lib64/libmysqlclient_r.so,
which is OK, but our test for pthread_mutex_lock uses nm on the library,
which is simply broken. It is not an object file, but rather a shared
object, so it simply imports other shared objects.
The second problem is that for PostgreSQL, it is using /usr/lib/libpq.a, which
does show pthread_mutex_lock, but in my opinion, we should
probably /usr/lib/libpq.so, and obviously if we do the test will fail.
I think there is one or more bugs hidden here ...
Kern
On Thursday 17 December 2009 08:11:09 Bruno Friedmann wrote:
> Hello Kern, and thank you for your precious comment
> about the mysql pthread_mutex_lock availability.
>
> I cut all none interesting part. And add what I've found
> I'm working now on openSUSE 11.2 and I want to extend the bacula.spec of
> Scott to include the su112 tag and a way to build with and without
> batch-insert-enabled binary.
>
> After several check, none of the rpm build have batch insert inside.
>
> So to be sure, I've reget the tar.gz original sources try to reconfigure
> with simple flags
>
> ./configure --enable-smartalloc --with-tcp-wrappers --with-libiconv-prefix
> --with-libintl-prefix --enable-largefile --with-x --enable-ipv6
> --enable-acl --enable-xattr --enable-batch-insert --with-python --with-qwt
> --enable-gnome --enable-bat --with-sqlite3 ( or --with-mysql or
> --with-postgresql )
>
> In every try I only get Enable Batch insert No
>
> Configuration on Thu Dec 17 07:59:22 CET 2009:
>
> Host: i686-pc-linux-gnu -- suse 11.2
> Bacula version: Bacula 3.0.2 (18 July 2009)
> Source code location: .
> Install binaries: /sbin
> Install libraries: /usr/lib
> Install config files: /etc/bacula
> Scripts directory: /etc/bacula
> Archive directory: /tmp
> Working directory: /var/bacula/working
> PID directory: /var/run
> Subsys directory: /var/lock/subsys
> Man directory: ${datarootdir}/man
> Data directory: /usr/share
> Plugin directory: /usr/lib
> C Compiler: gcc Linux)
> C++ Compiler: /usr/bin/g++ Linux)
> Compiler flags: -g -O2 -Wall -fno-strict-aliasing
> -fno-exceptions -fno-rtti Linker flags:
> Libraries: -lpthread -ldl
> Statically Linked Tools: no
> Statically Linked FD: no
> Statically Linked SD: no
> Statically Linked DIR: no
> Statically Linked CONS: no
> Database type: MySQL
> Database port:
> Database lib: -L/usr/lib -lmysqlclient_r -lz
> Database name: bacula
> Database user: bacula
>
> Job Output Email: r...@localhost
> Traceback Email: r...@localhost
> SMTP Host Address: localhost
>
> Director Port: 9101
> File daemon Port: 9102
> Storage daemon Port: 9103
>
> Director User:
> Director Group:
> Storage Daemon User:
> Storage DaemonGroup:
> File Daemon User:
> File Daemon Group:
>
> SQL binaries Directory /usr/bin
>
> Large file support: yes
> Bacula conio support: yes -lncurses
> readline support: no
> TCP Wrappers support: yes -lwrap
> TLS support: yes
> Encryption support: yes
> ZLIB support: yes
> enable-smartalloc: yes
> enable-lockmgr: no
> bat support: yes -L/usr/lib
> enable-gnome: yes Version 2.x
> enable-bwx-console: no
> enable-tray-monitor: no
> client-only: no
> build-dird: yes
> build-stored: yes
> ACL support: yes
> XATTR support: yes
> Python support: yes -lpthread -ldl -lutil -lm -lpython2.6
> -lutil -lrt Batch insert enabled: no
>
>
> I put here the pg_config and mysql_config results
> mysql_config
> Usage: /usr/bin/mysql_config [OPTIONS]
> Options:
> --cflags [-I/usr/include/mysql -fomit-frame-pointer
> -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables
> -fasynchronous-unwind-tables -g -DPIC -fPIC -DUNDEF_HAVE_INITGROUPS
> -fno-strict-aliasing -DUNIV_LINUX] --include
> [-I/usr/include/mysql]
> --libs [-L/usr/lib/mysql -lmysqlclient -lz -lcrypt -lnsl
> -lm -L/usr/lib -L/usr/lib64 -lssl -lcrypto] --libs_r
> [-L/usr/lib/mysql -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -lpthread
> -L/usr/lib -L/usr/lib64 -lssl -lcrypto]
> --plugindir [/usr/lib/mysql/plugin]
> --socket [/var/run/mysql/mysql.sock]
> --port [0]
> --version [5.1.36]
> --libmysqld-libs [-L/usr/lib/mysql -lmysqld -ldl -lz -lpthread
> -lcrypt -lnsl -lm -lpthread -lwrap -lrt -L/usr/lib -L/usr/lib64 -lssl
> -lcrypto]
>
> pg_config
> BINDIR = /usr/bin
> DOCDIR = /usr/share/doc/packages/postgresql
> HTMLDIR = /usr/share/doc/packages/postgresql
> INCLUDEDIR = /usr/include/pgsql
> PKGINCLUDEDIR = /usr/include/pgsql
> INCLUDEDIR-SERVER = /usr/include/pgsql/server
> LIBDIR = /usr/lib
> PKGLIBDIR = /usr/lib/postgresql
> LOCALEDIR = /usr/share/locale
> MANDIR = /usr/share/man
> SHAREDIR = /usr/share/postgresql
> SYSCONFDIR = /usr/etc/postgresql
> PGXS = /usr/lib/postgresql/pgxs/src/makefiles/pgxs.mk
> CONFIGURE = '--prefix=/usr' '--libdir=/usr/lib' '--bindir=/usr/bin'
> '--includedir=/usr/include/pgsql' '--datadir=/usr/share/postgresql'
> '--mandir=/usr/share/man' '--docdir=/usr/share/doc/packages'
> '--disable-rpath' '--enable-nls' '--enable-thread-safety'
> '--enable-integer-datetimes' '--without-python' '--without-perl'
> '--without-tcl' '--with-openssl' '--with-pam' '--with-krb5' '--with-gssapi'
> '--with-ldap' '--with-libxml' '--with-libxslt'
> '--with-system-tzdata=/usr/share/zoneinfo' 'CFLAGS=-fomit-frame-pointer
> -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector
> -funwind-tables -fasynchronous-unwind-tables '
> CC = gcc
> CPPFLAGS = -D_GNU_SOURCE -I/usr/include/libxml2
> CFLAGS = -fomit-frame-pointer -fmessage-length=0 -O2 -Wall
> -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables
> -fasynchronous-unwind-tables -Wall -Wmissing-prototypes -Wpointer-arith
> -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv
> CFLAGS_SL = -fpic
> LDFLAGS = -Wl,--as-needed
> LDFLAGS_SL =
> LIBS = -lpgport -lxslt -lxml2 -lpam -lssl -lcrypto -lkrb5 -lcom_err
> -lgssapi_krb5 -lz -lreadline -lcrypt -ldl -lm VERSION = PostgreSQL 8.4.1
>
>
> The two main database are quiet new 5.1 and 8.4.1.
>
> I really don't understand why I didn't get batch enable insert to on
> openSUSE 11.2, and need some advice to find why and/or where to look. So
> perharps i could complain in bugzilla.novell.com
>
>
> ps : I'm using the default new desktop kernel (mainly preempt+pae)
> uname -a
> Linux c-3po 2.6.31.5-0.1-desktop #1 SMP PREEMPT 2009-10-26 15:49:03 +0100
> i686 i686 i386 GNU/Linux
>
>
>
> rest of mail is for archive.
>
> On 12/16/2009 08:24 AM, Kern Sibbald wrote:
> > Hello Bruno,
> >
> > On Wednesday 16 December 2009 07:18:48 Bruno Friedmann wrote:
> >> On 12/14/2009 10:00 PM, John Jorgensen wrote:
> >>>>>>>> "bruno" == Bruno Friedmann <[email protected]> writes:
> >>>
> >>> bruno> ok I've try and only get this on the two proposed
> >>>
> >>> bruno> strings /usr/sbin/bacula-dir | grep batch
> >>> bruno> _Z27db_write_batch_file_recordsP3JCR
> >>> bruno> _Z23db_open_batch_connexionP3JCRP4B_DB
> >>>
> >>> I think that the string that Eric suggested searching for might
> >>> have changed from one release of bacula to another.
> >>>
> >>> So in case you're as far behind the times as our site (we're
> >>> still running Bacula 2.4.3) ...
> >>>
> >>> While I'm pretty sure our old server is doing batch inserts,
> >>> grep'ing for "batch" produces only:
> >>>
> >>> 558 backup $strings /usr/local/bacula/sbin/bacula-dir | grep -i batch
> >>> Cannot select %s in batch mode.
> >>>
> >>> Searching for "temporary" hits though:
> >>>
> >>> 559 backup $strings /usr/local/bacula/sbin/bacula-dir | grep -i
> >>> temporary CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER NOT NULL,
> >>> PurgedFiles SMALLINT, FileSetId INTEGER, JobFiles INTEGER, JobStatus
> >>> char(1)) CREATE TEMPORARY TABLE temp (JobId INTEGER NOT NULL,JobTDate
> >>> BIGINT,ClientId INTEGER,Level CHAR,JobFiles INTEGER,JobBytes
> >>> BIGINT,StartTime TEXT,VolumeName TEXT,StartFile INTEGER,VolSessionId
> >>> INTEGER,VolSessionTime INTEGER) CREATE TEMPORARY TABLE temp1 (JobId
> >>> INTEGER NOT NULL,JobTDate BIGINT) 560 backup $
> >>>
> >>> As does looking for the message that I see in bconsole while the
> >>> directory is unbatching the attributes:
> >>>
> >>> 560 backup $strings /usr/local/bacula/sbin/bacula-dir | grep -i
> >>> inserting Inserting first %s res: %s index=%d
> >>> Inserting %s res: %s index=%d pass=%d
> >>> Dir inserting Attributes
> >>
> >> Thanks John on a 3.0.3 build the temporary search give more instructions
> >> strings /usr/sbin/bacula-dir | grep -i temporary
> >> CREATE TEMPORARY TABLE cpy_tmp AS SELECT MIN(JobId) AS JobId FROM Job
> >> WHERE Type='%c' AND ( PriorJobId IN (%s) OR PriorJobId IN ( SELECT
> >> PriorJobId FROM Job WHERE JobId IN (%s) AND Type='B' ) ) GROUP BY
> >> PriorJobId
> >>
> >> and in libbacsql.so
> >>
> >> strings libbacsql.so.1.0.0 | grep -i temporary
> >> CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER UNSIGNED NOT NULL,
> >> PurgedFiles TINYINT, FileSetId INTEGER UNSIGNED, JobFiles INTEGER
> >> UNSIGNED, JobStatus BINARY(1))
> >> CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER NOT NULL,
> >> PurgedFiles SMALLINT, FileSetId INTEGER, JobFiles INTEGER, JobStatus
> >> char(1)) CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER UNSIGNED
> >> NOT NULL, PurgedFiles TINYINT, FileSetId INTEGER UNSIGNED, JobFiles
> >> INTEGER UNSIGNED, JobStatus CHAR)
> >> CREATE TEMPORARY TABLE temp (JobId INTEGER UNSIGNED NOT NULL,JobTDate
> >> BIGINT UNSIGNED,ClientId INTEGER UNSIGNED,Level CHAR,JobFiles INTEGER
> >> UNSIGNED,JobBytes BIGINT UNSIGNED,StartTime TEXT,VolumeName
> >> TEXT,StartFile INTEGER UNSIGNED,VolSessionId INTEGER
> >> UNSIGNED,VolSessionTime INTEGER UNSIGNED)
> >> CREATE TEMPORARY TABLE temp (JobId INTEGER NOT NULL,JobTDate
> >> BIGINT,ClientId INTEGER,Level CHAR,JobFiles INTEGER,JobBytes
> >> BIGINT,StartTime TEXT,VolumeName TEXT,StartFile INTEGER,VolSessionId
> >> INTEGER,VolSessionTime INTEGER) CREATE TEMPORARY TABLE temp1 (JobId
> >> INTEGER UNSIGNED NOT NULL,JobTDate BIGINT UNSIGNED) CREATE TEMPORARY
> >> TABLE temp1 (JobId INTEGER NOT NULL,JobTDate BIGINT)
> >>
> >> So in this case I can deduct that there's batch-insert enable ..
> >
> > None of the above look to me like the Batch Insert code. I am not the
> > expert, but these are not the SQL statements that Eric said would be
> > present in Batch Insert.
> >
> >> BUT when I check against a package newly build with
> >> --disable-batch-inssert confirmed by the result of config
> >>
> >> Configuration on Wed Dec 16 06:33:02 CET 2009:
> >>
> >> Host: i686-pc-linux-gnu -- suse 11.2
> >> Bacula version: Bacula 3.0.3 (18 October 2009)
> >> Source code location: .
> >> Install binaries: /usr/sbin
> >> Install libraries: /usr/lib
> >> Install config files: /etc/bacula
> >> Scripts directory: /usr/lib/bacula
> >> Archive directory: /tmp
> >> Working directory: /var/lib/bacula
> >> PID directory: /var/run
> >> Subsys directory: /var/lock/subsys
> >> Man directory: /usr/share/man
> >> Data directory: /usr/share
> >> Plugin directory: /usr/lib/bacula
> >> C Compiler: gcc Linux)
> >> C++ Compiler: /usr/bin/g++ Linux)
> >> Compiler flags: -O2 -g -msse -mmmx -march=i586 -mtune=i686
> >> -funroll-loops -mfpmath=sse -fmessage-length=0 -fno-strict-aliasing
> >> -pipe -fstack-protector -Wall -D_FORTIFY_SOURCE=0 -fno-strict-aliasing
> >> -fno-exceptions -fno-rtti
> >>
> >> Linker flags: -I/usr/lib -L/usr/lib/termcap
> >>
> >> Libraries: -lpthread -ldl
> >>
> >> Statically Linked Tools: no
> >>
> >> Statically Linked FD: no
> >>
> >> Statically Linked SD: no
> >>
> >> Statically Linked DIR: no
> >>
> >> Statically Linked CONS: no
> >>
> >> Database type: MySQL
> >>
> >> Database port:
> >>
> >> Database lib: -L/usr/lib -lmysqlclient_r -lz
> >>
> >> Database name: bacula
> >>
> >> Database user: bacula
> >>
> >>
> >> Job Output Email: r...@localhost
> >> Traceback Email: r...@localhost
> >> SMTP Host Address: localhost
> >>
> >> Director Port: 9101
> >> File daemon Port: 9102
> >> Storage daemon Port: 9103
> >>
> >> Director User: bacula
> >> Director Group: bacula
> >> Storage Daemon User: bacula
> >> Storage DaemonGroup: disk
> >> File Daemon User: root
> >> File Daemon Group: bacula
> >>
> >> SQL binaries Directory /usr/bin
> >>
> >> Large file support: yes
> >> Bacula conio support: yes -ltermcap
> >> readline support: no
> >> TCP Wrappers support: yes -lwrap
> >> TLS support: yes
> >> Encryption support: yes
> >> ZLIB support: yes
> >> enable-smartalloc: yes
> >> enable-lockmgr: no
> >> bat support: no
> >> enable-gnome: no
> >> enable-bwx-console: no
> >> enable-tray-monitor: no
> >> client-only: no
> >> build-dird: yes
> >> build-stored: yes
> >> ACL support: yes
> >> XATTR support: yes
> >> Python support: yes -lpthread -ldl -lutil -lm -lpython2.6
> >> -lutil -lrt Batch insert enabled: no
> >>
> >>
> >> I'm getting the same string inside bacula-dir and libbacsql.so ?
> >>
> >> CREATE TEMPORARY TABLE cpy_tmp AS SELECT MIN(JobId) AS JobId FROM Job
> >> WHERE Type='%c' AND ( PriorJobId IN (%s) OR PriorJobId IN ( SELECT
> >> PriorJobId FROM Job WHERE JobId IN (%s) AND Type='B' ) ) GROUP BY
> >> PriorJobId
> >>
> >> strings libbacsql.so.1.0.0 | grep -i temporary
> >> CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER UNSIGNED NOT NULL,
> >> PurgedFiles TINYINT, FileSetId INTEGER UNSIGNED, JobFiles INTEGER
> >> UNSIGNED, JobStatus BINARY(1))
> >> CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER NOT NULL,
> >> PurgedFiles SMALLINT, FileSetId INTEGER, JobFiles INTEGER, JobStatus
> >> char(1)) CREATE TEMPORARY TABLE DelCandidates (JobId INTEGER UNSIGNED
> >> NOT NULL, PurgedFiles TINYINT, FileSetId INTEGER UNSIGNED, JobFiles
> >> INTEGER UNSIGNED, JobStatus CHAR)
> >> CREATE TEMPORARY TABLE temp (JobId INTEGER UNSIGNED NOT NULL,JobTDate
> >> BIGINT UNSIGNED,ClientId INTEGER UNSIGNED,Level CHAR,JobFiles INTEGER
> >> UNSIGNED,JobBytes BIGINT UNSIGNED,StartTime TEXT,VolumeName
> >> TEXT,StartFile INTEGER UNSIGNED,VolSessionId INTEGER
> >> UNSIGNED,VolSessionTime INTEGER UNSIGNED)
> >> CREATE TEMPORARY TABLE temp (JobId INTEGER NOT NULL,JobTDate
> >> BIGINT,ClientId INTEGER,Level CHAR,JobFiles INTEGER,JobBytes
> >> BIGINT,StartTime TEXT,VolumeName TEXT,StartFile INTEGER,VolSessionId
> >> INTEGER,VolSessionTime INTEGER) CREATE TEMPORARY TABLE temp1 (JobId
> >> INTEGER UNSIGNED NOT NULL,JobTDate BIGINT UNSIGNED) CREATE TEMPORARY
> >> TABLE temp1 (JobId INTEGER NOT NULL,JobTDate BIGINT)
> >>
> >>
> >> So there's two solutions, the strings are always present, or there's
> >> somewhere a default in sources or build ....
> >>
> >> Eric can you recheck about the string on a 3.0.3 pure git trunk ?
> >
> > It is pretty clear from the code that there will be differences in the
> > SQL statements that are in the binary files when Batch Insert is on and
> > when it is off. If you are getting the same code with Batch Insert on
> > and off, then the problem lies in MySQL not having the multithread option
> > turned on when it was built (i.e. it has no references to
> > pthread_mutex_lock in the MySQL client library that Bacula links in). If
> > that is the case, batch insert will be flagged as "no" in the output of
> > ./configure.
> >
> > If you are getting the same strings in both bacula-dir and libbacsql.so,
> > then you are mixing code. That is you have built bacula-dir statically
> > and dynamically. You probably need to do "make uninstall" between make
> > installs, but make uninstall is so seldom used, I am not 100% sure it
> > handles uninstalling the shared objects.
> >
> > The CREATE TEMPORARY TABLE cpy_tmp should appear only in bacula-dir and
> > not in libbacsq.so
> >
> >> What make me in doubt is that bacula-dir/libbacsql with ou without batch
> >> enabled have the same md5sum ? b9156e6cb7902024c790add0291216bc
> >> bacula-dir (without bi)
> >> b9156e6cb7902024c790add0291216bc /usr/sbin/bacula-dir
> >> f01fe8fb42648dae586762b44987bfe1 libbacsql.so.1.0.0 (without bi)
> >> f01fe8fb42648dae586762b44987bfe1 /usr/lib/libbacsql.so.1.0.0
> >>
> >> What are the result for you ?
> >
> > Given what I wrote above, you are clearly not comparing Bacula with and
> > without Batch Insert.
> >
> > Regards,
> >
> > Kern
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
