Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hi Otto, On Sun, Oct 04, 2020 at 12:39:06PM +0300, Otto Kekäläinen wrote: > So far I've only done > https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/3759ce315280328098d4ed8032e91bdb295f98bc > and I am not sure what the ultimate solution here is. Please advise if > you have an opinion on this. I do think that this is the way to go. Let's see whether anyone has any practical need to change this. If we ever revisit this, the first thing to look at is mariadb_config. In order to ever mark it M-A:same, mariadb_config must be removed. This is a huge task. Not for the faint of heart and maybe not worth the trouble. What I'll do however is remove users of mariadb_config. I'll migrate them to pkg-config. For instance, a patch to qtbase-opensource-src is pending about this. Once this work has advanced, we might reconsider. Helmut
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hello! > Keep in mind however what the goal is. Whenever you think about > multiarching a -dev package, the only use case is cross building. But > most of the time, cross building never requires marking -dev packages > multiarch. It just makes it more convenient for developers. So really, > removing the M-A is a simple stop gap solution that works for 90% of the > cases. > > Please spend your precious time elsewhere if possible and just drop the > M-A:same. Working on fixing multi-arch on branch https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commits/bugfix/multi-arch So far I've only done https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/3759ce315280328098d4ed8032e91bdb295f98bc and I am not sure what the ultimate solution here is. Please advise if you have an opinion on this.
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
I've removed the multi-arch: same for libmariadbd-dev in https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/173b5ffbd40e448ecc5fc47312fa23ea30e1f0b5 but either way I do it Lintian keeps complaining and I get confused of what is the final and best solution for mariadb-10.5 on this front. Merge requests / patches are welcome.
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hi Otto, On Wed, Sep 30, 2020 at 12:14:20AM +0300, Otto Kekäläinen wrote: > If I move just the header file but not all of > /usr/include/mariadb/server, will there be issues with paths at some > point? Likely yes. For using these headers, you specify a custo -I flag (that likely comes from pkg-config or something similar). You'd need to insert another -I flag into every consumer. > I don't have any sample program to build and test against > libmariadbd-dev and I am not a C/C++ coder so this starts to be beyond > my level. You'd likely have to test this with every reverse dependency of which there seem to be around 200. Keep in mind however what the goal is. Whenever you think about multiarching a -dev package, the only use case is cross building. But most of the time, cross building never requires marking -dev packages multiarch. It just makes it more convenient for developers. So really, removing the M-A is a simple stop gap solution that works for 90% of the cases. Please spend your precious time elsewhere if possible and just drop the M-A:same. Helmut
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
> $ wget
> http://deb.debian.org/debian/pool/main/m/mariadb-10.5/libmariadbd-dev_10.5.5-1_amd64.deb
> ...
> $ wget
> http://deb.debian.org/debian/pool/main/m/mariadb-10.5/libmariadbd-dev_10.5.5-1_arm64.deb
> ...
> $ dpkg-deb -x libmariadbd-dev_10.5.5-1_amd64.deb amd64
> $ dpkg-deb -x libmariadbd-dev_10.5.5-1_arm64.deb arm64
> $ diff -u a*64/usr/include/mariadb/server/my_config.h
> --- amd64/usr/include/mariadb/server/my_config.h2020-09-25
> 18:56:59.0 +0200
> +++ arm64/usr/include/mariadb/server/my_config.h2020-09-25
> 18:56:59.0 +0200
$ diff -u a*64/usr/include/mariadb/server/my_config.h | diffstat
my_config.h | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
OK, so they have custom contents based on arch, but just that header
file, nothing else.
> The usual solution to fix this is moving the header to a multiarch
> location (i.e. below /usr/include/${DEB_HOST_MULTIARCH}).
If I move just the header file but not all of
/usr/include/mariadb/server, will there be issues with paths at some
point?
I don't have any sample program to build and test against
libmariadbd-dev and I am not a C/C++ coder so this starts to be beyond
my level.
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hi Otto,
On Tue, Sep 29, 2020 at 11:46:26PM +0300, Otto Kekäläinen wrote:
> Multiarch-hinter says: libmariadbd-dev conflicts on
> /usr/include/mariadb/server/my_config.h on any two of amd64, arm64,
> armel, armhf, and 5 more
>
> I don't understand this, since my_config.h is supposed to be an ASCII
> file and identical to all archs.
>
> Running in Debian unstable to verify the file contents:
>
> apt install libmariadbd-dev
> cat /usr/include/mariadb/server/my_config.h
$ wget
http://deb.debian.org/debian/pool/main/m/mariadb-10.5/libmariadbd-dev_10.5.5-1_amd64.deb
...
$ wget
http://deb.debian.org/debian/pool/main/m/mariadb-10.5/libmariadbd-dev_10.5.5-1_arm64.deb
...
$ dpkg-deb -x libmariadbd-dev_10.5.5-1_amd64.deb amd64
$ dpkg-deb -x libmariadbd-dev_10.5.5-1_arm64.deb arm64
$ diff -u a*64/usr/include/mariadb/server/my_config.h
--- amd64/usr/include/mariadb/server/my_config.h2020-09-25
18:56:59.0 +0200
+++ arm64/usr/include/mariadb/server/my_config.h2020-09-25
18:56:59.0 +0200
@@ -105,9 +105,9 @@
/* #undef HAVE_CRC32_VPMSUM */
/* Support ARMv8 crc + crypto */
-/* #undef HAVE_ARMV8_CRC */
-/* #undef HAVE_ARMV8_CRYPTO */
-/* #undef HAVE_ARMV8_CRC_CRYPTO_INTRINSICS */
+#define HAVE_ARMV8_CRC 1
+#define HAVE_ARMV8_CRYPTO 1
+#define HAVE_ARMV8_CRC_CRYPTO_INTRINSICS 1
/* Does "struct timespec" have a "sec" and "nsec" field? */
/* #undef HAVE_TIMESPEC_TS_SEC */
@@ -395,8 +395,8 @@
#define STACK_DIRECTION -1
#define SYSTEM_TYPE "debian-linux-gnu"
-#define MACHINE_TYPE "x86_64"
-#define DEFAULT_MACHINE "x86_64"
+#define MACHINE_TYPE "aarch64"
+#define DEFAULT_MACHINE "aarch64"
/* #undef HAVE_DTRACE */
#define SIGNAL_WITH_VIO_CLOSE 1
$
The usual solution to fix this is moving the header to a multiarch
location (i.e. below /usr/include/${DEB_HOST_MULTIARCH}).
The next best solution is dropping Multi-Arch: same from libmariadb-dev.
Do note that Multi-Arch: same is not required for using libmariadb-dev
for cross building.
Hope this helps
Helmut
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
> > W dependency-is-not-multi-archified libmariadb-dev-compat depends on > > libmariadb-dev (multi-arch: no) > > W dependency-is-not-multi-archified libmariadbd-dev depends on > > libmariadb-dev (multi-arch: no) > > W dependency-is-not-multi-archified mariadb-plugin-gssapi-client > > depends on mariadb-client-10.5 (multi-arch: no) > > I fear that there is little you can do here. The hinter likely told you > that you can mark libmariadb-dev-compat Multi-Arch: same. Lintian told > you that Multi-Arch: same is useless since libmariadb-dev is not thus > marked. Both are correct. The idea of the hinter is that by > parallelizing the addition of M-A:same, we get everything marked quicker > while having some useless (but correct) markings in the interim. The > lintian stance is that useless markings issue a warning. Multiarch-hinter says: libmariadbd-dev conflicts on /usr/include/mariadb/server/my_config.h on any two of amd64, arm64, armel, armhf, and 5 more I don't understand this, since my_config.h is supposed to be an ASCII file and identical to all archs. Running in Debian unstable to verify the file contents: apt install libmariadbd-dev cat /usr/include/mariadb/server/my_config.h
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hi Otto, On Tue, Sep 29, 2020 at 08:59:49PM +0300, Otto Kekäläinen wrote: > Tested patch in > https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/2a8304051a3107fe828314b66b33aca7ecb863d4 > > Builds went fine, so I'll add this commit on top of the master branch. Cool. My build environment must be buggy then. In any case, this change makes cross building mariadb a lot easier. > The topic of cross-building tangents the multi-arch definitions in the > control file which currently are not correct and I am not sure on how > to correctly solve it without regressions: > > W dependency-is-not-multi-archified libmariadb-dev-compat depends on > libmariadb-dev (multi-arch: no) > W dependency-is-not-multi-archified libmariadbd-dev depends on > libmariadb-dev (multi-arch: no) > W dependency-is-not-multi-archified mariadb-plugin-gssapi-client > depends on mariadb-client-10.5 (multi-arch: no) I fear that there is little you can do here. The hinter likely told you that you can mark libmariadb-dev-compat Multi-Arch: same. Lintian told you that Multi-Arch: same is useless since libmariadb-dev is not thus marked. Both are correct. The idea of the hinter is that by parallelizing the addition of M-A:same, we get everything marked quicker while having some useless (but correct) markings in the interim. The lintian stance is that useless markings issue a warning. Helmut
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Thanks! Tested patch in https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/2a8304051a3107fe828314b66b33aca7ecb863d4 Builds went fine, so I'll add this commit on top of the master branch. The topic of cross-building tangents the multi-arch definitions in the control file which currently are not correct and I am not sure on how to correctly solve it without regressions: W dependency-is-not-multi-archified libmariadb-dev-compat depends on libmariadb-dev (multi-arch: no) W dependency-is-not-multi-archified libmariadbd-dev depends on libmariadb-dev (multi-arch: no) W dependency-is-not-multi-archified mariadb-plugin-gssapi-client depends on mariadb-client-10.5 (multi-arch: no)
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hi Otto,
On Mon, Sep 28, 2020 at 10:24:16PM +0300, Otto Kekäläinen wrote:
> > While looking into it anyway, I was wondering why you are running cmake
> > directly instead of running it through dh_auto_configure. Can you shed
> > light on that question? Using dh_auto_configure would automatically pass
> > -DCMAKE_SYSTEM_NAME=..., which is required for cross compiling mariadb.
>
> The cmake line has been manually defined in the debian/rules files
> since the beginning of time and mysql-5.0 packaging or something.
> There are so many customizations – I don't know if we can dismantle
> it.
Yeah, but that seems like "historical cruft" to me. I'm attaching a
patch that makes it use dh_auto_configure. Unfortunately, when I build
it with that patch, I get test suite failures about "debug_sync" being
unknown. I'm not sure whether that is caused by my build environment or
the change. Can you make that work?
For amd64, the generated cmake invocation becomes:
install -d builddir
cd builddir && cmake -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_SYSCONFDIR=/etc
-DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON "-GUnix Makefiles"
-DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu
"-DCOMPILATION_COMMENT=Debian unstable " -DMYSQL_SERVER_SUFFIX=-1.1
-DSYSTEM_TYPE=debian-linux-gnu -DBUILD_CONFIG=mysql_release -DWITH_SSL=bundled
-DPLUGIN_TOKUDB=NO -DPLUGIN_CASSANDRA=NO -DPLUGIN_AWS_KEY_MANAGEMENT=NO
-DPLUGIN_COLUMNSTORE=NO -DWITH_INNODB_SNAPPY=ON -DDEB=Debian ..
The stuff that comes before -DCOMPILATION_COMMENT=... is inserted by
debhelper.
Helmut
diff --minimal -Nru mariadb-10.5-10.5.5/debian/changelog
mariadb-10.5-10.5.5/debian/changelog
--- mariadb-10.5-10.5.5/debian/changelog2020-09-25 18:56:59.0
+0200
+++ mariadb-10.5-10.5.5/debian/changelog2020-09-28 22:22:58.0
+0200
@@ -1,3 +1,10 @@
+mariadb-10.5 (1:10.5.5-1.1) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+ * Use dh_auto_configure.
+
+ -- Helmut Grohne Mon, 28 Sep 2020 22:22:58 +0200
+
mariadb-10.5 (1:10.5.5-1) unstable; urgency=medium
* New upstream version 10.5.5 (Closes: #968895)
diff --minimal -Nru mariadb-10.5-10.5.5/debian/rules
mariadb-10.5-10.5.5/debian/rules
--- mariadb-10.5-10.5.5/debian/rules2020-09-17 22:17:28.0 +0200
+++ mariadb-10.5-10.5.5/debian/rules2020-09-28 22:22:57.0 +0200
@@ -22,9 +22,6 @@
RELEASE := $(shell lsb_release -r -s) # Use changelog based DEB_DISTRIBUTION
instead?
TMP:=$(CURDIR)/debian/tmp
-CC := $(DEB_HOST_GNU_TYPE)-gcc
-CXX := $(DEB_HOST_GNU_TYPE)-g++
-
# According to Debian Policy version 4.2.0 builds should be as verbose as
# possible unless 'terse' is specifically passed.
ifeq (,$(filter terse,$(DEB_BUILD_OPTIONS)))
@@ -94,18 +91,14 @@
# Remove -DWITH_SSL=bundle if you want to use system OpenSSL, otherwise
# the server will use YaSSL and Connector/C will use GnuTLS.
# Don't build ColumnStore, not mature enough for Debian yet.
- mkdir -p $(BUILDDIR) && cd $(BUILDDIR) && \
- sh -c
'PATH=$${MYSQL_BUILD_PATH:-"/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin"} \
- CC=${CC} \
- CXX=${CXX} \
- NO_UPDATE_BUILD_VERSION=1 \
- cmake -DCMAKE_INSTALL_PREFIX=/usr \
+
PATH=$${MYSQL_BUILD_PATH:-"/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin"} \
+ NO_UPDATE_BUILD_VERSION=1 \
+ dh_auto_configure --builddirectory=$(BUILDDIR) -- \
$(CMAKEFLAGS) \
$(if $(filter
$(DEB_BUILD_ARCH),$(DEB_HOST_ARCH)),,-DIMPORT_EXECUTABLES=$(CURDIR)/builddir-native/import_executables.cmake)
\
-DCOMPILATION_COMMENT="$(DEB_VENDOR) $(RELEASE)" \
-DMYSQL_SERVER_SUFFIX="-$(DEB_VERSION_REVISION)" \
-DSYSTEM_TYPE="debian-$(DEB_HOST_GNU_SYSTEM)" \
- -DCMAKE_SYSTEM_PROCESSOR=$(DEB_HOST_ARCH) \
-DBUILD_CONFIG=mysql_release \
-DWITH_SSL=bundled \
-DPLUGIN_TOKUDB=NO \
@@ -113,7 +106,7 @@
-DPLUGIN_AWS_KEY_MANAGEMENT=NO \
-DPLUGIN_COLUMNSTORE=NO \
-DWITH_INNODB_SNAPPY=ON \
- -DDEB=$(DEB_VENDOR) ..'
+ -DDEB=$(DEB_VENDOR)
# This is needed, otherwise 'make test' will run before binaries have been
built
override_dh_auto_build:
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
> I've started looking into it. I noticed a few embedded code copies. I've > filed #971005 about readline. Bundling an ssl library seems like a very > bad idea to me. The outcome of these will have significant impact on how > cross compilation can be implemented. I've filed issues upstream that they should build against the system gnutls and system wolfssl. I believe the server now builds against the system gnutls but there are still bundled codes in the sources. > While looking into it anyway, I was wondering why you are running cmake > directly instead of running it through dh_auto_configure. Can you shed > light on that question? Using dh_auto_configure would automatically pass > -DCMAKE_SYSTEM_NAME=..., which is required for cross compiling mariadb. The cmake line has been manually defined in the debian/rules files since the beginning of time and mysql-5.0 packaging or something. There are so many customizations – I don't know if we can dismantle it.
Bug#920365: [debian-mysql] Bug#920365: Bug#920365: mariadb_config: improve cross compilation support
Hello Helmut! Would you be kind and check out mariadb-10.5 currently in Debian experimental, and report what the status is and what changes would still be needed? Thanks!
Bug#920365: [debian-mysql] Bug#920365: mariadb_config: improve cross compilation support
> > OK, but for MariaDB the > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=707750 does not > > apply anymore thanks to > > https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/b50ee2eb0bdca49583cf447ded46f799198cd4bf > > Great news. As far a s I can see, it still applies to mariadb-10.3, but > not to mariadb-10.4. The metadata of the bug correctly reflects that. > Once mariadb-10.4 hits unstable, I'll likely look into it. If you > backport this change to mariadb-10.3, please tell me explicitly. I will not do any development for mariadb-10.3 anymore, only maintain it in stable and oldstable. mariadb-10.4 is now in experimental and will soon be uploaded to unstable. I will ping you when that happens. Thanks for your help! - Otto
Bug#920365: [debian-mysql] Bug#920365: mariadb_config: improve cross compilation support
Hi Otto, On Mon, Mar 02, 2020 at 02:54:05PM +0200, Otto Kekäläinen wrote: > OK, now I get it. I could have the script in debian/additions etc and > run the above step in the rules file. Thank you. > > Of course, doing so breaks cross building mariadb itself. However that's > > known to be broken due to #707750, which hasn't shown any progress in 6 > > years and likely won't show any progress in the next 6 years. > > OK, but for MariaDB the > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=707750 does not > apply anymore thanks to > https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/b50ee2eb0bdca49583cf447ded46f799198cd4bf Great news. As far a s I can see, it still applies to mariadb-10.3, but not to mariadb-10.4. The metadata of the bug correctly reflects that. Once mariadb-10.4 hits unstable, I'll likely look into it. If you backport this change to mariadb-10.3, please tell me explicitly. Helmut
Bug#920365: [debian-mysql] Bug#920365: mariadb_config: improve cross compilation support
Hello! > > I am happy to do these changes for you in the Debian package, but I > > wasn't quite able to follow what script do you mean is the > > replacement? > > The bug submission has a shell script attached. > > $ ./mysql_config_generator.sh debian/libmariadb-dev/usr/bin/mariadb_config > > debian/mariadb_config.new > $ chmod +x debian/mariadb_config.new > $ mv debian/mariadb_config.new debian/libmariadb-dev/usr/bin/mariadb_config OK, now I get it. I could have the script in debian/additions etc and run the above step in the rules file. > Of course, doing so breaks cross building mariadb itself. However that's > known to be broken due to #707750, which hasn't shown any progress in 6 > years and likely won't show any progress in the next 6 years. OK, but for MariaDB the https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=707750 does not apply anymore thanks to https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/b50ee2eb0bdca49583cf447ded46f799198cd4bf
Bug#920365: [debian-mysql] Bug#920365: mariadb_config: improve cross compilation support
Hi Otto, On Tue, Aug 27, 2019 at 11:47:17AM +0200, Otto Kekäläinen wrote: > Hello! Apparently, I missed your reply. Sorry for the delay. > I am happy to do these changes for you in the Debian package, but I > wasn't quite able to follow what script do you mean is the > replacement? The bug submission has a shell script attached. $ ./mysql_config_generator.sh debian/libmariadb-dev/usr/bin/mariadb_config > debian/mariadb_config.new $ chmod +x debian/mariadb_config.new $ mv debian/mariadb_config.new debian/libmariadb-dev/usr/bin/mariadb_config $ Of course, doing so breaks cross building mariadb itself. However that's known to be broken due to #707750, which hasn't shown any progress in 6 years and likely won't show any progress in the next 6 years. Helmut
Bug#920365: [debian-mysql] Bug#920365: mariadb_config: improve cross compilation support
Hello! > The only thing that mariadb upstream could do here is officially > deprecate mariadb_config. I'm not sure whether you want that though. > > > pkg-config just works and you already have the script? > > Yeah, I'm asking for integrating it into the Debian package. The Debian > package could replace the shipped mariadb_config ELF executable with the > generated shell script. Doing so would make a number of downstream > packages that don't use pkg-config cross buildable. I am happy to do these changes for you in the Debian package, but I wasn't quite able to follow what script do you mean is the replacement? Currently the Debian package ships mariadb_config (libmariadb-dev.install:usr/bin/mariadb_config, and mysql_config is a symlink to it). I looking at build products in mariadb-10.3 I don't see that alternative scripts. What is the name/path?
Bug#920365: mariadb_config: improve cross compilation support
Hi Sergei, On Fri, Aug 16, 2019 at 07:29:02PM +0200, Sergei Golubchik wrote: > So, there's nothing for us (upstream) to do here? Correct, I think this is not an upstream issue. It's a packaging issue. The only thing that mariadb upstream could do here is officially deprecate mariadb_config. I'm not sure whether you want that though. > pkg-config just works and you already have the script? Yeah, I'm asking for integrating it into the Debian package. The Debian package could replace the shipped mariadb_config ELF executable with the generated shell script. Doing so would make a number of downstream packages that don't use pkg-config cross buildable. Helmut
Bug#920365: mariadb_config: improve cross compilation support
Hi, Helmut! So, there's nothing for us (upstream) to do here? pkg-config just works and you already have the script? Regards, Sergei On Aug 16, Helmut Grohne wrote: > Hi Sergei, > > On Fri, Aug 16, 2019 at 05:41:14PM +0200, Sergei Golubchik wrote: > > Ok. Two thoughts. First, libmariadb comes with mariadb.pc pkg-config > > file. Does it have all you need to build? If not, we can, perhaps, add > > missing info to it. > > The pkg-config part works great for cross compilation. There is nothing > you need to change here. It really just works. The problem is that not > everything uses pkg-config. Some downstreams prefer using mariadb_config > or more commonly mysql_config. Deprecating mysql_config and telling > everyone to use pkg-config could be part of a long-term solution and > I've done that to some packages (e.g. collectd, courier-authlib, > lighttpd, rsyslog, sphinxsearch, ulogd2), but that's a lot of effort to > get there. > > One package that is particularly hard to convert to using pkg-config is > mysql-connector-c++. See > https://sources.debian.org/src/mysql-connector-c++/1.1.12-4/FindMySQL.cmake/. > Please try converting this to pkg-config. > > > Second, binary mariadb_config does nothing else but printing a set > > of string. If there's no other choice, we could consider providing a > > shell script with the same functionality too. > > Exactly. The script I attached to the bug submission converts > mariadb_config into a shell script that behaves exactly the same. You > point it at mariadb_config and you get back a shell script that > behaves exactly the same way. The major difference here is that the > shell script will work for cross compiling downstreams that happen to > not use pkg-config. > > Helmut
Bug#920365: mariadb_config: improve cross compilation support
Hi Sergei, On Fri, Aug 16, 2019 at 05:41:14PM +0200, Sergei Golubchik wrote: > Ok. Two thoughts. First, libmariadb comes with mariadb.pc pkg-config > file. Does it have all you need to build? If not, we can, perhaps, add > missing info to it. The pkg-config part works great for cross compilation. There is nothing you need to change here. It really just works. The problem is that not everything uses pkg-config. Some downstreams prefer using mariadb_config or more commonly mysql_config. Deprecating mysql_config and telling everyone to use pkg-config could be part of a long-term solution and I've done that to some packages (e.g. collectd, courier-authlib, lighttpd, rsyslog, sphinxsearch, ulogd2), but that's a lot of effort to get there. One package that is particularly hard to convert to using pkg-config is mysql-connector-c++. See https://sources.debian.org/src/mysql-connector-c++/1.1.12-4/FindMySQL.cmake/. Please try converting this to pkg-config. > Second, binary mariadb_config does nothing else but printing a set of > string. If there's no other choice, we could consider providing a shell > script with the same functionality too. Exactly. The script I attached to the bug submission converts mariadb_config into a shell script that behaves exactly the same. You point it at mariadb_config and you get back a shell script that behaves exactly the same way. The major difference here is that the shell script will work for cross compiling downstreams that happen to not use pkg-config. Helmut
Bug#920365: mariadb_config: improve cross compilation support
Hi, Helmut! On Aug 13, Helmut Grohne wrote: > > On Sat, Jan 26, 2019 at 07:37:14PM +0100, Sergei Golubchik wrote: > > I would say that when cross-compiling mariadb_config should be built > > for the host architecture. > > That is the status quo, but I guess you're confusing terms here. > Please refer to man dpkg-architecture for what build/host/target mean. > > > You'll get "Exec format error" if you run'll run it on the target > > architecture, but that should not be a problem, as you will not need > > to run mariadb_config on the target architecture. Because > > mariadb_config is a helper tool to build applications, and you build > > on the host architecture for the target architecture. You don't need > > mariadb_config to use libmariadb. > > I'm going to assume that whenever you write "target" you actually mean > "host" and whenever you write "host" you actually mean "build". > Otherwise this paragraph doesn't make any sense at all. Yes, that's what I mean, thanks. I didn't know debian has a defined terminology for this. > I kinda agree that you'd want to have mariadb_config for the build > architecture, but that's not what is currently happening nor is it > reasonably achievable. Currently, mariadb_config is shipped with > libmariadb-dev and you need libmariadb-dev for the host architecture > (for linking it). Thus mariadb_config is a host architecture > executable and cannot be run. Ok. Two thoughts. First, libmariadb comes with mariadb.pc pkg-config file. Does it have all you need to build? If not, we can, perhaps, add missing info to it. Second, binary mariadb_config does nothing else but printing a set of string. If there's no other choice, we could consider providing a shell script with the same functionality too. Regards, Sergei VP of MariaDB Server Engineering and secur...@mariadb.org
Bug#920365: mariadb_config: improve cross compilation support
Hi Sergei,
Please be aware that the Debian BTS does not automatically Cc
submitters. I found your reply by chance.
On Sat, Jan 26, 2019 at 07:37:14PM +0100, Sergei Golubchik wrote:
> I would say that when cross-compiling mariadb_config should be built for
> the host architecture.
That is the status quo, but I guess you're confusing terms here. Please
refer to man dpkg-architecture for what build/host/target mean.
> You'll get "Exec format error" if you run'll run it on the target
> architecture, but that should not be a problem, as you will not need to
> run mariadb_config on the target architecture. Because mariadb_config is
> a helper tool to build applications, and you build on the host
> architecture for the target architecture. You don't need mariadb_config
> to use libmariadb.
I'm going to assume that whenever you write "target" you actually mean
"host" and whenever you write "host" you actually mean "build".
Otherwise this paragraph doesn't make any sense at all.
I kinda agree that you'd want to have mariadb_config for the build
architecture, but that's not what is currently happening nor is it
reasonably achievable. Currently, mariadb_config is shipped with
libmariadb-dev and you need libmariadb-dev for the host architecture
(for linking it). Thus mariadb_config is a host architecture executable
and cannot be run.
Now how would one change that? We'd have to add a new package (say
libmariadb-dev-bin) which contains mariadb_config. Now you'd somehow
install libmariadb-dev for the host architecture (for linking) and
libmariadb-dev-bin for the build architecture (for running
mariadb_config). Unfortunately, the behaviour of mariadb_config is
architecture-dependent as it emits different paths and flags for
different architectures. So you actually need one mariadb_config for
every *pair* of architectures: One is the architecture you want to run
it on and the other is the architecture you want it to tell you things
about. So libmariadb-dev-bin would much rather be
libmariadb-dev-bin-${architecture}. And the number of mariadb_config you
have to build explodes. With 10 release architectures, you build 100
mariadb_config executables. Adding 10 ports architectures and you have
no less than 400 mariadb_config executables. This is going to be messy.
Very messy.
> Do you mean that you've built libmariadb on one (host) architecture,
> copied to another (target) architecture. And then you want to build
> various applications (that use libmariadb) directly on the target
> architecture? That's confusing, if your target architecture has a
> working development environment, why not to build libmariadb there too?
Presently, I'm going to assume that one somehow built libmariadb and
copied it over to another system, yes. However the intention is to
eventually cross build libmariadb itself. The typical use case is
building a customized Debian derivative for embedded systems: You can
use some packages as is, but you need to modify others. Given that your
embedded system is slow, you cross build on a fast X86 machine.
> I'm afraid, I don't understand your use case.
I'm working at all ends concurrently. I don't have to wait for
libmariadb to be cross buildable before I can start working on building
libmariadb's reverse dependencies. The goal is to cross build OS images
from Debian (similar to Yocto/Buildroot/PtxDist). Let's see whether we
get there. This generator is an intermediate step to make progress on
reverse dependencies.
Helmut
Bug#920365: mariadb_config: improve cross compilation support
Hi, Helmut! I don't think I understand what's happening. I would say that when cross-compiling mariadb_config should be built for the host architecture. You'll get "Exec format error" if you run'll run it on the target architecture, but that should not be a problem, as you will not need to run mariadb_config on the target architecture. Because mariadb_config is a helper tool to build applications, and you build on the host architecture for the target architecture. You don't need mariadb_config to use libmariadb. Do you mean that you've built libmariadb on one (host) architecture, copied to another (target) architecture. And then you want to build various applications (that use libmariadb) directly on the target architecture? That's confusing, if your target architecture has a working development environment, why not to build libmariadb there too? I'm afraid, I don't understand your use case. Regards, Sergei On Jan 24, Otto Kekäläinen wrote: > Package: libmariadb-dev > Version: 1:10.3.12-1 > Severity: wishlist > > Hi, > > I'm seeing that very many packages fail to cross build when using > mysql/mariadb. Many of those packages use mysql_config (rarely > mariadb_config). During cross building, there are essentially two ways > to use mysql_config: Either you have the build architecture > mariadb_config. In this case, you can run it, but the values it outputs > are wrong. Or you have the host architecture mariadb_config. In this > case, running mariadb_config fails with "Exec format error", because it > is an ELF binary. In neither case, do you get useful results. > > This is very unfortunate and we've had the exact same situation with > postgresql/pg_config. Then Christoph (Cced) came up with a crazy idea: > Run pg_config at build time, capture all of its output and generate a > shell script from it. Then he turned the packaged pg_config into a perl > script. How does that help? When running the host architecture > pg_config, it is run via /usr/bin/perl, which is a build architecture > executable. The "Exec format error" goes away. One gets the right > results. > > Can we do the same for mariadb_config? I'm attaching a generator script. > If you run sh mysql_config_generator.sh path/to/mariadb_config, it will > output a new shell script to stdout. This new shell script should behave > exactly the same as the original mariadb_config except that it works for > cross compilation. > > The key here is turning mariadb_config from an ELF executable into some > interpreted script. That's what makes it useful for cross compilation. > > What do you think about this?
Bug#920365: mariadb_config: improve cross compilation support
Re: Helmut Grohne 2019-01-24 <20190124190511.ga30...@alf.mars> > This is very unfortunate and we've had the exact same situation with > postgresql/pg_config. Then Christoph (Cced) came up with a crazy idea: > Run pg_config at build time, capture all of its output and generate a > shell script from it. Then he turned the packaged pg_config into a perl > script. How does that help? When running the host architecture > pg_config, it is run via /usr/bin/perl, which is a build architecture > executable. The "Exec format error" goes away. One gets the right > results. Fwiw, the PostgreSQL hack is there: https://salsa.debian.org/postgresql/postgresql/blob/11/debian/pg_config.pl The __DATA__ section there is a stub, it get replaced by the real data at build time. https://salsa.debian.org/postgresql/postgresql/blob/11/debian/rules#L187-191 Christoph
Bug#920365: mariadb_config: improve cross compilation support
Package: libmariadb-dev Version: 1:10.3.12-1 Severity: wishlist Hi, I'm seeing that very many packages fail to cross build when using mysql/mariadb. Many of those packages use mysql_config (rarely mariadb_config). During cross building, there are essentially two ways to use mysql_config: Either you have the build architecture mariadb_config. In this case, you can run it, but the values it outputs are wrong. Or you have the host architecture mariadb_config. In this case, running mariadb_config fails with "Exec format error", because it is an ELF binary. In neither case, do you get useful results. This is very unfortunate and we've had the exact same situation with postgresql/pg_config. Then Christoph (Cced) came up with a crazy idea: Run pg_config at build time, capture all of its output and generate a shell script from it. Then he turned the packaged pg_config into a perl script. How does that help? When running the host architecture pg_config, it is run via /usr/bin/perl, which is a build architecture executable. The "Exec format error" goes away. One gets the right results. Can we do the same for mariadb_config? I'm attaching a generator script. If you run sh mysql_config_generator.sh path/to/mariadb_config, it will output a new shell script to stdout. This new shell script should behave exactly the same as the original mariadb_config except that it works for cross compilation. The key here is turning mariadb_config from an ELF executable into some interpreted script. That's what makes it useful for cross compilation. What do you think about this? Helmut mysql_config_generator.sh Description: Bourne shell script
