Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On 12/23/20 6:18 PM, Michael Orlitzky wrote: Using pkg-config has a related problem. If lua-5.1 is eselected and if the upstream build system runs $(pkg-config ... lua), it's going to get the information for lua-5.1, even if you're trying to build against lua-5.2. So, first I have to patch upstream to use pkg-config, and then I have to hack it in Gentoo to avoid using pkg-config. Sorry, I was wrong about this. Patching the upstream build system to support pkg-config does work if you use the eclasses.
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On 12/23/20 4:51 PM, Mart Raudsepp wrote: Ühel kenal päeval, K, 23.12.2020 kell 07:49, kirjutas Michael Orlitzky: AC_SEARCH_LIBS([whatever], [lua], [lua_found="yes"]) How do I pass the name "lua5.2" to that, without hacking configure.ac and running autoreconf? The only option that comes to mind is to build the entire project with LIBS="-llua5.2", but that's not right if only some of the executables are supposed to be linked with lua. You either fix upstream to use pkg-config, or you set up the cache variable for your configure call to tell configure what the answer is, without making it do the work wrong. Using pkg-config has a related problem. If lua-5.1 is eselected and if the upstream build system runs $(pkg-config ... lua), it's going to get the information for lua-5.1, even if you're trying to build against lua-5.2. So, first I have to patch upstream to use pkg-config, and then I have to hack it in Gentoo to avoid using pkg-config. Overriding the cached check variable does work, but you have to override every single call to AC_SEARCH_LIBS(function...) with ac_cv_search_ which depends highly on the implementation details of configure.ac. This is roughly as invasive and hard to maintain as patching, and should just not be the default way to link against a library IMO.
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
Ühel kenal päeval, K, 23.12.2020 kell 07:49, kirjutas Michael Orlitzky: > AC_SEARCH_LIBS([whatever], [lua], [lua_found="yes"]) > > How do I pass the name "lua5.2" to that, without hacking configure.ac > and running autoreconf? The only option that comes to mind is to > build > the entire project with LIBS="-llua5.2", but that's not right if only > some of the executables are supposed to be linked with lua. You either fix upstream to use pkg-config, or you set up the cache variable for your configure call to tell configure what the answer is, without making it do the work wrong. Should be ac_cv_search_whatever=something in this case. sys-apps/grep ebuild has an example. I'm not sure what package this is to look or test closer. It doesn't appear to be OpenDKIM which is discussed in this thread elsewhere. Mart signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On 12/23/20 8:35 AM, Jaco Kroon wrote: Michael, I'm busy disecting what Marek has done for asterisk as I need to make that work for multiple versions of net-misc/asterisk-16.14.0-r100, not merely the one he did it for - perhaps that would be helpful for you as well? I don't know lua at all. Anyway, I'm on IRC as jkroon if you'd like to exchange notes. I don't particularly like the patch Marek did as I'll NEVER be able to push that upstream. The patch will work for us, but as a policy I highly prefer patches which I can get unstreamed such that we don't need to carry them more than one or two versions. Agreed. This is more of a system library packaging issue than anything to do with Lua specifically. I'd prefer to patch upstream to keep it's behaviour of detecting a lua version, but have a --with-lua(version)? type argument that can take an optional argument which will then be the version, but --with-* generally takes a prefix where the include and lib folders can be found ... and I'd prefer to depend on pkg-config as primary and fallback to the ./configure mess which tries to guess at things ... This is exactly what I'm trying to get across. Support for multiple versions of libraries in weird locations is not a new thing. Generally, you can put CPPFLAGS="-I/foo/bar" and LIBS="-L/baz/quux" before running ./configure and everything will just work with your header files and libraries in the non-standard paths /foo/bar and /baz/quux, respectively. Those paths take precedence over the defaults (if used correctly...), so you can use them to override the default version of a library and compile/link against a specific copy if you need to do that. Likewise, one could prepend a path to PKG_CONFIG_PATH to override the version that will be detected via pkg-config. The problem that we've gotten ourselves into is that we've copied Debian by not only relocating the library, but renaming it (and its *.pc file) entirely. There is no good way to tell an autotools build system that you've renamed a library completely, and that it should prefer the renamed version over the standard one if the standard one also exists. If you want to build against the eselected version of Lua on Gentoo, then eselect-lua makes everything OK. It creates symlinks from the standard names to the nonstandard ones, and everything just works. But if you want to build against a specific, not-eselected version of Lua? You have to tell the build system what to do. The eselected version lives in the correct locations (via those symlinks), so the build system is going to want to find that first. If the eselected version is not the one you want to build against, oops. Then, since the version you *do* want to link against has the wrong name, you have to tell the build system to link against it somehow. You can't simply override the PKG_CONFIG_PATH, because our *.pc files have the wrong names. You can't just override the library search path, because our libraries have the wrong names. The work I've done so far for OpenDKIM does nothing to address these larger problems. If lua.pc is on the system, it will get used first, regardless of the version you actually want to build against. AFAIK there's no good way around this without patching. If the libraries (and pkg-config files?) were installed to subdirectories instead, then the standard method of overrides could be used to build against a specific version, rather than requiring every upstream build system to support our weird layout.
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
Michael, I'm busy disecting what Marek has done for asterisk as I need to make that work for multiple versions of net-misc/asterisk-16.14.0-r100, not merely the one he did it for - perhaps that would be helpful for you as well? I don't know lua at all. Anyway, I'm on IRC as jkroon if you'd like to exchange notes. I don't particularly like the patch Marek did as I'll NEVER be able to push that upstream. The patch will work for us, but as a policy I highly prefer patches which I can get unstreamed such that we don't need to carry them more than one or two versions. I'd prefer to patch upstream to keep it's behaviour of detecting a lua version, but have a --with-lua(version)? type argument that can take an optional argument which will then be the version, but --with-* generally takes a prefix where the include and lib folders can be found ... and I'd prefer to depend on pkg-config as primary and fallback to the ./configure mess which tries to guess at things ... Kind Regards, Jaco On 2020/12/23 14:49, Michael Orlitzky wrote: > On 12/23/20 4:09 AM, Marek Szuba wrote: >> >> I think what you are looking for is lua_get_shared_lib() from >> lua-utils.eclass. We have already got ebuilds in the tree which use >> it. >> > > Knowing the library name only helps if I patch the build system; > that's what I'm getting at. The few packages where this works use > CMake, and CMake already tries to guess[0] the name of the lua library > (thanks Debian) and so it has a pre-defined variable to store the result. > > Not all build systems are going to work like that. Suppose I know that > the name of the lua library is "lua5.2". An autotools build system is > going to run something like, > > AC_SEARCH_LIBS([whatever], [lua], [lua_found="yes"]) > > How do I pass the name "lua5.2" to that, without hacking configure.ac > and running autoreconf? The only option that comes to mind is to build > the entire project with LIBS="-llua5.2", but that's not right if only > some of the executables are supposed to be linked with lua. > > For contrast, if the lua library was stored in /usr/lib64/lua5.2, then > I could just set LIBS="-L/usr/lib64/lua5.2" and let ./configure do its > thing. > > > [0] https://github.com/Kitware/CMake/blob/master/Modules/FindLua.cmake >
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On 12/23/20 4:09 AM, Marek Szuba wrote: I think what you are looking for is lua_get_shared_lib() from lua-utils.eclass. We have already got ebuilds in the tree which use it. Knowing the library name only helps if I patch the build system; that's what I'm getting at. The few packages where this works use CMake, and CMake already tries to guess[0] the name of the lua library (thanks Debian) and so it has a pre-defined variable to store the result. Not all build systems are going to work like that. Suppose I know that the name of the lua library is "lua5.2". An autotools build system is going to run something like, AC_SEARCH_LIBS([whatever], [lua], [lua_found="yes"]) How do I pass the name "lua5.2" to that, without hacking configure.ac and running autoreconf? The only option that comes to mind is to build the entire project with LIBS="-llua5.2", but that's not right if only some of the executables are supposed to be linked with lua. For contrast, if the lua library was stored in /usr/lib64/lua5.2, then I could just set LIBS="-L/usr/lib64/lua5.2" and let ./configure do its thing. [0] https://github.com/Kitware/CMake/blob/master/Modules/FindLua.cmake
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On December 23, 2020 2:56:05 AM UTC, Michael Orlitzky wrote: >One last design issue that I ran into during the migration. > >The slotted lua ebuilds install the headers into subdirectories like >/usr/include/lua5.2, but otherwise with their upstream names. The >libraries, on the other hand, all get installed directly to $libdir.. >but with NON-standard names like liblua5.2.a (as opposed to simply >liblua.a). > >This makes it impossible to build against a specific version of Lua >without using pkg-config. The usual way would be to add >/usr/include/lua5.2 to the compiler's include path, and $libdir/lua5.2 >to its library path (via -I and -L)... but there's no way to tell the >build system to look for a library of an entirely different name. >Things >like AC_SEARCH_LIBS are going to try -llua, and that's it. I think what you are looking for is lua_get_shared_lib() from lua-utils.eclass. We have already got ebuilds in the tree which use it. -- MS
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On 12/22/20 6:15 PM, Marek Szuba wrote: > Dear all, > > As of right now, the blocking slotted-lua tracker depends on 5 (FIVE) open > bugs. The still-to-be migrated packages are: > > app-metrics/collectd - ebuild under review of the maintainer > games-strategy/megaglest - no response from maintainers (games@g.o) > mail-filter/opendkim - committed ebuild needs one extra fix > sci-geosciences/osm2pgsql - no response from maintainers (sci-geo@g.o) > www-apps/cgit - to be migrated by bman > > Therefore, the unmasking of slotted Lua + all the ebuilds migrated to Lua > eclasses will take place on Christmas Eve, i.e. this Thursday, around noon > UTC. > > -- > Marecki Congratulations marecki@ ! Thanks for all your work (and also others, conikost, williamh, others who I am forgetting :P). As with any large enough change, lets hope this works in a chirstmas miracle. :D Cheers, Aisha
Re: [gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
On 12/22/20 6:15 PM, Marek Szuba wrote: Dear all, mail-filter/opendkim - committed ebuild needs one extra fix One last design issue that I ran into during the migration. The slotted lua ebuilds install the headers into subdirectories like /usr/include/lua5.2, but otherwise with their upstream names. The libraries, on the other hand, all get installed directly to $libdir.. but with NON-standard names like liblua5.2.a (as opposed to simply liblua.a). This makes it impossible to build against a specific version of Lua without using pkg-config. The usual way would be to add /usr/include/lua5.2 to the compiler's include path, and $libdir/lua5.2 to its library path (via -I and -L)... but there's no way to tell the build system to look for a library of an entirely different name. Things like AC_SEARCH_LIBS are going to try -llua, and that's it.
[gentoo-dev] Slotted Lua + revdeps to be unmasked on Christmas Eve
Dear all, As of right now, the blocking slotted-lua tracker depends on 5 (FIVE) open bugs. The still-to-be migrated packages are: app-metrics/collectd - ebuild under review of the maintainer games-strategy/megaglest - no response from maintainers (games@g.o) mail-filter/opendkim - committed ebuild needs one extra fix sci-geosciences/osm2pgsql - no response from maintainers (sci-geo@g.o) www-apps/cgit - to be migrated by bman Therefore, the unmasking of slotted Lua + all the ebuilds migrated to Lua eclasses will take place on Christmas Eve, i.e. this Thursday, around noon UTC. -- Marecki
Re: [gentoo-dev] Slotted Lua: 2020-12-04 status update
On 2020-12-09 16:21, Michael Orlitzky wrote: LUA_DEPS itself will not but the change of LUA_SINGLE_TARGET in the package in question will, same way other packages can be rebuilt on USE-flag changes. So lua has inherited the python approach of requiring everyone to use portage? =/ I don't know about requiring everyone to use Portage but Lua eclasses *are* indeed strongly based on Python ones - as openly admitted both in eclassdoc of the three Lua eclasses and during their public review on this mailing list. -- MS OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] Slotted Lua: 2020-12-04 status update
On 12/9/20 10:10 AM, Marek Szuba wrote: LUA_DEPS itself will not but the change of LUA_SINGLE_TARGET in the package in question will, same way other packages can be rebuilt on USE-flag changes. So lua has inherited the python approach of requiring everyone to use portage? =/
Re: [gentoo-dev] Slotted Lua: 2020-12-04 status update
On 2020-12-09 15:56, Michael Orlitzky wrote: I think the slotted lua ebuilds should be doing some eselect-lua stuff in pkg_postinst() and pkg_postrm(). For example: * I have lua-5.1 and lua-5.2 installed * Lua-5.2 is eselected * I uninstall lua-5.2 * Now /usr/lib64/pkgconfig/lua.pc is a dangling symlink I've already mentioned this to WilliamH, it is expected to be done before the unmasking of slotted Lua. Meanwhile (i.e. during work on migrating packages so that they among other things respect the current values of LUA_TARGETS and LUA_SINGLE_TARGET), not having any "current" Lua versions makes it easier to catch looking for Lua where they shouldn't. Related to the above: is there a way to trigger rebuilds of consumers when the "single implementation" changes? If I have a package that links against liblua and if I upgrade from lua-5.1 to lua-5.2, then my package should rebuild. I don't think $LUA_DEPS from lua-single.eclass can do that? LUA_DEPS itself will not but the change of LUA_SINGLE_TARGET in the package in question will, same way other packages can be rebuilt on USE-flag changes. -- Marecki OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] Slotted Lua: 2020-12-04 status update
On 12/7/20 9:11 AM, Marek Szuba wrote: On 2020-12-04 13:16, Marek Szuba wrote: Since a week ago the number of open bugs blocking the slotted-Lua tracker has been reduced from 119 to under 80. Updated count as of a few minutes ago: 64 open tickets! Full list: https://dev.gentoo.org/~marecki/open_blocking_lua_eclass_bugs-20201207135752.txt I think the slotted lua ebuilds should be doing some eselect-lua stuff in pkg_postinst() and pkg_postrm(). For example: * I have lua-5.1 and lua-5.2 installed * Lua-5.2 is eselected * I uninstall lua-5.2 * Now /usr/lib64/pkgconfig/lua.pc is a dangling symlink Related to the above: is there a way to trigger rebuilds of consumers when the "single implementation" changes? If I have a package that links against liblua and if I upgrade from lua-5.1 to lua-5.2, then my package should rebuild. I don't think $LUA_DEPS from lua-single.eclass can do that?
Re: [gentoo-dev] Slotted Lua: 2020-12-04 status update
On 2020-12-04 13:16, Marek Szuba wrote: Since a week ago the number of open bugs blocking the slotted-Lua tracker has been reduced from 119 to under 80. Updated count as of a few minutes ago: 64 open tickets! Full list: https://dev.gentoo.org/~marecki/open_blocking_lua_eclass_bugs-20201207135752.txt -- Marecki OpenPGP_signature Description: OpenPGP digital signature
[gentoo-dev] Slotted Lua: 2020-12-04 status update
Dear everyone, Since a week ago the number of open bugs blocking the slotted-Lua tracker has been reduced from 119 to under 80. A few of these have been either made no longer dependent on dev-lang/lua, last-rited, or moved to a separate tracker (it is linked to the old one under See Also) owing to the fact they or their Lua support had been masked to begin with - but I am happy to say that most of them have indeed been migrated to Lua eclasses. This includes most (if not all) multi-impl packages. Many thanks to everyone who has helped with this effort, with particular to ConiKost for his *massive* offensive against multi-impl packages populating the category dev-lua. At this point I would like to strongly encourage everyone who has got any open "please migrate to Lua eclasses" tickets assigned to make an effort to address them within the next couple of weeks. These should (almost?) all be single-impl by now and therefore quite straightforward to migrate, and should you need inspiration we have by now got a lot successful migrations to lua-single in three tree covering a wide range of build systems. Last but not least, you can always ask for advice via IRC in #gentoo-lua. On a related note, please make sure that any and all unmigrated ebuilds of your packages depending on dev-lang/lua explicitly request slot 0 of that package. This will help spare our users some nasty conflicts once slotted Lua has been unmasked. At this point I strongly feel the end of migration to slotted Lua is very much in sight. Let's do this! -- Marecki
[gentoo-dev] Slotted Lua: eclass migration status
Let me begin by giving my sincere thanks to everyone who has already
taken time in the last several weeks to either migrate their
Lua-dependent packages to lua{,-single}.eclass or otherwise made sure
said packages will not block migration to slotted dev-lang/lua. Your
efforts have been VERY much appreciated!
On the less positive side, the slotted-Lua tracker bug [1] has still got
119 open "migrate to lua eclasses" tickets attached to it. You can find
the complete list at [2]. While the migration of some of depends on
various Lua modules which have not themselves been migrated (from a
glance at the dependency tree, it seems dev-lua/lpeg and
dev-lua/LuaBitOp are the ones we need the most urgently), the vast
majority of these packages depend on dev-lang/lua only i.e. could
already be migrated.
[1] https://bugs.gentoo.org/657722
[2] https://dev.gentoo.org/~marecki/open_lua_eclass_bugs-20201128184637.txt
--
MS
OpenPGP_signature
Description: OpenPGP digital signature
[gentoo-dev] Slotted Lua: let's get this done!
Dear everyone,
tl;dr: This is an appeal to everyone in Gentoo who maintains packages
depending on dev-lang/lua, dev-lang/luajit or dev-lua/*: please migrate
your packages to lua.eclass (ones which have to be installed for
multiple Lua implementations, i.e. most likely Lua modules) and
lua-single.eclass (those that simply link against liblua or are
otherwise bound to a single implementation)! We are, if I say so myself
as the author of these eclasses, finally REALLY close now to unleashing
slotted Lua on the general public and if we make that final push, we
might finally put this long-overdue matter to rest.
Details
In addition to my packages, I have just migrated all but one [1] m-n
packages currently in the tree which depend on dev-lang/lua, to the new
eclasses. This has turned out to be much easier than I anticipated - in
quite a few number of single-impl cases it Just Worked (TM), multi-impl
requires a bit more typing but in general has worked easily too, and
even for packages using CMake it was usually the simple matter of
patching the build scripts to look for Lua as
find_package(Lua ${LUA_VERSION} EXACT REQUIRED)
and setting LUA_VERSION=$(ver_cut 1-2 $(lua_get_version)) in
src_configure. For anyone who prefers to learn by example, you can look
at e.g.:
=www-servers/lighttpd-1.4.55-r100 - single-impl, autoconf-based, Lua
support controlled by USE=lua, requires an explicit call to
lua-single_pkg_setup due to having defined its own pkg_setup;
=media-gfx/darktable-3.2.1-r1 - single-impl, cmake-based but no patching
required because it is locked by upstream to Lua5.3, Lua support
controlled USE=lua, slotted Lua pulled in by USE=system-lua;
=net-analyzer/suricata-6.0.0-r100 - single-impl, autoconf-based, Lua
support controlled by USE=lua, explicitly supports LuaJIT (i.e. there is
an upstream configure option distinguishing that from PUC Lua), has been
migrated from USE=luajit to LUA_SINGLE_TARGET=luajit;
=dev-lua/messagepack/messagepack-0.3.2-r100 - multi-impl, no build
system (it's a pure-Lua module), installs a different version for lua5-3
than for other implementations
Anyway, this leaves us with about [2] 120 packages depending directly on
dev-lang/lua (23 of them in dev-lua so likely multi-implementation
candidates, the rest most likely single-impl) and just under 30 ones
depending on something from the category dev-lua (which do include 6
lacking maintainers, which I'll migrate once their dependencies have
been taken care of). I'll likely start opening please-migrate tickets
for them on Friday, i.e. once my latest proposed change set to the
eclasses (which will likely be needed by e.g. x11-wm/awesome) has been
merged.
We can do this, guys! And do let us [3] know should you run into
problems looking at already-migrated ebuilds hasn't helped you with.
[1] www-apache/mod_security, which cannot currently be bumped to EAPI 7
because Apache eclasses do not support it
[2] I haven't excluded last-rited maintained packages from the search
[3] i.e. people haunting #gentoo-lua on Freenode, preferably via said
channel
--
MS
OpenPGP_0xD4945FFDD7FE7E37_and_old_rev.asc
Description: application/pgp-keys
OpenPGP_signature
Description: OpenPGP digital signature
[gentoo-dev] slotted lua
All, I'm trying again because I didn't see my last msg come back. Someone mentioned issues with slotted lua on a thread earlier but didn't give any details. What are the issues that you have found? are there open bugs for them? I would like to take a look. Thanks, William signature.asc Description: PGP signature
