Re: Proposal to turn off AOT in clojure-build-system
Hello,
I’ve been following along with this discussion, as well as a
discussion on Clojureverse, and thought it might be helpful to
pull together some threads and design decisions around Clojure’s
behavior.
Clojure is designed to ship libraries as source artifacts, not
bytecode ("pretty much all other Clojure libraries ... are all
source code by design[1]."; "Clojure is ... a source-first
language[2]"), and the view of the community is that shipping AOT
artifacts "is an anti-pattern[1]." Clojure library JARs are more
akin to source tarballs than binaries. The original design and
intent of Clojure’s AOT compiler is to compile "just a few
things... for the interop case" or "Everything... For the
'Application delivery', 'Syntax check', and 'reflection warnings'
cases[3]."
Clojure’s compiler is transitive and "does not support separate
compilation"[3], meaning when a namespace is compiled, anything it
uses is compiled and emitted with it. This is the crux of why
mixing AOT and non-AOT code is troublesome: it causes dependency
diamonds, where the AOT’d library contains a duplicate, older
version of code used elsewhere in the project.
The Clojure reference on compiling[4] gives some reasons you might
want to AOT: "To deliver your application without source," "To
speed up application startup," "To generate named classes for use
by Java," "To create an application that does not need runtime
bytecode generation and custom classloaders." Note that there’s
no mention of compiling libraries for any reason; only
applications.
When AOT is used "for the interop case," it’s typical to AOT only
those namespaces[5], not the entire library.
Shipping AOT-compiled Clojure libraries has caused real and very
weird and hard-to-debug problems in the past:
https://clojure.atlassian.net/browse/CLJ-1886?focusedCommentId=15290
https://github.com/clj-commons/byte-streams/issues/68 and
https://clojure.atlassian.net/browse/CLJ-1741
Clojure doesn’t have guarantees around ABI stability[6][7]. To
date, most ABI changes have been additive, but there are no
guarantees that the ABI will be compatible from any one version of
Clojure to any other. The understanding of the Clojure community
is that the design of the current compiler can’t offer a stable
ABI[8] at all. Because nobody in the Clojure community AOTs
intermediate (that is, library) code, this hasn’t been a problem
and is unlikely to change.
"Clojure tries very hard to provide source compatibility but not
bytecode compatibility across versions[9]."
Correctly handling the ABI concerns — which Guix currently does
not do — would result in a combinatorial explosion of Clojure
packages should multiple versions of Clojure ever be available in
Guix at the same time. For example, if someone wanted to package
Clojure 1.12.0-alpha9, you’d need to duplicate every package
taking Clojure as an input so they use the correct version. While
ABI breakage has been rare thus far, it seems likely that it’ll
occur at some point; perhaps if Clojure reaches version 2.0.0. If
Guix disables AOT for Clojure libraries, we have source
compatibility, and the AOT/ABI problems are moot.
Clojure’s compiler is non-deterministic[10]: the same compiler can
will produce different bytecode for the same input across multiple
runs. I’m not sure if this is a problem for Guix at this point in
time, but it seems out of line with Guix expectations for
compilation generally.
Opinions follow:
If we’re taking votes, mine is to *not* AOT Clojure libraries,
both for the technical reasons laid out in, and also for the
social reason of not violating the principle of least surprise. I
understand that Guix and Clojure have very different approaches,
and some balance must be struck. However, the lack of ABI
guarantees, the compiler’s behavior, the promise of source
compatibility, and matching the expectation of the audience these
tools are meant for all convince me that disabling AOT is the
right course here.
AOT’ing Clojure applications (which means, more or less, "the
Clojure tooling") is desirable, and should be maintained.
— Ian
[1]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/8
[2]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[3]: https://clojure.org/reference/compilation
[4]:
https://archive.clojure.org/design-wiki/display/design/Transitive%2BAOT%2BCompilation.html
[5]: https://clojure.org/guides/deps_and_cli#aot_compilation
[6]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[7]:
https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
[8]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/4
[9]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/18
Re: (Lx)Qt team in Guix
Hello, 宋文武 writes: [...] > Hello, I have just pushed 2 commits to remove qt.scm from the scope of > lxqt and add myself to the qt team. > > Happy weekend! Yay! Thank you, and happy weekend to you as well! -- Thanks, Maxim
Re: (Lx)Qt team in Guix
Hi, On Fri, 08 Mar 2024 19:05:57 +0800, Andreas Enge wrote: > > Hello, > > Am Thu, Mar 07, 2024 at 08:46:12PM -0500 schrieb Maxim Cournoyer: > > I think the reason qt.scm is part of the lxqt team is historical; lxqt-team > > predates qt-team. We should probably just remove qt.scm from > > lxqt-team's scope. What do you think? > > > > I'd keep both team separated; I'm interested in maintaining qt upgrades > > but not much lxqt. > > that sounds good to me as well; and I would then encourage 宋文武 and > Hilton to join the Qt team, too. I don't know about Qt... My existing commits to qt.scm are all fixes to package definitions, so joining the Qt team doesn't sound suitable. Thanks
Re: [PATCH 2/2] gnu: grub: Modernize.
Hi Fabio, Fabio Natali writes: > On 2024-03-03, 17:42 +0100, Josselin Poiret wrote: >> From: Josselin Poiret >> >> * gnu/packages/bootloaders.scm (grub-minimal, grub-coreboot, grub-efi32, >> grub-hybrid): Use G-Exps. >> (grub-efi): Use G-Exps. Also use OVMF to test. > > Hi Josselin, > > This is brilliant, thanks. > > I'll see how far I can go with this and get back to you. I might reach > out to you on IRC, maybe, in case I need some guidance? > > Thanks, cheers, Fabio. Of course, I'm known as jpoiret there. I use a bouncer, so even if I'm online I might not be available, but will definitely read your message later if that's the case. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: on the bug tracker (Re: Guix Days: Patch flow discussion)
-- Giovanni Biscuolo writes: > Hi Josselin, > > Josselin Poiret writes: > > [...] > >> One thing I would like to get rid of though is debbugs. > > given that a significant part of the Guix infrastructure is provided by > the GNU project, including the bug/issue tracker, and I don't think that > GNU will replace https://debbugs.gnu.org/ (or the forge, Savannah) with > something else, I suggest to concentrate the Guix community efforts in > giving contributors better user interfaces to Debbugs, e.g Mumi (web and > CLI) instead of trying to get rid of it. We can re-use some other tool as well, no need to code our own in Guile though. Also note that the choice of bug-tracking tool is not benign, as most other tools (Mumi, but also QA) have to adapt to its idiosyncrasies. > In other words: the "problem" it's not the tool, it's the *interface*. I don't agree: see below for the two examples of where the tool design itself is problematic. > Please also consider that if (I hope not) the Guix would decide to adopt > a different bug/issue tracker system then Someome™ will have to > administrate it, and currently there are other pieces of core > infrastructure that need more resources, e.g. QA. > > Speaking of interface features, I simply *love* the email based > interface provided by Debbugs [1]; also the web UI is not bad, and the Mumi > one (https://issues.guix.gnu.org/) it's even better. I prefer a public-inbox web frontend for browsing emails personally. >> It causes a lot of pain for everyone, eg. when sending patchsets, it >> completely breaks modern email because it insists on rewriting >> DMARC-protected headers, thus needing to also rewrite "From:" to avoid >> DMARC errors. > > I don't understand what "completely breaks modern email" means: please > could you point me to a document where this issue/bug is documented? I don't think there's a "public" discussion of this, but I've conversed with the mailman maintainers in private about this: because Debbugs relies on modifying the mail message itself, it clashes with modern email features that authentify received emails. DKIM authentifies a subset of mail headers by signing them with a private key, among which the "From" header is mandatory. DMARC is a policy mechanism for domains that can enforce DKIM (and/or SPF) for all mails with a corresponding "From:". Together, they prevent email impersonation. So Debbugs modifies a protected header (e.g. Subject) -> DKIM fails -> if DMARC is set to enforce DKIM, DMARC fails -> mail is not received. To avoid this issue, Mailman (the mailing daemon overseeing the debbugs-managed MLs) also rewrites the From: header of such mails to a debbugs-controlled domain, so that DKIM headers can be re-signed and a different DMARC policy be used. This is why you often see "via guix-patches" in From: headers, and if you look at the email it's a generic one. I've also mentioned this before, but not being able to simply send a full patchset without having to go through hoops is, put simply, a problem. Even prospective contributors used to email-based workflows will be turned off by this. In general, the problem with Debbugs is that it insists on *taking over* emails, rather than just consuming them. >> I've been following the Linux kernel development a bit closer this past >> year, and while there are things that need to improve (like knowing the >> status of a patchset in a maintainer's tree), they at least have a lot >> of tools that I think we should adopt more broadly: > > you mention: b4/lei and patchwork but they are not bug/issue trackers. Yes, I was mentioning general tools. > I personally like the idea that the bug/issue tracker is _embedded_ > (integrated?) in the DVCS used by the project, Git in Guix case. > > For this reason I find Tissue https://tissue.systemreboot.net/ an > interesting project for *public* issue/bug tracking systems, also > because Tissue is _not_ discussion-oriented: this means that > discussions are managed "somewhere else", because «It's much better to > have a clear succinct actionable issue report. This way, the issue > tracker is a list of clear actionable items rather than a mess of > unreproducible issues.» [2] I like this separation as well, and Tissue seemed interesting the last time I looked at it. I liked how it uses standard tools and can be consulted/modified off-line. It basically checks all of the boxes :) Best, -- Josselin Poiret signature.asc Description: PGP signature
