Bug#877418: dh-strip-nondeterminism: kills clojure performance

On Fri 2017-10-06 10:00:27 +0100, Chris Lamb wrote: > If it were hardcoded into the filenames, one wouldn't need to do > anything onerous, eg. > > -rw-r--r-- 1 0 Oct 6 09:56 > helloworld.adc83b19e793491b1c6ea0fd8b46cd9f32e592fc.class > -rw-r--r-- 1 0 Oct 6 09:56 >

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi dkg, > And there are more questions too: what if multiple source files > contributed to the creation of the compiled artifact (e.g. "include" > directives)? Hm, that's an excellent point. > You can also imagine a compilation regime that detects changes to a file > (e.g. via inotify) and

Bug#877418: dh-strip-nondeterminism: kills clojure performance

On Thu 2017-10-05 10:56:46 +0100, Chris Lamb wrote: > I'd also be curious to know why you think *more* than one second could > ever be needed here. I think I'm mising something. some filesystems have a resolution > 1s :( http://www.ntfs.com/exfat-comparison.htm shows that FAT32 has a 2s

Bug#877418: dh-strip-nondeterminism: kills clojure performance

On Wed 2017-10-04 19:45:49 +0100, Chris Lamb wrote: > *Very* quick thoughts here: could some variant of a) be merged > upstream…? Perhaps upstream could move to a hash-based system instead > of using timestamps? eg. encoding the SHA1 of the file in the filename. I'm thinking about this problem

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Chris Lamb writes: > I don't quite get what you mean I'm afraid. Filesystem ordering (at least > via readdir/listdir, etc.) is non-deterministic. Can you explain it to me > another way? (...or quite likely I'm not describing things all that well.) In Clojure's case, I'd think

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Rob, > Or rather, if Clojure's only looking at the timestamps in the jar file, > then those may have a known (fixed) resolution, and so we'd just need to > make sure that the .clj files are at least that much older than the > corresponding .class files inside the jar. Right; that's: > > b)

Bug#877418: dh-strip-nondeterminism: kills clojure performance

> Chris Lamb writes: >> […] assumes a filesystem with 1s mtime resolution. > Mmm, which is a completely fair assumption. See also: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804339 ! While I did mention filesystem timestamps on IRC as an example, and they are

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi; I'm the upstream maintainer of Leiningen, a Clojure application being packaged for Debian. I would strongly vote for adjusting the timestamps of .clj files to be older than the corresponding .class files. I don't know enough about filesystem timestamp granularity to comment on the wisdom of

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Elana, > Hi all, just catching up on this thread. No problem, great to see more people adding their thoughts! :) > […] assumes a filesystem with 1s mtime resolution. Mmm, which is a completely fair assumption. See also: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804339 ! > Having

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi all, just catching up on this thread. FWIW I agree with Apollon; as rlb pointed out on IRC, we introduce a potential race condition when we don't recompile when the timestamps are equal. Quoting him... The scenario I was thinking of is "clock ticks over to 1001s, I compile foo.clj

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Vincent, > FWIW, I agree with Apollon. >= is better than > as the resolution of the > timestamp can be coarced. Mm, I agree. Also, as strip-nondeterminism should really "go away" in the medium- to long- term, I'd rather avoid adding ad-hoc modifications (especially ones so ugly) for each

Bug#877418: dh-strip-nondeterminism: kills clojure performance

❦ 4 octobre 2017 08:03 +0100, Chris Lamb  : >> Fixing our clojure package would only solve the issue for Debian. > > Sure, but why don't we patch Debian's version of clojure whilst we wait > for upstream to "catch up"? :-) FWIW, I agree with Apollon. >= is better than > as

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Emmanuel, > Fixing our clojure package would only solve the issue for Debian. Sure, but why don't we patch Debian's version of clojure whilst we wait for upstream to "catch up"? :-) Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org /

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Le 3/10/2017 à 19:49, Chris Lamb a écrit : > So, did you see Apollon's remarks to this bug? I guess the messages were delayed, I didn't see them yesterday. > Very happy to rollback the changes to strip-nondeterminism that > implement b) if we go with a) in the end; I haven't uploaded yet. > >

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Emmanuel, > > a) We patch clojure with ">=" (and send it upstream, etc. etc.) > > > > b) We make strip-nondetermism subtract 1 second from the .clj files' > > target modification times so it matches with the existing ">". > > I thought about b) too but this is definitely a clojure

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Le 3/10/2017 à 10:32, Chris Lamb a écrit : > Great stuff! So, we have two options as I see it: > > a) We patch clojure with ">=" (and send it upstream, etc. etc.) > > b) We make strip-nondetermism subtract 1 second from the .clj files' > target modification times so it matches with

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Apollon, > Thanks for fixing this. Just a small comment: the comment in [1] > should probably say "to always be older than .class", instead of "to always be younger". Good idea; pushed in:

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Chris, On Tue, 03 Oct 2017 15:00:11 +0100 Chris Lamb wrote: > tags 877418 + pending > thanks > > > Setting the mtime of .clj files one second earlier than .class > > should Do The Right Thing™. > > Thanks. I've just pushed the following: > > >

Bug#877418: dh-strip-nondeterminism: kills clojure performance

tags 877418 + pending thanks > Setting the mtime of .clj files one second earlier than .class > should Do The Right Thing™. Thanks. I've just pushed the following: https://anonscm.debian.org/git/reproducible/strip-nondeterminism.git/commit/?id=99af63bec965d924275d53f4db90f9853e4db8a7

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Chris, On Tue, 03 Oct 2017 09:32:39 +0100 Chris Lamb wrote: > Hi Emmanuel, > > > I eventually found this check performed in the load() method of RT.java: > > > > if((classURL != null && > > (cljURL == null > > || lastModified(classURL, classfile) >

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Hi Emmanuel, > I eventually found this check performed in the load() method of RT.java: > > if((classURL != null && > (cljURL == null > || lastModified(classURL, classfile) > lastModified(cljURL, > scriptfile))) > > Changing '>' with '>=' fixes the issue. Great stuff! So, we

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Chris Lamb writes: > Chris Lamb wrote: > >> > I noticed that Debian's clojure-1.8.0.jar had terrible performance as >> > compared to both the upstream jar >> >> Oh boy, this sounds fun! > > There's no obvious reason at this point why this performance regression is > limited to

Bug#877418: dh-strip-nondeterminism: kills clojure performance

On Mon, 02 Oct 2017 12:00:36 +0100 Chris Lamb wrote: > There's no obvious reason at this point why this performance regression is > limited to Clojure, unless — hopefully — it's related to the .clj files? > > ie. this could be affecting the performance of all Java applications

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Chris Lamb wrote: > > I noticed that Debian's clojure-1.8.0.jar had terrible performance as > > compared to both the upstream jar > > Oh boy, this sounds fun! There's no obvious reason at this point why this performance regression is limited to Clojure, unless — hopefully — it's related to the

Bug#877418: dh-strip-nondeterminism: kills clojure performance

tags 877418 + confirmed thanks Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-

Bug#877418: dh-strip-nondeterminism: kills clojure performance

tags 877418 + confirmed thanks Hi Rob, > I noticed that Debian's clojure-1.8.0.jar had terrible performance as > compared to both the upstream jar Oh boy, this sounds fun! I can confirm it is due to strip-nondeterminism. In particular, the part that sets the last modified date of the .jar

Bug#877418: dh-strip-nondeterminism: kills clojure performance

Package: dh-strip-nondeterminism Version: 0.034-1 I noticed that Debian's clojure-1.8.0.jar had terrible performance as compared to both the upstream jar and one built manually via the "mvn package" or ant process, and after some investigation, I think I've tracked it down to