Re: LLVM port(s) take very long time to package
On Fri, Apr 14, 2017 at 04:42:35PM +0200, Michael Gmelin wrote: > > On 14. Apr 2017, at 16:22, Alexey Dokuchaevwrote: > > On Wed, Apr 12, 2017 at 12:06:42PM +0200, Michael Gmelin wrote: > >>> On 12. Apr 2017, at 11:37, Alexey Dokuchaev wrote: > >>> ... > >>> Other alternative to PKG_NOCOMPRESS=1 could be PKGSUFFIX=.tbz > >>> (pkg-static create -f tbz ...), will play with that as well. > >> > >> I don't think bzip will buy you much in terms of performance, > > > > Surprisingly, it actually did: > > > > $ time make package PKG_SUFX=.tbz > > ===> Building package for llvm40-4.0.0_2 > > 565.215u 5.316s 9:34.18 99.3% 7899+502k 1+3020io 0pf+0w > > ^^^ > > Single-threaded bzip2 was faster than multi-threaded xz [...] > > of larger package of course: > > Interesting, I'll tried that on our high-frequency builders. Cool, please share your numbers with us once you have them. Meanwhile, I've patched my tinderbox to package with PKG_NOCOMPRESS=1 and then use `archivers/pbzip2' to compress the file. This cut down the time to package `devel/llvm40' from previous ~50 minutes to just three without touching any libarchive(3) or pkg(8) code. I guess I can live with that until libarchive(3) and/or pkg(8) gain proper support of multicore-aware efficient modern compression algorithm(s). ./danfe ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
> On 14. Apr 2017, at 16:22, Alexey Dokuchaevwrote: > > On Wed, Apr 12, 2017 at 12:06:42PM +0200, Michael Gmelin wrote: >>> On 12. Apr 2017, at 11:37, Alexey Dokuchaev wrote: >>> ... >>> Other alternative to PKG_NOCOMPRESS=1 could be PKGSUFFIX=.tbz (pkg-static >>> create -f tbz ...), will play with that as well. >> >> I don't think bzip will buy you much in terms of performance, > > Surprisingly, it actually did: > > $ time make package PKG_SUFX=.tbz > ===> Building package for llvm40-4.0.0_2 > 565.215u 5.316s 9:34.18 99.3% 7899+502k 1+3020io 0pf+0w >^^^ > Single-threaded bzip2 was faster than multi-threaded xz, at the expense > of larger package of course: > > $ ls -s1 llvm40-4.0.0_2.t* > 386752 llvm40-4.0.0_2.tbz > 286816 llvm40-4.0.0_2.txz >> Interesting, I'll tried that on our high-frequency builders. -m ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
On Wed, Apr 12, 2017 at 12:06:42PM +0200, Michael Gmelin wrote: > > On 12. Apr 2017, at 11:37, Alexey Dokuchaevwrote: > > ... > > Other alternative to PKG_NOCOMPRESS=1 could be PKGSUFFIX=.tbz (pkg-static > > create -f tbz ...), will play with that as well. > > I don't think bzip will buy you much in terms of performance, Surprisingly, it actually did: $ time make package PKG_SUFX=.tbz ===> Building package for llvm40-4.0.0_2 565.215u 5.316s 9:34.18 99.3% 7899+502k 1+3020io 0pf+0w ^^^ Single-threaded bzip2 was faster than multi-threaded xz, at the expense of larger package of course: $ ls -s1 llvm40-4.0.0_2.t* 386752 llvm40-4.0.0_2.tbz 286816 llvm40-4.0.0_2.txz > supporting something like lz4 would be useful. Or, like vsevolod@ had mentioned, Zstandard. But I'm afraid it would take some time before we'd arrive there. ./danfe ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
On Wed, Apr 12, 2017 at 04:12:45PM +0200, Jan Beich wrote: > Michael Gmelinwrites: > > It's probably busy compressing the package. Try setting > > PKG_NOCOMPRESS=1 in /etc/make.conf (works ok when just building a > > package with make, didn't test with poudriere[0]). When creating > > packages, the current compression algorithm can't take advantage of > > multiple cores, so it's kind of a bottleneck on modern machines. > > Nah, libarchive just defaults to 1 thread instead of using hw.ncpu. > The tricky part is propagating the fix into old releases or setting > compression options from within pkg(8). Nice, I've rebuilt pkg(8) against patched libarchive, and run `make package' for devel/llvm40: $ make ... $ time make package ===> Building package for llvm40-4.0.0_2 2029.111u 6.614s 11:03.25 306.9%7884+502k 4+2240io 1pf+0w $ xzcat -t llvm40-4.0.0_2.txz ; echo $? 0 It went down from ~50 minutes to eleven on a 4-core APU. I've filed an issue on GitHub: https://github.com/freebsd/pkg/issues/1564 ./danfe ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
Sorry, seems it was my error. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
The patch does not apply. Only the first file is patched, the two others are rejected. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
Michael Gmelinwrites: >> On 12. Apr 2017, at 05:43, Alexey Dokuchaev wrote: >> >> Hi there porters, >> >> It had been annoying me for a while, so I figured I try asking here: >> LLVM ports (particularly, llvm39-3.9.1) take abnormally long time to >> package, e.g. right now it took 43 minutes (tmpfs-backed build area, >> pkg-1.10.0_2.txz, AMD A8 APU @1900 MHz, plenty of RAM, no swap). >> >> pkg was eating 100% (of a single core) during that time, so it looks >> like it's actually busy (not waiting for I/O or something), but with >> what? And can it be fixed? LLVM ports are real PITA to (re)build, >> and while ccache helps to greatly reduce the build times, 3/4-hour >> package time still cripples it. >> > > It's probably busy compressing the package. Try setting > PKG_NOCOMPRESS=1 in /etc/make.conf (works ok when just building a > package with make, didn't test with poudriere[0]). When creating > packages, the current compression algorithm can't take advantage of > multiple cores, so it's kind of a bottleneck on modern machines. Nah, libarchive just defaults to 1 thread instead of using hw.ncpu. The tricky part is propagating the fix into old releases or setting compression options from within pkg(8). Index: lib/libarchive/config_freebsd.h === --- lib/libarchive/config_freebsd.h (revision 316719) +++ lib/libarchive/config_freebsd.h (working copy) @@ -139,6 +139,7 @@ #define HAVE_LONG_LONG_INT 1 #define HAVE_LSTAT 1 #define HAVE_LUTIMES 1 +#define HAVE_LZMA_STREAM_ENCODER_MT 1 #define HAVE_MBRTOWC 1 #define HAVE_MEMMOVE 1 #define HAVE_MEMORY_H 1 Index: contrib/libarchive/libarchive/archive_write_add_filter_xz.c === --- contrib/libarchive/libarchive/archive_write_add_filter_xz.c (revision 316719) +++ contrib/libarchive/libarchive/archive_write_add_filter_xz.c (working copy) @@ -152,7 +152,11 @@ common_setup(struct archive_write_filter *f) } f->data = data; data->compression_level = LZMA_PRESET_DEFAULT; +#ifdef HAVE_LZMA_STREAM_ENCODER_MT + data->threads = lzma_cputhreads(); +#else data->threads = 1; +#endif f->open = _compressor_xz_open; f->close = archive_compressor_xz_close; f->free = archive_compressor_xz_free; Index: contrib/libarchive/libarchive/archive_write_set_format_xar.c === --- contrib/libarchive/libarchive/archive_write_set_format_xar.c (revision 316719) +++ contrib/libarchive/libarchive/archive_write_set_format_xar.c (working copy) @@ -384,7 +384,11 @@ archive_write_set_format_xar(struct archive *_a) /* Set default compression type, level, and number of threads. */ xar->opt_compression = GZIP; xar->opt_compression_level = 6; +#ifdef HAVE_LZMA_STREAM_ENCODER_MT + xar->opt_threads = lzma_cputhreads(); +#else xar->opt_threads = 1; +#endif a->format_data = xar; signature.asc Description: PGP signature
Re: LLVM port(s) take very long time to package
> On 12. Apr 2017, at 11:37, Alexey Dokuchaevwrote: > > On Wed, Apr 12, 2017 at 10:43:14AM +0200, Michael Gmelin wrote: >>> On 12. Apr 2017, at 05:43, Alexey Dokuchaev wrote: >>> ... >>> pkg was eating 100% (of a single core) during that time, so it looks >>> like it's actually busy (not waiting for I/O or something), but with >>> what? And can it be fixed? LLVM ports are real PITA to (re)build, >>> and while ccache helps to greatly reduce the build times, 3/4-hour >>> package time still cripples it. >> >> It's probably busy compressing the package. Try setting PKG_NOCOMPRESS=1 >> in /etc/make.conf (works ok when just building a package with make, >> didn't test with poudriere[0]). When creating packages, the current >> compression algorithm can't take advantage of multiple cores, so it's >> kind of a bottleneck on modern machines. > > That is my current best guest as well, but I could not confirm it because > /usr/local/sbin/pkg-static is stripped by default and I could not see any > function names when I attached gdb(1) to it. Will rebuild pkg(8) shortly > WITH_DEBUG to get more information. > > Other alternative to PKG_NOCOMPRESS=1 could be PKGSUFFIX=.tbz (pkg-static > create -f tbz ...), will play with that as well. I don't think bzip will buy you much in terms of performance, supporting something like lz4 would be useful. -m ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
On Wed, Apr 12, 2017 at 10:43:14AM +0200, Michael Gmelin wrote: > > On 12. Apr 2017, at 05:43, Alexey Dokuchaevwrote: > > ... > > pkg was eating 100% (of a single core) during that time, so it looks > > like it's actually busy (not waiting for I/O or something), but with > > what? And can it be fixed? LLVM ports are real PITA to (re)build, > > and while ccache helps to greatly reduce the build times, 3/4-hour > > package time still cripples it. > > It's probably busy compressing the package. Try setting PKG_NOCOMPRESS=1 > in /etc/make.conf (works ok when just building a package with make, > didn't test with poudriere[0]). When creating packages, the current > compression algorithm can't take advantage of multiple cores, so it's > kind of a bottleneck on modern machines. That is my current best guest as well, but I could not confirm it because /usr/local/sbin/pkg-static is stripped by default and I could not see any function names when I attached gdb(1) to it. Will rebuild pkg(8) shortly WITH_DEBUG to get more information. Other alternative to PKG_NOCOMPRESS=1 could be PKGSUFFIX=.tbz (pkg-static create -f tbz ...), will play with that as well. Thanks, ./danfe ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: LLVM port(s) take very long time to package
> On 12. Apr 2017, at 05:43, Alexey Dokuchaevwrote: > > Hi there porters, > > It had been annoying me for a while, so I figured I try asking here: > LLVM ports (particularly, llvm39-3.9.1) take abnormally long time to > package, e.g. right now it took 43 minutes (tmpfs-backed build area, > pkg-1.10.0_2.txz, AMD A8 APU @1900 MHz, plenty of RAM, no swap). > > pkg was eating 100% (of a single core) during that time, so it looks > like it's actually busy (not waiting for I/O or something), but with > what? And can it be fixed? LLVM ports are real PITA to (re)build, > and while ccache helps to greatly reduce the build times, 3/4-hour > package time still cripples it. > It's probably busy compressing the package. Try setting PKG_NOCOMPRESS=1 in /etc/make.conf (works ok when just building a package with make, didn't test with poudriere[0]). When creating packages, the current compression algorithm can't take advantage of multiple cores, so it's kind of a bottleneck on modern machines. -m [0] probably not, but it seems like bdrewery is working on this https://github.com/freebsd/poudriere/issues/459 ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
LLVM port(s) take very long time to package
Hi there porters, It had been annoying me for a while, so I figured I try asking here: LLVM ports (particularly, llvm39-3.9.1) take abnormally long time to package, e.g. right now it took 43 minutes (tmpfs-backed build area, pkg-1.10.0_2.txz, AMD A8 APU @1900 MHz, plenty of RAM, no swap). pkg was eating 100% (of a single core) during that time, so it looks like it's actually busy (not waiting for I/O or something), but with what? And can it be fixed? LLVM ports are real PITA to (re)build, and while ccache helps to greatly reduce the build times, 3/4-hour package time still cripples it. ./danfe ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"