Re: [gentoo-user] long compiles
You may also want to take a look at "distcc", with which you can set up compiler farms; this can be even combined with "ccache": https://wiki.gentoo.org/wiki/Distcc#With_ccache -Ramon On 11/09/2023 23:46, Alan McKinnon wrote: On Mon, Sep 11, 2023 at 11:23 PM Michael wrote: On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote: > On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick wrote: > > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote: > > > chromium has been building since 10:14, it's now 21:16 and still going > > > so 9 hours at least on this machine to build a browser - almost as bad > > > as openoffice at it's worst (regularly took 12 hours). Nodejs also took > > > a while, but I didn't record time. > > > > Chromium is definitely the worst, and strangely variable. The last few > > compiles have taken between 6 and 14 hours. Since it takes longer than > > everything else to build, it is usually compiling on its own, so parallel > > emerges aren't a factor. > > > > Qtwebengine is also bad, not surprising as it is a cut down Chromium. > > Emerging world with --exclude then timing build to coincide with sleep > > helps, although I haven't quite reached the age where I need 14 hours of > > sleep a day. > > > > > > -- > > Neil Bothwick > > > > If it isn't broken, I can fix it. > > Yup, that jibes with what I see. Oh well, just means that the need for > overnight compiles did not go away haha > > Thanks to every one else that replied too - everyone said much the same > thing so I figured one replay to rule them all was the best way > > > Alan As the old saying goes, "there ain't no substitute to cubic inches". Moar cores and moar RAM is almost always the solution, but with laptops and older PCs in general overnight builds soon become inevitable. Selectively reducing jobs and adding swap, or for packages like rust placing /var/tmp/portage on the disk becomes necessary. A solution I use for older/smaller laptops is to build binaries on a more powerful PC and emerge these in turn on the weaker PCs. There's also the option of using bin alternatives where available, e.g. google-chrome, firefox-bin, libreoffice-bin. Finally, there is a small scale project to provide systemd based binaries as an alternative to building your own: https://wiki.gentoo.org/wiki/Experimental_binary_package_host As it turns out this laptop is the most powerful machine I have available, my large collection of previous work laptops are getting older and older. Although, I *could* create a ginormous build host on one of the virtualization clusters at work hahaha :-) That link looks interesting, I'll check it out, thanks! -- Alan McKinnon alan dot mckinnon at gmail dot com -- GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF OpenPGP_0x155BE26413E699BF.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-user] long compiles
On Mon, Sep 11, 2023 at 11:23 PM Michael wrote: > On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote: > > On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick > wrote: > > > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote: > > > > chromium has been building since 10:14, it's now 21:16 and still > going > > > > so 9 hours at least on this machine to build a browser - almost as > bad > > > > as openoffice at it's worst (regularly took 12 hours). Nodejs also > took > > > > a while, but I didn't record time. > > > > > > Chromium is definitely the worst, and strangely variable. The last few > > > compiles have taken between 6 and 14 hours. Since it takes longer than > > > everything else to build, it is usually compiling on its own, so > parallel > > > emerges aren't a factor. > > > > > > Qtwebengine is also bad, not surprising as it is a cut down Chromium. > > > Emerging world with --exclude then timing build to coincide with sleep > > > helps, although I haven't quite reached the age where I need 14 hours > of > > > sleep a day. > > > > > > > > > -- > > > Neil Bothwick > > > > > > If it isn't broken, I can fix it. > > > > Yup, that jibes with what I see. Oh well, just means that the need for > > overnight compiles did not go away haha > > > > Thanks to every one else that replied too - everyone said much the same > > thing so I figured one replay to rule them all was the best way > > > > > > Alan > > As the old saying goes, "there ain't no substitute to cubic inches". Moar > cores and moar RAM is almost always the solution, but with laptops and > older > PCs in general overnight builds soon become inevitable. Selectively > reducing > jobs and adding swap, or for packages like rust placing /var/tmp/portage > on > the disk becomes necessary. > > A solution I use for older/smaller laptops is to build binaries on a more > powerful PC and emerge these in turn on the weaker PCs. > > There's also the option of using bin alternatives where available, e.g. > google-chrome, firefox-bin, libreoffice-bin. > > Finally, there is a small scale project to provide systemd based binaries > as > an alternative to building your own: > > https://wiki.gentoo.org/wiki/Experimental_binary_package_host > As it turns out this laptop is the most powerful machine I have available, my large collection of previous work laptops are getting older and older. Although, I *could* create a ginormous build host on one of the virtualization clusters at work hahaha :-) That link looks interesting, I'll check it out, thanks! -- Alan McKinnon alan dot mckinnon at gmail dot com
Re: [gentoo-user] long compiles
On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote: > On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick wrote: > > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote: > > > chromium has been building since 10:14, it's now 21:16 and still going > > > so 9 hours at least on this machine to build a browser - almost as bad > > > as openoffice at it's worst (regularly took 12 hours). Nodejs also took > > > a while, but I didn't record time. > > > > Chromium is definitely the worst, and strangely variable. The last few > > compiles have taken between 6 and 14 hours. Since it takes longer than > > everything else to build, it is usually compiling on its own, so parallel > > emerges aren't a factor. > > > > Qtwebengine is also bad, not surprising as it is a cut down Chromium. > > Emerging world with --exclude then timing build to coincide with sleep > > helps, although I haven't quite reached the age where I need 14 hours of > > sleep a day. > > > > > > -- > > Neil Bothwick > > > > If it isn't broken, I can fix it. > > Yup, that jibes with what I see. Oh well, just means that the need for > overnight compiles did not go away haha > > Thanks to every one else that replied too - everyone said much the same > thing so I figured one replay to rule them all was the best way > > > Alan As the old saying goes, "there ain't no substitute to cubic inches". Moar cores and moar RAM is almost always the solution, but with laptops and older PCs in general overnight builds soon become inevitable. Selectively reducing jobs and adding swap, or for packages like rust placing /var/tmp/portage on the disk becomes necessary. A solution I use for older/smaller laptops is to build binaries on a more powerful PC and emerge these in turn on the weaker PCs. There's also the option of using bin alternatives where available, e.g. google-chrome, firefox-bin, libreoffice-bin. Finally, there is a small scale project to provide systemd based binaries as an alternative to building your own: https://wiki.gentoo.org/wiki/Experimental_binary_package_host signature.asc Description: This is a digitally signed message part.
[gentoo-user] Anyone used openmediavault with LVM?
Howdy, As some know, I like LVM. The Truenas box serves a purpose with zfs but I am more familiar with LVM and using zfs is sort of confusing me because they do similar things in similar ways but are different. Each time I want to do something, I have to figure it out again, sometimes ask for help. As long as I don't need to change anything, it works great. ;-) I found something called openmediavault, OMV. It is here: https://www.openmediavault.org/ On the features page, it lists LVM as a plugin. From what I read, it doesn't seem to have a default tool for managing hard drives, it seems you have to pick one. This leads to me to questions. It is based on Debian, never used it but have read it is fairly easy, been around a long time and is usually very stable. Seems to be a server type distro. So far, I kinda like the idea of this. I'd have to redo my backups again but hey, I been there before. At least if I do switch, I'll be using a tool that I'm pretty good at. I think Alan M suggested this ages ago. Could have been Neil. LVM is likely the best thing I ever used except for Linux itself. :-D Anyone use OMV before? Does it work similar to Truenas but able to have other tools installed? Anyone use LVM on this thing? If nothing else, was it stable and dependable? I have to say, Truenas has been rock solid. Never so much as a hiccup. It just boots and runs until I shut it down. I suspect OMV would be the same but never hurts to ask. Thanks. Dale :-) :-)
Re: [gentoo-user] long compiles
On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick wrote: > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote: > > > chromium has been building since 10:14, it's now 21:16 and still going > > so 9 hours at least on this machine to build a browser - almost as bad > > as openoffice at it's worst (regularly took 12 hours). Nodejs also took > > a while, but I didn't record time. > > Chromium is definitely the worst, and strangely variable. The last few > compiles have taken between 6 and 14 hours. Since it takes longer than > everything else to build, it is usually compiling on its own, so parallel > emerges aren't a factor. > > Qtwebengine is also bad, not surprising as it is a cut down Chromium. > Emerging world with --exclude then timing build to coincide with sleep > helps, although I haven't quite reached the age where I need 14 hours of > sleep a day. > > > -- > Neil Bothwick > > If it isn't broken, I can fix it. > Yup, that jibes with what I see. Oh well, just means that the need for overnight compiles did not go away haha Thanks to every one else that replied too - everyone said much the same thing so I figured one replay to rule them all was the best way Alan -- Alan McKinnon alan dot mckinnon at gmail dot com
Re: [gentoo-user] long compiles
On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote: > chromium has been building since 10:14, it's now 21:16 and still going > so 9 hours at least on this machine to build a browser - almost as bad > as openoffice at it's worst (regularly took 12 hours). Nodejs also took > a while, but I didn't record time. Chromium is definitely the worst, and strangely variable. The last few compiles have taken between 6 and 14 hours. Since it takes longer than everything else to build, it is usually compiling on its own, so parallel emerges aren't a factor. Qtwebengine is also bad, not surprising as it is a cut down Chromium. Emerging world with --exclude then timing build to coincide with sleep helps, although I haven't quite reached the age where I need 14 hours of sleep a day. -- Neil Bothwick If it isn't broken, I can fix it. pgpb9yNuiaxzs.pgp Description: OpenPGP digital signature
Re: [gentoo-user] long compiles
qtwebengine! yes that one took forever also. It also said my 16G of RAM was smaller than the 16G it needed. Weird. Anyways I enabled a swapfile and left it to run overnight Alan On Mon, Sep 11, 2023 at 9:31 PM Dale wrote: > Alan McKinnon wrote: > > After my long time away from Gentoo, I thought perhaps some packages that > always took ages to compile would have improved. I needed to change to > ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did > and let emerge do it's thing. > > chromium has been building since 10:14, it's now 21:16 and still going so > 9 hours at least on this machine to build a browser - almost as bad as > openoffice at it's worst (regularly took 12 hours). Nodejs also took a > while, but I didn't record time. > > > What other packages have huge build times? > > -- > Alan McKinnon > alan dot mckinnon at gmail dot com > > > > I have some software you don't likely use that takes a while but one that > is common is qtwebengine or something. If it's not that one, it's qtweb > something. It takes about 4 hours, sometimes 5 or so. > > I think the software takes longer to compile so that we will build new > rigs. ROFL > > Dale > > :-) :-) > -- Alan McKinnon alan dot mckinnon at gmail dot com
Re: [gentoo-user] long compiles
In addition to the reference to "qlop": $ qlop ungoogled-chromium | tail 2022-08-04T19:58:22 >>> www-client/ungoogled-chromium: 9:06:54 2022-08-05T14:27:44 >>> www-client/ungoogled-chromium: 16:19:06 2022-08-25T11:45:37 >>> www-client/ungoogled-chromium: 8:01:54 2022-09-01T10:03:19 >>> www-client/ungoogled-chromium: 15:27:22 2022-09-06T16:29:49 >>> www-client/ungoogled-chromium: 9:46:16 2022-09-14T17:48:16 >>> www-client/ungoogled-chromium: 9:30:29 2022-10-08T03:40:44 >>> www-client/ungoogled-chromium: 1:52:16 2022-10-21T17:58:43 >>> www-client/ungoogled-chromium: 9:24:55 2022-12-16T17:47:27 >>> www-client/ungoogled-chromium: 20:56:08 2023-03-20T14:12:02 <<< www-client/ungoogled-chromium: 2s Since I am using "ccache", the compilation time is sometimes doubled. ZzZzZzzz -Ramon On 11/09/2023 21:42, Ramon Fischer wrote: Hi Alan, just quick and dirty, I am too tired for formalities. :) The following list contains packages, that may be too big for tmpfs and are most probably very time consuming to compile: $ < /etc/portage/package.env/no_tmpfs.conf # custom - 20181121 - rfischer: list packages, which are too big for tmpfs #app-editors/neovim no_tmpfs.conf #app-emulation/qemu-kv no_tmpfs.conf #app-office/libreoffice no_tmpfs.conf #dev-db/mysql no_tmpfs.conf #dev-java/icedtea no_tmpfs.conf #dev-lang/ghc no_tmpfs.conf #dev-lang/ghc no_tmpfs.conf #dev-lang/mono no_tmpfs.conf #dev-lang/rust no_tmpfs.conf #dev-lang/spidermonkey no_tmpfs.conf #dev-libs/libpcre no_tmpfs.conf #dev-qt/qtwebengine no_tmpfs.conf #throttle_make_emerge.conf #mail-client/thunderbird no_tmpfs.conf #media-libs/opencv no_tmpfs.conf #media-libs/opencv no_tmpfs.conf #net-libs/nodejs no_tmpfs.conf #net-misc/openssh no_tmpfs.conf #sci-libs/tensorflow no_tmpfs.conf #sys-apps/iproute2 no_tmpfs.conf #sys-devel/clang no_tmpfs.conf #sys-devel/gcc no_tmpfs.conf #www-client/chromium no_tmpfs.conf #throttle_make_emerge.conf #www-client/firefox no_tmpfs.conf #www-client/ungoogled-chromium no_tmpfs.conf #throttle_make_emerge.conf See also: * https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs#Considering_tmpfs.27_size * https://wiki.gentoo.org/wiki/Q_applets#Extracting_information_from_emerge_logs_.28qlop.29 Sleeps away. -Ramon On 11/09/2023 21:19, Alan McKinnon wrote: After my long time away from Gentoo, I thought perhaps some packages that always took ages to compile would have improved. I needed to change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did and let emerge do it's thing. chromium has been building since 10:14, it's now 21:16 and still going so 9 hours at least on this machine to build a browser - almost as bad as openoffice at it's worst (regularly took 12 hours). Nodejs also took a while, but I didn't record time. What other packages have huge build times? -- Alan McKinnon alan dot mckinnon at gmail dot com -- GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF OpenPGP_0x155BE26413E699BF.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-user] long compiles
Hi Alan, just quick and dirty, I am too tired for formalities. :) The following list contains packages, that may be too big for tmpfs and are most probably very time consuming to compile: $ < /etc/portage/package.env/no_tmpfs.conf # custom - 20181121 - rfischer: list packages, which are too big for tmpfs #app-editors/neovim no_tmpfs.conf #app-emulation/qemu-kv no_tmpfs.conf #app-office/libreoffice no_tmpfs.conf #dev-db/mysql no_tmpfs.conf #dev-java/icedtea no_tmpfs.conf #dev-lang/ghc no_tmpfs.conf #dev-lang/ghc no_tmpfs.conf #dev-lang/mono no_tmpfs.conf #dev-lang/rust no_tmpfs.conf #dev-lang/spidermonkey no_tmpfs.conf #dev-libs/libpcre no_tmpfs.conf #dev-qt/qtwebengine no_tmpfs.conf #throttle_make_emerge.conf #mail-client/thunderbird no_tmpfs.conf #media-libs/opencv no_tmpfs.conf #media-libs/opencv no_tmpfs.conf #net-libs/nodejs no_tmpfs.conf #net-misc/openssh no_tmpfs.conf #sci-libs/tensorflow no_tmpfs.conf #sys-apps/iproute2 no_tmpfs.conf #sys-devel/clang no_tmpfs.conf #sys-devel/gcc no_tmpfs.conf #www-client/chromium no_tmpfs.conf #throttle_make_emerge.conf #www-client/firefox no_tmpfs.conf #www-client/ungoogled-chromium no_tmpfs.conf #throttle_make_emerge.conf See also: * https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs#Considering_tmpfs.27_size * https://wiki.gentoo.org/wiki/Q_applets#Extracting_information_from_emerge_logs_.28qlop.29 Sleeps away. -Ramon On 11/09/2023 21:19, Alan McKinnon wrote: After my long time away from Gentoo, I thought perhaps some packages that always took ages to compile would have improved. I needed to change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did and let emerge do it's thing. chromium has been building since 10:14, it's now 21:16 and still going so 9 hours at least on this machine to build a browser - almost as bad as openoffice at it's worst (regularly took 12 hours). Nodejs also took a while, but I didn't record time. What other packages have huge build times? -- Alan McKinnon alan dot mckinnon at gmail dot com -- GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF OpenPGP_0x155BE26413E699BF.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-user] long compiles
Chromium and qtwebengine have the longest build times that I have encountered On Tue, 12 Sep 2023 at 1:01 AM, Dale wrote: > Alan McKinnon wrote: > > After my long time away from Gentoo, I thought perhaps some packages that > always took ages to compile would have improved. I needed to change to > ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did > and let emerge do it's thing. > > chromium has been building since 10:14, it's now 21:16 and still going so > 9 hours at least on this machine to build a browser - almost as bad as > openoffice at it's worst (regularly took 12 hours). Nodejs also took a > while, but I didn't record time. > > > What other packages have huge build times? > > -- > Alan McKinnon > alan dot mckinnon at gmail dot com > > > > I have some software you don't likely use that takes a while but one that > is common is qtwebengine or something. If it's not that one, it's qtweb > something. It takes about 4 hours, sometimes 5 or so. > > I think the software takes longer to compile so that we will build new > rigs. ROFL > > Dale > > :-) :-) >
Re: [gentoo-user] long compiles
Alan McKinnon wrote: > After my long time away from Gentoo, I thought perhaps some packages > that always took ages to compile would have improved. I needed to > change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So > that's what I did and let emerge do it's thing. > > chromium has been building since 10:14, it's now 21:16 and still going > so 9 hours at least on this machine to build a browser - almost as bad > as openoffice at it's worst (regularly took 12 hours). Nodejs also > took a while, but I didn't record time. > > > What other packages have huge build times? > > -- > Alan McKinnon > alan dot mckinnon at gmail dot com I have some software you don't likely use that takes a while but one that is common is qtwebengine or something. If it's not that one, it's qtweb something. It takes about 4 hours, sometimes 5 or so. I think the software takes longer to compile so that we will build new rigs. ROFL Dale :-) :-)
[gentoo-user] long compiles
After my long time away from Gentoo, I thought perhaps some packages that always took ages to compile would have improved. I needed to change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did and let emerge do it's thing. chromium has been building since 10:14, it's now 21:16 and still going so 9 hours at least on this machine to build a browser - almost as bad as openoffice at it's worst (regularly took 12 hours). Nodejs also took a while, but I didn't record time. What other packages have huge build times? -- Alan McKinnon alan dot mckinnon at gmail dot com
