[gentoo-user] ssh from linux to Windows
I installed openssh server on Windows 11 and tried to ssh to it using the id_rsa.pub key but I didn't have luck. I copied the key to .ssh\authorized_keys file. On linux the last line ending with "\" on Windows Notepad replaces it with the "+" sign. ssh with password is working but windows doesn't recognize the public key or maybe it is wrong directory C:\Users\Garry Server\.ssh\authorized_keys -- Thelma
Re: [gentoo-user] Emerge load again
On Sunday, 7 January 2024 00:54:12 GMT Adam Carter wrote: > > > So if it's consistently gcc that collapses to two threads, then > > > something (maybe explicit settings, maybe dependencies, maybe yadda > > > yadda) is telling make that only two jobs can run at the same time else > > > they'll trip over each other. > > > > > > Could be a dev has hard-coded the "two jobs" rule to make those random > > > crashes go away :-) Or maybe they found the problem, and that's why only > > > two jobs can run in parallel. > > > > Not so. As I said last time: 'if I set -distcc and -j12 -l12, I get 12 > > threads > > in parallel'. > > Have you checked you're not limiting jobs in /etc/distcc/hosts? ie no '/2' > after the IP address? $ cat /etc/distcc/hosts localhost/12 -- Regards, Peter.
Re: [gentoo-user] Emerge load again
> > > So if it's consistently gcc that collapses to two threads, then > > something (maybe explicit settings, maybe dependencies, maybe yadda > > yadda) is telling make that only two jobs can run at the same time else > > they'll trip over each other. > > > > Could be a dev has hard-coded the "two jobs" rule to make those random > > crashes go away :-) Or maybe they found the problem, and that's why only > > two jobs can run in parallel. > > Not so. As I said last time: 'if I set -distcc and -j12 -l12, I get 12 > threads > in parallel'. > Have you checked you're not limiting jobs in /etc/distcc/hosts? ie no '/2' after the IP address?
Re: [gentoo-user] Genlop wonky again
On Saturday, 6 January 2024 19:28:05 GMT Wols Lists wrote: > Statistics is one of those areas where, if you don't know what you're > doing and you use the wrong maths, then you are going to get stupid results. > > "Statistics tell you how to get from A to B. What they don't tell you is > that you're all at C". I took a module on statistics in my Open University maths degree 40-odd years ago. I was bemused. They seemed to say that the subject was founded on two basic principles; then they proceeded to define each of them in terms of the other. I'm still waiting for the entire edifice to come crashing down around our ears. :) -- Regards, Peter.
Re: [gentoo-user] Emerge load again
On Saturday, 6 January 2024 19:31:59 GMT Wols Lists wrote: > On 06/01/2024 17:52, Peter Humphrey wrote: > >> In other cases, there may be a hundred separate tasks, make fires off a > >> hundred tasks shared amongst all the resource it can find, and sits back > >> and waits. > > > > And that's how the very first installation goes, with single-host distcc. > > Then, when it gets to gcc, it collapses to 2 threads and everything > > gained so far is lost many-fold. (I set USE=-fortran to avoid pointless > > recompilation, since nothing needs it here.) > > So if it's consistently gcc that collapses to two threads, then > something (maybe explicit settings, maybe dependencies, maybe yadda > yadda) is telling make that only two jobs can run at the same time else > they'll trip over each other. > > Could be a dev has hard-coded the "two jobs" rule to make those random > crashes go away :-) Or maybe they found the problem, and that's why only > two jobs can run in parallel. Not so. As I said last time: 'if I set -distcc and -j12 -l12, I get 12 threads in parallel'. -- Regards, Peter.
Re: [gentoo-user] Emerge load again
On 06/01/2024 17:52, Peter Humphrey wrote: In other cases, there may be a hundred separate tasks, make fires off a hundred tasks shared amongst all the resource it can find, and sits back and waits. And that's how the very first installation goes, with single-host distcc. Then, when it gets to gcc, it collapses to 2 threads and everything gained so far is lost many-fold. (I set USE=-fortran to avoid pointless recompilation, since nothing needs it here.) So if it's consistently gcc that collapses to two threads, then something (maybe explicit settings, maybe dependencies, maybe yadda yadda) is telling make that only two jobs can run at the same time else they'll trip over each other. Could be a dev has hard-coded the "two jobs" rule to make those random crashes go away :-) Or maybe they found the problem, and that's why only two jobs can run in parallel. Cheers, Wol
Re: [gentoo-user] Genlop wonky again
On 06/01/2024 17:59, Peter Humphrey wrote: On Saturday, 6 January 2024 16:21:30 GMT Wols Lists wrote: ... it's nothing to do with more power or whatever, it's down to simple statistics. If genloop guesses the statistical spread wrongly, it's going to mess up its estimates. Aren't you exaggerating genlop's complexity? I wasn't aware of any use of statistics in it, other than a simple arithmetic mean to estimate the time remaining. It certainly seems to do that, anyway. Other than a simple arithmetic mean !!! Other than a simple arithmetic mean !!! If that's the case, you've just confirmed my statement - genloop is almost certainly using the wrong statistics for the job !!! If you take the average (arithmetic mean) of a power-law (exponential decay) distribution, your results are going to be garbage. Statistics is one of those areas where, if you don't know what you're doing and you use the wrong maths, then you are going to get stupid results. "Statistics tell you how to get from A to B. What they don't tell you is that you're all at C". Cheers, Wol
Re: [gentoo-user] Genlop wonky again
On Saturday, 6 January 2024 16:26:49 GMT Daniel Pielmeier wrote: > Am 5. Januar 2024 23:51:39 UTC schrieb Peter Humphrey : > >Hello list, > > > >I've just had some strange output from genlop on my 16-thread i5 box, thus: > > > ># genlop -t libreoffice | /bin/grep minute > > > > merge time: 37 minutes and 38 seconds. > > merge time: 52 minutes and 59 seconds. > > merge time: 46 minutes and 17 seconds. > > > ># genlop -c > > > > Currently merging 11 out of 11 > > > > * app-office/libreoffice-7.5.9.2 > > > > current merge time: 4 minutes and 3 seconds. > > ETA: 1 hour, 4 minutes and 24 seconds. > > > >### Then, once the update finished: > > > ># genlop -t libreoffice | /bin/grep minute > > > > merge time: 37 minutes and 38 seconds. > > merge time: 52 minutes and 59 seconds. > > merge time: 46 minutes and 17 seconds. > > merge time: 38 minutes and 40 seconds. > > > >I know genlop is, shall we say, not perfect, but how can it be so grossly > >wrong as that? > > > >I have this in make.conf, and it hasn't changed since I built the machine: > > > >grep '\-j' /etc/portage/make.conf > >EMERGE_DEFAULT_OPTS="--jobs --load-average=12 > >MAKEOPTS="-j12 -l12" > > There are not by chance binary merges which took less than a minute? That > might explain the differences. That would skew the prediction downwards, not up. > What is the output wihout the grep or filtering by merge time instead. The same. -- Regards, Peter.
Re: [gentoo-user] Genlop wonky again
On Saturday, 6 January 2024 16:21:30 GMT Wols Lists wrote: > ... it's nothing to do with more power or whatever, it's down to simple > statistics. If genloop guesses the statistical spread wrongly, it's > going to mess up its estimates. Aren't you exaggerating genlop's complexity? I wasn't aware of any use of statistics in it, other than a simple arithmetic mean to estimate the time remaining. It certainly seems to do that, anyway. > If you have a double-peak distribution, with a large short-lived peak, > and a small long-lived peak, you can get some weird results, especially > if you have assumed a bell curve (almost always wrong) or an exponential > decay (which is generally, NOT ALWAYS, a good choice). I doubt it does any of that. -- Regards, Peter.
Re: [gentoo-user] Emerge load again
On Saturday, 6 January 2024 15:28:53 GMT Wols Lists wrote: > As far as I'm aware, there's no mystery. On a single machine you get the > exact same thing ... it's all down to parallelism. > > Make asks itself "how many separate tasks can I do at the same time, > which won't interfere with each other". In gcc's case, the answer > appears to be two. It doesn't matter how much resource is available, > make can only make use of two cores. Yet, if I set -distcc and -j12 -l12, I get 12 threads in parallel. That's the mystery. > In other cases, there may be a hundred separate tasks, make fires off a > hundred tasks shared amongst all the resource it can find, and sits back > and waits. And that's how the very first installation goes, with single-host distcc. Then, when it gets to gcc, it collapses to 2 threads and everything gained so far is lost many-fold. (I set USE=-fortran to avoid pointless recompilation, since nothing needs it here.) > Think of a hundred compile jobs all running at the same time, but then > the linker is invoked, and you can only have the one linker running, > after all the compile jobs have finished. I hadn't thought of that - another thing to consider. > And this is a HARD problem, I haven't seen it recently, but there used > to be plenty of threads about hard-to-debug compile failures that went > away with -j1. The obvious cause was two compile jobs being set off in > parallel, when in reality one depended on the other, and things messed up. I haven't either - seen it recently. -- Regards, Peter.
[gentoo-user]
Re: [gentoo-user] Genlop wonky again
Re: [gentoo-user] Genlop wonky again
On 1/6/24 11:21, Wols Lists wrote: On 06/01/2024 16:12, John Blinka wrote: And it doesn’t actually take 2x longer - the new estimate is just grossly wrong. I presume that the old estimate was also wrong. And it's nothing to do with more power or whatever, it's down to simple statistics. If genloop guesses the statistical spread wrongly, it's going to mess up its estimates. If you have a double-peak distribution, with a large short-lived peak, and a small long-lived peak, you can get some weird results, especially if you have assumed a bell curve (almost always wrong) or an exponential decay (which is generally, NOT ALWAYS, a good choice). Cheers, Wol I think there is a slightly deeper question also involved. First, I'll assume (safe or not) that genlop's assumption of total build time for a package depends solely on the previous build times, with all the foibles Wol implies in that. However, that estimate then gets adjusted as the build progresses. Clearly, experience shows up that the estimated remaining time is NOT simply the estimated build time minus the time spent so far, except possibly when an emerge is only for one package. What else contributes to that estimate? If that adjustment includes using the number of other builds going on at the same time, and their original and estimated build times, I can see lots of opportunity for shenanigans Jack.
Re: [gentoo-user] Genlop wonky again
Am 5. Januar 2024 23:51:39 UTC schrieb Peter Humphrey : >Hello list, > >I've just had some strange output from genlop on my 16-thread i5 box, thus: > ># genlop -t libreoffice | /bin/grep minute > merge time: 37 minutes and 38 seconds. > merge time: 52 minutes and 59 seconds. > merge time: 46 minutes and 17 seconds. > ># genlop -c > > Currently merging 11 out of 11 > > * app-office/libreoffice-7.5.9.2 > > current merge time: 4 minutes and 3 seconds. > ETA: 1 hour, 4 minutes and 24 seconds. > >### Then, once the update finished: > ># genlop -t libreoffice | /bin/grep minute > merge time: 37 minutes and 38 seconds. > merge time: 52 minutes and 59 seconds. > merge time: 46 minutes and 17 seconds. > merge time: 38 minutes and 40 seconds. > >I know genlop is, shall we say, not perfect, but how can it be so grossly >wrong as that? > >I have this in make.conf, and it hasn't changed since I built the machine: > >grep '\-j' /etc/portage/make.conf >EMERGE_DEFAULT_OPTS="--jobs --load-average=12 >MAKEOPTS="-j12 -l12" > There are not by chance binary merges which took less than a minute? That might explain the differences. What is the output wihout the grep or filtering by merge time instead. -- Best regards Daniel
Re: [gentoo-user] Genlop wonky again
On 06/01/2024 16:12, John Blinka wrote: And it doesn’t actually take 2x longer - the new estimate is just grossly wrong. I presume that the old estimate was also wrong. And it's nothing to do with more power or whatever, it's down to simple statistics. If genloop guesses the statistical spread wrongly, it's going to mess up its estimates. If you have a double-peak distribution, with a large short-lived peak, and a small long-lived peak, you can get some weird results, especially if you have assumed a bell curve (almost always wrong) or an exponential decay (which is generally, NOT ALWAYS, a good choice). Cheers, Wol
Re: [gentoo-user] Genlop wonky again
On Sat, Jan 6, 2024 at 3:56 AM Wols Lists wrote: > On 06/01/2024 00:54, John Blinka wrote: > > I’ve often found that it gives one estimate when multiple packages are > > being built, then a much longer estimate for still-in-progress builds > > once some of the builds have finished. > > > > That result defies common sense. Less remaining work has to take less, > > not more (much more), time. > > Common sense isn't common and, well, often doesn't make sense. > > If there's a bunch of small builds skewing the "time per build" estimate > down, as they drop off the list the estimated time per build will go up, > and if the skew is serious enough it can even make the total estimated > time go up ... I don’t follow you. What is the source of this “skew”? Why should more available processing power/less load cause builds to run more slowly? I’d really like to understand your point. I have observed what I reported above many times, often when there are 2 builds running, a long one and a shorter one. Once the shorter one ends , the longer one’s time estimate via genlop increases , sometimes by 2x. And it doesn’t actually take 2x longer - the new estimate is just grossly wrong. Invoking skew or common sense being uncommon/wrong doesn’t change my and the original poster’s observations that genlop sometimes gives really bad time estimates. Something’s not right. Respectfully John >
Re: [gentoo-user] Emerge load again
On 29/11/2023 12:06, Peter Humphreey wrote: The contribution of distcc isn't clear to me yet, as I said before. Sometimes it's the bee's knees; other times it might just as well not be there. I don't like mysteries... As far as I'm aware, there's no mystery. On a single machine you get the exact same thing ... it's all down to parallelism. Make asks itself "how many separate tasks can I do at the same time, which won't interfere with each other". In gcc's case, the answer appears to be two. It doesn't matter how much resource is available, make can only make use of two cores. In other cases, there may be a hundred separate tasks, make fires off a hundred tasks shared amongst all the resource it can find, and sits back and waits. Think of a hundred compile jobs all running at the same time, but then the linker is invoked, and you can only have the one linker running, after all the compile jobs have finished. And this is a HARD problem, I haven't seen it recently, but there used to be plenty of threads about hard-to-debug compile failures that went away with -j1. The obvious cause was two compile jobs being set off in parallel, when in reality one depended on the other, and things messed up. Cheers, Wol
Re: [gentoo-user] Emerge load again
On Saturday, 6 January 2024 11:44:20 GMT Michael wrote: > On Wednesday, 29 November 2023 12:06:15 GMT Peter Humphreey wrote: > > On Wednesday, 29 November 2023 10:26:36 GMT Michael wrote: > > > Here's my hypothesis explaining your own observation with libreoffice. > > > As > > > a package or more finished emerging, libreoffice's turn comes up. Soon > > > libreoffice starts to execute make jobs, but any of the following may > > > apply: > > > > > > 1. There are only 4 out of 30 jobs available, because other packages are > > > already using 26, throughout your window of observation. > > > > Nope. Nothing else in progress. > > > > > 2. Libreoffice sequencing of make jobs is mostly linear with succeeding > > > make jobs waiting on output from their predecessors. > > > > That's possible, but it doesn't seem likely with such a huge code base. > > And > > why four processes, specifically and consistently? > > > > > 3. Libreoffice source code is not optimised for high parallelism - I > > > recall > > > when it was hardcoded at -j1 just a few years ago. Before this > > > restriction > > > was added, any bug reporters were advised to try again after limiting > > > make > > > to -j1. > > > > Yes, that was common to many packages for a long time because of > > incomplete > > optimisation. > > > > > Next time I'm building libreoffice on a beefier system I'll keep an eye > > > out > > > for the number of jobs to see what it gets up to. > > > > That would help, yes. > > OK, I eventually got around to it. I am observing right now LO is building > with as many as 24 jobs: > > top - 11:14:59 up 2:19, 2 users, load average: 24.46, 23.15, 9.51 > Tasks: 474 total, 25 running, 449 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.2 us, 5.6 sy, 94.0 ni, 0.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 > st > MiB Mem : 64217.1 total, 50028.6 free, 6233.7 used, 7954.9 buff/cache > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 54333.4 avail Mem > > I don't use distcc. The make -j25 -l24.8 I have specified is respected. Interesting. Thanks. > > The contribution of distcc isn't clear to me yet, as I said before. > > Sometimes it's the bee's knees; other times it might just as well not be > > there. I don't like mysteries... :) I've decided to ditch distcc altogether. During the very first build, what it grants with one hand it takes away double with the other - lots of tiny jobs all started together, but then gcc is sompiled with just two threads. That just-two happens on at least two different machines (not just separate; different). The position is no better in regular maintenance: no matter how many /make/ tasks are needed, I get just two threads compiling at a time. (I'm referring to the single-host arrangement I mentioned at the start.) I'm baffled, and I don't like it; I much prefer understanding to mystery. -- Regards, Peter.
Re: [gentoo-user] Emerge load again
On Wednesday, 29 November 2023 12:06:15 GMT Peter Humphreey wrote: > On Wednesday, 29 November 2023 10:26:36 GMT Michael wrote: > > Here's my hypothesis explaining your own observation with libreoffice. As > > a package or more finished emerging, libreoffice's turn comes up. Soon > > libreoffice starts to execute make jobs, but any of the following may > > apply: > > > > 1. There are only 4 out of 30 jobs available, because other packages are > > already using 26, throughout your window of observation. > > Nope. Nothing else in progress. > > > 2. Libreoffice sequencing of make jobs is mostly linear with succeeding > > make jobs waiting on output from their predecessors. > > That's possible, but it doesn't seem likely with such a huge code base. And > why four processes, specifically and consistently? > > > 3. Libreoffice source code is not optimised for high parallelism - I > > recall > > when it was hardcoded at -j1 just a few years ago. Before this > > restriction > > was added, any bug reporters were advised to try again after limiting make > > to -j1. > > Yes, that was common to many packages for a long time because of incomplete > optimisation. > > > Next time I'm building libreoffice on a beefier system I'll keep an eye > > out > > for the number of jobs to see what it gets up to. > > That would help, yes. OK, I eventually got around to it. I am observing right now LO is building with as many as 24 jobs: top - 11:14:59 up 2:19, 2 users, load average: 24.46, 23.15, 9.51 Tasks: 474 total, 25 running, 449 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 5.6 sy, 94.0 ni, 0.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 64217.1 total, 50028.6 free, 6233.7 used, 7954.9 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 54333.4 avail Mem I don't use distcc. The make -j25 -l24.8 I have specified is respected. > The contribution of distcc isn't clear to me yet, as I said before. > Sometimes it's the bee's knees; other times it might just as well not be > there. I don't like mysteries... :) signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Genlop wonky again
On 06/01/2024 00:54, John Blinka wrote: I’ve often found that it gives one estimate when multiple packages are being built, then a much longer estimate for still-in-progress builds once some of the builds have finished. That result defies common sense. Less remaining work has to take less, not more (much more), time. Common sense isn't common and, well, often doesn't make sense. If there's a bunch of small builds skewing the "time per build" estimate down, as they drop off the list the estimated time per build will go up, and if the skew is serious enough it can even make the total estimated time go up ... Cheers, Wol
