Re: is macports getting rusty?

[Ryan Schmidt]

On Nov 30, 2021, at 14:47, Christopher Nielsen wrote:

> Back in May, @ryandesign mentioned that we have the option of upgrading our 
> buildbot farm with more powerful CPUs. Said farm is currently hosted on 
> Xserves, and since the CPUs are socketed, it’s reasonably straightforward.

The new CPU generation is available in hex-core versions which would be a 50% 
improvement over the quad-core CPUs we have now. And the new CPU generation is 
made using a newer process with smaller traces, which will result in faster 
performance at the same speed rating.

It's not entirely as straightforward as just replacing the CPUs though, because 
the newer CPU generation is not compatible with Xserve firmware. It would 
require replacing the Xserve firmware with Mac Pro firmware, which is possible 
at the expense of some Xserve functionality, which is an acceptable tradeoff.

There's also the matter that the Xserve requires lidless CPUs which reduces the 
number of places from which one can purchase a CPU since not everybody sells 
lidless versions, or requires one to purchase a standard CPU and delid the CPU 
oneself or send it to someone else to have them do it for a fee.

I also still haven't made a decision with regard to which CPU speed to seek 
out. The Xserve shipped in three CPU speed variants but Intel offers something 
like a dozen different CPU speeds. The choice is constrained by the power 
consumption of each different speed CPU and the thermal limits of the Xserve.

Some of our builds spend more time on disk tasks than CPU tasks. We could 
increase the speed with which we can produce builds by optimizing mpbb to avoid 
unnecessary disk work, including unnecessary activations and deactivations of 
dependencies. See https://trac.macports.org/ticket/62621 for some thoughts on 
that.

RE: is macports getting rusty?

[Christopher Nielsen]

> The real improvement would be more buildbots, but that would take either a 
> way of trusting volunteer buildbots (and that they were managed properly to 
> produce fully correct results, and quite possibly dedicated to the purpose to 
> avoid anything that would conflict or interfere), or an actual budget, or 
> donated servers and server farm space, power, cooling, etc.

Back in May, @ryandesign mentioned that we have the option of upgrading our 
buildbot farm with more powerful CPUs. Said farm is currently hosted on 
Xserves, and since the CPUs are socketed, it’s reasonably straightforward.

Ultimately I’m hoping that Apple might donate one or more rack-mount MacPros 
though, as a few of those fully-loaded beasts would provide a nice upgrade.

But overall, our present setup still works quite well. And most of the time, 
the buildbots are able to keep up with the flurry of commits that occur each 
day.

> Another improvement might be more people qualified, trusted, and authorized 
> to babysit the buildbots, and/or automatic tickets on failed builds, since if 
> a build on a buildbot fails, it's either a problem with the buildbot or with 
> the port itself; and in the latter case, those building themselves will also 
> have the problem.

The good news is that there are already a number of us who monitor the 
buildbots, as well as port builds in-general. Time is spent informally, on an 
as-available basis, and there are certainly times when failures go unnoticed. 
But for heavily-used ports, build failures are generally caught quickly.

Re: is macports getting rusty?

[Chris Jones]

To first order builds are done on a first-committed-first-built basis. 
The time a build takes is not something known up front so prioritizing 
basis on this would anyway not really be feasible.


Yes, the builders must install the dependencies of a port before a given 
port can be built. If one of those deps also has an update in the queue, 
later on than the current build, but needed for the build then it will 
be done 'early'. so in a sense it is correct that commonly used 
dependencies will tend to get built a bit faster than they might 
otherwise get done.


'Normally' though the builders keep up with the pace of commits, so the 
build queue is reasonable short. It can happen that some commit might 
happen that requires a lot of ports to get built, or a big build (like 
gcc, clang or rust) comes along that causes a small backlog to build up. 
These are usually cleared quite quickly (by which I mean a few days).


Another cause of a backlog is when a new builder is added, like was done 
recently for macOS12. The builders in this case have to work their way 
through all the various deps, so a backlog can build up that takes a 
while to clear.


Chris

On 29/11/2021 2:32 pm, Richard L. Hamilton wrote:
I would expect that the buildbots also need to satisfy dependencies, and 
thus, heavily-depended on builds would tend to be done earlier, 
regardless of whether or not they were large and slow; and if they are 
large and slow, they're arguably delaying the building of even more 
smaller ports that don't depend on them, which is not necessarily a gain.


If I'm right about ports being built also by the buildbots in dependency 
order, any further tweak to build order would likely have minimal benefits.


The real improvement would be more buildbots, but that would take either 
a way of trusting volunteer buildbots (and that they were managed 
properly to produce fully correct results, and quite possibly dedicated 
to the purpose to avoid anything that would conflict or interfere), or 
an actual budget, or donated servers and server farm space, power, 
cooling, etc. Another improvement might be more people qualified, 
trusted, and authorized to babysit the buildbots, and/or automatic 
tickets on failed builds, since if a build on a buildbot fails, it's 
either a problem with the buildbot or with the port itself; and in the 
latter case, those building themselves will also have the problem.


On Nov 29, 2021, at 09:15, Bill Cole 
> wrote:


On 2021-11-29 at 08:32:46 UTC-0500 (Mon, 29 Nov 2021 13:32:46 +)
Chris Jones mailto:jon...@hep.phy.cam.ac.uk>>
is rumored to have said:

If you find yourself during an port update building rust from source, 
either just let it finish, or if you don't want your machine tied up 
building rust for some period, just cancel the build and try again 
later on, as as others have pointed out eventually the buildbots will 
provide the binary tarball for it and thus you will just pick that 
once its available.


This raises a question that I cannot find a clear answer to:

How are builds prioritized on the buildbots?

I would hope that in an ideal world, some priority is given to large, 
slow, and heavily-depended-upon builds. It's also easier to say that 
than to actually implement it, but it would help address a real pain 
point in using MacPorts.



--
Bill Cole
b...@scconsult.com  or billc...@apache.org 


(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire



--
eMail:mailto:rlha...@smart.net 

Re: is macports getting rusty?

[Richard L. Hamilton]

I would expect that the buildbots also need to satisfy dependencies, and thus, 
heavily-depended on builds would tend to be done earlier, regardless of whether 
or not they were large and slow; and if they are large and slow, they're 
arguably delaying the building of even more smaller ports that don't depend on 
them, which is not necessarily a gain.

If I'm right about ports being built also by the buildbots in dependency order, 
any further tweak to build order would likely have minimal benefits.

The real improvement would be more buildbots, but that would take either a way 
of trusting volunteer buildbots (and that they were managed properly to produce 
fully correct results, and quite possibly dedicated to the purpose to avoid 
anything that would conflict or interfere), or an actual budget, or donated 
servers and server farm space, power, cooling, etc. Another improvement might 
be more people qualified, trusted, and authorized to babysit the buildbots, 
and/or automatic tickets on failed builds, since if a build on a buildbot 
fails, it's either a problem with the buildbot or with the port itself; and in 
the latter case, those building themselves will also have the problem.

> On Nov 29, 2021, at 09:15, Bill Cole 
>  wrote:
> 
> On 2021-11-29 at 08:32:46 UTC-0500 (Mon, 29 Nov 2021 13:32:46 +)
> Chris Jones 
> is rumored to have said:
> 
>> If you find yourself during an port update building rust from source, either 
>> just let it finish, or if you don't want your machine tied up building rust 
>> for some period, just cancel the build and try again later on, as as others 
>> have pointed out eventually the buildbots will provide the binary tarball 
>> for it and thus you will just pick that once its available.
> 
> This raises a question that I cannot find a clear answer to:
> 
> How are builds prioritized on the buildbots?
> 
> I would hope that in an ideal world, some priority is given to large, slow, 
> and heavily-depended-upon builds. It's also easier to say that than to 
> actually implement it, but it would help address a real pain point in using 
> MacPorts.
> 
> 
> -- 
> Bill Cole
> b...@scconsult.com or billc...@apache.org
> (AKA @grumpybozo and many *@billmail.scconsult.com addresses)
> Not Currently Available For Hire
> 

-- 
eMail:  mailto:rlha...@smart.net

Re: is macports getting rusty?

[Bill Cole]

On 2021-11-29 at 08:32:46 UTC-0500 (Mon, 29 Nov 2021 13:32:46 +)
Chris Jones 
is rumored to have said:

If you find yourself during an port update building rust from source, 
either just let it finish, or if you don't want your machine tied up 
building rust for some period, just cancel the build and try again 
later on, as as others have pointed out eventually the buildbots will 
provide the binary tarball for it and thus you will just pick that 
once its available.


This raises a question that I cannot find a clear answer to:

How are builds prioritized on the buildbots?

I would hope that in an ideal world, some priority is given to large, 
slow, and heavily-depended-upon builds. It's also easier to say that 
than to actually implement it, but it would help address a real pain 
point in using MacPorts.



--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

Re: is macports getting rusty?

[Chris Jones]

Give the developers of rust itself a piece of your mind if you like, 
over how slow it is to compile. Good luck with that - I suspect to be 
THAT slow (although really? compiling a C compiler suite plus library 
isn't exactly fast either) they might be doing some things by brute 
force that they haven't got any other good way to do - like running a 
serious test suite, as just one example. And of course they're the ones 
that suffer most over how slow it is to compile, since they have to 
rebuild rust every time they want to test some seemingly minor change. 
So I suspect they're already doing what can to speed it up that they 
don't think sacrifices any other concern.


Actually building rust does NOT take that much longer than any other 
compiler tool kit, such as gcc or LLVM(clang) . This is in part because 
rust itself is based on LLVM and internally the build compiles its own 
LLVM instance before going on to build the rust specific parts on top of 
that. This accounts from probably something like 60-70% of the total 
build time.


If you find yourself during an port update building rust from source, 
either just let it finish, or if you don't want your machine tied up 
building rust for some period, just cancel the build and try again later 
on, as as others have pointed out eventually the buildbots will provide 
the binary tarball for it and thus you will just pick that once its 
available.


Chris

Re: is macports getting rusty?

[Richard L. Hamilton]

It is _possible_ to write reliable, secure code in C or even assembler. But for 
large complex software, it's darn unlikely; there's too many ways to mess up, 
even for capable and meticulous programmers, and most, even if bright and 
useful, aren't _that_ good.

Rust is supposed to eliminate a number of common types of problems, while still 
producing efficient code. So of course more projects will adopt it, and rewrite 
at least critical parts to use it (or something similar).

MacPorts is mainly a collection of software from elsewhere, and the mechanisms 
to automate installation and updates. That does not provide any control over 
such decisions of individual projects; MacPorts can perhaps feed bug fixes back 
to projects, but they don't have to accept them, and would not be likely to 
listen to complaints about depending on rust because of how slow it is to update

Give the developers of rust itself a piece of your mind if you like, over how 
slow it is to compile. Good luck with that - I suspect to be THAT slow 
(although really? compiling a C compiler suite plus library isn't exactly fast 
either) they might be doing some things by brute force that they haven't got 
any other good way to do - like running a serious test suite, as just one 
example. And of course they're the ones that suffer most over how slow it is to 
compile, since they have to rebuild rust every time they want to test some 
seemingly minor change. So I suspect they're already doing what can to speed it 
up that they don't think sacrifices any other concern.

A better question might be why aren't you getting/using one of the pre-built 
binary packages? On https://packages.macports.org/rust/ 
 I see for the latest rust version:

rust-1.56.1_3.darwin_13.x86_64.tbz2 2021-11-24 11:01101M
rust-1.56.1_3.darwin_14.x86_64.tbz2 2021-11-24 15:46101M
rust-1.56.1_3.darwin_15.x86_64.tbz2 2021-11-24 17:20101M
rust-1.56.1_3.darwin_16.x86_64.tbz2 2021-11-24 14:04101M
rust-1.56.1_3.darwin_17.x86_64.tbz2 2021-11-24 14:01101M
rust-1.56.1_3.darwin_18.x86_64.tbz2 2021-11-24 15:54101M
rust-1.56.1_3.darwin_19.x86_64.tbz2 2021-11-24 15:52101M
rust-1.56.1_3.darwin_20.arm64.tbz2  2021-11-25 15:0694M
rust-1.56.1_3.darwin_20.x86_64.tbz2 2021-11-24 10:38100M
rust-1.56.1_3.darwin_21.arm64.tbz2  2021-11-24 08:1594M
rust-1.56.1_3.darwin_21.x86_64.tbz2 2021-11-29 05:47100M

Look at the dates: they don't get pre-built instantly, so if you're updating 
early, you won't benefit from the pre-built version. There also appears to be a 
single defined variant (+debug) - the pre-built packages will only be used if 
that variant is NOT requested. See below for equivalent macOS versions.

Darwin version  macOS version
 1  10.0 Cheetah
 5  10.1 Puma (later minor versions, at any rate)
 6  10.2 Jaguar
 7  10.3 Panther
 8  10.4 Tiger
 9  10.5 Leopard
10  10.6 Snow Leopard
11  10.7 Lion
12  10.8 Mountain Lion
13  10.9 Mavericks
14  10.10 Yosemite
15  10.11 El Capitan
16  10.12 Sierra
17  10.13 High Sierra
18  10.14 Mojave
19  10.15 Catalina
20  11 Big Sur
21  12 Monterey



> On Nov 28, 2021, at 23:37, Kastus Shchuka  wrote:
> 
> Dear macports users,
> 
> Recently, more and more ports began to depend on rust and cargo.
> 
> Maybe rust is a wonderful language that will solve all problems of the world. 
> I just wonder, if it is so good, why it takes forever and a day (literally)  
> to compile? I've never seen anything taking that long to build. 
> 
> I've been using graphviz port for over 10 years, I guess. I had to delete it 
> today.
> 
> graphviz depends on gd2. gd2 depends on libheif. libheif depends on rav1e. 
> Now rav1e started depending on cargo-c, nasm, clang-13, cargo.
> An attempt to upgrade rav1e launched a build of cargo-c which I had to kill 
> as I did not have luxary to wait for tens of hours for it to finish.
> 
> I either have to keep outdated ports or stop using them and delete. 
> Unfortunately, the usable surface of macports started shrinking for me (or 
> should I call it "rusting"?). 
> 
> Another example is py-cryptography, which now requires rust to build. Until 
> binary package was made available, it took me over a day to upgrade 
> py-cryptography. 
> 
> I also now have a broken ImageMagic because its dependency chain pulls in 
> rust. And the list goes on and on.
> 
> I doubt people who rushed rust into macports are going to reconsider their 
> decisions. I am just sharing my experience with this "rusting"
> 
> Thank you 

Re: is macports getting rusty?

[Mark Anderson]

I think this is more a reflection of the open source landscape than
anything we're doing.

py-cryptography, for example, is 8% Rust, according to Github. I don't know
when they added it, but the latest version needs it. Same goes for rav1e.
It's one-third Rust.

Along with Go, you're going to see more and more Rust. I think the only way
around that is waiting for a binary package.

—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Sun, Nov 28, 2021 at 11:37 PM Kastus Shchuka  wrote:

> Dear macports users,
>
> Recently, more and more ports began to depend on rust and cargo.
>
> Maybe rust is a wonderful language that will solve all problems of the
> world. I just wonder, if it is so good, why it takes forever and a day
> (literally)  to compile? I've never seen anything taking that long to
> build.
>
> I've been using graphviz port for over 10 years, I guess. I had to delete
> it today.
>
> graphviz depends on gd2. gd2 depends on libheif. libheif depends on rav1e.
> Now rav1e started depending on cargo-c, nasm, clang-13, cargo.
> An attempt to upgrade rav1e launched a build of cargo-c which I had to
> kill as I did not have luxary to wait for tens of hours for it to finish.
>
> I either have to keep outdated ports or stop using them and delete.
> Unfortunately, the usable surface of macports started shrinking for me (or
> should I call it "rusting"?).
>
> Another example is py-cryptography, which now requires rust to build.
> Until binary package was made available, it took me over a day to upgrade
> py-cryptography.
>
> I also now have a broken ImageMagic because its dependency chain pulls in
> rust. And the list goes on and on.
>
> I doubt people who rushed rust into macports are going to reconsider their
> decisions. I am just sharing my experience with this "rusting"
>
> Thank you for reading.
>
> -Kastus

is macports getting rusty?

[Kastus Shchuka]

Dear macports users,

Recently, more and more ports began to depend on rust and cargo.

Maybe rust is a wonderful language that will solve all problems of the world. I 
just wonder, if it is so good, why it takes forever and a day (literally)  to 
compile? I've never seen anything taking that long to build. 

I've been using graphviz port for over 10 years, I guess. I had to delete it 
today.

graphviz depends on gd2. gd2 depends on libheif. libheif depends on rav1e. Now 
rav1e started depending on cargo-c, nasm, clang-13, cargo.
An attempt to upgrade rav1e launched a build of cargo-c which I had to kill as 
I did not have luxary to wait for tens of hours for it to finish.

I either have to keep outdated ports or stop using them and delete. 
Unfortunately, the usable surface of macports started shrinking for me (or 
should I call it "rusting"?). 

Another example is py-cryptography, which now requires rust to build. Until 
binary package was made available, it took me over a day to upgrade 
py-cryptography. 

I also now have a broken ImageMagic because its dependency chain pulls in rust. 
And the list goes on and on.

I doubt people who rushed rust into macports are going to reconsider their 
decisions. I am just sharing my experience with this "rusting"

Thank you for reading.

-Kastus