Re: llvm-9.0 on 10.6.8 shows red -- but it builds

[Joshua Root]

On 2021-5-18 15:53 , Mojca Miklavec wrote:

Sure, there's also a problem that Joshua pointed out that needs
addressing, but even that fix wouldn't solve this particular instance.


That was just an example of one improvement that could be made. It's 
been discussed before on the mailing list that it would be nice if the 
status indicators also took into account whether binaries are available 
(and other information like known_fail).


- Josh

Re: llvm-9.0 on 10.6.8 shows red -- but it builds

[Mojca Miklavec]

On Tue, 18 May 2021 at 07:15, Ken Cunningham wrote:
>
> People are quoting 
> 
>   the lack of llvm-9.0 for SnowLeopard on macports as a sign of the demise of 
> older systems, and it does for some reason show red here:
>
> https://ports.macports.org/port/llvm-9.0/builds
>
> but it builds fine, along with most later llvms and all of the earlier ones, 
> and it exists on the packages server in all archs:

To me it looks like it was broken for a very short period of time on
all systems from 10.6 to 10.9 (tons of ports were broken).
https://build.macports.org/builders/ports-10.6_x86_64-watcher/builds/9907

It must have been fixed some 30 hours later judging from the
timestamps. I see that Ryan scheduled a rebuild of many failed jobs in
https://build.macports.org/builders/ports-10.6_x86_64-watcher/builds/9920
including llvm-9.0, but llvm-9.0 is missing on that list of subports.

I suspect that's because llvm-9.0 must have been the default compiler
for one of the ports built somewhere inbetween those two portwatcher
jobs (logs for portbuilder are gone already, so we cannot investigate
those any longer), and it was built and uploaded successfully as part
of building another unrelated ports, but we most likely don't account
for a broken compiler (as a port dependency) being built explicitly in
a standalone portbuilder job.

(There are only 12 portwatcher jobs to investigate if we wanted to
find which port was that ;)

That probably counts as a bug in the Tcl script that returns the list
of subports to be built.

> If anyone knows how to make the ports.macports.org website show it in green, 
> that would be appreciated.

The fastest one would be by revbumping that port ;)

The whole logic for showing the port as green currently requires
having an entry of a successful port build in a database. But
apparently we never had such a build, and you cannot force build a
port that already exists on the package server either.

Sure, there's also a problem that Joshua pointed out that needs
addressing, but even that fix wouldn't solve this particular instance.

Mojca

(My feature request for the webapp would be to allow an explicit URL
to show older builds of a port. But it makes sense to deploy the new
site first.)

Re: llvm-9.0 on 10.6.8 shows red -- but it builds

[Joshua Root]

On 2021-5-18 15:15 , Ken Cunningham wrote:
People are quoting 
> 
  the lack of llvm-9.0 for SnowLeopard on macports as a sign of the 
demise of older systems, and it does for some reason show red here:


https://ports.macports.org/port/llvm-9.0/builds 



but it builds fine, along with most later llvms and all of the earlier 
ones, and it exists on the packages server in all archs:


http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls.darwin_10.x86_64.tbz2 



http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls.darwin_10.i386.tbz2 



http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls+universal.darwin_10.i386-x86_64.tbz2 



If anyone knows how to make the ports.macports.org 
 website show it in green, that would be 
appreciated.


The status indicator isn't very sophisticated, it simply shows the 
success or failure of the last build it knows about for the port on that 
platform. If you click on the Build Information tab, you can see that 
indeed the last known build on 10.6 failed (due to dependencies 
failing). For whatever reason, the later build that obviously succeeded 
is not in the database.


There are a number of improvements to the display that have been 
suggested, e.g. 
, and I think 
some of them are implemented in the updated version of the web app 
which, last I heard, Arjun and Mojca are working to deploy.


- Josh

llvm-9.0 on 10.6.8 shows red -- but it builds

[Ken Cunningham]

People are quoting 
>
  the lack of llvm-9.0 for SnowLeopard on macports as a sign of the demise of 
older systems, and it does for some reason show red here:

https://ports.macports.org/port/llvm-9.0/builds 


but it builds fine, along with most later llvms and all of the earlier ones, 
and it exists on the packages server in all archs:

http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls.darwin_10.x86_64.tbz2
 


http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls.darwin_10.i386.tbz2
 


http://packages.macports.org/llvm-9.0/llvm-9.0-9.0.1_1+emulated_tls+universal.darwin_10.i386-x86_64.tbz2
 


If anyone knows how to make the ports.macports.org  
website show it in green, that would be appreciated.

Thanks,

Ken

Framing the MacPorts discussion

[David Gilman]

Over the past month or so there has been a few threads on the
direction of the project, what can be improved and what the vision
should be. I wanted to summarize my two cents because I haven't been
otherwise participating and hopefully this can adjust the community
mindset towards what the problems are and frame their solutions.

TL; DR
- We're all package maintainers so MacPorts is structurally going to
be short staffed on CI (builder) work because CI is a different set of
skills and interests. Implement a structural solution like
positions/delegation to volunteers to explicitly work on CI
- Cloud hardware too expensive to consider
- In lieu of cloud, design CI that can be run on trusted volunteer's
home/office donated machines
- Consider Linux/QEMU for builder virtualization

As a user of MacPorts I greatly value having binaries available. It
would be incredible if the only packages I build are the ones that I'm
trying to maintain. There have been a few threads where users thought
binary builds weren't available or were rarely available so it seems
like it's in high demand. It is worth it to prioritize CI changes that
allow for more binaries to be available.

MacPorts has always been short on labor for the CI system. We're still
using the roots of a buildbot setup that was originally built on
Apple's dime, if my history is correct. I don't think this should
surprise any of us, though. We're all here, involved with MacPorts and
port maintenance, because we're interested in the packaging side of
things. We're not people with the time or inclination to work on CI
tasks. That's not a bad thing, but it's something that the
organizational structure should account for in order to keep stuff
maintained. Because this is a structural problem I think the mindset
of the MacPorts community is that a fix to the issue is going to
involve structural changes.

I think the way out of this is to recruit volunteers explicitly to
work on CI tasks. Trying to get package maintainers to cover CI work
is, and has been, getting blood from a stone. On the MacPorts
administration side there needs to be a bit of work put into figuring
out what the formal relationship and expectations are between CI
volunteers and the MacPorts leadership. There are lots of ways forward
there and lots of good ways to structure it. There is going to have to
be some delegation and the relationship needs to handle that.

For the CI work that does initial builds of PRs, I unfortunately think
the only technical solution here is GitHub Actions and third party
services like it. There's a plague of cryptocurrency miners looking to
exploit these systems and nothing a MacPorts volunteer team can build
will win in that game of cat and mouse. So for the future of PR CI I
think the conversation needs to be around using and maintaining those
third party solutions.

There's a thread on here about raising money to pay for CI (for
builders). I'd like to remind everyone that cloud bills are damn
expensive and I don't think MacPorts has the dedicated user base to
donate to keep that CI alive. For the vast majority of humans on the
planet a monthly donation of $20-$40 USD is a massive expenditure, and
I would be surprised if we could get a reliable $500 a month out of
everyone for MacPorts. That $500 a month can get you some cloud
computing but not a ton, and likely not what we need for reliable CI.
It's more likely than not we'd have trouble getting donations to keep
the lights on, it would be a constant struggle to fund raise, and the
whole issue around accepting donations and incorporation and that
stuff. I encourage everyone involved to write off the idea of doing
cloud CI for MacPorts builders.

Not being able to use cloud technology also hurts the ability to
recruit CI volunteers, so this tradeoff needs to be made explicit up
front. But hopefully some people are interested in a challenge.

So if a recurring cash donation is a struggle for individual donors I
think the way to get hardware for CI is a bit of a psychological hack:
ask people to buy/donate build machines and run them out of their own
offices or houses. My gut is that individuals are willing to spend
more (or at all) if it's a one time thing and it's tied to a concrete,
physical item. There's also probably enough used and aged hardware
amongst the MacPorts community that could be donated to make up a good
enough build farm. The operators of the machines are also likely
comfortable donating the electricity cost as it'll just be lumped in
with the rest of their bill. And for new machines an achievable
Kickstarter of $1000-$2000 could probably get enough donors and would
result in a fine build server.

This has a few implementation challenges. The CI system will have to
be built to work with a build farm of machines of varying reliability
and speed. Only trusted members of the community would be allowed to
host. The CI design would also need to have network isolation between
the host's personal/corporate 

Re: Homebrew Disables OSXFuse ports

[Perry E. Metzger]

This is one reasonable compromise, given that a binary installer exists 
from the (combative, not very reasonable) developer if one wants it. One 
could then just leave the dependent ports and tell people to install the 
binary themselves if they want it.


That said, I think it would be a boon to the community to have a version 
of FUSE that would work for the long term going forward and which was 
open source.


Perry

On 5/17/21 20:27, Mark Anderson wrote:
I agree with Perry, partly because the developer seems combative and I 
worry what happens if he decides to stop working on it one day.


But I also don't want to remove all the ports - this kinda leads into 
why I want cask like functionality, but then again, who would notice 
if they just wanted to install ssh-fuse or something.


My other issue is we have two ports - which I think Ryan indicated is 
kinda confusing/bad. That one seems easiest to fix.


—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Mon, May 17, 2021 at 7:06 PM Ryan Schmidt  
wrote:


On May 17, 2021, at 08:44, Perry E. Metzger wrote:

> On 5/17/21 01:44, Ryan Schmidt wrote:
>> On May 16, 2021, at 21:49, Mark Anderson wrote:
>>
>>> Given some of our recent back and forth, I found this
interesting: https://github.com/Homebrew/homebrew-core/pull/74812
>> Meh. They have different priorities than we do. No reason for
us to follow what they do.
>
> In this instance, though, I think their reasoning is correct.

So you would like MacPorts to delete all ports that depend on
osxfuse, and all ports that depend on those ports, and so on?

Each of those ports was added to MacPorts because someone wanted
it. What are they to do once we delete them?

Re: Homebrew Disables OSXFuse ports

[Perry E. Metzger]

On 5/17/21 19:06, Ryan Schmidt wrote

Meh. They have different priorities than we do. No reason for us to follow what 
they do.

In this instance, though, I think their reasoning is correct.

So you would like MacPorts to delete all ports that depend on osxfuse, and all 
ports that depend on those ports, and so on?

Each of those ports was added to MacPorts because someone wanted it. What are 
they to do once we delete them?


The FUSE implementation itself already has a binary installer (and 
indeed, we cannot build it from source since there are none), and so if 
one wants it, it's easy to install without our infrastructure.


I've suggested elsewhere that it's a good idea to probably take the last 
open source version of FUSE for MacOS and figure out how to get it to 
work without needing a kernel extension given that those are on the way out.


Anyway, this is just one person's opinion. I'm not going to yell very 
loud if it stays, but it feels icky to me.


Perry

Re: Becoming a legal entity and accepting donations (was: Re: Buildbot Performance)

[Mark Anderson]

Something like the Software Freedom Conservancy was something I was hoping
existed - I don't know if anyone else has ever heard of them, but I bet
they could help.

Yeah, I was looking at mac1.metal instances which are surprisingly cheap
for macs, but still pretty expensive.

—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Mon, May 17, 2021 at 9:14 PM Andrew Janke  wrote:

> Software Freedom Conservancy exists largely to help FLOSS orgs do this
> sort of thing safely and conveniently, while retaining independent
> governance. I believe Homebrew had a good experience with them, and
> Buildbot itself is a member. Was that one of the options considered when
> this question came up before?
>
> https://sfconservancy.org/
>
> Amazon AWS supports FLOSS projects by providing free AWS Promotional
> Credits to FLOSS projects. It looks like you could maybe use these to buy
> mac1.metal EC2 instances, and that would give you full-machine root access.
> But I dunno; those are expensive instances and the provisioning level and
> conversion process seems a little complicated. The AWS mac1.metal VMs are
> Intel-based, not Apple Silicon.
>
>
> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
>
> I dunno how much credits Amazon would want to give you, but MacPorts is a
> name with some pull, and Amazon has deep pockets. Couldn't hurt to ask?
>
> Cheers,
> Andrew
>
> On 5/17/21 8:22 PM, Mark Anderson wrote:
>
> Yeah, I was thinking of the US as well, and I meant non-profit, which
> doesn't have tax deductible donations but is assumed to not make money. The
> problem is there is a lot of work around becoming a legal entity and
> accepting donations or whatever. I honestly have no idea how much work
> exactly - but certainly not zero. I have no idea how non-US entities work.
>
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage 
> GitHub Profile 
>
>
>
> On Mon, May 17, 2021 at 2:02 AM Ryan Schmidt 
> wrote:
>
>> On May 16, 2021, at 14:46, Mark Anderson wrote:
>>
>> > I keep wondering if we became like a not-for-profit If we could get
>> someone like MacStadium or Amazon or something to donate server time to us.
>> Or accept donations from Github sponsorship. I could look into what that
>> would take, although it might be way more trouble than it's worth. I think
>> my current corp lawyer knows non-profit law - I could bring it up next time
>> I see them.
>>
>> MacStadium already donates the use of an Apple Silicon Mac mini to us. I
>> am not aware of whether Amazon offers free persistent Mac servers with root
>> access to open source projects.
>>
>> Accepting donations through GitHub Sponsors or any other means would, I
>> suspect, require the formation of a legal entity for MacPorts, which would
>> be the owner of the business bank account we would probably have to open.
>> We've discussed becoming a legal entity a few times over the years but it
>> hasn't been done. If we do it, my preference would be for MacPorts to be a
>> U.S. entity, since I am in the U.S. and since MacPorts was started by Apple
>> and is for the benefit of Apple users and Apple is a U.S. company. A
>> different suggestion was that we should join an existing free software
>> organization and leave all the legalities up to them, and funnel donations
>> through them. I don't think that idea was supported by everyone so that
>> didn't happen either.
>>
>> If we accepted donations, we would have to develop guidelines for how the
>> donations could be spent.
>>
>> Being recognized as a not-for-profit is a whole 'nother can of worms.
>> First one has to form a legal entity, then one has to apply to be
>> recognized as a not-for-profit (which incurs additional fees) and make a
>> case for why that should be, a process which can take years, and the answer
>> to the application could be no. For example there was increased scrutiny of
>> non-profit organization applications in the field of open source software
>> in 2010; see https://opensource.org/node/840. That's what I recall from
>> researching the process in the U.S. It may differ in other countries.
>>
>>
>

Re: Becoming a legal entity and accepting donations (was: Re: Buildbot Performance)

[Andrew Janke]

Software Freedom Conservancy exists largely to help FLOSS orgs do this
sort of thing safely and conveniently, while retaining independent
governance. I believe Homebrew had a good experience with them, and
Buildbot itself is a member. Was that one of the options considered when
this question came up before?

https://sfconservancy.org/

Amazon AWS supports FLOSS projects by providing free AWS Promotional
Credits to FLOSS projects. It looks like you could maybe use these to
buy mac1.metal EC2 instances, and that would give you full-machine root
access. But I dunno; those are expensive instances and the provisioning
level and conversion process seems a little complicated. The AWS
mac1.metal VMs are Intel-based, not Apple Silicon.

https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/

I dunno how much credits Amazon would want to give you, but MacPorts is
a name with some pull, and Amazon has deep pockets. Couldn't hurt to ask?

Cheers,
Andrew

On 5/17/21 8:22 PM, Mark Anderson wrote:
> Yeah, I was thinking of the US as well, and I meant non-profit, which
> doesn't have tax deductible donations but is assumed to not make
> money. The problem is there is a lot of work around becoming a legal
> entity and accepting donations or whatever. I honestly have no idea
> how much work exactly - but certainly not zero. I have no idea how
> non-US entities work.
>
> —Mark
> ___
> Mark E. Anderson mailto:m...@macports.org>>
> MacPorts Trac WikiPage 
> GitHub Profile 
>
>
>
> On Mon, May 17, 2021 at 2:02 AM Ryan Schmidt  > wrote:
>
> On May 16, 2021, at 14:46, Mark Anderson wrote:
>
> > I keep wondering if we became like a not-for-profit If we could
> get someone like MacStadium or Amazon or something to donate
> server time to us. Or accept donations from Github sponsorship. I
> could look into what that would take, although it might be way
> more trouble than it's worth. I think my current corp lawyer knows
> non-profit law - I could bring it up next time I see them.
>
> MacStadium already donates the use of an Apple Silicon Mac mini to
> us. I am not aware of whether Amazon offers free persistent Mac
> servers with root access to open source projects.
>
> Accepting donations through GitHub Sponsors or any other means
> would, I suspect, require the formation of a legal entity for
> MacPorts, which would be the owner of the business bank account we
> would probably have to open. We've discussed becoming a legal
> entity a few times over the years but it hasn't been done. If we
> do it, my preference would be for MacPorts to be a U.S. entity,
> since I am in the U.S. and since MacPorts was started by Apple and
> is for the benefit of Apple users and Apple is a U.S. company. A
> different suggestion was that we should join an existing free
> software organization and leave all the legalities up to them, and
> funnel donations through them. I don't think that idea was
> supported by everyone so that didn't happen either.
>
> If we accepted donations, we would have to develop guidelines for
> how the donations could be spent.
>
> Being recognized as a not-for-profit is a whole 'nother can of
> worms. First one has to form a legal entity, then one has to apply
> to be recognized as a not-for-profit (which incurs additional
> fees) and make a case for why that should be, a process which can
> take years, and the answer to the application could be no. For
> example there was increased scrutiny of non-profit organization
> applications in the field of open source software in 2010; see
> https://opensource.org/node/840 .
> That's what I recall from researching the process in the U.S. It
> may differ in other countries.
>

Re: Homebrew Disables OSXFuse ports

[Ruben Di Battista]

I personally feel that the developer has the right of having their own
opinion, and rightly so. Lots of commercial software do not contribute back
to the foundation made by open source projects and the fact that it is
taken for granted that someone is working free on such important piece of
software makes me uncomfortable.

If the developer decides to stop working on the code, and the code stops
working on some machines, then maybe deleting the package has a reason.

At this point, I think Macports should keep shipping osxfuse and related
ports in binary form. But we already faced this discussion... 


On Tue, 18 May 2021, 02:27 Mark Anderson,  wrote:

> I agree with Perry, partly because the developer seems combative and I
> worry what happens if he decides to stop working on it one day.
>
> But I also don't want to remove all the ports - this kinda leads into why
> I want cask like functionality, but then again, who would notice if they
> just wanted to install ssh-fuse or something.
>
> My other issue is we have two ports - which I think Ryan indicated is
> kinda confusing/bad. That one seems easiest to fix.
>
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage 
> GitHub Profile 
>
>
>
> On Mon, May 17, 2021 at 7:06 PM Ryan Schmidt 
> wrote:
>
>> On May 17, 2021, at 08:44, Perry E. Metzger wrote:
>>
>> > On 5/17/21 01:44, Ryan Schmidt wrote:
>> >> On May 16, 2021, at 21:49, Mark Anderson wrote:
>> >>
>> >>> Given some of our recent back and forth, I found this interesting:
>> https://github.com/Homebrew/homebrew-core/pull/74812
>> >> Meh. They have different priorities than we do. No reason for us to
>> follow what they do.
>> >
>> > In this instance, though, I think their reasoning is correct.
>>
>> So you would like MacPorts to delete all ports that depend on osxfuse,
>> and all ports that depend on those ports, and so on?
>>
>> Each of those ports was added to MacPorts because someone wanted it. What
>> are they to do once we delete them?
>>
>>

Re: Homebrew Disables OSXFuse ports

[Mark Anderson]

I agree with Perry, partly because the developer seems combative and I
worry what happens if he decides to stop working on it one day.

But I also don't want to remove all the ports - this kinda leads into why I
want cask like functionality, but then again, who would notice if they just
wanted to install ssh-fuse or something.

My other issue is we have two ports - which I think Ryan indicated is kinda
confusing/bad. That one seems easiest to fix.

—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Mon, May 17, 2021 at 7:06 PM Ryan Schmidt 
wrote:

> On May 17, 2021, at 08:44, Perry E. Metzger wrote:
>
> > On 5/17/21 01:44, Ryan Schmidt wrote:
> >> On May 16, 2021, at 21:49, Mark Anderson wrote:
> >>
> >>> Given some of our recent back and forth, I found this interesting:
> https://github.com/Homebrew/homebrew-core/pull/74812
> >> Meh. They have different priorities than we do. No reason for us to
> follow what they do.
> >
> > In this instance, though, I think their reasoning is correct.
>
> So you would like MacPorts to delete all ports that depend on osxfuse, and
> all ports that depend on those ports, and so on?
>
> Each of those ports was added to MacPorts because someone wanted it. What
> are they to do once we delete them?
>
>

Re: Becoming a legal entity and accepting donations (was: Re: Buildbot Performance)

[Mark Anderson]

Yeah, I was thinking of the US as well, and I meant non-profit, which
doesn't have tax deductible donations but is assumed to not make money. The
problem is there is a lot of work around becoming a legal entity and
accepting donations or whatever. I honestly have no idea how much work
exactly - but certainly not zero. I have no idea how non-US entities work.

—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Mon, May 17, 2021 at 2:02 AM Ryan Schmidt 
wrote:

> On May 16, 2021, at 14:46, Mark Anderson wrote:
>
> > I keep wondering if we became like a not-for-profit If we could get
> someone like MacStadium or Amazon or something to donate server time to us.
> Or accept donations from Github sponsorship. I could look into what that
> would take, although it might be way more trouble than it's worth. I think
> my current corp lawyer knows non-profit law - I could bring it up next time
> I see them.
>
> MacStadium already donates the use of an Apple Silicon Mac mini to us. I
> am not aware of whether Amazon offers free persistent Mac servers with root
> access to open source projects.
>
> Accepting donations through GitHub Sponsors or any other means would, I
> suspect, require the formation of a legal entity for MacPorts, which would
> be the owner of the business bank account we would probably have to open.
> We've discussed becoming a legal entity a few times over the years but it
> hasn't been done. If we do it, my preference would be for MacPorts to be a
> U.S. entity, since I am in the U.S. and since MacPorts was started by Apple
> and is for the benefit of Apple users and Apple is a U.S. company. A
> different suggestion was that we should join an existing free software
> organization and leave all the legalities up to them, and funnel donations
> through them. I don't think that idea was supported by everyone so that
> didn't happen either.
>
> If we accepted donations, we would have to develop guidelines for how the
> donations could be spent.
>
> Being recognized as a not-for-profit is a whole 'nother can of worms.
> First one has to form a legal entity, then one has to apply to be
> recognized as a not-for-profit (which incurs additional fees) and make a
> case for why that should be, a process which can take years, and the answer
> to the application could be no. For example there was increased scrutiny of
> non-profit organization applications in the field of open source software
> in 2010; see https://opensource.org/node/840. That's what I recall from
> researching the process in the U.S. It may differ in other countries.
>
>

Re: Buildbot Performance

[Jason Liu]

On Mon, May 17, 2021 at 7:54 PM Christopher Nielsen <
masc...@rochester.rr.com> wrote:

>
> Pinning our buildbot VMs to specific NUMA nodes can result in starvation,
> when multiple VMs assigned to a given node are all busy. That would also
> result in underutilization of the other node, if VMs assigned to that are
> idle.
>
> Does that make sense?


What you say does make sense, but I believe that what I am suggesting is
different than what I think you are describing. I'm not suggesting that the
guests get pinned to different NUMA nodes. Instead, I'm suggesting that all
guests get pinned to the same 6 nodes. AFAICT, the starvation situation
that you're describing can occur when some of the guests get pinned to a
different set of NUMA nodes. If all of the guests are pinned to the same
nodes, then theoretically, those guests should behave as if they all have
equal access to the same pool of physical cores. There shouldn't be any
underutilization of any particular node, because the node has been pinned
for all of the guests; this means that the hypervisor should be in control
of scheduling the utilization of the node among all of the guests which
have that node pinned.

-- 
Jason Liu


On Mon, May 17, 2021 at 7:54 PM Christopher Nielsen <
masc...@rochester.rr.com> wrote:

> Well, my recommendation for our setup, is to avoid pinning.
>
> Why? For several reasons:
> * Out-of-the-box, ESX/ESXi makes a best-effort attempt to schedule all
> vCPUs for a given VM, on a single NUMA node.
> * Even when that’s not possible, the hypervisor schedules VM vCPUs to
> hyperthread pairs.
>
> Pinning our buildbot VMs to specific NUMA nodes can result in starvation,
> when multiple VMs assigned to a given node are all busy. That would also
> result in underutilization of the other node, if VMs assigned to that are
> idle.
>
> Does that make sense?
>
> > On 2021-05-17-M, at 19:34, Jason Liu  wrote:
> >
> > If the guests on a virtual server are exerting a heavy enough load that
> the virtual host is not able to obtain the resources it needs, then the
> entire system's performance, both physical and virtual, can be affected.
> I'm not claiming to be familiar enough with the specifics of the situation
> to claim that this is what's happening, but if it is, then using CPU
> pinning can help. It's basically analogous to the situation where you
> overcommit the memory too much, and the virtual host isn't able to have
> enough memory to do the tasks it needs to.
> >
> > On the virtual servers which I set up for my customers, I have the
> hypervisor set up to automatically pin all but 2 cores. So for example, on
> an 8 core machine, I pin cores 0-5 for all of the VM settings, so that none
> of the guests are able to use cores 6 and 7. This effectively removes 2
> cores from the pool of CPUs available for the guest to use, which means
> that the virtual host will always have those 2 cores available to it at all
> times. This allows my customer to run, on average, 10 guests on the virtual
> server simultaneously, and everything stays performant, even under heavy
> vCPU loads.
> >
> > The other thing I do is that I tell my customers to never overcommit on
> memory. My rule of thumb is that the sum of all simultaneously running
> guests should never exceed the amount of physical RAM minus 4 GB. So, on a
> server with 128 GB of RAM, the sum of the memory allocated to the guests
> running simultaneously should never add up to more than 124 GB of memory.
> >
> >> On Mon, May 17, 2021 at 6:24 PM Ryan Schmidt 
> wrote:
> >>
> >>> On May 17, 2021, at 13:13, Jason Liu wrote:
> >>>
> >>> Regarding CPU overcommitment: Are the virtual hosts doing any sort of
> CPU pinning? Many virtualization products have the ability to specify which
> of the pCPU cores a guest is allowed to use. As far as I can remember,
> products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
> >>
> >> Nope, nothing like that is set up. Is there any reason why we would
> want that?
>

Re: Buildbot Performance

[Christopher Nielsen]

Well, my recommendation for our setup, is to avoid pinning.

Why? For several reasons:
* Out-of-the-box, ESX/ESXi makes a best-effort attempt to schedule all vCPUs 
for a given VM, on a single NUMA node.
* Even when that’s not possible, the hypervisor schedules VM vCPUs to 
hyperthread pairs.

Pinning our buildbot VMs to specific NUMA nodes can result in starvation, when 
multiple VMs assigned to a given node are all busy. That would also result in 
underutilization of the other node, if VMs assigned to that are idle.

Does that make sense?

> On 2021-05-17-M, at 19:34, Jason Liu  wrote:
> 
> If the guests on a virtual server are exerting a heavy enough load that the 
> virtual host is not able to obtain the resources it needs, then the entire 
> system's performance, both physical and virtual, can be affected. I'm not 
> claiming to be familiar enough with the specifics of the situation to claim 
> that this is what's happening, but if it is, then using CPU pinning can help. 
> It's basically analogous to the situation where you overcommit the memory too 
> much, and the virtual host isn't able to have enough memory to do the tasks 
> it needs to.
> 
> On the virtual servers which I set up for my customers, I have the hypervisor 
> set up to automatically pin all but 2 cores. So for example, on an 8 core 
> machine, I pin cores 0-5 for all of the VM settings, so that none of the 
> guests are able to use cores 6 and 7. This effectively removes 2 cores from 
> the pool of CPUs available for the guest to use, which means that the virtual 
> host will always have those 2 cores available to it at all times. This allows 
> my customer to run, on average, 10 guests on the virtual server 
> simultaneously, and everything stays performant, even under heavy vCPU loads.
> 
> The other thing I do is that I tell my customers to never overcommit on 
> memory. My rule of thumb is that the sum of all simultaneously running guests 
> should never exceed the amount of physical RAM minus 4 GB. So, on a server 
> with 128 GB of RAM, the sum of the memory allocated to the guests running 
> simultaneously should never add up to more than 124 GB of memory.
> 
>> On Mon, May 17, 2021 at 6:24 PM Ryan Schmidt  wrote:
>> 
>>> On May 17, 2021, at 13:13, Jason Liu wrote:
>>> 
>>> Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU 
>>> pinning? Many virtualization products have the ability to specify which of 
>>> the pCPU cores a guest is allowed to use. As far as I can remember, 
>>> products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
>> 
>> Nope, nothing like that is set up. Is there any reason why we would want 
>> that?

Re: Buildbot Performance

[Jason Liu]

>
> We don’t want any type of pinning, as that will further exacerbate the
> situation.
>

Why would that be? Do the virtual servers have a low number of physical
cores or something?

-- 
Jason Liu


On Mon, May 17, 2021 at 6:26 PM Christopher Nielsen <
masc...@rochester.rr.com> wrote:

> We don’t want any type of pinning, as that will further exacerbate the
> situation.
>
> > On 2021-05-17-M, at 18:24, Ryan Schmidt  wrote:
> >
> >> On May 17, 2021, at 13:13, Jason Liu wrote:
> >>
> >> Regarding CPU overcommitment: Are the virtual hosts doing any sort of
> CPU pinning? Many virtualization products have the ability to specify which
> of the pCPU cores a guest is allowed to use. As far as I can remember,
> products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
> >
> > Nope, nothing like that is set up. Is there any reason why we would want
> that?
>

Re: Buildbot Performance

[Jason Liu]

If the guests on a virtual server are exerting a heavy enough load that the
virtual host is not able to obtain the resources it needs, then the entire
system's performance, both physical and virtual, can be affected. I'm not
claiming to be familiar enough with the specifics of the situation to claim
that this is what's happening, but if it is, then using CPU pinning can
help. It's basically analogous to the situation where you overcommit the
memory too much, and the virtual host isn't able to have enough memory to
do the tasks it needs to.

On the virtual servers which I set up for my customers, I have the
hypervisor set up to automatically pin all but 2 cores. So for example, on
an 8 core machine, I pin cores 0-5 for all of the VM settings, so that none
of the guests are able to use cores 6 and 7. This effectively removes 2
cores from the pool of CPUs available for the guest to use, which means
that the virtual host will always have those 2 cores available to it at all
times. This allows my customer to run, on average, 10 guests on the virtual
server simultaneously, and everything stays performant, even under heavy
vCPU loads.

The other thing I do is that I tell my customers to never overcommit on
memory. My rule of thumb is that the sum of all simultaneously running
guests should never exceed the amount of physical RAM minus 4 GB. So, on a
server with 128 GB of RAM, the sum of the memory allocated to the guests
running simultaneously should never add up to more than 124 GB of memory.

-- 
Jason Liu


On Mon, May 17, 2021 at 6:24 PM Ryan Schmidt 
wrote:

> On May 17, 2021, at 13:13, Jason Liu wrote:
>
> > Regarding CPU overcommitment: Are the virtual hosts doing any sort of
> CPU pinning? Many virtualization products have the ability to specify which
> of the pCPU cores a guest is allowed to use. As far as I can remember,
> products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
>
> Nope, nothing like that is set up. Is there any reason why we would want
> that?
>
>

Re: Buildbot Performance

[Christopher Nielsen]

Hmmm, perhaps I responded a bit too quickly, as there can be some performance 
benefit to pinning to a specific NUMA node (CPU socket). Particularly if our 
VMs were only running with only four vCPUs each.

So to qualify my statement: Given the configuration we’re running - VMs with 
eight vCPUs/each, along with significant CPU overcommitment - I wouldn’t 
recommend using pinning.

> On 2021-05-17-M, at 18:26, Christopher Nielsen  
> wrote:
> 
> We don’t want any type of pinning, as that will further exacerbate the 
> situation.
> 
>> On 2021-05-17-M, at 18:24, Ryan Schmidt  wrote:
>> 
>>> On May 17, 2021, at 13:13, Jason Liu wrote:
>>> 
>>> Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU 
>>> pinning? Many virtualization products have the ability to specify which of 
>>> the pCPU cores a guest is allowed to use. As far as I can remember, 
>>> products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
>> 
>> Nope, nothing like that is set up. Is there any reason why we would want 
>> that?

Re: Buildbot Performance

[Christopher Nielsen]

Sounds good.

And perhaps that’s a good long-term plan: If we can install additional memory 
in the Xserves, such that all buildbot VMs can be allocated 9 or 10 GB each, 
that should help across-the-board.

Combine that with an upgrade to six-core Westmere Xeons, and our build times 
will increase noticeably.

When can we get those upgrades?  ;-)

> On 2021-05-17-M, at 19:00, Ryan Schmidt  wrote:
> 
> It is not problematic. Per my earlier explanation of which VM is on which 
> host, 10.8, 10.11, and 10.14 are currently together on a host that has 32 GB 
> RAM. If I set them all to 8 GB RAM, then 2.x GB RAM are not being used for 
> anything. I improved the performance of the 10.11 and 10.14 VMs and made it 
> less likely for them to swap by assigning the additional available 1 GB RAM 
> to each.

Re: Homebrew Disables OSXFuse ports

[Ryan Schmidt]

On May 17, 2021, at 08:44, Perry E. Metzger wrote:

> On 5/17/21 01:44, Ryan Schmidt wrote:
>> On May 16, 2021, at 21:49, Mark Anderson wrote:
>> 
>>> Given some of our recent back and forth, I found this interesting: 
>>> https://github.com/Homebrew/homebrew-core/pull/74812
>> Meh. They have different priorities than we do. No reason for us to follow 
>> what they do.
> 
> In this instance, though, I think their reasoning is correct.

So you would like MacPorts to delete all ports that depend on osxfuse, and all 
ports that depend on those ports, and so on?

Each of those ports was added to MacPorts because someone wanted it. What are 
they to do once we delete them?

Re: Buildbot Performance

[Ryan Schmidt]

On May 17, 2021, at 17:42, Christopher Nielsen wrote:

> Also, I see two buildbot VMs with 9 GB of memory allocated:
> 
> OS X El Capitan v10.11.6 (15G22010)
> Xcode v8.2.1 (8C1002)
> Apple LLVM version 8.0.0 (clang-800.0.42.1)
> Architecture: x86_64
> C++ library: libc++
> CPU: 8 ⨉ 2.15 GHz
> RAM: 9 GB
> Boot date: 2021-05-01T21:54:00Z
> 
> macOS Mojave v10.14.6 (18G9028)
> Xcode v10.3 (10G8)
> Apple LLVM version 10.0.1 (clang-1001.0.46.4)
> Architecture: x86_64
> C++ library: libc++
> CPU: 8 ⨉ 2.15 GHz
> RAM: 9 GB
> Boot date: 2021-05-01T21:54:00Z
> 
> This is problematic, if we’re already exceeding the physical memory available 
> [when also factoring in the needs of the hypervisor].

It is not problematic. Per my earlier explanation of which VM is on which host, 
10.8, 10.11, and 10.14 are currently together on a host that has 32 GB RAM. If 
I set them all to 8 GB RAM, then 2.x GB RAM are not being used for anything. I 
improved the performance of the 10.11 and 10.14 VMs and made it less likely for 
them to swap by assigning the additional available 1 GB RAM to each.

Re: Buildbot Performance

[Christopher Nielsen]

Also, I see two buildbot VMs with 9 GB of memory allocated:

OS X El Capitan v10.11.6 (15G22010)
Xcode v8.2.1 (8C1002)
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Architecture: x86_64
C++ library: libc++
CPU: 8 ⨉ 2.15 GHz
RAM: 9 GB
Boot date: 2021-05-01T21:54:00Z

macOS Mojave v10.14.6 (18G9028)
Xcode v10.3 (10G8)
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Architecture: x86_64
C++ library: libc++
CPU: 8 ⨉ 2.15 GHz
RAM: 9 GB
Boot date: 2021-05-01T21:54:00Z

This is problematic, if we’re already exceeding the physical memory available 
[when also factoring in the needs of the hypervisor].


> On 2021-05-17-M, at 18:30, Christopher Nielsen  
> wrote:
> 
> If you total up the memory allocated across all VMs, is there at least one or 
> two GB free for the hypervisor?
> 
>> On 2021-05-17-M, at 18:26, Ryan Schmidt  wrote:
>> 
>>> On May 17, 2021, at 07:36, Christopher Nielsen wrote:
>>> 
>>> As for overcommitment: I’m simply suggesting that we reduce the number of 
>>> vCPUs per builder, from eight to six.
>> 
>> And I'm suggesting that doing so will slow things down in those situations 
>> when only one or two VMs on a host are busy.
>> 
>> If I reduce the number of vCPUs from 8 to 6, then there would be 2 GB extra 
>> RAM per VM in each server that I could move elsewhere.
>> 
>>> Otherwise, the formula MacPorts base uses to calculate builds jobs and such 
>>> is fine as-is, and I’m not advocating that we change it.

Re: Buildbot Performance

[Ryan Schmidt]

On May 17, 2021, at 17:30, Christopher Nielsen wrote:

> If you total up the memory allocated across all VMs, is there at least one or 
> two GB free for the hypervisor?

The hypervisor reserves 4.x GB for itself.

Re: Buildbot Performance

[Christopher Nielsen]

If you total up the memory allocated across all VMs, is there at least one or 
two GB free for the hypervisor?

> On 2021-05-17-M, at 18:26, Ryan Schmidt  wrote:
> 
>> On May 17, 2021, at 07:36, Christopher Nielsen wrote:
>> 
>> As for overcommitment: I’m simply suggesting that we reduce the number of 
>> vCPUs per builder, from eight to six.
> 
> And I'm suggesting that doing so will slow things down in those situations 
> when only one or two VMs on a host are busy.
> 
> If I reduce the number of vCPUs from 8 to 6, then there would be 2 GB extra 
> RAM per VM in each server that I could move elsewhere.
> 
>> Otherwise, the formula MacPorts base uses to calculate builds jobs and such 
>> is fine as-is, and I’m not advocating that we change it.

Re: Buildbot Performance

[Christopher Nielsen]

We don’t want any type of pinning, as that will further exacerbate the 
situation.

> On 2021-05-17-M, at 18:24, Ryan Schmidt  wrote:
> 
>> On May 17, 2021, at 13:13, Jason Liu wrote:
>> 
>> Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU 
>> pinning? Many virtualization products have the ability to specify which of 
>> the pCPU cores a guest is allowed to use. As far as I can remember, products 
>> like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
> 
> Nope, nothing like that is set up. Is there any reason why we would want that?

Re: Buildbot Performance

[Ryan Schmidt]

On May 17, 2021, at 07:36, Christopher Nielsen wrote:

> As for overcommitment: I’m simply suggesting that we reduce the number of 
> vCPUs per builder, from eight to six.

And I'm suggesting that doing so will slow things down in those situations when 
only one or two VMs on a host are busy.

If I reduce the number of vCPUs from 8 to 6, then there would be 2 GB extra RAM 
per VM in each server that I could move elsewhere.


> Otherwise, the formula MacPorts base uses to calculate builds jobs and such 
> is fine as-is, and I’m not advocating that we change it.

Re: Buildbot Performance

[Ryan Schmidt]

On May 17, 2021, at 13:13, Jason Liu wrote:

> Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU 
> pinning? Many virtualization products have the ability to specify which of 
> the pCPU cores a guest is allowed to use. As far as I can remember, products 
> like KVM and ESXi can do CPU pinning, while VirtualBox cannot.

Nope, nothing like that is set up. Is there any reason why we would want that?

Re: Buildbot Performance

[Jason Liu]

On Sun, May 16, 2021 at 3:38 PM Ryan Schmidt 
wrote:

>
>
> On May 16, 2021, at 09:48, Christopher Nielsen wrote:
>

>> Upgrading them to six-core Xeons would absolutely help, for sure. But I’m
>> quite certain that we could also improve the situation, by reducing the
>> level of CPU overcommitment. And reducing the vCPUs per VM would help, as
>> we simply don’t have the physical CPU power to support eight/VM.
>>
>
> When you say reduce CPU overcommitment, by what means are you referring
> to, other than reducing CPUs per VM?


Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU
pinning? Many virtualization products have the ability to specify which of
the pCPU cores a guest is allowed to use. As far as I can remember,
products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.

-- 
Jason Liu


On Sun, May 16, 2021 at 3:38 PM Ryan Schmidt 
wrote:

>
>
> On May 16, 2021, at 09:48, Christopher Nielsen wrote:
>
> > In terms of the ratio of vCPUs to GB of RAM, 1:1 isn’t totally
> unreasonable. However, we should also reserve 2 GB of RAM for the OS,
> including the disk cache. So perhaps 6 vCPUs would be a better choice.
>
> MacPorts base hasn't ever considered reserving any RAM for the OS. I
> cannot confirm that any changes are needed to the formula of how many jobs
> we start based on how much RAM is available, but if there are, then they
> should be agreed upon and made in base first; then the buildbot setup can
> be adjusted accordingly.
>
>
> > As for the total physical CPUs available on our Xserves, here’s the rub:
> While hyperthreading does provide some benefit, best-case it generally only
> provides 50% more headroom. And sometimes it’s as low as 25%.
>
> I'm aware.
>
>
> > So if we assume best-case, our Xserve’s only provide the processing
> power of 12 CPU cores, when accounting for hyperthreading. So even if only
> two builders are active, we’re already well overcommitted on CPU. And with
> three or more going, I’d bet the hypervisor is spending more time on
> scheduling and pre-emption, than actual processing time.
>
> I cannot confirm or deny your claims about the hypervisor.
>
>
> > By way of comparison, I’m running on a modest 2008-era MacPro, with only
> eight physical CPU cores… and no hyper threading. Plus the Xeons in my
> MacPro are one major generation behind the Nehalem-based CPUs on our
> Xserves. Yet, my port build times are anywhere from 2x to 10x faster than
> we’re seeing on our builders. (And no, that’s not an exaggeration.)
>
> Are you looking at the time to build just the port, or are you including
> the time to install all the dependencies? Because on your system, you
> already have the dependencies installed, or if not, they just need to be
> downloaded and installed once. On the buildbot, on the other hand,
> dependencies are deactivated between builds, and are sometimes activated
> and deactivated many times before the main port is built. This can result
> in a significant and in my opinion an unnecessarily large amount of time
> being taken for dealing with dependencies first. I believe this can be
> improved upon. For example, instead of deactivating *all* active ports
> between each build and between each dependency, we could develop a way to
> deactive only the active ports that are not needed by the next build or
> dependency. This would have the added benefit of reducing disk wear, which
> has already been a problem before. See
> https://trac.macports.org/ticket/62621
>
> It is definitely expected that any individual VM will take longer to build
> if other VMs on the same host are also busy. Reducing the number of CPUs
> each VM has will just ensure that the builds take longer, even if other VMs
> on the same host are not busy.
>
>
> > So we need to do something, as the buildbots simply can’t keep up.
>
> I think they've been keeping up fine, except when a bunch of huge things
> need to be built, which I find understandable.
>
>
> > Upgrading them to six-core Xeons would absolutely help, for sure. But
> I’m quite certain that we could also improve the situation, by reducing the
> level of CPU overcommitment. And reducing the vCPUs per VM would help, as
> we simply don’t have the physical CPU power to support eight/VM.
>
> When you say reduce CPU overcommitment, by what means are you referring
> to, other than reducing CPUs per VM?
>
>

Re: Homebrew Disables OSXFuse ports

[Perry E. Metzger]

On 5/17/21 01:44, Ryan Schmidt wrote:

On May 16, 2021, at 21:49, Mark Anderson wrote:


Given some of our recent back and forth, I found this interesting: 
https://github.com/Homebrew/homebrew-core/pull/74812

Meh. They have different priorities than we do. No reason for us to follow what 
they do.


In this instance, though, I think their reasoning is correct.

Re: Homebrew Disables OSXFuse ports

[Perry E. Metzger]

They seem to share my opinion.

On 5/16/21 22:49, Mark Anderson wrote:
Given some of our recent back and forth, I found this interesting:Â 
https://github.com/Homebrew/homebrew-core/pull/74812

Re: Homebrew Disables OSXFuse ports

[Andrew Janke]

On 5/17/21 1:44 AM, Ryan Schmidt wrote:
> On May 16, 2021, at 21:49, Mark Anderson wrote:
>
>> Given some of our recent back and forth, I found this interesting: 
>> https://github.com/Homebrew/homebrew-core/pull/74812
> Meh. They have different priorities than we do. No reason for us to follow 
> what they do.
>

+1 to this. I'm a sometimes user of some of these OSXFuse-using
projects, and would like to see them continue to be supported in some
package manager.

Cheers,
Andrew

Re: Buildbot Performance

[Christopher Nielsen]

It used to be that you had to pay for vCenter, to get deep insights into VM 
performance. But ‘esxtop' is certainly a great starting point. And perhaps it 
does provide enough info, to get an idea of where time is being spent.

> On 2021-05-17-M, at 02:55, Daniel J. Luke  wrote:
> 
> I'm not an ESXi expert but after a quick look through some of their 
> documentation it looks like there's an `esxtop` command that can show some 
> information. More info here: https://kb.vmware.com/s/article/1005362 (some 
> google searches seem to indicate that when the oversubscription is a problem, 
> it's usually because ESXi is waiting for 'enough' cores to be available to 
> start all of the vCPUs - and that this used to be much worse in older ESXi 
> versions, but can still be a problem). This post also has information that 
> looks useful: https://www.heroix.com/blog/vmware-vcpu-over-allocation/ 
> 

Ryan, you’re right, it wasn’t bad before. And let me clarify that none of my 
comments were meant to criticize our setup; rather, I’m simply trying to 
provide some ideas to potentially improve build times.

However, things have definitely slowed down with one of the Xserve’s out of the 
mix. And if we could get back to four, with builders spread evenly amongst the 
hardware, that would help.

As for overcommitment: I’m simply suggesting that we reduce the number of vCPUs 
per builder, from eight to six. Otherwise, the formula MacPorts base uses to 
calculate builds jobs and such is fine as-is, and I’m not advocating that we 
change it.

> On 2021-05-16-S, at 15:38, Ryan Schmidt  wrote:
> 
>> So we need to do something, as the buildbots simply can’t keep up.
> 
> I think they've been keeping up fine, except when a bunch of huge things need 
> to be built, which I find understandable.
> 
>> Upgrading them to six-core Xeons would absolutely help, for sure. But I’m 
>> quite certain that we could also improve the situation, by reducing the 
>> level of CPU overcommitment. And reducing the vCPUs per VM would help, as we 
>> simply don’t have the physical CPU power to support eight/VM.
> 
> When you say reduce CPU overcommitment, by what means are you referring to, 
> other than reducing CPUs per VM?

Re: Becoming a legal entity and accepting donations (was: Re: Buildbot Performance)

[Mojca Miklavec]

On Mon, 17 May 2021 at 10:39, Ruben Di Battista wrote:
>
> Just as a side note, here in France I just created a non-profit association 
> for a project I'm working on related to the organization of an event, and the 
> process is almost free and reasonably fast. In a matter of few weeks we had 
> the association published on the official governmental gazette and a bank 
> account, also free of keeping charges.
>
> Same thing in Italy.

We have a non-profit in France and I would say that it's
extreeemely easy to run one there compared to almost any other
country.
The only downside is that one needs a person physically located there
who's willing to do some stuff occasionally, but it's orders of
magnitude less paperwork than in any other country that we
investigated about.

In Italy it gets a bit more complicated in my opinion (but still quite ok).

Another huge benefit of having an org in Europe is that organizing
meetings in Europe is much easier and cheaper with an European bank
account.
(That said, with Covid and everything, I'm not sure how likely we are
to organize further in-person meetings any time soon.)

If we are to organize a meeting here in Europe, having a legal entity
in the USA hardly helps. It's expensive to collect money for the fee,
and also expensive to pay the hotel & other costs. Also, signing any
contracts with the accommodation facility becomes a lot more tricky.

(Last time we organized the meeting via an Italian non-profit.)

> I perfectly agree with the will of having the legal entity in US,

It's also perfectly fine to run two non-profits in two countries at
the same time. But of course each one brings its own time commitment
and real costs (bank account etc.).

> but I think the process in Europe might be less expensive at least, probably 
> faster.

I believe that we discussed a while ago that having a non-profit in
the US would cost us some 1000 USD just for the bookkeeping or so (not
sure whether that price was per year or per month, but either of those
is a non-trivial amount of money unless you really have a lot of
income).

In France we simply have board members that do all the roles on a
volunteer basis.
Of course, if we had a large traffic (lots of donations and expenses)
that might not have been viable, but then we could also easily afford
to pay for one.
The important part is that it's easy enough to do the job that one
doesn't need to be a trained professional to do it.

(What I'm slightly more worried about is the manpower to run a
non-profit, no matter how little work that is.)

Mojca

Re: Homebrew Disables OSXFuse ports

[Eric Gallager via macports-dev]

I like having Fuse ports available in MacPorts, I just wish they
didn't break as often; see the following bugs:
https://trac.macports.org/ticket/59316
https://trac.macports.org/ticket/62226

On Mon, May 17, 2021 at 4:55 AM Ruben Di Battista
 wrote:
>
> I also agree with Ryan. Osxfuse is a dependency on some projects that are 
> very useful and shipping it in binary form is very therefore useful too.
>
> On Mon, 17 May 2021, 07:44 Ryan Schmidt,  wrote:
>>
>> On May 16, 2021, at 21:49, Mark Anderson wrote:
>>
>> > Given some of our recent back and forth, I found this interesting: 
>> > https://github.com/Homebrew/homebrew-core/pull/74812
>>
>> Meh. They have different priorities than we do. No reason for us to follow 
>> what they do.
>>

Re: Homebrew Disables OSXFuse ports

[Ruben Di Battista]

I also agree with Ryan. Osxfuse is a dependency on some projects that are
very useful and shipping it in binary form is very therefore useful too.

On Mon, 17 May 2021, 07:44 Ryan Schmidt,  wrote:

> On May 16, 2021, at 21:49, Mark Anderson wrote:
>
> > Given some of our recent back and forth, I found this interesting:
> https://github.com/Homebrew/homebrew-core/pull/74812
>
> Meh. They have different priorities than we do. No reason for us to follow
> what they do.
>
>

Re: Becoming a legal entity and accepting donations (was: Re: Buildbot Performance)

[Ruben Di Battista]

Just as a side note, here in France I just created a non-profit association
for a project I'm working on related to the organization of an event, and
the process is almost free and reasonably fast. In a matter of few weeks we
had the association published on the official governmental gazette and a
bank account, also free of keeping charges.

Same thing in Italy.

I perfectly agree with the will of having the legal entity in US, but I
think the process in Europe might be less expensive at least, probably
faster.


On Mon, 17 May 2021, 08:02 Ryan Schmidt,  wrote:

> On May 16, 2021, at 14:46, Mark Anderson wrote:
>
> > I keep wondering if we became like a not-for-profit If we could get
> someone like MacStadium or Amazon or something to donate server time to us.
> Or accept donations from Github sponsorship. I could look into what that
> would take, although it might be way more trouble than it's worth. I think
> my current corp lawyer knows non-profit law - I could bring it up next time
> I see them.
>
> MacStadium already donates the use of an Apple Silicon Mac mini to us. I
> am not aware of whether Amazon offers free persistent Mac servers with root
> access to open source projects.
>
> Accepting donations through GitHub Sponsors or any other means would, I
> suspect, require the formation of a legal entity for MacPorts, which would
> be the owner of the business bank account we would probably have to open.
> We've discussed becoming a legal entity a few times over the years but it
> hasn't been done. If we do it, my preference would be for MacPorts to be a
> U.S. entity, since I am in the U.S. and since MacPorts was started by Apple
> and is for the benefit of Apple users and Apple is a U.S. company. A
> different suggestion was that we should join an existing free software
> organization and leave all the legalities up to them, and funnel donations
> through them. I don't think that idea was supported by everyone so that
> didn't happen either.
>
> If we accepted donations, we would have to develop guidelines for how the
> donations could be spent.
>
> Being recognized as a not-for-profit is a whole 'nother can of worms.
> First one has to form a legal entity, then one has to apply to be
> recognized as a not-for-profit (which incurs additional fees) and make a
> case for why that should be, a process which can take years, and the answer
> to the application could be no. For example there was increased scrutiny of
> non-profit organization applications in the field of open source software
> in 2010; see https://opensource.org/node/840. That's what I recall from
> researching the process in the U.S. It may differ in other countries.
>
>

Re: Buildbot Performance

[Daniel J. Luke]

On May 17, 2021, at 2:03 AM, Ryan Schmidt  wrote:
> On May 16, 2021, at 17:57, Daniel J. Luke wrote:
>> On May 16, 2021, at 10:48 AM, Christopher Nielsen wrote:
>>> I’d bet the hypervisor is spending more time on scheduling and pre-emption, 
>>> than actual processing time.
>> 
>> This is something we could actually measure, though, right? Then we don't 
>> have to just speculate (and if we do determine that a config change needs to 
>> happen, we can use the actual measurements to help us optimize the 
>> configuration).
> 
> What would be your suggested methodology?

I'm not an ESXi expert but after a quick look through some of their 
documentation it looks like there's an `esxtop` command that can show some 
information. More info here: https://kb.vmware.com/s/article/1005362 (some 
google searches seem to indicate that when the oversubscription is a problem, 
it's usually because ESXi is waiting for 'enough' cores to be available to 
start all of the vCPUs - and that this used to be much worse in older ESXi 
versions, but can still be a problem). This post also has information that 
looks useful: https://www.heroix.com/blog/vmware-vcpu-over-allocation/

-- 
Daniel J. Luke

Re: Buildbot Performance

[Ryan Schmidt]

On May 16, 2021, at 17:57, Daniel J. Luke wrote:

> On May 16, 2021, at 10:48 AM, Christopher Nielsen wrote:
>> I’d bet the hypervisor is spending more time on scheduling and pre-emption, 
>> than actual processing time.
> 
> This is something we could actually measure, though, right? Then we don't 
> have to just speculate (and if we do determine that a config change needs to 
> happen, we can use the actual measurements to help us optimize the 
> configuration).

What would be your suggested methodology?

Re: Becoming a legal entity and accepting donations (was: Re: Buildbot Performance)

[Ryan Schmidt]

On May 16, 2021, at 14:46, Mark Anderson wrote:

> I keep wondering if we became like a not-for-profit If we could get someone 
> like MacStadium or Amazon or something to donate server time to us. Or accept 
> donations from Github sponsorship. I could look into what that would take, 
> although it might be way more trouble than it's worth. I think my current 
> corp lawyer knows non-profit law - I could bring it up next time I see them.

MacStadium already donates the use of an Apple Silicon Mac mini to us. I am not 
aware of whether Amazon offers free persistent Mac servers with root access to 
open source projects.

Accepting donations through GitHub Sponsors or any other means would, I 
suspect, require the formation of a legal entity for MacPorts, which would be 
the owner of the business bank account we would probably have to open. We've 
discussed becoming a legal entity a few times over the years but it hasn't been 
done. If we do it, my preference would be for MacPorts to be a U.S. entity, 
since I am in the U.S. and since MacPorts was started by Apple and is for the 
benefit of Apple users and Apple is a U.S. company. A different suggestion was 
that we should join an existing free software organization and leave all the 
legalities up to them, and funnel donations through them. I don't think that 
idea was supported by everyone so that didn't happen either.

If we accepted donations, we would have to develop guidelines for how the 
donations could be spent.

Being recognized as a not-for-profit is a whole 'nother can of worms. First one 
has to form a legal entity, then one has to apply to be recognized as a 
not-for-profit (which incurs additional fees) and make a case for why that 
should be, a process which can take years, and the answer to the application 
could be no. For example there was increased scrutiny of non-profit 
organization applications in the field of open source software in 2010; see 
https://opensource.org/node/840. That's what I recall from researching the 
process in the U.S. It may differ in other countries.