Re: Homebrew Disables OSXFuse ports

[Ryan Schmidt]

On May 17, 2021, at 21:00, Perry E. Metzger wrote:

> On 5/17/21 19:06, Ryan Schmidt wrote
 Meh. They have different priorities than we do. No reason for us to follow 
 what they do.
>>> In this instance, though, I think their reasoning is correct.
>> So you would like MacPorts to delete all ports that depend on osxfuse, and 
>> all ports that depend on those ports, and so on?
>> 
>> Each of those ports was added to MacPorts because someone wanted it. What 
>> are they to do once we delete them?
> 
> The FUSE implementation itself already has a binary installer (and indeed, we 
> cannot build it from source since there are none), and so if one wants it, 
> it's easy to install without our infrastructure.

As I said there are many ports in MacPorts that depend on fuse. Thus, we must 
offer fuse as a port in MacPorts if we want to continue to have those ports. 
MacPorts ports are not, as a rule, supposed to require the user to install 
other things outside of MacPorts.


> I've suggested elsewhere that it's a good idea to probably take the last open 
> source version of FUSE for MacOS and figure out how to get it to work without 
> needing a kernel extension given that those are on the way out.

If someone is interested in forking osxfuse/macfuse or any other software to 
improve it in a different direction from the original developer, that's their 
prerogative.

Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)

[Ryan Schmidt]

On May 19, 2021, at 12:38, Andrew Janke wrote:

> I have a small stack of Mac Minis I got to use as a buildbot farm for 
> Octave.app; I might be able to have them pull double duty for MacPorts 
> depending on your change volume.


On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote:

> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite 
> often idle.
> I'm not privy with how our build system work, but if we could get to a point 
> where agents can be added, stopped, throttled, trusted members of our 
> community could volunteer the computational power they have at their disposal 
> without fully dedicating a machine.
> In my specific case: I'm happy to offer VMs on that machine to volunteer 
> computational resources.


On May 20, 2021, at 08:20, Ben Greenfield wrote:

> I can definitely donate the facilities if not the talent.
> 
> I have a symmetrical fiber connection and a static ip. I also have battery 
> backup.
> I’m in the final weeks of making the building legal and I haven’t configured 
> the final network set-up for the building. I was going to set-up a vlan on my 
> hp procurve switch.
> I’m still shopping for a router to run OPNsense I think.
> 
>  I have been a mac sysadmin long time.


There seem to be a lot of people suddenly volunteering hardware for our build 
system. First, thank you; I didn't know we had people interested in that.

Our build system has never been designed to accommodate external hardware. It 
has always been designed as a centralized system controlled by one 
administrator. When it was first set up in 2011-12 it was under the control of 
our Apple administrator at macOS forge. I became the macOS forge administrator 
temporarily in late 2015, and MacPorts left macOS forge in late 2016 as that 
service shut down, and I recreated the buildbot system on my own hardware and 
have run it since then.

We now have one external Apple Silicon build machine hosted at another data 
center, but it's still under my exclusive control so that I can keep everything 
working together.

There are currently many situations where the build system gets into a state 
that requires manual intervention. Because I control all the machines, I'm able 
to make those fixes and get things back up and running quickly.

We currently have all the builders we need: one for each OS version / arch 
combination. The system was never designed to have more than that. If for 
example we added a second macOS 11 / x86_64 builder, there could be confusion 
and problems if the two machines have different OS / Xcode / command line tools 
/ java versions installed.

There are security issues to consider. The binaries produced by our buildbot 
workers are signed on the master with our private key. This is our "seal of 
approval" that says we believe these binaries to be good and safe. Users trust 
that. If we start allowing other people to run build machines, then we have the 
problem that we do not know for certain whether those other build machines are 
free of malware or other problems. We would be signing binaries for 
distribution to users without being certain of their safety or correctness. I'm 
not very comfortable with that.

Why is this discussion happening? Why do people think we need more hardware? If 
we need more or faster CPUs or more memory, I can make those changes to the 
hardware I already manage.

Re: Becoming a legal entity and accepting donations

[Ryan Schmidt]

On May 17, 2021, at 06:13, Mojca Miklavec wrote:

> On Mon, 17 May 2021 at 10:39, Ruben Di Battista wrote:
>> 
>> Just as a side note, here in France I just created a non-profit association 
>> for a project I'm working on related to the organization of an event, and 
>> the process is almost free and reasonably fast. In a matter of few weeks we 
>> had the association published on the official governmental gazette and a 
>> bank account, also free of keeping charges.
>> 
>> Same thing in Italy.
> 
> We have a non-profit in France and I would say that it's
> extreeemely easy to run one there compared to almost any other
> country.
> The only downside is that one needs a person physically located there
> who's willing to do some stuff occasionally, but it's orders of
> magnitude less paperwork than in any other country that we
> investigated about.
> 
> In Italy it gets a bit more complicated in my opinion (but still quite ok).
> 
> Another huge benefit of having an org in Europe is that organizing
> meetings in Europe is much easier and cheaper with an European bank
> account.
> (That said, with Covid and everything, I'm not sure how likely we are
> to organize further in-person meetings any time soon.)
> 
> If we are to organize a meeting here in Europe, having a legal entity
> in the USA hardly helps. It's expensive to collect money for the fee,
> and also expensive to pay the hotel & other costs. Also, signing any
> contracts with the accommodation facility becomes a lot more tricky.
> 
> (Last time we organized the meeting via an Italian non-profit.)
> 
>> I perfectly agree with the will of having the legal entity in US,
> 
> It's also perfectly fine to run two non-profits in two countries at
> the same time. But of course each one brings its own time commitment
> and real costs (bank account etc.).
> 
>> but I think the process in Europe might be less expensive at least, probably 
>> faster.
> 
> I believe that we discussed a while ago that having a non-profit in
> the US would cost us some 1000 USD just for the bookkeeping or so (not
> sure whether that price was per year or per month, but either of those
> is a non-trivial amount of money unless you really have a lot of
> income).
> 
> In France we simply have board members that do all the roles on a
> volunteer basis.
> Of course, if we had a large traffic (lots of donations and expenses)
> that might not have been viable, but then we could also easily afford
> to pay for one.
> The important part is that it's easy enough to do the job that one
> doesn't need to be a trained professional to do it.
> 
> (What I'm slightly more worried about is the manpower to run a
> non-profit, no matter how little work that is.)

I know you've suggested starting the MacPorts entity in Europe before. Maybe 
others disagree, but I personally would prefer for MacPorts to be a U.S. 
entity. Apple is a U.S. company, and the U.S. is a big country, and my 
assumption, though I don't have data to back it up, is that most Mac users are 
located in the U.S. Therefore to make it easiest for U.S. Mac users to donate 
to MacPorts, having a U.S. entity would be preferable.

Your reasoning about ease of supporting a European MacPorts meeting doesn't 
convince me. There have been MacPorts meetings in Europe before, yes, but 
there's no reason why there might not also be MacPorts meetings in other 
countries. And though we have before, there's no particular reason why MacPorts 
should financially support any IRL meeting of MacPorts users. This latest 
discussion about becoming an entity arose because we are discussing accepting 
donations for the purchase of hardware or cloud services to support our build 
infrastructure. A message in this thread posited that we would not be able to 
collect enough donations to cover such costs. If that's so, and if we want to 
allocate what money we do have toward infrastructure, then we wouldn't have 
extra money to support future meetings.

I don't see any value, especially at this point of not having any entity, of 
considering starting multiple entities in different countries.

In my previous message about the difficulty of starting a non-profit, I think I 
left out that I was thinking of a 501(c)3 tax-deductible non-profit. It's 
certainly easier to start an entity and just call it a non-profit without it 
being tax-deductible. And working to gain tax-deductible status, if that's 
possible for us, is something that could happen later.

Re: Framing the MacPorts discussion

[Ryan Schmidt]

On May 19, 2021, at 10:01, David Gilman wrote:

> but maybe the right mindset here is to do the best with what we've got.

We do, and have, for over ten years.

Re: Framing the MacPorts discussion

[Ryan Schmidt]

On May 19, 2021, at 07:56, Bjarne D Mathiesen wrote:

> Ryan Schmidt wrote:
>> I personally am not comfortable adding other build machines to our buildbot 
>> system that I do not control. When I control the machines, I know what is 
>> installed on them and that they are set up correctly. Having build machines 
>> located outside of my local network also poses additional challenges, as 
>> I've learned by having our Apple Silicon build machine outside of my 
>> network, challenges which I would prefer to minimize, not increase.
>> 
>> We currently use one build machine per OS version / arch, and have the 
>> hardware needed to do that. Adding more hardware such that we have more than 
>> one build machine per OS version / arch is not something our buildbot system 
>> was ever designed to accommodate, and would introduce problems.
> 
> So we are more-or-less single-point-of-failure regarding the buildbots
> !?

The buildbot system has always been a single point of failure, as has almost 
every other aspect of MacPorts infrastructure, with the exception of our 
mirrors. We have one mailing list server. One main repository. One main web 
server.

Buildbot is not critical infrastructure, in that it is nice to have but the 
world does not end if it is down for a short period of time. People can still 
get any packages that were built before from our distributed network of mirror 
servers, and if a binary is not available for whatever the user wants to 
install, then it can be built from source on their system.


> Personally, I'ld deem it advisable to have at least 3 complete
> systems geographically dispersed.

That seems completely infeasible to me.

As far as the buildmaster goes, Buildbot is designed to have just one. There is 
a more complicated optional multiple-master mode available in the latest 
Buildbot, but AFAIK it's intended for each master to handle separate tasks. For 
example, one master handles the web interface while a second master handles 
scheduling. I intend to use this mode in the new buildbot setup.

As for the workers, our system is designed with the assumption that there is 
one worker for each OS version / arch. We already have this with our existing 
hardware. Adding more hardware running additional copies of the workers we 
already have will introduce complications. If we want to do that, we would have 
to modify mpbb and the buildbot configuration to accommodate those 
complications. We would have to make the system much more robust against 
failures than it currently is. Right now, it works because one person (me) is 
able to intervene to fix problems that arise.

There are also security considerations in allowing additional individuals to 
run the build machines that produce our officially signed binaries.


>> 
>> Using Linux and commodity hardware is not applicable because it the macOS 
>> EULA only permits running macOS on Apple hardware, as we currently do.
> 
> So it's not the proposed software stack that's the problem, but the
> hardware stack. Then how about an old 2010/12 MacPro with all the bells
> and whistles ? I've been able to boot a 2010 MacPro w/ 256GB RAM under
> Ubuntu without any problems.

We already have four 2009 Xserves. Their CPU and memory could be upgraded if 
needed.

One of these is offline at the moment due to hardware failure. It can be 
repaired. If necessary, I'll repair it.

Re: Error: Failed to build snort

[Ryan Schmidt]

On May 19, 2021, at 19:22, Bjarne D Mathiesen wrote:

> #=> port outdated
> The following installed ports are outdated:
> snort  2.9.17_0 < 2.9.17.1_0
> root@MacPro 02:05:26 /Library/LaunchDaemons
> #=> port -cuNp upgrade snort
> --->  Computing dependencies for snort
> --->  Fetching archive for snort
> --->  Attempting to fetch snort-2.9.17.1_0+if_en0.darwin_19.x86_64.tbz2
> from https://cph.dk.packages.macports.org/snort
> --->  Attempting to fetch snort-2.9.17.1_0+if_en0.darwin_19.x86_64.tbz2
> from https://nue.de.packages.macports.org/snort
> --->  Attempting to fetch snort-2.9.17.1_0+if_en0.darwin_19.x86_64.tbz2
> from https://fra.de.packages.macports.org/snort
> --->  Fetching distfiles for snort
> --->  Verifying checksums for snort
> --->  Extracting snort
> --->  Configuring snort
> Warning: Configuration logfiles contain indications of
> -Wimplicit-function-declaration; check that features were not
> accidentally disabled:
>  pcap_lex_destroy: found in snort-2.9.17.1/config.log
> --->  Building snort
> Error: Failed to build snort: command execution failed
> Error: See
> /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_snort/snort/main.log
> for details.
> Error: Follow https://guide.macports.org/#project.tickets if you believe
> there is a bug.
> 
> 
> main.log here : https://macports.mathiesen.info/logs/net/snort/main.log

Bug reports should be filed in the issue tracker.

Re: Why is binary avoided for Xcode MPI wrappers?

[Ryan Schmidt]

On May 20, 2021, at 11:51, Eric Borisch wrote:

> It was an issue where when the xcode versions were not aligned on the builder 
> and user that the resulting binary package did not work correctly.
> 
> I really can't speak to if this is still an issue we would run into; we could 
> always try re-enabling pre-compiled usage and see if anyone runs into it 
> again. (And try to do a better job of documenting what the issue is this time 
> if it does.)

We have no tickets about the previous problem to study?


> It is possible that rev-upgrade will catch the issue (if it still occurs) as 
> well; I don't recall if that was available at the time.

rev-upgrade only checks library linkage. If you're saying that this software 
links with a library inside Xcode, and that the install name of that library 
varies by Xcode version such that the software linked with the library in one 
version of Xcode fails with a library file not found error when Xcode is 
upgraded to a new version, then yes rev-upgrade would detect that. Otherwise, 
no it would not.

Re: Why is binary avoided for Xcode MPI wrappers?

[Eric Borisch]

It was an issue where when the xcode versions were not aligned on the
builder and user that the resulting binary package did not work correctly.

I really can't speak to if this is still an issue we would run into; we
could always try re-enabling pre-compiled usage and see if anyone runs into
it again. (And try to do a better job of documenting what the issue is this
time if it does.)

It is possible that rev-upgrade will catch the issue (if it still occurs)
as well; I don't recall if that was available at the time.

 - Eric

Re: Why is binary avoided for Xcode MPI wrappers?

[Christopher Nielsen]

It’s quite possible that those binaries will work without issue. But without 
further testing and validation - across all supported macOS releases (10.6 
through Big Sur) - we simply can’t say for sure.

That being said, if our LLVM/toolchain experts (KenC, JeremyHu, etc) think it’s 
safe, we can allow those binaries to be used.

Thoughts?


> On 2021-05-20-T, at 12:30, Christopher Chavez  wrote:
> 
> The MPI wrapper ports (openmpi, mpich) for Xcode avoid binary archives from 
> builders. The stated reason (currently in the mpiutil 1.0 portgroup) is that 
> the builder and user may have different Xcode versions. This goes back at 
> least to https://github.com/macports/macports-ports/commit/7c6fc1faa9 in 
> 2014; not aware if discussed further elsewhere.
> 
> I am curious: is this still an issue (whether for the original reasons and/or 
> others)?
> 
> Consider what happens if Xcode is updated—either on the builder, or on a 
> user’s machine where they’ve locally built MPI wrapper ports: would the MPI 
> wrapper ports need to be rebuilt? If not, then avoiding the binary archive 
> from the builder seems needlessly restrictive. (If so, however, I’m not aware 
> whether the ports detect they need to be rebuilt and prompt the user to do 
> so.)

Why is binary avoided for Xcode MPI wrappers?

[Christopher Chavez]

The MPI wrapper ports (openmpi, mpich) for Xcode avoid binary archives from 
builders. The stated reason (currently in the mpiutil 1.0 portgroup) is that 
the builder and user may have different Xcode versions. This goes back at least 
to https://github.com/macports/macports-ports/commit/7c6fc1faa9 in 2014; not 
aware if discussed further elsewhere.

I am curious: is this still an issue (whether for the original reasons and/or 
others)?

Consider what happens if Xcode is updated—either on the builder, or on a user’s 
machine where they’ve locally built MPI wrapper ports: would the MPI wrapper 
ports need to be rebuilt? If not, then avoiding the binary archive from the 
builder seems needlessly restrictive. (If so, however, I’m not aware whether 
the ports detect they need to be rebuilt and prompt the user to do so.)

Christopher A. Chavez

Re: Framing the MacPorts discussion

[Ben Greenfield via macports-dev]

I can definitely donate the facilities if not the talent.

I have a symmetrical fiber connection and a static ip. I also have battery 
backup.
I’m in the final weeks of making the building legal and I haven’t configured 
the final network set-up for the building. I was going to set-up a vlan on my 
hp procurve switch.
I’m still shopping for a router to run OPNsense I think.

 I have been a mac sysadmin long time.

happy to accommodate.


Ben


> On May 19, 2021, at 11:01 AM, David Gilman  wrote:
> 
> Three distributed machines is a great goal, I agree with you, but maybe the 
> right mindset here is to do the best with what we've got. If we have three or 
> more trusted volunteers and the CI can support it let's do it, if we have 
> fewer, we'll live with it.
> 
> On Wed, May 19, 2021, 8:57 AM Bjarne D Mathiesen  > wrote:
> 
> 
> Ryan Schmidt wrote:
> > I personally am not comfortable adding other build machines to our buildbot 
> > system that I do not control. When I control the machines, I know what is 
> > installed on them and that they are set up correctly. Having build machines 
> > located outside of my local network also poses additional challenges, as 
> > I've learned by having our Apple Silicon build machine outside of my 
> > network, challenges which I would prefer to minimize, not increase.
> > 
> > We currently use one build machine per OS version / arch, and have the 
> > hardware needed to do that. Adding more hardware such that we have more 
> > than one build machine per OS version / arch is not something our buildbot 
> > system was ever designed to accommodate, and would introduce problems.
> 
> So we are more-or-less single-point-of-failure regarding the buildbots
> !? Personally, I'ld deem it advisable to have at least 3 complete
> systems geographically dispersed.
> 
> > 
> > Using Linux and commodity hardware is not applicable because it the macOS 
> > EULA only permits running macOS on Apple hardware, as we currently do.
> 
> So it's not the proposed software stack that's the problem, but the
> hardware stack. Then how about an old 2010/12 MacPro with all the bells
> and whistles ? I've been able to boot a 2010 MacPro w/ 256GB RAM under
> Ubuntu without any problems.
> 
> -- 
> Bjarne D Mathiesen
> Korsør ; Danmark ; Europa
> ---
> denne besked er skrevet i et totalt M$-frit miljø
> OpenCore + macOS 10.15.7 Catalina
> MacPro 2010 ; 2 x 3,46 GHz 6-Core Intel Xeon ; 256 GB 1333 MHz DDR3 ECC
> ATI Radeon RX 590 8 GB

Re: Framing the MacPorts discussion

[Enrico Maria Crisostomo]

Hi all,

I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is
quite often idle.
I'm not privy with how our build system work, but if we could get to a
point where agents can be added, stopped, throttled, trusted members of our
community could volunteer the computational power they have at their
disposal without fully dedicating a machine.
In my specific case: I'm happy to offer VMs on that machine to volunteer
computational resources.

On Wed, May 19, 2021 at 7:38 PM Andrew Janke  wrote:

> I have a small stack of Mac Minis I got to use as a buildbot farm for
> Octave.app; I might be able to have them pull double duty for MacPorts
> depending on your change volume.
>
> Cheers,
> Andrew
>
> On 5/19/21 1:16 PM, Mark Anderson wrote:
>
> Yeah - we are certainly short staffed everywhere - I try to add more and
> more of my time to the project but aside from my ports, I'm still in
> learning mode digging through all the asciidoc and tcl and everything. I'm
> trying to build some tools to help me, but again, more time.
>
> Once we move to the latest build bot, we might want to see if we can get
> other volunteers to host some hardware, but the problem is, we're going to
> have to hit up ebay or something and the infrastructure will be tougher.
>
> I'm honestly not sure how we can manage to staff up more at all - I mean
> this is a FOSS problem all over for non-company sponsored projects.
>
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage 
> GitHub Profile 
>
>
>
> On Tue, May 18, 2021 at 2:35 AM Ryan Schmidt 
> wrote:
>
>> MacPorts is short-staffed in all areas, not just infrastructure.
>>
>> Our Buildbot system works. It produces all the binaries we are able to.
>>
>> Our buildbot system was already substantially redesigned when we took it
>> over from Apple in 2016 and will be substantially redesigned again as we
>> upgrade to the latest version of the buildbot software.
>>
>> We already have a small infrastructure team who are interested in working
>> on improving the buildbot system and our other infrastructure, and do so.
>>
>> We already use GitHub Actions for CI. We cannot use it to replace
>> buildbot because it only offers recent OS versions and because it does not
>> offer persistent machines.
>>
>> I personally am not comfortable adding other build machines to our
>> buildbot system that I do not control. When I control the machines, I know
>> what is installed on them and that they are set up correctly. Having build
>> machines located outside of my local network also poses additional
>> challenges, as I've learned by having our Apple Silicon build machine
>> outside of my network, challenges which I would prefer to minimize, not
>> increase.
>>
>> We currently use one build machine per OS version / arch, and have the
>> hardware needed to do that. Adding more hardware such that we have more
>> than one build machine per OS version / arch is not something our buildbot
>> system was ever designed to accommodate, and would introduce problems.
>>
>> Using Linux and commodity hardware is not applicable because it the macOS
>> EULA only permits running macOS on Apple hardware, as we currently do.
>
>
>