Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

Just for the record here, I'm signing off the "cask" port issue, and will leave 
it you to sort out what you want to do, if anything.

I won't review or commit any cask port submissions, and I have no opinion about 
them other to say that I don't use them myself.

Best,

Ken

Re: "cask" ports just keep on rolling in...

[Clemens Lang]

Hi Ken,

On Sun, Feb 07, 2021 at 07:59:39AM -0800, Ken Cunningham wrote:
> > Now, we're apparently shipping binaries compiled by other people
> > with other people's toolchains. When I previously installed things
> > from MacPorts, I knew that I'd either compile those with my own
> > toolchain locally, or that they had been compiled on MacPorts'
> > buildbots. Repackaging binaries breaks that assumption and adds
> > additional trusted third parties. If such parties were infiltrated
> > by supply chain attacks such as Xcode Ghost, we'd now ship malware
> > via 'port install'.
> 
> Now that the reality of what this really means comes to the fore,
> people are starting the be more vocal about their thoughts on it. This
> is good. These ports have been coming in, with no plan so far.

I simply haven't followed the other thread. There are about a hundred
messages in there, and as you surely have noticed, I no longer have the
time to contribute as much and/or read such long threads :/


> So new policy coming then?
> 
> As I stated at the very beginning months ago, with no plan or
> guidance, these just keep coming in. I am not championing this, but
> raising the flag here that there is a potential problem.
> 
> If it took a slightly more obvious message about what this really
> meant to get everyone to notice, I guess I accept that.

I think we should document a proposal somewhere in the wiki, invite for
comments, and then once a consensus is reached, that becomes our policy.

MacPorts has usually made such decisions by consensus rather than
benevolent dictatorship. I recall only one instance in our history where
we have strayed from this path, and that was the migration to GitHub,
which was planned and prototyped by a smaller group of people before it
was announced. We've never had a formal process to make such policy
decisions, but maybe it is time do introduce one by example now.

Ken, you're obviously aware of the potential issues with shipping
binaries. Would you find the time to write a proposal policy of how
MacPorts should handle this and ask for comments?

I personally see three potential alternatives:

(a) don't accept binary ports except for the cases where Apple's signing
requirements or the buildsystem complexity do not allow us to build our
own.

(b) accept binary ports in the main tree with a clear naming scheme
(unfortunately I don't think the variant proposal is enough, it's simply
not obvious enough what you're getting and that you're trusting
additional third parties by installing those)

(c) create a separate tree that contains binaries and require users that
want those to add said tree to sources.conf (potentially with some
usability improvements like providing a command to automatically do
that).


The last proposal might require some work on base and potentially the
CI, but I think it could be done with limited effort. My personal
preference is a > c > b, but since I'm no longer that involved in
MacPorts development that's a decision I'd leave to the people that are
more active contributors.


HTH,
-- 
Clemens

Re: "cask" ports just keep on rolling in...

[Ryan Schmidt]

On Feb 7, 2021, at 09:59, Ken Cunningham wrote:

> On Feb 7, 2021, at 03:17, Clemens Lang wrote:
> 
>> Hi Ken,
>> 
>>> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
>>> although I was concerned about getting this pattern right before we
>>> had too many of these to fix, it does seem the admins feel there's
>>> really no issue to worry about here.
>> 
>> I don't like this tone, Ken. "The admins" have as much obligation to
>> provide infrastructure as anybody else in this project, which is none.
> 
> It wasn't meant to be a tone. Just stating the exact facts. 
> 

>> If you feel repackaging binary archives is a thing MacPorts should
>> support better, please invest the time to come up with patches that do
>> this, or find somebody that will.
> 
> I did wonder about that, started some thought about how that might proceed, 
> but the clear message was the current system is fine and there is nothing of 
> significance that needs fixing.

To clarify a couple things... The managers (Josh, Rainer and myself) are here 
to guide the project. We've been here longer than most so we can provide 
historical context about why things are done the way they're done and recommend 
consistency in how we do things. But our opinions are not meant to be 
directives that can never be broken. We've granted commit access to you and 
many others who have demonstrated competency and understanding of MacPorts 
traditions and we trust you to make sound judgements when it comes to making 
changes in MacPorts.

I did not mean to convey that I "feel there's really no issue to worry about 
here" or that "the current system is fine and there is nothing of significance 
that needs fixing". What I meant to convey was that *based on the concerns you 
had raised so far* in this thread, which I addressed in turn, I did not 
understand why you feel that the current situation is an "unworkable disaster". 
You declined to elaborate, saying "without ever trying it, you couldn't know". 
I've already spent hours on this thread that I didn't plan on spending, and I 
don't want to spend more time trying your portfile. If you believe your 
portfile exposes a flaw in MacPorts base or some other need for changes to 
base, please describe it.


>>> So I guess we just open the gate and let them in. There is no
>>> recommendation for a requirement for a naming convention or other
>>> identifier.
>> 
>> Personally, I don't like this trend at all. It always used to be
>> MacPorts' policy to compile from source except in cases where Apple's
>> limitations made this impossible (e.g. because valid signatures with a
>> developer certificate were required and an ad-hoc signature would not
>> work).
>> 
>> Now, we're apparently shipping binaries compiled by other people with
>> other people's toolchains. When I previously installed things from
>> MacPorts, I knew that I'd either compile those with my own toolchain
>> locally, or that they had been compiled on MacPorts' buildbots.
>> Repackaging binaries breaks that assumption and adds additional trusted
>> third parties. If such parties were infiltrated by supply chain attacks
>> such as Xcode Ghost, we'd now ship malware via 'port install'.
> 
> Now that the reality of what this really means comes to the fore, people are 
> starting the be more vocal about their thoughts on it. This is good. These 
> ports have been coming in, with no plan so far.

I thought that binary ports were exactly for what Clemens says above: "cases 
where Apple's limitations made [compiling from source] impossible" (or at least 
sufficiently difficult). I've said many times that I don't have a problem with 
that, and we've long had such ports in MacPorts already (e.g. for closed-source 
software like oracle-instantclient and libxl; osxfuse should be one too). But 
if we're proposing a more general acceptance of precompiled software, even if 
we could build it from source, then I think it's reasonable to ask why we would 
want to allow that.


> So new policy coming then?
> 
> As I stated at the very beginning months ago, with no plan or guidance, these 
> just keep coming in. I am not championing this, but raising the flag here 
> that there is a potential problem.
> 
> If it took a slightly more obvious message about what this really meant to 
> get everyone to notice, I guess I accept that.

It's not going to be me or the other managers who comes down from high with an 
edict telling everyone else how to handle this. I for one have demonstrated 
that I don't yet understand what the concerns are, so I'm not yet equipped to 
advise what to do. I recommend interested parties have a discussion, as in this 
mailing list thread, to come to an agreement about what the issues are and how 
each of them should be addressed. I will warn you that I may not personally be 
participating in this discussion, which does not mean the issues are not 
important, just that we all have to decide what we have time to work on, and I 
may not 

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

> On Feb 7, 2021, at 03:17, Clemens Lang  wrote:
> 
> Hi Ken,
> 
>> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
>> although I was concerned about getting this pattern right before we
>> had too many of these to fix, it does seem the admins feel there's
>> really no issue to worry about here.
> 
> I don't like this tone, Ken. "The admins" have as much obligation to
> provide infrastructure as anybody else in this project, which is none.

It wasn't meant to be a tone. Just stating the exact facts. 

> 
> If you feel repackaging binary archives is a thing MacPorts should
> support better, please invest the time to come up with patches that do
> this, or find somebody that will.
> 

I did wonder about that, started some thought about how that might proceed, but 
the clear message was the current system is fine and there is nothing of 
significance that needs fixing.

> 
>> So I guess we just open the gate and let them in. There is no
>> recommendation for a requirement for a naming convention or other
>> identifier.
> 
> Personally, I don't like this trend at all. It always used to be
> MacPorts' policy to compile from source except in cases where Apple's
> limitations made this impossible (e.g. because valid signatures with a
> developer certificate were required and an ad-hoc signature would not
> work).
> 
> Now, we're apparently shipping binaries compiled by other people with
> other people's toolchains. When I previously installed things from
> MacPorts, I knew that I'd either compile those with my own toolchain
> locally, or that they had been compiled on MacPorts' buildbots.
> Repackaging binaries breaks that assumption and adds additional trusted
> third parties. If such parties were infiltrated by supply chain attacks
> such as Xcode Ghost, we'd now ship malware via 'port install'.

Now that the reality of what this really means comes to the fore, people are 
starting the be more vocal about their thoughts on it. This is good. These 
ports have been coming in, with no plan so far.


> 
> I do know that we have recently started making more and more exceptions
> to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
> these approaches to a broken build of a very outdated version, but I'd
> like to argue that we should keep asking ourselves the question "should
> we really trust this person's toolchain" before merging such ports and
> keep pushing for builds from source where those are feasible.
> 
> Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
> source code that was used to compile a certain binary. Our distfiles
> server does that automatically for everything that's compiled from
> source, but repackaging a binary that contains compiled GPL or LGPL code
> puts us in legal muddy water very quickly.
> 
> -- 
> Clemens



All true.

So new policy coming then?

As I stated at the very beginning months ago, with no plan or guidance, these 
just keep coming in. I am not championing this, but raising the flag here that 
there is a potential problem.

If it took a slightly more obvious message about what this really meant to get 
everyone to notice, I guess I accept that.

K

Re: "cask" ports just keep on rolling in...

[Jonathan Stickel]

On 2/7/21 04:48 , macports-dev-requ...@lists.macports.org wrote:

Hi Ken,

On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:

although I was concerned about getting this pattern right before we
had too many of these to fix, it does seem the admins feel there's
really no issue to worry about here.

I don't like this tone, Ken. "The admins" have as much obligation to
provide infrastructure as anybody else in this project, which is none.

If you feel repackaging binary archives is a thing MacPorts should
support better, please invest the time to come up with patches that do
this, or find somebody that will.



If my interject, as a long-time user and observer of open-source 
software and occasional contributor, this bit of conflict is a common 
theme. There is always a tension between "doing it right" and "getting 
it done". Ken seems to be leaning towards "getting it done". I 
appreciate this a lot. I've been frustrated, at times, over the years by 
languishing broken ports, especially when I offered patches/PRs that 
received no response. What to do when there isn't consensus on a 
particular issue or a new policy, like this one about binary-only ports? 
"Doing it right" then means doing nothing at all. When volunteers are 
willing to work on something but face this sentiment, they tend to stop 
contributing. I don't know the answers but urge caution before things 
get personal.


Regarding binary ports, I guess I don't understand what is the point. 
Why not require users install the upstream binaries themselves and use 
`path:` dependencies for things that depend on them? I know this isn't 
ideal because it isn't automated, but I consider that a secondary issue.


Regards,
Jonathan

Re: "cask" ports just keep on rolling in...

[Ruben Di Battista]

If I can express my opinion, I agree with Clemens.

In my opinion binary only ports should be allowed only for a very
restricted subset, where the effort is justified by the extreme complexity
of the build (and I'm also I'm not sure about that even, if someone managed
to build it, we should be able too ☺) or the port bering binary only (e.g.
osxfuse).



On Sun, 7 Feb 2021, 12:17 Clemens Lang,  wrote:

> Hi Ken,
>
> On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
> > although I was concerned about getting this pattern right before we
> > had too many of these to fix, it does seem the admins feel there's
> > really no issue to worry about here.
>
> I don't like this tone, Ken. "The admins" have as much obligation to
> provide infrastructure as anybody else in this project, which is none.
>
> If you feel repackaging binary archives is a thing MacPorts should
> support better, please invest the time to come up with patches that do
> this, or find somebody that will.
>
>
> > So I guess we just open the gate and let them in. There is no
> > recommendation for a requirement for a naming convention or other
> > identifier.
>
> Personally, I don't like this trend at all. It always used to be
> MacPorts' policy to compile from source except in cases where Apple's
> limitations made this impossible (e.g. because valid signatures with a
> developer certificate were required and an ad-hoc signature would not
> work).
>
> Now, we're apparently shipping binaries compiled by other people with
> other people's toolchains. When I previously installed things from
> MacPorts, I knew that I'd either compile those with my own toolchain
> locally, or that they had been compiled on MacPorts' buildbots.
> Repackaging binaries breaks that assumption and adds additional trusted
> third parties. If such parties were infiltrated by supply chain attacks
> such as Xcode Ghost, we'd now ship malware via 'port install'.
>
> I do know that we have recently started making more and more exceptions
> to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
> these approaches to a broken build of a very outdated version, but I'd
> like to argue that we should keep asking ourselves the question "should
> we really trust this person's toolchain" before merging such ports and
> keep pushing for builds from source where those are feasible.
>
> Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
> source code that was used to compile a certain binary. Our distfiles
> server does that automatically for everything that's compiled from
> source, but repackaging a binary that contains compiled GPL or LGPL code
> puts us in legal muddy water very quickly.
>
> --
> Clemens
>

Re: "cask" ports just keep on rolling in...

[Clemens Lang]

Hi Ken,

On Sat, Feb 06, 2021 at 11:35:58PM -0800, Ken Cunningham wrote:
> although I was concerned about getting this pattern right before we
> had too many of these to fix, it does seem the admins feel there's
> really no issue to worry about here.

I don't like this tone, Ken. "The admins" have as much obligation to
provide infrastructure as anybody else in this project, which is none.

If you feel repackaging binary archives is a thing MacPorts should
support better, please invest the time to come up with patches that do
this, or find somebody that will.


> So I guess we just open the gate and let them in. There is no
> recommendation for a requirement for a naming convention or other
> identifier.

Personally, I don't like this trend at all. It always used to be
MacPorts' policy to compile from source except in cases where Apple's
limitations made this impossible (e.g. because valid signatures with a
developer certificate were required and an ad-hoc signature would not
work).

Now, we're apparently shipping binaries compiled by other people with
other people's toolchains. When I previously installed things from
MacPorts, I knew that I'd either compile those with my own toolchain
locally, or that they had been compiled on MacPorts' buildbots.
Repackaging binaries breaks that assumption and adds additional trusted
third parties. If such parties were infiltrated by supply chain attacks
such as Xcode Ghost, we'd now ship malware via 'port install'.

I do know that we have recently started making more and more exceptions
to this rule, e.g. for Java and Haskell, and I'm guilty of preferring
these approaches to a broken build of a very outdated version, but I'd
like to argue that we should keep asking ourselves the question "should
we really trust this person's toolchain" before merging such ports and
keep pushing for builds from source where those are feasible.

Also, keep in mind that some licenses (GPL, LGPL) require us to ship the
source code that was used to compile a certain binary. Our distfiles
server does that automatically for everything that's compiled from
source, but repackaging a binary that contains compiled GPL or LGPL code
puts us in legal muddy water very quickly.

-- 
Clemens

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

although I was concerned about getting this pattern right before we had too 
many of these to fix, it does seem the admins feel there's really no issue to 
worry about here.

So I guess we just open the gate and let them in. There is no recommendation 
for a requirement for a naming convention or other identifier.

I have gone ahead and committed the ones in the PR queue I found there. Feel 
free to use those as a template.

K

> On Feb 6, 2021, at 18:16, Mark Anderson  wrote:
> 
> Yeah, it seems like a lot of the stuff is in place, but we just need some 
> tweaks. I like the idea of a portgroup like BinaryOnly or something, but what 
> else needs to happen?
> 
> I'd be more than happy to help, what needs to be done? Should we maybe take 
> to Trac to get a to-do and proposal going?
>  
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage
> GitHub Profile
> 
> 
> 
>> On Sat, Feb 6, 2021 at 3:28 AM Ken Cunningham 
>>  wrote:
>> 
>> 
>> > On Feb 5, 2021, at 10:00 PM, Ryan Schmidt  wrote:
>> > 
>> > 
>> > 
>> > On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
>> > 
>> >> With no plan, we’ll just keep getting more and more of these.
>> > 
>> > I'm not aware of a huge influx of these, but I'm also not really paying 
>> > attention to the PR queue. And I'm not intending to get drawn into this 
>> > discussion of binary ports again.
>> 
>> The last discussion didn’t get anywhere past the appropriate naming scheme 
>> :> We never even got into implementation details.
>> 
>> >> This kind of port just repackages the DMG into many tgz archives.
>> >> 
>> >> It’s wasteful. People want them.
>> >> 
>> >> What we should have instead of this is some kinda tech that
>> >> 
>> >> 1. downloads the DMG
>> >> 2. installs the app
>> >> 3. records some way of uninstalling everything
>> > 
>> > MacPorts already does all these things... The submitted Portfile works 
>> > fine, presumably. There's no need to reinvent the existing MacPorts 
>> > functionality that does all these things.
>> > 
>> 
>> Well, IMHO there is, but I’ve looked at it quite a bit.
>> 
>> Look back a month or so and see my post about a “cask” port for SageMath for 
>> an idea of why this won’t work (well) in general.
>> 
>> For trivially simple ports, a few MB or less for a *.app that copies into 
>> /Applications/MacPorts — yeah, who cares?
>> 
>> > 
>> >> What we have instead is a repackaging of the DMG into many, identical, 
>> >> system-specific archive bundles.
>> >> 
>> >> Yuk.
>> > 
>> > If your objection is that we waste server space storing several copies of 
>> > the same thing, then we could invent a new way for a portfile to indicate 
>> > that it does not want binary archives stored on the server. It could be a 
>> > new separate keyword or a new pseudo license type, like we already have 
>> > for "nomirror" which tells the buildbot not to mirror the distfiles.
>> > A port like the one we're talking about in the above PR could set e.g. 
>> > "license GPL-2 nopackage", and buildbot could be modified to understand 
>> > that this means that it should not upload the packages.
>> 
>> Yes, that would help, if these binary ports start getting to any size.
>> 
>> > 
>> > MacPorts base would still try to download the nonexistent package. 
>> > MacPorts base currently does not use license values except to display to 
>> > the user and has no idea if a port is distributable. Distributability is 
>> > handled by a separate script. It should be integrated into base so that 
>> > base can know if something could have been distributed, and if it could 
>> > not have, then it shouldn't even try to download it. 
>> > https://trac.macports.org/ticket/60315
>> > 
>> 
>> Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup 
>> could set “archive_sites” to “” I guess, perhaps more easily.
>> 
>> > We might also want to modify the MacPorts base command line "-b" binary 
>> > only mode to allow installing these "nopackage" ports rather than error 
>> > out as it would currently do.
>> > 
>> > If the few steps like disabling the configure and build phases and adding 
>> > the hypothetical "nopackage" to the license line are too much work for a 
>> > portfile submitter to do, a portgroup could be created that does those 
>> > things. But you have often complained about "magic" portgroups doing 
>> > things you didn't know about or didn't expect, so there is something to be 
>> > said for ports doing what they need to do explicitly, when it's not so 
>> > many steps, and when it's not always the same steps needing to be done. 
>> > For example, surely each port will still need to specify what the archs of 
>> > the binary files are, what license they're under, what type of distfile it 
>> > is and where it is, and which files inside needs to be copied where.
>> > 
>> 
>> It’s not that it’s too much work. It’s that these things are very simple, 
>> and people submit them but don’t know all these details you mention.
>> 
>> 

Re: "cask" ports just keep on rolling in...

[Mark Anderson]

Yeah, it seems like a lot of the stuff is in place, but we just need some
tweaks. I like the idea of a portgroup like BinaryOnly or something, but
what else needs to happen?

I'd be more than happy to help, what needs to be done? Should we maybe take
to Trac to get a to-do and proposal going?

—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Sat, Feb 6, 2021 at 3:28 AM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

>
>
> > On Feb 5, 2021, at 10:00 PM, Ryan Schmidt 
> wrote:
> >
> >
> >
> > On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
> >
> >> With no plan, we’ll just keep getting more and more of these.
> >
> > I'm not aware of a huge influx of these, but I'm also not really paying
> attention to the PR queue. And I'm not intending to get drawn into this
> discussion of binary ports again.
>
> The last discussion didn’t get anywhere past the appropriate naming scheme
> :> We never even got into implementation details.
>
> >> This kind of port just repackages the DMG into many tgz archives.
> >>
> >> It’s wasteful. People want them.
> >>
> >> What we should have instead of this is some kinda tech that
> >>
> >> 1. downloads the DMG
> >> 2. installs the app
> >> 3. records some way of uninstalling everything
> >
> > MacPorts already does all these things... The submitted Portfile works
> fine, presumably. There's no need to reinvent the existing MacPorts
> functionality that does all these things.
> >
>
> Well, IMHO there is, but I’ve looked at it quite a bit.
>
> Look back a month or so and see my post about a “cask” port for SageMath
> for an idea of why this won’t work (well) in general.
>
> For trivially simple ports, a few MB or less for a *.app that copies into
> /Applications/MacPorts — yeah, who cares?
>
> >
> >> What we have instead is a repackaging of the DMG into many, identical,
> system-specific archive bundles.
> >>
> >> Yuk.
> >
> > If your objection is that we waste server space storing several copies
> of the same thing, then we could invent a new way for a portfile to
> indicate that it does not want binary archives stored on the server. It
> could be a new separate keyword or a new pseudo license type, like we
> already have for "nomirror" which tells the buildbot not to mirror the
> distfiles.
> > A port like the one we're talking about in the above PR could set e.g.
> "license GPL-2 nopackage", and buildbot could be modified to understand
> that this means that it should not upload the packages.
>
> Yes, that would help, if these binary ports start getting to any size.
>
> >
> > MacPorts base would still try to download the nonexistent package.
> MacPorts base currently does not use license values except to display to
> the user and has no idea if a port is distributable. Distributability is
> handled by a separate script. It should be integrated into base so that
> base can know if something could have been distributed, and if it could not
> have, then it shouldn't even try to download it.
> https://trac.macports.org/ticket/60315
> >
>
> Yes. That leverages our existing tools in a nice way. Or a “cask”
> PortGroup could set “archive_sites” to “” I guess, perhaps more easily.
>
> > We might also want to modify the MacPorts base command line "-b" binary
> only mode to allow installing these "nopackage" ports rather than error out
> as it would currently do.
> >
> > If the few steps like disabling the configure and build phases and
> adding the hypothetical "nopackage" to the license line are too much work
> for a portfile submitter to do, a portgroup could be created that does
> those things. But you have often complained about "magic" portgroups doing
> things you didn't know about or didn't expect, so there is something to be
> said for ports doing what they need to do explicitly, when it's not so many
> steps, and when it's not always the same steps needing to be done. For
> example, surely each port will still need to specify what the archs of the
> binary files are, what license they're under, what type of distfile it is
> and where it is, and which files inside needs to be copied where.
> >
>
> It’s not that it’s too much work. It’s that these things are very simple,
> and people submit them but don’t know all these details you mention.
>
> And —
>
> this won’t work for larger ports too well (see SageMath message for an
> extreme example)
> we don’t want to run rev-upgrade on them (surprises, surprises, surprises
> — eg SageMath again)
> we want to be able to run a pkg-installer into some destination…. and
> nobody except three people know how to do that right
> we want to be able to uninstall all the cruft the software installs in
> ~/Library, ~/Preferences, etc easily
>
> So — it’s doable. What we do now is — meh.
>
> IF we are going to do it, we should do it right.
>
> And — we STILL have no naming scheme, so a user will have NO IDEA if he’s
> downloading 

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

true, without ever trying it, you couldn't know

Re: "cask" ports just keep on rolling in...

[Ryan Schmidt]

On Feb 6, 2021, at 09:17, Ken Cunningham wrote:

> those are the small things.
> 
> I stopped when the broad strokes, as it is now, were a unworkable disaster.

I don't understand which aspects are an unworkable disaster. As I said, based 
on what you've said so far, it seems like everything should work fine the way 
it is now, albeit in a possibly non-optimal way due to some excessive disk use 
during install.

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

those are the small things.

I stopped when the broad strokes, as it is now, were a unworkable disaster.

K

Re: "cask" ports just keep on rolling in...

[Ryan Schmidt]

On Feb 6, 2021, at 04:15, Ken Cunningham wrote:

> try the sagemath one if you like. It's all set, for a newish system at 
> present:
> 
> https://github.com/kencu/myports/blob/master/binary/sagemath-binary/Portfile
> 
> It is a stress test, for sure -- and I fully admit you may know simple tweaks 
> to make it much more efficient.

I did see that portfile in the previous discussion. I don't feel a need to try 
it myself. The portfile doesn't look overly more complicated than most ports.

Some tweaks could of course be made to the portfile.

To save some time and disk space usage, the app could be moved to the destroot 
rather than copied.

Something should be done to ensure that the port errors out on earlier OS 
versions on which it won't work. This is not a binary-port-specific issue; 
we've long wanted MacPorts base to be improved to handle this better.

If the binary is x86_64 only supported_archs should reflect that. If the binary 
is arm64/x86_64 universal then the universal variant should be enabled and for 
nonuninversal builds the "other" arch should be removed from all files with 
lipo. The libxl port gives an example. This could be tedious if there are many 
files, and could be a candidate for including in a hypothetical portgroup.

You're currently setting version conditionally. version is a required keyword; 
it must be set in all cases or portindex will probably fail.

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

try the sagemath one if you like. It's all set, for a newish system at present:

https://github.com/kencu/myports/blob/master/binary/sagemath-binary/Portfile

It is a stress test, for sure -- and I fully admit you may know simple tweaks 
to make it much more efficient.

Ken

Re: "cask" ports just keep on rolling in...

[Ryan Schmidt]

On Feb 6, 2021, at 02:28, Ken Cunningham wrote:

>>> This kind of port just repackages the DMG into many tgz archives.
>>> 
>>> It’s wasteful. People want them.
>>> 
>>> What we should have instead of this is some kinda tech that
>>> 
>>> 1. downloads the DMG
>>> 2. installs the app
>>> 3. records some way of uninstalling everything
>> 
>> MacPorts already does all these things... The submitted Portfile works fine, 
>> presumably. There's no need to reinvent the existing MacPorts functionality 
>> that does all these things.
> 
> Well, IMHO there is, but I’ve looked at it quite a bit.
> 
> Look back a month or so and see my post about a “cask” port for SageMath for 
> an idea of why this won’t work (well) in general.

I had not seen that part of the previous discussion because I stopped following 
it, but going back and searching for "sagemath" in those messages now, I still 
don't see any reason why what MacPorts can already do would not work for these 
types of ports.

In that example, the sagemath dmg was 1.5GB and the installed size was 4GB. 
This is a stupid size for a software package to be, but that's up to its 
developers. Hopefully there are not more than one or two prospective ports that 
are so exorbitantly large, so I would not let this extreme example dictate what 
we do.

The complaint was that MacPorts extracts that 4GB into the work directory, then 
compresses the contents of the destroot into a package, then activates the 
package, using unnecessary time and disk space. Let's start by acknowledging 
that this works just fine, so it does not need to be changed. It is true that 
this takes a little time and temporarily disk space, but as you know the reason 
why MacPorts does this is so that the user can deactivate and later re-activate 
the port as needed.

It was proposed there that MacPorts should instead somehow use the original dmg 
distfile as the archive, skipping the normal extract, patch, configure, build, 
destroot, install (create package) and activate (extract package) steps. Based 
on what I've read so far, that doesn't sound like a great idea, since you'd be 
replacing a large amount of already-working MacPorts base code with new code 
that you'd have to test.

You're either requiring the user to keep that distfile around in order to be 
able to reactivate the port (which would be highly unexpected), or you have 
MacPorts copy the distfile into the software directory, which could be done, 
but then the port has to do all of that stuff during the activate phase, has to 
contain code that knows what files to pull off the dmg and where to put them at 
activate time, losing the safety net that destroot provides, making development 
of the port more difficult. When developing a port you want to be able to "sudo 
port destroot" and then check the contents of destroot to make sure everything 
is correct, and you want to be able to "sudo port clean" to delete it all with 
the assurance that nothing outside of the work directory was touched. We 
already have that today; I don't see how the proposed new mechanism would allow 
that to continue to be the case.

A point was raised that the original dmg is compressed already, therefore 
keeping and using it was desirable, but I have to point out that dmgs can be 
uncompressed, or could be compressed with less-efficient methods than our bz2 
(though, granted, these days most of them probably are compressed, some 
probably with more-efficient methods like lzma). There's also the possibility 
that the original dmg contains additional files that we don't care about, so 
requiring the user keep the original dmg around might be requiring them to keep 
unneeded files around.

Another suggestion was raised there of preventing distfiles from being 
mirrored. I see no reason why that should be tied to this type of port. On the 
contrary, if we don't provide packages, then we absolutely want to mirror 
distfiles (license permitting, of course) for all the usual reasons, such as in 
case the original site is offline. With most ports, most users use our packages 
and it doesn't matter if the distfile is inaccessible. But with these ports, if 
we are talking about not providing our own packages and forcing all users to 
use the original distfile, then it very much matters if the distfile is 
inaccessible.


> It’s not that it’s too much work. It’s that these things are very simple, and 
> people submit them but don’t know all these details you mention.

They won't know about the hypothetical new portgroup either unless they read 
about it or we tell them about it. So it comes down to a matter of keeping our 
documentation current and readable, and I'm the first to admit that our 
documentation is not in the best shape that it could be in.


> this won’t work for larger ports too well (see SageMath message for an 
> extreme example)

> we don’t want to run rev-upgrade on them (surprises, surprises, surprises — 
> eg SageMath again)

This might be a valid 

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

> On Feb 5, 2021, at 10:00 PM, Ryan Schmidt  wrote:
> 
> 
> 
> On Feb 5, 2021, at 13:00, Ken Cunningham wrote:
> 
>> With no plan, we’ll just keep getting more and more of these.
> 
> I'm not aware of a huge influx of these, but I'm also not really paying 
> attention to the PR queue. And I'm not intending to get drawn into this 
> discussion of binary ports again.

The last discussion didn’t get anywhere past the appropriate naming scheme :> 
We never even got into implementation details.

>> This kind of port just repackages the DMG into many tgz archives.
>> 
>> It’s wasteful. People want them.
>> 
>> What we should have instead of this is some kinda tech that
>> 
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
> 
> MacPorts already does all these things... The submitted Portfile works fine, 
> presumably. There's no need to reinvent the existing MacPorts functionality 
> that does all these things.
> 

Well, IMHO there is, but I’ve looked at it quite a bit.

Look back a month or so and see my post about a “cask” port for SageMath for an 
idea of why this won’t work (well) in general.

For trivially simple ports, a few MB or less for a *.app that copies into 
/Applications/MacPorts — yeah, who cares?

> 
>> What we have instead is a repackaging of the DMG into many, identical, 
>> system-specific archive bundles.
>> 
>> Yuk.
> 
> If your objection is that we waste server space storing several copies of the 
> same thing, then we could invent a new way for a portfile to indicate that it 
> does not want binary archives stored on the server. It could be a new 
> separate keyword or a new pseudo license type, like we already have for 
> "nomirror" which tells the buildbot not to mirror the distfiles.
> A port like the one we're talking about in the above PR could set e.g. 
> "license GPL-2 nopackage", and buildbot could be modified to understand that 
> this means that it should not upload the packages.

Yes, that would help, if these binary ports start getting to any size.

> 
> MacPorts base would still try to download the nonexistent package. MacPorts 
> base currently does not use license values except to display to the user and 
> has no idea if a port is distributable. Distributability is handled by a 
> separate script. It should be integrated into base so that base can know if 
> something could have been distributed, and if it could not have, then it 
> shouldn't even try to download it. https://trac.macports.org/ticket/60315
> 

Yes. That leverages our existing tools in a nice way. Or a “cask” PortGroup 
could set “archive_sites” to “” I guess, perhaps more easily.

> We might also want to modify the MacPorts base command line "-b" binary only 
> mode to allow installing these "nopackage" ports rather than error out as it 
> would currently do.
> 
> If the few steps like disabling the configure and build phases and adding the 
> hypothetical "nopackage" to the license line are too much work for a portfile 
> submitter to do, a portgroup could be created that does those things. But you 
> have often complained about "magic" portgroups doing things you didn't know 
> about or didn't expect, so there is something to be said for ports doing what 
> they need to do explicitly, when it's not so many steps, and when it's not 
> always the same steps needing to be done. For example, surely each port will 
> still need to specify what the archs of the binary files are, what license 
> they're under, what type of distfile it is and where it is, and which files 
> inside needs to be copied where.
> 

It’s not that it’s too much work. It’s that these things are very simple, and 
people submit them but don’t know all these details you mention.

And —

this won’t work for larger ports too well (see SageMath message for an extreme 
example)
we don’t want to run rev-upgrade on them (surprises, surprises, surprises — eg 
SageMath again)
we want to be able to run a pkg-installer into some destination…. and nobody 
except three people know how to do that right
we want to be able to uninstall all the cruft the software installs in 
~/Library, ~/Preferences, etc easily

So — it’s doable. What we do now is — meh. 

IF we are going to do it, we should do it right.

And — we STILL have no naming scheme, so a user will have NO IDEA if he’s 
downloading an app from some website on the internet rather than something 
build by MacPorts.

And I think we should have a good way of identifying them, whatever it is. And 
yes, I still think identifying them by using a “+prebuilt” variant name is not 
the way to do it, if for no other reason than that might propagate down through 
all kinds of ports as “+prebuilt” and nobody wants that, it carries over from 
installation to upgrade and nobody wants that, and it is not (IMHO) logical to 
have something specified as “-prebuilt” if there is no “-prebuilt” and on and 
on and on for why I think this is just not clear enough …

but 

Re: "cask" ports just keep on rolling in...

[Ryan Schmidt]

On Feb 5, 2021, at 13:00, Ken Cunningham wrote:

> With no plan, we’ll just keep getting more and more of these.

I'm not aware of a huge influx of these, but I'm also not really paying 
attention to the PR queue. And I'm not intending to get drawn into this 
discussion of binary ports again.


> 
> 
> This kind of port just repackages the DMG into many tgz archives.
> 
> It’s wasteful. People want them.
> 
> What we should have instead of this is some kinda tech that
> 
> 1. downloads the DMG
> 2. installs the app
> 3. records some way of uninstalling everything

MacPorts already does all these things... The submitted Portfile works fine, 
presumably. There's no need to reinvent the existing MacPorts functionality 
that does all these things.


> What we have instead is a repackaging of the DMG into many, identical, 
> system-specific archive bundles.
> 
> Yuk.

If your objection is that we waste server space storing several copies of the 
same thing, then we could invent a new way for a portfile to indicate that it 
does not want binary archives stored on the server. It could be a new separate 
keyword or a new pseudo license type, like we already have for "nomirror" which 
tells the buildbot not to mirror the distfiles.

A port like the one we're talking about in the above PR could set e.g. "license 
GPL-2 nopackage", and buildbot could be modified to understand that this means 
that it should not upload the packages.

MacPorts base would still try to download the nonexistent package. MacPorts 
base currently does not use license values except to display to the user and 
has no idea if a port is distributable. Distributability is handled by a 
separate script. It should be integrated into base so that base can know if 
something could have been distributed, and if it could not have, then it 
shouldn't even try to download it. https://trac.macports.org/ticket/60315

We might also want to modify the MacPorts base command line "-b" binary only 
mode to allow installing these "nopackage" ports rather than error out as it 
would currently do.

If the few steps like disabling the configure and build phases and adding the 
hypothetical "nopackage" to the license line are too much work for a portfile 
submitter to do, a portgroup could be created that does those things. But you 
have often complained about "magic" portgroups doing things you didn't know 
about or didn't expect, so there is something to be said for ports doing what 
they need to do explicitly, when it's not so many steps, and when it's not 
always the same steps needing to be done. For example, surely each port will 
still need to specify what the archs of the binary files are, what license 
they're under, what type of distfile it is and where it is, and which files 
inside needs to be copied where.

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

The idea of MacPorts is to manage installing inter-related libraries and 
executables.

But people want to use it to install prebuilt binaries as well, like homebrew 
does for cask installs.

MP has not decided if it will or will not accept these, but they just keep 
rolling in more and more anyway.

Installing them as "ports" is kinda silly, and wasteful, but simple, practical, 
and slides them in under the radar.

Having an actual plan for these would be better.

If software _can_ be built, we want to do it that way, so (IMHO) your work is 
not wasted. 

We get many benefits, I believe, from building things ourselves.

Whether we do or don't accept binary "cask" installs ins not up to me -- but 
I'm just point out that they are coming in anyway so a plan would be good, esp 
for the multi-megabyte behemoths that are out there.


Ken




On 2021-02-05, at 2:58 PM, Jason Liu wrote:

> The destroot phase needs to — what — record the files that will be installed 
> in some kind of an index file? And also know about the stuff that gets 
> installed into ~/Library, etc.
> 
> You would probably need to get the list of files installed by the installer 
> by running either pkgutil --files or lsbom, if we're talking about a PKG 
> installer. (Or maybe it would be easier to just grab the bom file in its 
> entirety.) Typically running an ls -R on the .app bundle inside a DMG 
> installer would be sufficient for those kinds of installers.
> 
> But then this begs the question: what happens to all of the work that went 
> into the build-from-source packages? Wouldn't this end up rendering the 
> hundreds of hours I spent getting the Blender package to work a complete 
> waste?
> 
> -- 
> Jason Liu
> 
> 
> On Fri, Feb 5, 2021 at 5:31 PM Ken Cunningham 
> mailto:ken.cunningham.web...@gmail.com>> 
> wrote:
> What would we really need to do this properly?
> 
> The current fetch, checksum phases are OK.
> 
> The extract phase can be used as is for some software (simple copies), but 
> for other software we don’t want to extract it, we want to run the installer.
> 
> The configure and build phases need to be overridden and disappear.
> 
> The destroot phase needs to — what — record the files that will be installed 
> in some kind of an index file? And also know about the stuff that gets 
> installed into ~/Library, etc.
> 
> Then run the “cask” destroot without installing any files into the actual 
> destroot:
> 
> copy the apps and stuff
> or 
> run the installer
> or
> extract from the pkg
> or 
> … 
> 
> and then NOT have the entire software repackaged into a MP archive file to be 
> stored 12 different times…
> 
> Or some such plan
> 
> Best,
> 
> Ken
> 
> 
> 
> 
> 
>> On Feb 5, 2021, at 11:00 AM, Ken Cunningham > > wrote:
>> 
>> With no plan, we’ll just keep getting more and more of these.
>> 
>> > >
>> 
>> This kind of port just repackages the DMG into many tgz archives.
>> 
>> It’s wasteful. People want them.
>> 
>> What we should have instead of this is some kinda tech that
>> 
>> 1. downloads the DMG
>> 2. installs the app
>> 3. records some way of uninstalling everything
>> 
>> 
>> What we have instead is a repackaging of the DMG into many, identical, 
>> system-specific archive bundles.
>> 
>> Yuk.
>> 
>> Ken
>> 
> 

Re: "cask" ports just keep on rolling in...

[Jason Liu]

>
> The destroot phase needs to — what — record the files that will be
> installed in some kind of an index file? And also know about the stuff that
> gets installed into ~/Library, etc.
>

You would probably need to get the list of files installed by the installer
by running either pkgutil --files or lsbom, if we're talking about a PKG
installer. (Or maybe it would be easier to just grab the bom file in its
entirety.) Typically running an ls -R on the .app bundle inside a DMG
installer would be sufficient for those kinds of installers.

But then this begs the question: what happens to all of the work that went
into the build-from-source packages? Wouldn't this end up rendering the
hundreds of hours I spent getting the Blender package to work a complete
waste?

-- 
Jason Liu


On Fri, Feb 5, 2021 at 5:31 PM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

> What would we really need to do this properly?
>
> The current fetch, checksum phases are OK.
>
> The extract phase can be used as is for some software (simple copies), but
> for other software we don’t want to extract it, we want to run the
> installer.
>
> The configure and build phases need to be overridden and disappear.
>
> The destroot phase needs to — what — record the files that will be
> installed in some kind of an index file? And also know about the stuff that
> gets installed into ~/Library, etc.
>
> Then run the “cask” destroot without installing any files into the actual
> destroot:
>
> copy the apps and stuff
> or
> run the installer
> or
> extract from the pkg
> or
> …
>
> and then NOT have the entire software repackaged into a MP archive file to
> be stored 12 different times…
>
> Or some such plan
>
> Best,
>
> Ken
>
>
>
>
>
> On Feb 5, 2021, at 11:00 AM, Ken Cunningham <
> ken.cunningham.web...@gmail.com> wrote:
>
> With no plan, we’ll just keep getting more and more of these.
>
> 
>
> This kind of port just repackages the DMG into many tgz archives.
>
> It’s wasteful. People want them.
>
> What we should have instead of this is some kinda tech that
>
> 1. downloads the DMG
> 2. installs the app
> 3. records some way of uninstalling everything
>
>
> What we have instead is a repackaging of the DMG into many, identical,
> system-specific archive bundles.
>
> Yuk.
>
> Ken
>
>
>

Re: "cask" ports just keep on rolling in...

[Ken Cunningham]

What would we really need to do this properly?

The current fetch, checksum phases are OK.

The extract phase can be used as is for some software (simple copies), but for 
other software we don’t want to extract it, we want to run the installer.

The configure and build phases need to be overridden and disappear.

The destroot phase needs to — what — record the files that will be installed in 
some kind of an index file? And also know about the stuff that gets installed 
into ~/Library, etc.

Then run the “cask” destroot without installing any files into the actual 
destroot:

copy the apps and stuff
or 
run the installer
or
extract from the pkg
or 
… 

and then NOT have the entire software repackaged into a MP archive file to be 
stored 12 different times…

Or some such plan

Best,

Ken





> On Feb 5, 2021, at 11:00 AM, Ken Cunningham  
> wrote:
> 
> With no plan, we’ll just keep getting more and more of these.
> 
>  >
> 
> This kind of port just repackages the DMG into many tgz archives.
> 
> It’s wasteful. People want them.
> 
> What we should have instead of this is some kinda tech that
> 
> 1. downloads the DMG
> 2. installs the app
> 3. records some way of uninstalling everything
> 
> 
> What we have instead is a repackaging of the DMG into many, identical, 
> system-specific archive bundles.
> 
> Yuk.
> 
> Ken
> 

Re: "cask" ports just keep on rolling in...

[Blake Garner]

I also think this would be highly desirable. Would it be another portgroup
that needs to be written in tcl?

On Fri, Feb 5, 2021 at 11:01 AM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

> With no plan, we’ll just keep getting more and more of these.
>
> 
>
> This kind of port just repackages the DMG into many tgz archives.
>
> It’s wasteful. People want them.
>
> What we should have instead of this is some kinda tech that
>
> 1. downloads the DMG
> 2. installs the app
> 3. records some way of uninstalling everything
>
>
> What we have instead is a repackaging of the DMG into many, identical,
> system-specific archive bundles.
>
> Yuk.
>
> Ken
>
>

"cask" ports just keep on rolling in...

[Ken Cunningham]

With no plan, we’ll just keep getting more and more of these.

>

This kind of port just repackages the DMG into many tgz archives.

It’s wasteful. People want them.

What we should have instead of this is some kinda tech that

1. downloads the DMG
2. installs the app
3. records some way of uninstalling everything


What we have instead is a repackaging of the DMG into many, identical, 
system-specific archive bundles.

Yuk.

Ken