Re: Apple ARM binary codesign issue
As far as I understand, ad-hoc codesigning is not actually really meant to protect a file on disk because you can just ad-hoc sign again when you modify the file; instead it simplifies some of Apple’s own code because it removes the special case of a binary that doesn’t have a signature (which until now has had a number of quirks and extra checks throughout the operating system). A more cynical interpretation would be that Apple would like to flip the switch to “paid developer account-signed software only” at some point in the future, but every engineer has denied that this is the goal when asked so I guess that if this will happen it hopefully won’t be anytime soon. I am still unsure why ld adds a signature but strip and install_name_tool don’t reapply an ad-hoc signature to a signed binary that they modify. This might be worth filing a feedback for. > On Sep 22, 2020, at 15:24, Ken Cunningham > wrote: > > > On 2020-09-22, at 12:58 PM, Ryan Schmidt wrote: >> >> To me it seems unrealistic for Apple to suggest that an infinite number of >> open source projects, many of whose developers have never seen a Mac, should >> now add code to their build systems to codesign things on macOS. Apple made >> a point of stating during WWDC that they love open source software; imposing >> busy work on the open source community is not a good way to show that love. > > As I read it, the linker automatically codesigns the binary when you link, > which is usually the final step in the process. So nobody has to change > anything there. > > But if you later modify that final linked binary by stripping it (I guess ) > or changing the libraries around with install_name_tool (which I believe > MESON does to every single install :>) then you invalidate the signature, as > you should IMHO. > > I'm trying to imagine how Jeremy might prevent hackers from surreptitiously > modifying signed binaries with strip or install_name_tool (which is good) > while letting people modify signed binaries with strip or install_name_tool > without invalidating the signature -- I don't immediately see how you can > have it both ways. But maybe Jeremy has some trick that works for this I > can't think of. > > I won't be surprised if the solution is that you have to resign them after > doing that, though. > > Ken >
Re: Apple ARM binary codesign issue
On 2020-09-22, at 12:58 PM, Ryan Schmidt wrote: > > To me it seems unrealistic for Apple to suggest that an infinite number of > open source projects, many of whose developers have never seen a Mac, should > now add code to their build systems to codesign things on macOS. Apple made a > point of stating during WWDC that they love open source software; imposing > busy work on the open source community is not a good way to show that love. As I read it, the linker automatically codesigns the binary when you link, which is usually the final step in the process. So nobody has to change anything there. But if you later modify that final linked binary by stripping it (I guess ) or changing the libraries around with install_name_tool (which I believe MESON does to every single install :>) then you invalidate the signature, as you should IMHO. I'm trying to imagine how Jeremy might prevent hackers from surreptitiously modifying signed binaries with strip or install_name_tool (which is good) while letting people modify signed binaries with strip or install_name_tool without invalidating the signature -- I don't immediately see how you can have it both ways. But maybe Jeremy has some trick that works for this I can't think of. I won't be surprised if the solution is that you have to resign them after doing that, though. Ken
Re: Apple ARM binary codesign issue
> > I've read that number of duplicates is one of the ways they determine > issue priority internally. > I can confirm that this is indeed supposed to be the case. Our Apple engineering liaison at the university where I used to work as a sysadmin frequently repeated this during our Apple developer meetings. These Apple developer meetings were where the IT staff from various departments around campus gathered once a month to discuss issues with the latest betas and issues with software apps being used by faculty, staff, and students. One of the main purposes of these meetings were to bring up potential bugs in the betas, and to have as many of the meeting attendees go back and try to duplicate the bug, and then report it. In fact, we even passed around the titles of the bug reports to the mailing list, so that it could get recorded as a duplicate of the same issue. Even better, if the original person's bug report was lucky enough to get a bug report number in Apple's Radar bug tracker, the Apple liaison would look it up and pass it along to everyone at the meeting, so that we could reference the original bug report (again, to increase the duplicate count). -- Jason Liu On Tue, Sep 22, 2020 at 4:17 PM Joshua Root wrote: > On 2020-9-23 05:33 , Ryan Schmidt wrote: > > > > Send feedback through the Feedback Assistant app. > > Yes, everyone with any issues with Apple preview software should do this > early and often. I've read that number of duplicates is one of the ways > they determine issue priority internally. > > - Josh >
Re: Apple ARM binary codesign issue
On 2020-9-23 05:33 , Ryan Schmidt wrote: > > Send feedback through the Feedback Assistant app. Yes, everyone with any issues with Apple preview software should do this early and often. I've read that number of duplicates is one of the ways they determine issue priority internally. - Josh
Re: Apple ARM binary codesign issue
On Sep 22, 2020, at 14:52, Ken Cunningham wrote: > On 2020-09-22, at 11:58 AM, Ryan Schmidt wrote: >> >> I hope that Apple fixes their toolchain to work without such intervention. > > I believe this may ultimately come under the category of "intended > behaviour". To me it seems unrealistic for Apple to suggest that an infinite number of open source projects, many of whose developers have never seen a Mac, should now add code to their build systems to codesign things on macOS. Apple made a point of stating during WWDC that they love open source software; imposing busy work on the open source community is not a good way to show that love. Part of the problem is that I don't understand what codesigning is for. I understand that for distribution in the app store I would codesign an app with my identity. Apple can then revoke apps I've signed if it turns out I've released malicious software. Great, but here we're not talking about signing with an identity, we're talking about ad-hoc signing. All that proves is that the binary hasn't been modified. That's good; it'll protect against corrupted files getting used. But how would it stop malicious code? Couldn't the malicious party just codesign it again after changing it?
Re: Apple ARM binary codesign issue
On 2020-09-22, at 11:58 AM, Ryan Schmidt wrote: > > I hope that Apple fixes their toolchain to work without such intervention. > I believe this may ultimately come under the category of "intended behaviour". I can understand why they would want these changes to invalidate the signature. I suspect we'll have to find a way to re-sign them after making such changes. In addition, we may not find that automatically doing it by wrapping our strip or install_name_tool with codesigners is suitably secure. Ken
Re: Apple ARM binary codesign issue
On Sep 22, 2020, at 14:19, Michael Dickens wrote: > I have macOS 11.0beta7 installed : check! > > Compare / contrast ARM Mac versus MacBook Pro 16 : check! > > I have Xcode 12.2 beta installed : check! > > I've removed "/Library/Developer/CommandLineTools" : check! > > I hope that Apple fixes their toolchain to work without such intervention : > check! > > Do you know that best way we can complain to Apple to fix the toolchain? > > Still doesn't work for me ... those specific files are almost the only ones > that just don't respond to codesign. I'll try reinstalling from scratch ... > maybe ports built with Xcode 12.0 don't entirely work with those built using > 12.2beta? H I was getting the "codesign_allocate helper tool cannot be found or used" error earlier too; the only thing I think I changed since then is getting rid of the Xcode 12.0 beta CLT. Though I do have a couple different MacPorts installs and I was working on the other one before. Send feedback through the Feedback Assistant app. They have responded to one of mine recently.
Re: Apple ARM binary codesign issue
I have macOS 11.0beta7 installed : check! Compare / contrast ARM Mac versus MacBook Pro 16 : check! I have Xcode 12.2 beta installed : check! I've removed "/Library/Developer/CommandLineTools" : check! I hope that Apple fixes their toolchain to work without such intervention : check! Do you know that best way we can complain to Apple to fix the toolchain? Still doesn't work for me ... those specific files are almost the only ones that just don't respond to codesign. I'll try reinstalling from scratch ... maybe ports built with Xcode 12.0 don't entirely work with those built using 12.2beta? H Thx! - MLD On Tue, Sep 22, 2020, at 2:58 PM, Ryan Schmidt wrote: > > > On Sep 22, 2020, at 13:29, Michael Dickens wrote: > > > % codesign -v - --ignore-resources > > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > > invalid signature (code or signature have been modified) > > % sudo codesign -s - > > --preserve-metadata=identifier,entitlements,flags,runtime -f > > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > > replacing existing signature > > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > > the codesign_allocate helper tool cannot be found or used > > % which codesign > > /usr/bin/codesign > > % which codesign_allocate > > /usr/bin/codesign_allocate > > > > You need Xcode 12.2 beta, which you probably have, but also make sure > that you don't have old command line tools installed. I don't think the > Xcode 12.0 beta command line tools are compatible, and there isn't an > Xcode 12.2 beta command line tools yet. Delete > /Library/Developer/CommandLineTools. > > > % codesign -v - > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > invalid signature (code or signature have been modified) > In architecture: arm64 > % codesign -s - > --preserve-metadata=identifier,entitlements,flags,runtime -f > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > replacing existing signature > ... > % codesign -v - > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > valid on disk > mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > satisfies its Designated Requirement > % > > > > Mentioned as possible fixes were: (1) inserting MP strip and > > install_name_tool wrappers that sign the binary if the signature is broken; > > or (2) a new step in destroot_finish . > > I hope that Apple fixes their toolchain to work without such intervention. > >
Re: Apple ARM binary codesign issue
On Sep 22, 2020, at 13:29, Michael Dickens wrote: > % codesign -v - --ignore-resources > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > invalid signature (code or signature have been modified) > % sudo codesign -s - > --preserve-metadata=identifier,entitlements,flags,runtime -f > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > replacing existing signature > /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: > the codesign_allocate helper tool cannot be found or used > % which codesign > /usr/bin/codesign > % which codesign_allocate > /usr/bin/codesign_allocate > You need Xcode 12.2 beta, which you probably have, but also make sure that you don't have old command line tools installed. I don't think the Xcode 12.0 beta command line tools are compatible, and there isn't an Xcode 12.2 beta command line tools yet. Delete /Library/Developer/CommandLineTools. % codesign -v - mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: invalid signature (code or signature have been modified) In architecture: arm64 % codesign -s - --preserve-metadata=identifier,entitlements,flags,runtime -f mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: replacing existing signature ... % codesign -v - mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: valid on disk mp/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: satisfies its Designated Requirement % > Mentioned as possible fixes were: (1) inserting MP strip and > install_name_tool wrappers that sign the binary if the signature is broken; > or (2) a new step in destroot_finish . I hope that Apple fixes their toolchain to work without such intervention.
Apple ARM binary codesign issue
There has been some discussion about the recent change Apple made for macOS
11.0beta7 for ARM Mac only (-not- Intel Mac at this time); we in MP-land had
some on this PR < https://github.com/macports/macports-ports/pull/8328 >. As
pointed out, a better venue for discussion would be these lists.
The brief summary is that Apple is instituting code signing for all binaries
during linking as noted here <
https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11-universal-apps-beta-release-notes#Updates-in-macOS-Big-Sur-11-Universal-Apps-Beta-7
>. The signing is as-hoc and done automatically; it is invalidated if one
modifies the binary in any way ... for example using "strip", or
"install_name_tool" to change the name of a required library, or even the
self-ID name.
Many MacPorts installs use various post-linking means to tweak the resulting
binary, and hence these are not currently usable under macOS 11.0beta7 or more
recent on ARM Mac only; again: this change is NOT for Intel Mac -- at least not
at this time.
One can test whether the signing is valid via the command: "codesign -v FILE" ;
according to Apple one should use "codesign -v - --ignore-resources FILE"
to be more verbose as well as look at just the non-resource binary. Both work
in my testing.
In my initial testing, some of the binaries can be made to work via the command
"[sudo] codesign -s - --preserve-metadata=identifier,entitlements,flags,runtime
-f FILE" ... but, not all! For example:
{{{
% codesign -v - --ignore-resources
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8:
invalid signature (code or signature have been modified)
% sudo codesign -s - --preserve-metadata=identifier,entitlements,flags,runtime
-f /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8:
replacing existing signature
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8: the
codesign_allocate helper tool cannot be found or used
% which codesign
/usr/bin/codesign
% which codesign_allocate
/usr/bin/codesign_allocate
}}}
So in this case, "codesign_allocate" cannot be used for some non-obvious reason
(as it clearly is in the PATH and works for some other codesign efforts). The
same holds true for some of the Python libraries and some of the primary
executables -- but, not all!
Mentioned as possible fixes were: (1) inserting MP strip and install_name_tool
wrappers that sign the binary if the signature is broken; or (2) a new step in
destroot_finish .
Let's pick up the discussion here, and try to work out a fix in short order.
For those of us trying to do anything within MacPorts using ARM Mac, this new
feature is causing significant issues & needs to be dealt with so that we can
get back to real work instead of fighting macOS::codesign .
Cheers! - MLD
Re: Thoughts on switching our archive compression method
On Sep 22, 2020, at 13:28, Vincent Habchi wrote: >> (xz is 85% of bz2 size) >> (xz is 70% of bz2 size) >> (xz is 57% of bz2 size) >> >> So I think we could save ourselves and our mirror providers, CDN, and users >> some disk space and bandwidth by switching to xz. bz2 was the best available >> built-in compression on Mac OS X 10.6 when we started doing binary archives >> but there are better options now. > > I agree wholeheartedly, but since you mention modern compression algorithms, > isn’t there another one (or more) which would be yet more efficient than xz? Maybe, but if we could use something built into macOS, like xz on 10.9 and later, then we wouldn't need to bundle yet another third-party project (the decompression program) into MacPorts base.
Re: Thoughts on switching our archive compression method
> (xz is 85% of bz2 size) > (xz is 70% of bz2 size) > (xz is 57% of bz2 size) > > So I think we could save ourselves and our mirror providers, CDN, and users > some disk space and bandwidth by switching to xz. bz2 was the best available > built-in compression on Mac OS X 10.6 when we started doing binary archives > but there are better options now. I agree wholeheartedly, but since you mention modern compression algorithms, isn’t there another one (or more) which would be yet more efficient than xz? Vincent
Re: Thoughts on switching our archive compression method
On Sep 22, 2020, at 07:58, Vincent wrote: >> It would be nice if we could easily switch our precompiled archives from >> bzip2-compressed tarballs (tbz2) to better compression methods as they >> become available. For example, xz-compressed tarballs (txz) would be better >> today. OS X 10.9 Mavericks and later has built-in support for xz compression >> (https://trac.macports.org/ticket/56237) so we could use that to avoid >> needing to use the xz port or needing to bundle a copy of xz >> (https://trac.macports.org/ticket/52000). > > How much is gained by switching from tbz2 to txz? It varies of course but here are a few examples: $ bzcat /opt/local/var/macports/software/texlive-latex-extra/texlive-latex-extra-54584_0+doc.darwin_17.noarch.tbz2 | xz -9 > texlive-latex-extra-54584_0+doc.darwin_17.noarch.txz $ ls -l /opt/local/var/macports/software/texlive-latex-extra/texlive-latex-extra-54584_0+doc.darwin_17.noarch.tbz2 texlive-latex-extra-54584_0+doc.darwin_17.noarch.txz -rw-r--r-- 1 macports wheel 565981279 Aug 23 23:16 /opt/local/var/macports/software/texlive-latex-extra/texlive-latex-extra-54584_0+doc.darwin_17.noarch.tbz2 -rw-r--r-- 1 rschmidt wheel 481850300 Sep 22 08:56 texlive-latex-extra-54584_0+doc.darwin_17.noarch.txz (xz is 85% of bz2 size) $ bzcat /opt/local/var/macports/software/go/go-1.15.2_0.darwin_17.x86_64.tbz2 | xz -9 > go-1.15.2_0.darwin_17.x86_64.txz $ ls -l /opt/local/var/macports/software/go/go-1.15.2_0.darwin_17.x86_64.tbz2 go-1.15.2_0.darwin_17.x86_64.txz -rw-r--r-- 1 macports wheel 124756642 Sep 14 09:09 /opt/local/var/macports/software/go/go-1.15.2_0.darwin_17.x86_64.tbz2 -rw-r--r-- 1 rschmidt wheel 86893084 Sep 22 09:32 go-1.15.2_0.darwin_17.x86_64.txz (xz is 70% of bz2 size) $ bzcat /opt/local/var/macports/software/python38/python38-3.8.5_1+universal.darwin_17.i386-x86_64.tbz2 | xz -9 > python38-3.8.5_1+universal.darwin_17.i386-x86_64.txz $ ls -l /opt/local/var/macports/software/python38/python38-3.8.5_1+universal.darwin_17.i386-x86_64.tbz2 python38-3.8.5_1+universal.darwin_17.i386-x86_64.txz -rw-r--r-- 1 root wheel 28496603 Sep 6 01:16 /opt/local/var/macports/software/python38/python38-3.8.5_1+universal.darwin_17.i386-x86_64.tbz2 -rw-r--r-- 1 rschmidt wheel 16136624 Sep 22 09:36 python38-3.8.5_1+universal.darwin_17.i386-x86_64.txz (xz is 57% of bz2 size) So I think we could save ourselves and our mirror providers, CDN, and users some disk space and bandwidth by switching to xz. bz2 was the best available built-in compression on Mac OS X 10.6 when we started doing binary archives but there are better options now.
Thoughts on switching our archive compression method
It would be nice if we could easily switch our precompiled archives from bzip2-compressed tarballs (tbz2) to better compression methods as they become available. For example, xz-compressed tarballs (txz) would be better today. OS X 10.9 Mavericks and later has built-in support for xz compression (https://trac.macports.org/ticket/56237) so we could use that to avoid needing to use the xz port or needing to bundle a copy of xz (https://trac.macports.org/ticket/52000). Currently, users have to specify what archive type they want for each server listed in archive_sites.conf. The default archive_sites.conf fortunately has everything commented out, so users get the MacPorts built-in defaults, so we could change the behavior in the future and users would automatically get it. We could remove the compression type setting in archive_sites.conf. (Or there could be a new value of the compression type setting that means to do the following.) And instead of having an rmd160 file for each archive, have an information file which could be encoded in any convenient format such as json or plist or whatever we're using for the PortIndex. MacPorts would fetch the information file first, and it would contain a field stating the compression format (filename extension) for this archive and a field for its rmd160 signature. From this MacPorts can deduce what archive filename to download. It would also give us a place to store any additional information about archives that we might want to store in the future. If that were in place, along with support in base for decompressing xz without needing the xz port, then we could begin compressing new archives for 10.9 and later with xz without needing to recompress the entire collection of existing archives. Over time we can do that as well, of course.
