Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
> On Sep 19, 2016, at 04:18, René J.V. Bertin wrote:
>
> On Monday September 19 2016 03:46:32 Jeremy Huddleston Sequoia wrote:
>
>>> TMPDIR points into /var/folders.
>>
>> Yes, and it looks like those do get cleared out on boot.
>>
>> Note, however, that this particular case relates to the cache dir, not tmp
>> (C, not T).
>
> Oh, right, but there are icon caches in both C and T dirs:
>
> {{{
> %> sudo sh -c "du -hcs /var/folders/*/*/*/com.apple.IconServices"
> 81M/var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices
> 264K
> /var/folders/11/_bf842l54ngf42g66hkzslmwgp/T/com.apple.IconServices
> 82M/var/folders/99/yvy25dd94jn_ykbfkf1q4mzwgw/C/com.apple.IconServices
> 88M/var/folders/gf/wmgp2zv506dcsg7_cjxx69m4gr/C/com.apple.IconServices
> 234M
> /var/folders/j1/1439ppj08xj8h6006s6drbq0gs/C/com.apple.IconServices
> 11M/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/T/com.apple.IconServices
> 80M/var/folders/sd/pwbxpphw8xj3vb006s6drbq0gn/C/com.apple.IconServices
> 264K
> /var/folders/sd/pwbxpphw8xj3vb006s6drbq0gn/T/com.apple.IconServices
> 12K/var/folders/zz/zyxvpxvq6csfxvn_n0/C/com.apple.IconServices
> 0B/var/folders/zz/zyxvpxvq6csfxvn_n0/T/com.apple.IconServices
> 12K/var/folders/zz/zyxvpxvq6csfxvn_n0bh2w/C/com.apple.IconServices
> 0B/var/folders/zz/zyxvpxvq6csfxvn_n0bh2w/T/com.apple.IconServices
> 577Mtotal
> }}}
>
> Those are compressed sizes; uncompressed they take about 5.5Gb .
>
>
>>> How large is your own icon cache directory? And asking here is me
>>> investigating. I wouldn't really know where else to ask, Apple's
>>> support/discussions forum? Doesn't seem like the most appropriate place to
>>> discuss this kind of thing :)
>>
>> 6 MB for me. 1MB each for two others.
>
> Right, looks like something isn't right - or something was changed
> significantly since 10.9.
I suspect a change is at least part of it. On 10.9, I see
com.apple.IconServices that contain ISCacheTOC and *.iscachebmp files. On
10.10, I see com.apple.iconservices (notice the case difference) that contains
a single store.index file, and there is a global
/Library/Caches/com.apple.iconservices.store (which is 28 MB on my system).
> I do have a 277Mb large /opt/local/share/icons, but those shouldn't be picked
> up by Apple's IconServices (only a few icon theme directories have ever been
> opened in the Finder, and only by me, not any of the other users with big
> icon caches.
>
> R.
> ___
> macports-dev mailing list
> macports-dev@lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/macports-dev
smime.p7s
Description: S/MIME cryptographic signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Monday September 19 2016 03:46:32 Jeremy Huddleston Sequoia wrote:
> > TMPDIR points into /var/folders.
>
> Yes, and it looks like those do get cleared out on boot.
>
> Note, however, that this particular case relates to the cache dir, not tmp
> (C, not T).
Oh, right, but there are icon caches in both C and T dirs:
{{{
%> sudo sh -c "du -hcs /var/folders/*/*/*/com.apple.IconServices"
81M/var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices
264K/var/folders/11/_bf842l54ngf42g66hkzslmwgp/T/com.apple.IconServices
82M/var/folders/99/yvy25dd94jn_ykbfkf1q4mzwgw/C/com.apple.IconServices
88M/var/folders/gf/wmgp2zv506dcsg7_cjxx69m4gr/C/com.apple.IconServices
234M/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/C/com.apple.IconServices
11M/var/folders/j1/1439ppj08xj8h6006s6drbq0gs/T/com.apple.IconServices
80M/var/folders/sd/pwbxpphw8xj3vb006s6drbq0gn/C/com.apple.IconServices
264K/var/folders/sd/pwbxpphw8xj3vb006s6drbq0gn/T/com.apple.IconServices
12K/var/folders/zz/zyxvpxvq6csfxvn_n0/C/com.apple.IconServices
0B/var/folders/zz/zyxvpxvq6csfxvn_n0/T/com.apple.IconServices
12K/var/folders/zz/zyxvpxvq6csfxvn_n0bh2w/C/com.apple.IconServices
0B/var/folders/zz/zyxvpxvq6csfxvn_n0bh2w/T/com.apple.IconServices
577Mtotal
}}}
Those are compressed sizes; uncompressed they take about 5.5Gb .
> > How large is your own icon cache directory? And asking here is me
> > investigating. I wouldn't really know where else to ask, Apple's
> > support/discussions forum? Doesn't seem like the most appropriate place to
> > discuss this kind of thing :)
>
> 6 MB for me. 1MB each for two others.
Right, looks like something isn't right - or something was changed
significantly since 10.9.
I do have a 277Mb large /opt/local/share/icons, but those shouldn't be picked
up by Apple's IconServices (only a few icon theme directories have ever been
opened in the Finder, and only by me, not any of the other users with big icon
caches.
R.
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
> On Sep 19, 2016, at 03:31, René J.V. Bertin wrote: > > On Monday September 19 2016 03:02:38 Jeremy Sequoia wrote: > >> /var/tmp definitely isn't cleared though. > > TMPDIR points into /var/folders. Yes, and it looks like those do get cleared out on boot. Note, however, that this particular case relates to the cache dir, not tmp (C, not T). > Here's an example entry from macports's icon cache: > > %> sudo afsctool -vvv > /var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices/FFB06CC518D1C57091BDD19F39A692B6.iscachebmp > > /var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices/FFB06CC518D1C57091BDD19F39A692B6.iscachebmp: > File is HFS+ compressed. > File content type: dyn.age80w65dqfv0u3pcrz2a > File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): > 4194304 bytes / 4.2 MB (megabytes) / 4 MiB (mebibytes) > File size (compressed data fork - decmpfs xattr; reported size by Mac OS > 10.0-10.5 Finder): 112916 bytes / 115 KB (kilobytes) / 112 KiB (kibibytes) > File size (compressed data fork): 112932 bytes / 115 KB (kilobytes) / 112 KiB > (kibibytes) > Compression savings: 97.3% > Number of extended attributes: 0 > Total size of extended attribute data: 0 bytes > Approximate overhead of extended attributes: 536 bytes > Approximate total file size (compressed data fork + EA + EA overhead + file > overhead): 115488 bytes / 115 KB (kilobytes) / 113 KiB (kibibytes) > > There are a number of files of the same size, plus many smaller. I have no > idea how I can map this to an actual icon though. > > >>> Even if keeping track of who's entitled to access which bits of cached icon >>> info the cache agent could prune the cache, removing everything that hasn't >>> been accessed for more than a reasonably long time. >> >> Again, I think you are seeing something anomalous. You should investigate >> before jumping to conclusions. > > How large is your own icon cache directory? And asking here is me > investigating. I wouldn't really know where else to ask, Apple's > support/discussions forum? Doesn't seem like the most appropriate place to > discuss this kind of thing :) 6 MB for me. 1MB each for two others. # du -scm /private/var/folders/*/*/*/com.apple.iconservices 6 /private/var/folders/c1/7p4m5bx904392j49x1w2vyxhgn/C/com.apple.iconservices 1 /private/var/folders/zz/zyxvpxvq6csfxvn_n0/C/com.apple.iconservices 1 /private/var/folders/zz/zyxvpxvq6csfxvn_n0z07r/C/com.apple.iconservices 6 total smime.p7s Description: S/MIME cryptographic signature ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Monday September 19 2016 03:02:38 Jeremy Sequoia wrote: > /var/tmp definitely isn't cleared though. TMPDIR points into /var/folders. Here's an example entry from macports's icon cache: %> sudo afsctool -vvv /var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices/FFB06CC518D1C57091BDD19F39A692B6.iscachebmp /var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices/FFB06CC518D1C57091BDD19F39A692B6.iscachebmp: File is HFS+ compressed. File content type: dyn.age80w65dqfv0u3pcrz2a File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): 4194304 bytes / 4.2 MB (megabytes) / 4 MiB (mebibytes) File size (compressed data fork - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 112916 bytes / 115 KB (kilobytes) / 112 KiB (kibibytes) File size (compressed data fork): 112932 bytes / 115 KB (kilobytes) / 112 KiB (kibibytes) Compression savings: 97.3% Number of extended attributes: 0 Total size of extended attribute data: 0 bytes Approximate overhead of extended attributes: 536 bytes Approximate total file size (compressed data fork + EA + EA overhead + file overhead): 115488 bytes / 115 KB (kilobytes) / 113 KiB (kibibytes) There are a number of files of the same size, plus many smaller. I have no idea how I can map this to an actual icon though. > > Even if keeping track of who's entitled to access which bits of cached icon > > info the cache agent could prune the cache, removing everything that hasn't > > been accessed for more than a reasonably long time. > > Again, I think you are seeing something anomalous. You should investigate > before jumping to conclusions. How large is your own icon cache directory? And asking here is me investigating. I wouldn't really know where else to ask, Apple's support/discussions forum? Doesn't seem like the most appropriate place to discuss this kind of thing :) R ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
Sent from my iPhone... > On Sep 19, 2016, at 02:52, René J.V. Bertin wrote: > > On Monday September 19 2016 02:08:12 Jeremy Huddleston Sequoia wrote: > >>> The icon cache sits in the user's $TMPDIR, and AFAIK that directory is >>> emptied at boot. >> >> No. /tmp is emptied at boot, not /var/tmp and not /var/folders. > > Did that change after 10.9? I keep a Finder window open on $TMPDIR on one of > my "Spaces", and it's always empty after logging in to a new session (which I > only do after rebooting). I could just be wrong. I'll double check tomorrow. /var/tmp definitely isn't cleared though. >> Well, if you're able to update to 10.10+, launchd2 is much much nicer at >> helping triage such things... > > At some point I guess I will... (have to...) > >> There are ACL settings somewhere to control that. It's not based on uid. I >> don't recall what it is off the top of my head. > > OK, great, let me know if anything comes to mind that could help me in a > google search. > >>> , and when? Boot time? Or login, and if so, what kind of login? >> >> The user (background) session is created whenever the user does pretty much >> anything (ssh, cron, sudo -u, screen sharing, etc.). >> The gui (aqua) session is created wheneger the user logs in to the gui >> (screen sharing, console, etc). >> > > And the gui session is never taken down when the user logs of the gui? It is at some point. > Should one think of launchd as the equivalent of Linux's systemd, or does it > also have bits of DBus? Similar in concept I suppose but quite different in practice. I'd take launchd over DBus and systemd any time. > >> iconservicesagent processes? 2, one in the system domain and one in my >> user's aqua session: >> $ ps aux | grep iconservicesagent >> jeremy677 0.0 0.1 2918544 19908 ?? SMon09AM >> 0:08.85 /System/Library/CoreServices/iconservicesagent >> root 92 0.0 0.0 2516172492 ?? Ss Mon09AM >> 0:00.47 /System/Library/CoreServices/iconservicesagent >> > > How large is root's icon cache? Do you have any idea why the agent is running > for root? Because of the login manager, maybe? It's in the system session, not root's user session. That's the LaunchDaemon. >> Yes. That's a clear privacy violation. User A could use that to determine >> what applications User B has installed. If there happened to be some kind >> of image rendering exploit, User A could install an app private to them to >> cause User B to render compromised icons. Etc, etc, etc. > > And you're seriously thinking that user A couldn't figure that out otherwise, > like from looking at the output from ps (running ps in a cron job because > evidently no 2 users can be logged in to the gui at the same time). ps would only tell you what is running, not what is installed. User B should also not have their icon sets changes just because User A downloaded some. ew app. > Even if keeping track of who's entitled to access which bits of cached icon > info the cache agent could prune the cache, removing everything that hasn't > been accessed for more than a reasonably long time. Again, I think you are seeing something anomalous. You should investigate before jumping to conclusions. >> Have you looked at the cache to see exactly what the icons are that are >> taking up so much room? I suspect that might give you a clue as to how >> they're getting there in the first place. > > Well, I did take a look, but there' a LOT of files in there, and they all > seem to be quite small. That's backed up by the percentages in brackets from > afsctool; those are CPU loads. With 4 threads that should approach 400% if > there were only large files; the smaller the number, the more time the > algorithm spends waiting for I/O instead of compressing data. > > R. > > PS: > Aqua, seriously? Is that term still being used or is it just a hardcoded > string in a couple of places that was never changed? Because there isn't much > left of the original watery appearance; possibly only the default > selection/highlight colour ... :) ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Monday September 19 2016 02:08:12 Jeremy Huddleston Sequoia wrote: >> The icon cache sits in the user's $TMPDIR, and AFAIK that directory is >> emptied at boot. > >No. /tmp is emptied at boot, not /var/tmp and not /var/folders. Did that change after 10.9? I keep a Finder window open on $TMPDIR on one of my "Spaces", and it's always empty after logging in to a new session (which I only do after rebooting). >Well, if you're able to update to 10.10+, launchd2 is much much nicer at >helping triage such things... At some point I guess I will... (have to...) >There are ACL settings somewhere to control that. It's not based on uid. I >don't recall what it is off the top of my head. OK, great, let me know if anything comes to mind that could help me in a google search. >> , and when? Boot time? Or login, and if so, what kind of login? > >The user (background) session is created whenever the user does pretty much >anything (ssh, cron, sudo -u, screen sharing, etc.). >The gui (aqua) session is created wheneger the user logs in to the gui (screen >sharing, console, etc). > And the gui session is never taken down when the user logs of the gui? Should one think of launchd as the equivalent of Linux's systemd, or does it also have bits of DBus? >iconservicesagent processes? 2, one in the system domain and one in my user's >aqua session: >$ ps aux | grep iconservicesagent >jeremy677 0.0 0.1 2918544 19908 ?? SMon09AM 0:08.85 >/System/Library/CoreServices/iconservicesagent >root 92 0.0 0.0 2516172492 ?? Ss Mon09AM 0:00.47 >/System/Library/CoreServices/iconservicesagent > How large is root's icon cache? Do you have any idea why the agent is running for root? Because of the login manager, maybe? >Yes. That's a clear privacy violation. User A could use that to determine >what applications User B has installed. If there happened to be some kind of >image rendering exploit, User A could install an app private to them to cause >User B to render compromised icons. Etc, etc, etc. And you're seriously thinking that user A couldn't figure that out otherwise, like from looking at the output from ps (running ps in a cron job because evidently no 2 users can be logged in to the gui at the same time). Even if keeping track of who's entitled to access which bits of cached icon info the cache agent could prune the cache, removing everything that hasn't been accessed for more than a reasonably long time. >Have you looked at the cache to see exactly what the icons are that are taking >up so much room? I suspect that might give you a clue as to how they're >getting there in the first place. Well, I did take a look, but there' a LOT of files in there, and they all seem to be quite small. That's backed up by the percentages in brackets from afsctool; those are CPU loads. With 4 threads that should approach 400% if there were only large files; the smaller the number, the more time the algorithm spends waiting for I/O instead of compressing data. R. PS: Aqua, seriously? Is that term still being used or is it just a hardcoded string in a couple of places that was never changed? Because there isn't much left of the original watery appearance; possibly only the default selection/highlight colour ... :) ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
> On Sep 19, 2016, at 00:34, René J.V. Bertin wrote: > > On Sunday September 18 2016 19:23:01 Jeremy Huddleston Sequoia wrote: > Yes, of course it's possible. Don't. There's nothing special about the UID in this case that has anything to do with what you're seeing. >>> >>> So what does make the difference? >> >> What is "the difference" that you want? All users have a user session, and >> if logged in to the gui, they also have an aqua (gui) session. All users. > > As I said, there's an order of magnitude difference in the size of the icon > cache for users like root, and that of users like macports or ldap. Well, that has nothing to do with the username or uid. It has to do with what the user is doing. > The icon cache sits in the user's $TMPDIR, and AFAIK that directory is > emptied at boot. No. /tmp is emptied at boot, not /var/tmp and not /var/folders. > My machine has been up for 35 days, and in that time I certainly didn't log > any of the "incriminated" users into an aqua session. Well, if you're able to update to 10.10+, launchd2 is much much nicer at helping triage such things... > I did things as the macports user that could explain why an IconServiceAgent > was started for it, but that's it. FWIW, I removed the entries for a few > other users before reporting here, including avahi - those haven't come back > yet. > > A difference I could think of is a setting that tells the system not to let a > particular user log in to a gui session. > Which is why I thought of a UID<500; those at least don't show up in the > login manager. But maybe they can still be logged in if the login manager is > in traditional text (= no icon) mode? There are ACL settings somewhere to control that. It's not based on uid. I don't recall what it is off the top of my head. >>> Possibly a tool, but what could have that effect? AFAIK the icon cache is >>> chiefly used by the Finder, so anything that leverages the Finder might >>> cause an icon cache to be populated. >> >> It's marked as RunAtLoad on my system (Sierra). That would explain why it's >> running, even if nothing needs it... > > But RunAtLoad for whom For all users. > , and when? Boot time? Or login, and if so, what kind of login? The user (background) session is created whenever the user does pretty much anything (ssh, cron, sudo -u, screen sharing, etc.). The gui (aqua) session is created wheneger the user logs in to the gui (screen sharing, console, etc). > How many copies do you have running now? Of what? User domains? 11: $ launchctl print system | grep com.apple.xpc.launchd.domain.user com.apple.xpc.launchd.domain.user.235 com.apple.xpc.launchd.domain.user.502 com.apple.xpc.launchd.domain.user.200 com.apple.xpc.launchd.domain.user.260 com.apple.xpc.launchd.domain.user.55 com.apple.xpc.launchd.domain.user.92 com.apple.xpc.launchd.domain.user.501 com.apple.xpc.launchd.domain.user.212 com.apple.xpc.launchd.domain.user.89 com.apple.xpc.launchd.domain.user.202 com.apple.xpc.launchd.domain.user.0 iconservicesagent processes? 2, one in the system domain and one in my user's aqua session: $ ps aux | grep iconservicesagent jeremy677 0.0 0.1 2918544 19908 ?? SMon09AM 0:08.85 /System/Library/CoreServices/iconservicesagent root 92 0.0 0.0 2516172492 ?? Ss Mon09AM 0:00.47 /System/Library/CoreServices/iconservicesagent > What surprises me is that every user would have to need a personal icon cache > for every icon in the system. > I know user A could have an app installed with a doc type having a particular > icon that no other user on the system is likely to encounter, but would it > hurt if they did get to see the icon if they stumbled across such a document? > IOW, would it be bad if all file-to-icon associations were cached in a > central location, 1 copy for everyone? Yes. That's a clear privacy violation. User A could use that to determine what applications User B has installed. If there happened to be some kind of image rendering exploit, User A could install an app private to them to cause User B to render compromised icons. Etc, etc, etc. > Not only does that cache get big, it also consists of plenty of small files, > which makes its actual footprint on disk even larger (due to blocksize; I > think that hasn't changed with SSDs?). Have you looked at the cache to see exactly what the icons are that are taking up so much room? I suspect that might give you a clue as to how they're getting there in the first place. smime.p7s Description: S/MIME cryptographic signature ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Sunday September 18 2016 19:23:01 Jeremy Huddleston Sequoia wrote: >>> Yes, of course it's possible. Don't. There's nothing special about the >>> UID in this case that has anything to do with what you're seeing. >> >> So what does make the difference? > >What is "the difference" that you want? All users have a user session, and if >logged in to the gui, they also have an aqua (gui) session. All users. As I said, there's an order of magnitude difference in the size of the icon cache for users like root, and that of users like macports or ldap. The icon cache sits in the user's $TMPDIR, and AFAIK that directory is emptied at boot. My machine has been up for 35 days, and in that time I certainly didn't log any of the "incriminated" users into an aqua session. I did things as the macports user that could explain why an IconServiceAgent was started for it, but that's it. FWIW, I removed the entries for a few other users before reporting here, including avahi - those haven't come back yet. A difference I could think of is a setting that tells the system not to let a particular user log in to a gui session. Which is why I thought of a UID<500; those at least don't show up in the login manager. But maybe they can still be logged in if the login manager is in traditional text (= no icon) mode? >> Possibly a tool, but what could have that effect? AFAIK the icon cache is >> chiefly used by the Finder, so anything that leverages the Finder might >> cause an icon cache to be populated. > >It's marked as RunAtLoad on my system (Sierra). That would explain why it's >running, even if nothing needs it... But RunAtLoad for whom, and when? Boot time? Or login, and if so, what kind of login? How many copies do you have running now? What surprises me is that every user would have to need a personal icon cache for every icon in the system. I know user A could have an app installed with a doc type having a particular icon that no other user on the system is likely to encounter, but would it hurt if they did get to see the icon if they stumbled across such a document? IOW, would it be bad if all file-to-icon associations were cached in a central location, 1 copy for everyone? Not only does that cache get big, it also consists of plenty of small files, which makes its actual footprint on disk even larger (due to blocksize; I think that hasn't changed with SSDs?). R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
> On Sep 18, 2016, at 18:10, René J.V. Bertin wrote: > > On Sunday September 18 2016 17:24:55 Jeremy Huddleston Sequoia wrote: > >> That all looks as expected for someone on OS X 10.9. launchd2 in 10.10+ did >> away with per-user launchd. All domains are managed by the same process. > > Still, why would a background user without even an interactive shell be > running an IconServicesAgent? >> >> That's pretty large. I'm a bit surprised by that. >> >> My macports user's disk usage is about 90 MB, mostly for its pch cache. > > That fits with the size I'm seeing for root and _securityagent. 900Mb is > about what I see for users who have logged in at some point. > >>> Is it possible to create users with UID<500 despite those numbers being >>> "reserved by Apple"? >> >> Yes, of course it's possible. Don't. There's nothing special about the UID >> in this case that has anything to do with what you're seeing. > > So what does make the difference? What is "the difference" that you want? All users have a user session, and if logged in to the gui, they also have an aqua (gui) session. All users. > > >>> %> sudo afsctool -cfvv -8 -J4 /var/folders/*/*/C/com.apple.IconServices >> >> What is -J4? I don't see that option mentioned, and afsctool seems to bawk >> at it. > > That's an option from my parallel version: > https://github.com/RJVB/afsctool > > I think I'll have to set up a cron task or similar that runs the afsctool > command with an appropriate frequency. Since the run from this morning only > my own icon cache has increased to about 250Mb; still 1/4 or what it was > before compressing. > >> On Sunday September 18 2016 20:42:53 Brandon Allbery wrote: Some port somewhere likely builds the icon cache while building or "installing" (destrooting) since it "knows" that it would only be built by the person intending to use it and that nobody uses packaging on OS X. :/ >>> >>> ...or maybe some tool invoked during build. The former could be mitigated; >>> the latter may be harder. > > Possibly a tool, but what could have that effect? AFAIK the icon cache is > chiefly used by the Finder, so anything that leverages the Finder might cause > an icon cache to be populated. It's marked as RunAtLoad on my system (Sierra). That would explain why it's running, even if nothing needs it... > But come to think of it that cannot be my case; the non-privileged build > phases are run under my own ID on my system. > > I'm calling it a night for now ... > > R. > ___ > macports-dev mailing list > macports-dev@lists.macosforge.org > https://lists.macosforge.org/mailman/listinfo/macports-dev smime.p7s Description: S/MIME cryptographic signature ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Sunday September 18 2016 17:24:55 Jeremy Huddleston Sequoia wrote: > That all looks as expected for someone on OS X 10.9. launchd2 in 10.10+ did > away with per-user launchd. All domains are managed by the same process. Still, why would a background user without even an interactive shell be running an IconServicesAgent? > > That's pretty large. I'm a bit surprised by that. > > My macports user's disk usage is about 90 MB, mostly for its pch cache. That fits with the size I'm seeing for root and _securityagent. 900Mb is about what I see for users who have logged in at some point. > > Is it possible to create users with UID<500 despite those numbers being > > "reserved by Apple"? > > Yes, of course it's possible. Don't. There's nothing special about the UID > in this case that has anything to do with what you're seeing. So what does make the difference? > > %> sudo afsctool -cfvv -8 -J4 /var/folders/*/*/C/com.apple.IconServices > > What is -J4? I don't see that option mentioned, and afsctool seems to bawk > at it. That's an option from my parallel version: https://github.com/RJVB/afsctool I think I'll have to set up a cron task or similar that runs the afsctool command with an appropriate frequency. Since the run from this morning only my own icon cache has increased to about 250Mb; still 1/4 or what it was before compressing. > On Sunday September 18 2016 20:42:53 Brandon Allbery wrote: > > > Some port somewhere likely builds the icon cache while building or > > > "installing" (destrooting) since it "knows" that it would only be built > > > by > > > the person intending to use it and that nobody uses packaging on OS X. > > > :/ > > > > ...or maybe some tool invoked during build. The former could be mitigated; > > the latter may be harder. Possibly a tool, but what could have that effect? AFAIK the icon cache is chiefly used by the Finder, so anything that leverages the Finder might cause an icon cache to be populated. But come to think of it that cannot be my case; the non-privileged build phases are run under my own ID on my system. I'm calling it a night for now ... R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Sun, Sep 18, 2016 at 8:41 PM, Brandon Allbery wrote: > Some port somewhere likely builds the icon cache while building or > "installing" (destrooting) since it "knows" that it would only be built by > the person intending to use it and that nobody uses packaging on OS X. :/ ...or maybe some tool invoked during build. The former could be mitigated; the latter may be harder. -- brandon s allbery kf8nh sine nomine associates allber...@gmail.com ballb...@sinenomine.net unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
On Sun, Sep 18, 2016 at 8:24 PM, Jeremy Huddleston Sequoia < jerem...@apple.com> wrote: > > Anyway, I don't see any valid reason why the macports user would have a > fullblown icon cache. In my case each cache directory eats up about 900Mb. > Some may find that insignificant; I still consider it a waste of space. > > That's pretty large. I'm a bit surprised by that. Some port somewhere likely builds the icon cache while building or "installing" (destrooting) since it "knows" that it would only be built by the person intending to use it and that nobody uses packaging on OS X. :/ -- brandon s allbery kf8nh sine nomine associates allber...@gmail.com ballb...@sinenomine.net unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
> On Sep 18, 2016, at 04:25, René J.V. Bertin wrote:
>
> Hi,
>
> This isn't actually about codesigning but something that may have come up
> because of my testing in that domain.
>
> I got a low disk space warning (for an unrelated reason) and hunting down the
> space eater I noticed this:
>
> {{{
> %> id macports
> uid=502(macports) gid=503(macports)
> groups=503(macports),12(everyone),61(localaccounts),402(com.apple.sharepoint.group.1),403(com.apple.sharepoint.group.2),100(_lpoperator)
> %> ps -u 502
> UID PID TTY TIME CMD
> 502 1476 ?? 0:26.85 /sbin/launchd
> 502 1479 ?? 0:19.00 /usr/sbin/distnoted agent
> 502 3843 ?? 0:00.14 /usr/libexec/xpcd
> 502 3844 ?? 0:13.81 com.apple.IconServicesAgent
> 502 94723 ?? 0:00.06
> /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/md
> 502 95293 ?? 0:00.19 /usr/sbin/cfprefsd agent
> }}}
>
> I'm not sure if it's normal that macports has a launchd running
That all looks as expected for someone on OS X 10.9. launchd2 in 10.10+ did
away with per-user launchd. All domains are managed by the same process.
> or to what extent that has to do with my recent tests of using that user's
> keychain for codesigning. I started the Keychain Utility as user macports,
> maybe that's what launched launchd c.s.
> Anyway, I don't see any valid reason why the macports user would have a
> fullblown icon cache. In my case each cache directory eats up about 900Mb.
> Some may find that insignificant; I still consider it a waste of space.
That's pretty large. I'm a bit surprised by that.
My macports user's disk usage is about 90 MB, mostly for its pch cache.
> I also notice that users like root and _securityagent have *much* smaller
> icon cache directories. Does anyone know if there's a setting or other means
> to limit the information being cached by users that should never run
> full-blown sessions? Could it be related to the UID being larger or smaller
> than some symbolic value (often 500 on Unix)?
>
> Evidently the macports user isn't the only one that's concerned: any user
> added by MacPorts is (polkituser, polkitd, avahi, ldap, ...)
>
> Is it possible to create users with UID<500 despite those numbers being
> "reserved by Apple"?
Yes, of course it's possible. Don't. There's nothing special about the UID in
this case that has anything to do with what you're seeing.
>
> Thanks,
> René
>
> PS: this is what afsctool reports about my icon caches
> {{{
> %> sudo afsctool -cfvv -8 -J4 /var/folders/*/*/C/com.apple.IconServices
What is -J4? I don't see that option mentioned, and afsctool seems to bawk at
it.
> Adding
> /var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/8d/5ghf31k1063f1wcln60nd91hgv/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/99/yvy25dd94jn_ykbfkf1q4mzwgw/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/gf/wmgp2zv506dcsg7_cjxx69m4gr/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/j1/1439ppj08xj8h6006s6drbq0gs/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/sd/pwbxpphw8xj3vb006s6drbq0gn/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/zz/zyxvpxvq6csfxvn_n0/C/com.apple.IconServices to
> queue
> Adding
> /var/folders/zz/zyxvpxvq6csfxvn_n0bh2w/C/com.apple.IconServices to
> queue
> Starting 4 worker threads to process queue with 8617 items
> 11% [33.45%] .. 21% [33.95%] .. 32% [33.63%] .. 42% [31.99%] .. 52% [30.15%]
> .. 63% [29.08%] .. 73% [27.90%] .. 84% [27.30%] .. 94% [27.62%]
> Processed 8617 entries
> Total number of files: 14016
> Total number of file hard links: 0
> Total number of folders: 0
> Total number of folder hard links: 0
> Total number of items (number of files + number of folders): 14016
> Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 6306042064
> bytes / 6.32 GB (gigabytes) / 5.88 GiB (gibibytes)
> Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5
> Finder): 665214917 bytes / 678.6 MB (megabytes) / 647.2 MiB (mebibytes)
> Folder size (compressed): 676008333 bytes / 689.4 MB (megabytes) / 657.5 MiB
> (mebibytes)
> Compression savings: 89.3%
> Approximate total folder size (files + file overhead + folder overhead):
> 698299808 bytes / 698.3 MB (megabytes) / 666 MiB (mebibytes)
> }}}
>
> IOW, they compress about 10x. A real shame HFS doesn't have transparent,
> online compression!!
smime.p7s
Description: S/MIME cryptographic signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
codesigning, the macports user (and other users added by MacPorts) and diskspace (IconServicesAgent) [was Re: lldb ...]
Hi,
This isn't actually about codesigning but something that may have come up
because of my testing in that domain.
I got a low disk space warning (for an unrelated reason) and hunting down the
space eater I noticed this:
{{{
%> id macports
uid=502(macports) gid=503(macports)
groups=503(macports),12(everyone),61(localaccounts),402(com.apple.sharepoint.group.1),403(com.apple.sharepoint.group.2),100(_lpoperator)
%> ps -u 502
UID PID TTY TIME CMD
502 1476 ?? 0:26.85 /sbin/launchd
502 1479 ?? 0:19.00 /usr/sbin/distnoted agent
502 3843 ?? 0:00.14 /usr/libexec/xpcd
502 3844 ?? 0:13.81 com.apple.IconServicesAgent
502 94723 ?? 0:00.06
/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/md
502 95293 ?? 0:00.19 /usr/sbin/cfprefsd agent
}}}
I'm not sure if it's normal that macports has a launchd running or to what
extent that has to do with my recent tests of using that user's keychain for
codesigning. I started the Keychain Utility as user macports, maybe that's what
launched launchd c.s.
Anyway, I don't see any valid reason why the macports user would have a
fullblown icon cache. In my case each cache directory eats up about 900Mb. Some
may find that insignificant; I still consider it a waste of space.
I also notice that users like root and _securityagent have *much* smaller icon
cache directories. Does anyone know if there's a setting or other means to
limit the information being cached by users that should never run full-blown
sessions? Could it be related to the UID being larger or smaller than some
symbolic value (often 500 on Unix)?
Evidently the macports user isn't the only one that's concerned: any user added
by MacPorts is (polkituser, polkitd, avahi, ldap, ...)
Is it possible to create users with UID<500 despite those numbers being
"reserved by Apple"?
Thanks,
René
PS: this is what afsctool reports about my icon caches
{{{
%> sudo afsctool -cfvv -8 -J4 /var/folders/*/*/C/com.apple.IconServices
Adding /var/folders/11/_bf842l54ngf42g66hkzslmwgp/C/com.apple.IconServices
to queue
Adding /var/folders/8d/5ghf31k1063f1wcln60nd91hgv/C/com.apple.IconServices
to queue
Adding /var/folders/99/yvy25dd94jn_ykbfkf1q4mzwgw/C/com.apple.IconServices
to queue
Adding /var/folders/gf/wmgp2zv506dcsg7_cjxx69m4gr/C/com.apple.IconServices
to queue
Adding /var/folders/j1/1439ppj08xj8h6006s6drbq0gs/C/com.apple.IconServices
to queue
Adding /var/folders/sd/pwbxpphw8xj3vb006s6drbq0gn/C/com.apple.IconServices
to queue
Adding /var/folders/zz/zyxvpxvq6csfxvn_n0/C/com.apple.IconServices
to queue
Adding /var/folders/zz/zyxvpxvq6csfxvn_n0bh2w/C/com.apple.IconServices
to queue
Starting 4 worker threads to process queue with 8617 items
11% [33.45%] .. 21% [33.95%] .. 32% [33.63%] .. 42% [31.99%] .. 52% [30.15%]
.. 63% [29.08%] .. 73% [27.90%] .. 84% [27.30%] .. 94% [27.62%]
Processed 8617 entries
Total number of files: 14016
Total number of file hard links: 0
Total number of folders: 0
Total number of folder hard links: 0
Total number of items (number of files + number of folders): 14016
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 6306042064
bytes / 6.32 GB (gigabytes) / 5.88 GiB (gibibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5
Finder): 665214917 bytes / 678.6 MB (megabytes) / 647.2 MiB (mebibytes)
Folder size (compressed): 676008333 bytes / 689.4 MB (megabytes) / 657.5 MiB
(mebibytes)
Compression savings: 89.3%
Approximate total folder size (files + file overhead + folder overhead):
698299808 bytes / 698.3 MB (megabytes) / 666 MiB (mebibytes)
}}}
IOW, they compress about 10x. A real shame HFS doesn't have transparent, online
compression!!
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-16 10:18, Jeremy Huddleston Sequoia wrote: > Yeah, this contradicts what I'm seeing as expected. Given that > you've signed /opt/local/bin/ggdb with an entitlement, it should be > CS_RESTRICT which should imply CS_HARD. The lack of a code signature > would trigger !CS_VALID which would prevent the process from loading > the unsigned libraries. There is actually no entitlement data in the code-signature itself. The access is granted by embedding a Info.plist into the binary: $ otool -P /opt/local/bin/ggdb /opt/local/bin/ggdb: (__TEXT,__info_plist) section http://www.apple.com/DTDs/PropertyList-1.0.dtd";> CFBundleIdentifier org.gnu.gdb CFBundleName gdb CFBundleVersion 1.0 SecTaskAccess allowed debug Probably that is why these rules are not enforced? Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 10, 2016, at 17:56, Jeremy Sequoia wrote:
>
> On Sep 10, 2016, at 16:51, Rainer Müller wrote:
>
>> I got gdb to work now on Sierra now. In fact I did not even have to sign
>> any of the libraries it links to.
>>
>>
>> $ otool -L /opt/local/bin/ggdb |awk 'NR>1 {print $1}' \
>>|grep '^/opt/local' | xargs -I{} codesign -d -v {}
>> /opt/local/lib/libintl.8.dylib: code object is not signed at all
>> /opt/local/lib/libncurses.6.dylib: code object is not signed at all
>> /opt/local/lib/libz.1.dylib: code object is not signed at all
>> /opt/local/lib/libiconv.2.dylib: code object is not signed at all
>> /opt/local/lib/libexpat.1.dylib: code object is not signed at all
>>
>> $ /opt/local/bin/ggdb -q /opt/local/bin/curl
>> Reading symbols from /opt/local/bin/curl...(no debugging symbols
>> found)...done.
>> (gdb) r
>> Starting program: /opt/local/bin/curl
>> warning: unhandled dyld version (15)
>> curl: try 'curl --help' or 'curl --manual' for more information
>> [Inferior 1 (process 6964) exited with code 02]
>> (gdb) q
>
> Hmm. That isn't what I'd expect. Gonna need to check why that is. It looks
> like CS_RESTRICT isn't implying CS_HARD like I thought it should.
Yeah, this contradicts what I'm seeing as expected. Given that you've signed
/opt/local/bin/ggdb with an entitlement, it should be CS_RESTRICT which should
imply CS_HARD. The lack of a code signature would trigger !CS_VALID which
would prevent the process from loading the unsigned libraries.
Can you send me a tarball of your working
/opt/local/bin/ggdb
/opt/local/lib/libintl.8.dylib
/opt/local/lib/libncurses.6.dylib
/opt/local/lib/libz.1.dylib
/opt/local/lib/libiconv.2.dylib
/opt/local/lib/libexpat.1.dylib
+ all other recursive dependencies that those dylibs link against in /opt/local?
I want to take a closer look.
--Jeremy
smime.p7s
Description: S/MIME cryptographic signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
> On Sep 13, 2016, at 2:06 AM, René J.V. Bertin wrote: > > On Tuesday September 13 2016 01:39:41 Jeremy Huddleston Sequoia wrote: > >> That's just the nature of things. > > Yeah, life sucks and all that ... > >> Do you have 3rd party menu extras? IIRC, you're on 10.9, and 3rd party menu >> extras run in SystemUIServer. I frequently got SystemUIServer crashes >> thanks to iStat Menus. > > Indeed, I have a few. In fact, even Qt's "systray" icon thingy may rely on > menu extras, I never checked that. > > So, what crashes now due to iStat Menus? :) The newer versions are much more stable. I haven't seen the frequent crashes that I used to, but if they were to crash, they would just tear down their own separate process instead of all of SystemUIServer. =) --Jeremy ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Tuesday September 13 2016 01:39:41 Jeremy Huddleston Sequoia wrote: > That's just the nature of things. Yeah, life sucks and all that ... >Do you have 3rd party menu extras? IIRC, you're on 10.9, and 3rd party menu >extras run in SystemUIServer. I frequently got SystemUIServer crashes thanks >to iStat Menus. Indeed, I have a few. In fact, even Qt's "systray" icon thingy may rely on menu extras, I never checked that. So, what crashes now due to iStat Menus? :) R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
> On Sep 13, 2016, at 00:14, René J.V. Bertin wrote: > > On Monday September 12 2016 21:51:55 Jeremy Sequoia wrote: > > I can confirm that there *is* feedback on radars, though I do have to admit > that in my case it is usually one of 3 forms: > - works as intended > - please check if the issue still exists in the latest release (typically > months later) > - please install the latest OS X version and tell us if the bug still persists Yes, ADC often sends template responses for bugs that are reported against old OS versions or when fixed (or we believe are fixed) in various releases. > Esp. the latter response doesn't really help. I know that clinging to an old > OS X release isn't the ideal solution either, but I think that a different > approach would be more constructive as long as a version is still supported > and receives updates of whatever kind. Sometimes bugs are fixed in newer versions of the OS and not backported to older OS versions because they don't meet the criteria for an SU. Many bugs got fixed in Sierra that won't get fixed in an El Cap software update. That's just the nature of things. On the flip side, a significant percentage the radars that I receive have insufficient information for action, and I send them back with detailed instructions on providing me with a sysdiagnose, sample projects, etc, etc. I usually do so within days of receiving the radar, and they sometimes sit for months without action on the part of the originator. > There are software providers out there who tend to give the impression they > don't always take into consideration the fact that we (as in those who file > bug reports) don't work for them, and Apple aren't completely foreign to that. > > Anyway, a bit off-topic :) > > R. > > PS: no, I cannot think of any open radars I filed about issues that keep > nagging me. I may have filed one about a frequent SystemUIServer crash when > getting back to my session from the login screen, but I actually haven't seen > that crash for a bit now. Do you have 3rd party menu extras? IIRC, you're on 10.9, and 3rd party menu extras run in SystemUIServer. I frequently got SystemUIServer crashes thanks to iStat Menus. > ___ > macports-dev mailing list > macports-dev@lists.macosforge.org > https://lists.macosforge.org/mailman/listinfo/macports-dev smime.p7s Description: S/MIME cryptographic signature ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Monday September 12 2016 21:51:55 Jeremy Sequoia wrote: I can confirm that there *is* feedback on radars, though I do have to admit that in my case it is usually one of 3 forms: - works as intended - please check if the issue still exists in the latest release (typically months later) - please install the latest OS X version and tell us if the bug still persists Esp. the latter response doesn't really help. I know that clinging to an old OS X release isn't the ideal solution either, but I think that a different approach would be more constructive as long as a version is still supported and receives updates of whatever kind. There are software providers out there who tend to give the impression they don't always take into consideration the fact that we (as in those who file bug reports) don't work for them, and Apple aren't completely foreign to that. Anyway, a bit off-topic :) R. PS: no, I cannot think of any open radars I filed about issues that keep nagging me. I may have filed one about a frequent SystemUIServer crash when getting back to my session from the login screen, but I actually haven't seen that crash for a bit now. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
Sent from my iPhone... > On Sep 12, 2016, at 21:44, Brandon Allbery wrote: > > > On Tue, Sep 13, 2016 at 12:24 AM, Jeremy Sequoia wrote: >>> I just noticed this. Received wisdom "out here" is that ``nobody is >>> listening and nobody ever responds, so don't bother filing radars.'' >> >> That is very very very false. > > That may be, but I got paraphrases of that response from a number of > Mac-oriented projects and other third party developers. Nobody trusts your > bug reporting any more. Yes, that is a problem. Not filing bugs is not the solution to that problem, especially when there are engineers expressly reaching out for said radars. I can't help you if you won't let me help you :/ > > -- > brandon s allbery kf8nh sine nomine associates > allber...@gmail.com ballb...@sinenomine.net > unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Tue, Sep 13, 2016 at 12:24 AM, Jeremy Sequoia wrote: > I just noticed this. Received wisdom "out here" is that ``nobody is > listening and nobody ever responds, so don't bother filing radars.'' > > > That is very very very false. > That may be, but I got paraphrases of that response from a number of Mac-oriented projects and other third party developers. Nobody trusts your bug reporting any more. -- brandon s allbery kf8nh sine nomine associates allber...@gmail.com ballb...@sinenomine.net unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
Sent from my iPhone... > On Sep 12, 2016, at 20:57, Brandon Allbery wrote: > >> On Sat, Sep 10, 2016 at 12:14 PM, Jeremy Huddleston Sequoia >> wrote: >> Please file radars and point me to them, so I can make sure they get routed >> to the right place (likely as dupes, but dupes are very useful "votes" for >> bugs). > > I just noticed this. Received wisdom "out here" is that ``nobody is listening > and nobody ever responds, so don't bother filing radars.'' That is very very very false. > If you really are still using it, you have procedural and/or communication > issues. There is a radar for that as well, and I have some opinions on the matter, but it is not something that I have any control over. Really, filing radars is valuable even if they are just duplicates. They are cheap to file and value able to the development process. Yes, ADC is sometimes not the best middleman. If you are hitting an obstacle with ADC, reach out to Apple engineers that you know directly to enquire about them. We are all over forums, stackoverflow, twitter, etc. And we want to help make the products better for users and developers. If you don't file a radar about an issue, there is no guarantee that the right Apple engineer knows about it. I came across a stackoverflow question a few months ago with an answer that "it is a known problem that ... so the workaround is just ..." but it was never reported to Apple until I happened upon the post and filed a radar about it. The issue was fixed (internally) a few days after that, but had someone externally filed a radar, it would've been fixed even sooner. > > -- > brandon s allbery kf8nh sine nomine associates > allber...@gmail.com ballb...@sinenomine.net > unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Sat, Sep 10, 2016 at 12:14 PM, Jeremy Huddleston Sequoia < jerem...@apple.com> wrote: > Please file radars and point me to them, so I can make sure they get > routed to the right place (likely as dupes, but dupes are very useful > "votes" for bugs). I just noticed this. Received wisdom "out here" is that ``nobody is listening and nobody ever responds, so don't bother filing radars.'' If you really are still using it, you have procedural and/or communication issues. -- brandon s allbery kf8nh sine nomine associates allber...@gmail.com ballb...@sinenomine.net unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
Can you please file a radar and toss me the number. This might be somehow falling through the cracks with the new logging changes and didn't get noticed yet. Hopefully we can get it fixed in an update soon if there isn't a workaround. Sent from my iPhone... > On Sep 11, 2016, at 05:07, Rainer Müller wrote: > > On 2016-09-11 02:56, Jeremy Sequoia wrote: >>> By the way, as I did lots of trial and error, is there a way to get >>> debug output (from taskgated?) to see why task_for_pid() was denied? >> >> Is it not being logged? You should see it in the system log >> (Console.app, log collect, etc). > > That is where I looked. When it fails, all I see is: > > taskgatedMacOS error: -67062 > > I assumed there must be some switch to get more output, but maybe it is > only enabled Apple internal. > > Messages with the same error code also appear from taskgated when it > works, but also some other error codes. So this might even be unrelated > to the actual verification. When it works, I also get more output from > other components such as trustd and authd. > > Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-11 02:56, Jeremy Sequoia wrote: >> By the way, as I did lots of trial and error, is there a way to get >> debug output (from taskgated?) to see why task_for_pid() was denied? > > Is it not being logged? You should see it in the system log > (Console.app, log collect, etc). That is where I looked. When it fails, all I see is: taskgatedMacOS error: -67062 I assumed there must be some switch to get more output, but maybe it is only enabled Apple internal. Messages with the same error code also appear from taskgated when it works, but also some other error codes. So this might even be unrelated to the actual verification. When it works, I also get more output from other components such as trustd and authd. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Sunday September 11 2016 00:24:17 Jeremy Huddleston Sequoia wrote: >Well, you can certainly upgrade the storage after purchase if you want (cf >https://eshop.macsales.com/shop/ssd/owc/macbook-pro-retina-display/2013-2014-2015) Sure ... and that's not even 10x more expensive than a 1TB HDD, and probably not even half the price I paid for my MBP. However, sadly OWC didn't have this out when I bought my Thunderbolt dock: https://eshop.macsales.com/shop/Thunderbolt/Dock/OWC/Thunderbolt2-Dock/ As they say, finally a dock done right (esp. if it heats less than my Belkin) R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Sunday September 11 2016 00:24:17 Jeremy Huddleston Sequoia wrote: >The secure thing to do is not escalate until we need to, prompt, do our thing, >then throw away the privs when done. The problem with that is that we'd ask >for credentials 2-3 times during the install and only when we need them. The solution would be to do as sudo does: cache the password for a given period of time. >The less secure thing to do is just ask for credentials up front if anything >will need root access and then just promote/demote as needed. Which is already what happens during the stages that shouldnt be run as root. >Even though iOS shares much of the same codebase as macOS, it is not UNIX. >Apple has never certified iOS as UNIX. Well, my cats don't have an official certification that they're European race. Doesn't mean they aren't ;) You know, looks like a duck, talks like one, etc. The fact you can't sell it as Unix is a distinction that IMHO matters only to marketeers and lawyers... >FWIW, HDDs are much more flakey than SSDs. I've had to replace countless HDDs >in my life, but I've yet to see one of my SSDs reach the end of their lives. The thing with HDDs is that you can predict failure only when they start failing. There's of course a MTBF estimate, but for each individual pair of HDDs you could just as well see the new one fail almost immediately and the old one last years more as the opposite (given identical use). SDDs have the advantage of being much more resistant to mechanical wear and abuse, but then again it wasn't the HDD that failed in my Powerbook which accompanied me in 3 motorcycle accidents and thousands of kms. >Then you definitely weren't using a good SSD. =/ No, and I got a lemon at that too. >There's also an SDXC port there. You can get a micro SDXC card and one of the >many flush-insert adaptors Is that port really different from the SD card port I have in my 2011 MBP? I've never dared investing in a SDXC card but the more affordable ones of a reasonable size tend to be disappointingly sluggish in certain operations. Otherwise they'd probably make a good choice for putting a filesystem journal (or ZFS ZIL). R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 10, 2016, at 09:44, René J.V. Bertin wrote: > > On Saturday September 10 2016 08:45:38 Jeremy Huddleston Sequoia wrote: > >> Yes, for most cases we would probably not need root access at activation >> time, but for those cases you just listed, we would. We could do a pass >> over the bom to determine if root access is needed or not. The UI for >> handling such a case is non-trivial. Do we prompt once for the entire run >> of the process or escalate permissions for each port we encounter and then >> drop them again? If 8 ports are being installed and 4 require root, that's >> not a good user experience to prompt for password 4 times. > > I think it'd be up to the port to indicate if it requires root privileges > during the destroot. This is (implicitly) what happens with ports that create > a new user or group; there's a phase that requires root (configure, I think). > How's that handled at the UI level? I'm guessing that the phase will simply > fail for the port(s) that don't have the appropriate privileges. My point is not about the failure case but the working case. Ideally, the user would be able to do 'port -v -s install '. The secure thing to do is not escalate until we need to, prompt, do our thing, then throw away the privs when done. The problem with that is that we'd ask for credentials 2-3 times during the install and only when we need them. The less secure thing to do is just ask for credentials up front if anything will need root access and then just promote/demote as needed. The latter is a better user experience, and perhaps this can be the default but configurable for those that want more security. >>> I understand that the certificate catching is coupled to the file's inode >>> (whatever that is under HFS+), and the new copy indeed had a new inode. And >>> yet another inode after signing it. >> >> If you can reproduce this, please let me know and file a radar with details >> and let me know the number, so I can followup internally on it. I haven't >> seen such cases. The only issue I'm aware of is with overwriting an >> existing file (eg: using cp instead of mv). > > To make this reproducible I'd probably be looking at the following sequence? > 1) copy a binary from some original and ad-hoc sign it > 2) delete it > 3) copy the original once more and sign it with another certificate > > Correct me if I'm wrong, but I think that's what a `port -n upgrade --force` > boils down to if signing is done in the post-activate, no? > Is there any reason to expect that the certificate used in 3) must be a new > one not yet used to sign other executables? > >> iOS was never branded as UNIX. It is UNIX-like, but it was never UNIX. > > AFAIK it is or used to be based on OS X, evidently without the userland but > with a comparable kernel. Was that just a shortcut way of thinking of things? iOS has the same core os macOS. Much of everything below the UI layer is shared between the two platforms, but there are subtle differences, similar to how tvOS and watchOS are forked from iOS. Even though iOS shares much of the same codebase as macOS, it is not UNIX. Apple has never certified iOS as UNIX. >> I've got an internal 1TB SSD and it is more than enough for most of my >> needs. I mainly use USB for thumbsticks and my camera. In such cases, the >> throughput is bottlenecked on the attached device, not the bus. >> >> That being said, my server machine is a 2012 MacMini, and I've got a >> DroboMini and a LaCie RAID attached to it via thunderbolt, and the >> throughput and latency are more than adequate for my demanding needs. > > Thunderbolt external disks and 1TB SSDs are simply out of my budget. > >> For folks like you and me, it might be a minor concern being limited to 1TB, >> but we can certainly use an external SSD or an SDXC to increase storage as >> needed in the future. > > I see it as more than a minor concern. Not because I expect I'll ever need > more than 1TB internal space, but I would certainly not feel comfortable when > I know I cannot replace something that can fail that easily and unpredictable > as an SSD (or HDD for that matter). I don't recall but it's quite likely too > that I replaced the HDD in my current MBP with a Hitachi as soon as I > received the machine. FWIW, HDDs are much more flakey than SSDs. I've had to replace countless HDDs in my life, but I've yet to see one of my SSDs reach the end of their lives. > My only experience to date with SSDs was as an external. Connected over FW800 > (the fastest bus I had at the time) showed 0 gain over a good HDD ... and the > whole thing died catastrophically after not even 4 months. Then you definitely weren't using a good SSD. =/ > FWIW, that Clevo notebook I mentioned has an internal 2.5" disk (came with an > SSHD) but even at its price point it has a fashionable slot to add an SSD. > That's the kind of approach I like: adding expansion options in
Re: lldb ...
Sent from my iPhone...
> On Sep 10, 2016, at 16:51, Rainer Müller wrote:
>
> On 2016-09-10 17:52, Jeremy Huddleston Sequoia wrote:
>>> On OS X 10.10 Yosemite, signing only the ggdb binary was certainly
>>> enough. I cannot reproduce this on macOS 10.12 Sierra, so the
>>> requirements might have changed.
>>
>> 10.10 predates SIP and related hardening around ptrace(). That
>> version is so far in my rearview that I forget the details there,
>> sorry. I'll have to dig into it, but it certainly seems wrong to me
>> that a process could become privileged if it linked against unsigned
>> libraries.
>
> I would assume if we find a solution that passes the current
> restrictions on Sierra that will also work for older releases with less
> strict checking.
>
> I got gdb to work now on Sierra now. In fact I did not even have to sign
> any of the libraries it links to.
>
>
> $ otool -L /opt/local/bin/ggdb |awk 'NR>1 {print $1}' \
>|grep '^/opt/local' | xargs -I{} codesign -d -v {}
> /opt/local/lib/libintl.8.dylib: code object is not signed at all
> /opt/local/lib/libncurses.6.dylib: code object is not signed at all
> /opt/local/lib/libz.1.dylib: code object is not signed at all
> /opt/local/lib/libiconv.2.dylib: code object is not signed at all
> /opt/local/lib/libexpat.1.dylib: code object is not signed at all
>
> $ /opt/local/bin/ggdb -q /opt/local/bin/curl
> Reading symbols from /opt/local/bin/curl...(no debugging symbols
> found)...done.
> (gdb) r
> Starting program: /opt/local/bin/curl
> warning: unhandled dyld version (15)
> curl: try 'curl --help' or 'curl --manual' for more information
> [Inferior 1 (process 6964) exited with code 02]
> (gdb) q
Hmm. That isn't what I'd expect. Gonna need to check why that is. It looks
like CS_RESTRICT isn't implying CS_HARD like I thought it should.
>
> The main problem I encountered was that the setgid for the procmod group
> seems to interfere with the validation now. Once I removed that by
> changing the permissions to a regular 0755, I can use the code-signed
> ggdb just fine to debug other programs.
>
> By the way, as I did lots of trial and error, is there a way to get
> debug output (from taskgated?) to see why task_for_pid() was denied?
Is it not being logged? You should see it in the system log (Console.app, log
collect, etc).
>
> Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-10 17:52, Jeremy Huddleston Sequoia wrote:
>> On OS X 10.10 Yosemite, signing only the ggdb binary was certainly
>> enough. I cannot reproduce this on macOS 10.12 Sierra, so the
>> requirements might have changed.
>
> 10.10 predates SIP and related hardening around ptrace(). That
> version is so far in my rearview that I forget the details there,
> sorry. I'll have to dig into it, but it certainly seems wrong to me
> that a process could become privileged if it linked against unsigned
> libraries.
I would assume if we find a solution that passes the current
restrictions on Sierra that will also work for older releases with less
strict checking.
I got gdb to work now on Sierra now. In fact I did not even have to sign
any of the libraries it links to.
$ otool -L /opt/local/bin/ggdb |awk 'NR>1 {print $1}' \
|grep '^/opt/local' | xargs -I{} codesign -d -v {}
/opt/local/lib/libintl.8.dylib: code object is not signed at all
/opt/local/lib/libncurses.6.dylib: code object is not signed at all
/opt/local/lib/libz.1.dylib: code object is not signed at all
/opt/local/lib/libiconv.2.dylib: code object is not signed at all
/opt/local/lib/libexpat.1.dylib: code object is not signed at all
$ /opt/local/bin/ggdb -q /opt/local/bin/curl
Reading symbols from /opt/local/bin/curl...(no debugging symbols
found)...done.
(gdb) r
Starting program: /opt/local/bin/curl
warning: unhandled dyld version (15)
curl: try 'curl --help' or 'curl --manual' for more information
[Inferior 1 (process 6964) exited with code 02]
(gdb) q
The main problem I encountered was that the setgid for the procmod group
seems to interfere with the validation now. Once I removed that by
changing the permissions to a regular 0755, I can use the code-signed
ggdb just fine to debug other programs.
By the way, as I did lots of trial and error, is there a way to get
debug output (from taskgated?) to see why task_for_pid() was denied?
Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
> On Sep 10, 2016, at 09:38, Clemens Lang wrote: > > Hi, > > On Sat, Sep 10, 2016 at 09:14:16AM -0700, Jeremy Huddleston Sequoia wrote: >> No, the DYLD_INSERT_LIBRARIES approach is the right one here. >> Interested users would need to disable SIP. > > "Interested users" would be everybody who uses MacPorts. I'd vote > against telling all our users to disable SIP. It's a useful > security/safety feature and it even helps us because users can no longer > mess up their /usr/bin. No. "Interested users" would be everybody who wants to use this fakeroot implementation instead of sudo. Users would need to make the tradeoff decision for themselves as to whether they want SIP+sudo or !SIP+fakeroot. > I don't see why the kernel, dyld, or whoever strips the flags could not > just behave like running a copy of the binary at hand when it sees a > DYLD variable, i.e. do the workaround we're doing manually at the > moment. Because that'e exactly what the system binary bit is intended to prevent. The system binary bit on the executables is in place to ensure that binary is not interposed by third party code that can potentially degrade the integrity of the running system. >> It would be nice if a mechanism were in place to determine trust of >> certain libraries in DYLD_INSERT_LIBRARIES. > > So you're suggesting DYLD_INSERT_LIBRARIES on SIP-protected binaries > should only work if the inserted library is signed? How would that > improve anything? Are you suggesting every open source project out there > that uses library preloading now pays for a certificate and regularly > builds and releases binaries for macOS? Frankly, I don't see that > happening. IMO, there should be a way to establish a trust model such that certain binaries could be inserted (maybe with a particular entitlement on the dylib). That isn't possible right now. Please file radars to vote for this getting addressed in a future release. OSS projects are free to do what they want for the benefit of their customers. They can sign their code with an ADC-provided cert, a cert from another CA, or not at all and just instruct users on how to sign the binary themselves. >> Please file radars and point me to them, so I can make sure they get >> routed to the right place (likely as dupes, but dupes are very useful >> "votes" for bugs). > > Those tickets have been filed when SIP was introduced and > DYLD_INSERT_LIBRARIES stopped working. > Evidently, it wasn't important > enough to get fixed, so you'll forgive me if I have better things to do > with my time. All the radars that I am aware of are mainly focused around "DYLD_INSERT_LIBRARIES doesn't get passed through system binaries" not "DYLD_INSERT_LIBRARIES doesn't get honored by system binaries". The latter is what this is about. smime.p7s Description: S/MIME cryptographic signature ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Saturday September 10 2016 18:38:36 Clemens Lang wrote: > I don't see why the kernel, dyld, or whoever strips the flags could not > just behave like running a copy of the binary at hand when it sees a > DYLD variable, i.e. do the workaround we're doing manually at the > moment. Then what would be the point of stripping those variables? :) > > Please file radars and point me to them, so I can make sure they get > > routed to the right place (likely as dupes, but dupes are very useful > > "votes" for bugs). > > Those tickets have been filed when SIP was introduced and > DYLD_INSERT_LIBRARIES stopped working. Evidently, it wasn't important > enough to get fixed, so you'll forgive me if I have better things to do > with my time. Erm, surely it wouldn't take a lot of time to point Jeremy to those radars if you have a list of them somewhere? R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Saturday September 10 2016 08:45:38 Jeremy Huddleston Sequoia wrote: > Yes, for most cases we would probably not need root access at activation > time, but for those cases you just listed, we would. We could do a pass over > the bom to determine if root access is needed or not. The UI for handling > such a case is non-trivial. Do we prompt once for the entire run of the > process or escalate permissions for each port we encounter and then drop them > again? If 8 ports are being installed and 4 require root, that's not a good > user experience to prompt for password 4 times. I think it'd be up to the port to indicate if it requires root privileges during the destroot. This is (implicitly) what happens with ports that create a new user or group; there's a phase that requires root (configure, I think). How's that handled at the UI level? I'm guessing that the phase will simply fail for the port(s) that don't have the appropriate privileges. > > I understand that the certificate catching is coupled to the file's inode > > (whatever that is under HFS+), and the new copy indeed had a new inode. And > > yet another inode after signing it. > > If you can reproduce this, please let me know and file a radar with details > and let me know the number, so I can followup internally on it. I haven't > seen such cases. The only issue I'm aware of is with overwriting an existing > file (eg: using cp instead of mv). To make this reproducible I'd probably be looking at the following sequence? 1) copy a binary from some original and ad-hoc sign it 2) delete it 3) copy the original once more and sign it with another certificate Correct me if I'm wrong, but I think that's what a `port -n upgrade --force` boils down to if signing is done in the post-activate, no? Is there any reason to expect that the certificate used in 3) must be a new one not yet used to sign other executables? > iOS was never branded as UNIX. It is UNIX-like, but it was never UNIX. AFAIK it is or used to be based on OS X, evidently without the userland but with a comparable kernel. Was that just a shortcut way of thinking of things? > I've got an internal 1TB SSD and it is more than enough for most of my needs. > I mainly use USB for thumbsticks and my camera. In such cases, the > throughput is bottlenecked on the attached device, not the bus. > > That being said, my server machine is a 2012 MacMini, and I've got a > DroboMini and a LaCie RAID attached to it via thunderbolt, and the throughput > and latency are more than adequate for my demanding needs. Thunderbolt external disks and 1TB SSDs are simply out of my budget. > For folks like you and me, it might be a minor concern being limited to 1TB, > but we can certainly use an external SSD or an SDXC to increase storage as > needed in the future. I see it as more than a minor concern. Not because I expect I'll ever need more than 1TB internal space, but I would certainly not feel comfortable when I know I cannot replace something that can fail that easily and unpredictable as an SSD (or HDD for that matter). I don't recall but it's quite likely too that I replaced the HDD in my current MBP with a Hitachi as soon as I received the machine. My only experience to date with SSDs was as an external. Connected over FW800 (the fastest bus I had at the time) showed 0 gain over a good HDD ... and the whole thing died catastrophically after not even 4 months. FWIW, that Clevo notebook I mentioned has an internal 2.5" disk (came with an SSHD) but even at its price point it has a fashionable slot to add an SSD. That's the kind of approach I like: adding expansion options instead of removing them. R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
Hi, On Sat, Sep 10, 2016 at 09:14:16AM -0700, Jeremy Huddleston Sequoia wrote: > No, the DYLD_INSERT_LIBRARIES approach is the right one here. > Interested users would need to disable SIP. "Interested users" would be everybody who uses MacPorts. I'd vote against telling all our users to disable SIP. It's a useful security/safety feature and it even helps us because users can no longer mess up their /usr/bin. I don't see why the kernel, dyld, or whoever strips the flags could not just behave like running a copy of the binary at hand when it sees a DYLD variable, i.e. do the workaround we're doing manually at the moment. > It would be nice if a mechanism were in place to determine trust of > certain libraries in DYLD_INSERT_LIBRARIES. So you're suggesting DYLD_INSERT_LIBRARIES on SIP-protected binaries should only work if the inserted library is signed? How would that improve anything? Are you suggesting every open source project out there that uses library preloading now pays for a certificate and regularly builds and releases binaries for macOS? Frankly, I don't see that happening. > Please file radars and point me to them, so I can make sure they get > routed to the right place (likely as dupes, but dupes are very useful > "votes" for bugs). Those tickets have been filed when SIP was introduced and DYLD_INSERT_LIBRARIES stopped working. Evidently, it wasn't important enough to get fixed, so you'll forgive me if I have better things to do with my time. -- Clemens ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
> On Sep 10, 2016, at 05:22, Clemens Lang wrote:
>
> Hi,
>
> On Fri, Sep 09, 2016 at 01:59:50PM -0700, Jeremy Huddleston Sequoia wrote:
>>> As an aside, I'd be in favour of setting up MacPorts such that
>>> ${prefix} is owned by a ${macports_operator} who's got admin rights
>>> (= myself) and reserve use of actual root privilege to those few
>>> ports that require setting up SETUID/GETUID executables or that need
>>> to create users or groups.
>>
>> YES! We should not be needing to do such things as root. That is
>> 100% true, and I am in full support of moving away from that and only
>> using root for activate. We should be able to use fakeroot
>> (https://wiki.debian.org/FakeRoot) for destdir.
>
> Except that fakeroot uses library preloading, a technique that's more or
> less dead on modern OS X thanks to Apple's changes related to SIP:
> DYLD_INSERT_LIBRARIES is stripped for all binaries with the SIP bit set.
> Combine that with every binary in /usr/bin and /bin having the bit, and
> you'll end up the variable being removed on the first invocation of a
> shell (which is basically everywhere in the build systems of our ports).
>
> It can still be done with utter hacks (copying the binary into a file
> without the SIP bit and executing it from there, which we do for trace
> mode), but I have neither seen any other library preloading utility that
> used to work on OS X implement these changes nor convinced any of their
> developers to do so.
>
> Other approaches that would allow simulating permissions, such as Linux'
> user namespaces don't exist on OS X at all. I think it's pretty obvious
> that implementing new code that relies on library preloading is riding a
> dead horse on macOS.
No, the DYLD_INSERT_LIBRARIES approach is the right one here. Interested users
would need to disable SIP. It would be nice if a mechanism were in place to
determine trust of certain libraries in DYLD_INSERT_LIBRARIES. Please file
radars and point me to them, so I can make sure they get routed to the right
place (likely as dupes, but dupes are very useful "votes" for bugs).
Thanks,
Jeremy
smime.p7s
Description: S/MIME cryptographic signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 10, 2016, at 05:09, Rainer Müller wrote: > > On 2016-09-09 22:59, Jeremy Huddleston Sequoia wrote: >> >>> On Sep 9, 2016, at 04:38, René J.V. Bertin wrote: >>> >>> On Friday September 09 2016 12:10:05 Rainer Müller wrote: >>> >>> > different than your case either. Either way, the debugger and all > its dependencies need to be signed by a valid certificate. That does not seem to be the case. In my testing on OS X 10.10 Yosemite, it is enough to sign /opt/local/bin/ggdb with a trusted certificate to get it working. >> >> It requires the ggdb executable and all libraries it links against to be >> signed. The port is written such that it only links against Apple-provided >> executables, so that solves that dependency. > > No? > > $ otool -L /opt/local/bin/ggdb > /opt/local/bin/ggdb: > /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current > version 10.5.0) > /opt/local/lib/libncurses.6.dylib (compatibility version 6.0.0, current > version 6.0.0) > /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current > version 1.2.8) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 1213.0.0) > /opt/local/lib/libiconv.2.dylib (compatibility version 8.0.0, current > version 8.1.0) > /opt/local/lib/libexpat.1.dylib (compatibility version 8.0.0, current > version 8.2.0) > > > On OS X 10.10 Yosemite, signing only the ggdb binary was certainly > enough. I cannot reproduce this on macOS 10.12 Sierra, so > the requirements might have changed. 10.10 predates SIP and related hardening around ptrace(). That version is so far in my rearview that I forget the details there, sorry. I'll have to dig into it, but it certainly seems wrong to me that a process could become privileged if it linked against unsigned libraries. > Also on Sierra it looks like I can no longer give codesign a > certificate, which is not known and trusted to the system. > > Both of these facts would destroy my idea of signing with a self-signed > certificate, but requiring the user to add trust on the certificate. > > Rainer > ___ > macports-dev mailing list > macports-dev@lists.macosforge.org > https://lists.macosforge.org/mailman/listinfo/macports-dev smime.p7s Description: S/MIME cryptographic signature ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 10, 2016, at 02:15, René J.V. Bertin wrote:
>
> On Friday September 09 2016 13:59:50 Jeremy Huddleston Sequoia wrote:
>
>>> As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
>>> owned by a ${macports_operator} who's got admin rights (= myself) and
>>> reserve use of actual root privilege to those few ports that require
>>> setting up SETUID/GETUID executables or that need to create users or groups.
>>
>> YES! We should not be needing to do such things as root. That is 100%
>> true, and I am in full support of moving away from that and only using root
>> for activate. We should be able to use fakeroot
>> (https://wiki.debian.org/FakeRoot) for destdir.
>
> Why would we even require root for activation, except for the few exceptions
> that install items outside of ${prefix} or that install SETUID/GUID to
> another user/group?
Exactly, so we'll need root access at activation time for such things.
Yes, for most cases we would probably not need root access at activation time,
but for those cases you just listed, we would. We could do a pass over the bom
to determine if root access is needed or not. The UI for handling such a case
is non-trivial. Do we prompt once for the entire run of the process or
escalate permissions for each port we encounter and then drop them again? If 8
ports are being installed and 4 require root, that's not a good user experience
to prompt for password 4 times.
> It's already an implicit requirement that the MacPorts operator (the user who
> installs it and ports) be an admin user, and once ${prefix} is created
> there's no need for it and anything below it to require root access.
>
> I've run MacPorts for years like that, and only moved away from it very
> recently because I made an error (forgot a ${destroot}) that caused pollution
> of my ${prefix}. Of course the protection I gained with that move is very
> relative, and depends on the destroot step NOT being run as root.
>
> Would fakeroot work on OS X, including on versions that predate SIP/rootless?
There's absolutely nothing preventing something like that from working
technically. All it's doing is interposing FS syscalls like stat(), chown(),
chmod(), and storing that information as a shadow. The limitation that I see
is with executables that are restricted from using DYLD_INSERT_LIBRARIES for
the interposition (ie: the same problem we have with darwintrace). We would
need to use our own versions of cp, mv, chmod, chown, stat, install, etc
command line utilities instead of the system binaries.
> Funny btw, I trust Debian to have written a safe fakeroot implementation, but
> if you read the wiki you get the impression it's a dangerous little hacking
> tool, which could be misused easily e.g. to make any executable setuid root...
Such executables still need to be installed through real root access in order
to become "really" setuid root.
>> It's quite a bit more complicated than that. First off, these settings are
>> on by default but can be configured through SIP flags, boot args, etc.
>> There >are also many types of restrictions that have different effects.
>
> Hah, TMI :)
>
>> Because of the CS_HARD restriction, all libraries that are linked against
>> require a valid code signature.
>
> Out of curiosity, if an IDE were to use a proper lldb debugger implementation
> that uses liblldb rather than an existing external driver (lldb-mi, python,
> ...), will all of the IDE have to be signed or is that still a requirement
> only for the debugserver utility?
The process doing the debugging (the one that is calling ptrace(2)) is the one
that has these restrictions, ie: debugserver.
>> This is because you likely already launched the executable, so the old
>> signature for that particular inode was already cached. If you copied
>> debugserver somewhere else and then copied it back, it would have addressed
>> the problem for you.
>
> Presumable, but that's the point, it didn't. I tried that manually, but
> shouldn't reinstalling via MacPorts have taken care of that too? Because that
> does
> - delete the previous copy
> - install a new, unsigned copy
> - sign the new copy
>
> I understand that the certificate catching is coupled to the file's inode
> (whatever that is under HFS+), and the new copy indeed had a new inode. And
> yet another inode after signing it.
If you can reproduce this, please let me know and file a radar with details and
let me know the number, so I can followup internally on it. I haven't seen
such cases. The only issue I'm aware of is with overwriting an existing file
(eg: using cp instead of mv).
>>> I'm concerned about every step that takes OS X away from a regular Unix
>>> (underneath a nice and truly integrated desktop) and towards a locked-in OS
>>> like iOS.'
>>
>> Well macOS is still UNIX. We continue to verify that through continual
>> conformance testing. I don't expect that to ever
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Saturday September 10 2016 15:56:54 Clemens Lang wrote: > cat and rm deal with files, consequently trace mode needs its code to be > injected into them (as would fakeroot). Not copying them would mean > their unmodified copies run without the "sandbox" enabled, since > DYLD_INSERT_LIBRARIES wouldn't have any effect on them. Right, I didn't think this out far enough. > It's kept, but how would you do that? You cannot inject code into that > executable, and a port's build system specifies what shell scripts (for > example) get run. Not that it's going to be of much help, but shells and interpreters usually can read files that set environment variables. R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
Hi, On Sat, Sep 10, 2016 at 03:45:25PM +0200, René J.V. Bertin wrote: > But I see quite a few items on there that are unlikely to call other > applications. If this is only about preserving DYLD_LIBRARY_PRELOAD, > why do you need to treat utilities like cat or rm? cat and rm deal with files, consequently trace mode needs its code to be injected into them (as would fakeroot). Not copying them would mean their unmodified copies run without the "sandbox" enabled, since DYLD_INSERT_LIBRARIES wouldn't have any effect on them. > And what happens when you (re)set one of the tainted env. variables in > a shell or interpreter with the SIP bit set? Is it unset or filtered > out when you call another executable, even if that exec doesn't have > the SIP bit set? It's kept, but how would you do that? You cannot inject code into that executable, and a port's build system specifies what shell scripts (for example) get run. -- Clemens ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Saturday September 10 2016 15:15:54 Clemens Lang wrote: Hi, >There are no other solutions that I'm aware of on OS X. We discussed Sorry, I meant for Apple to develop something like fakeroot from the ground up. >FYI, I'm including a list of the files that are currently affected on my >system by the last 200-or-so runs of MacPorts at the end of this mail. Right ... >additional effort to support macOS for little gain. It's no secret that >macOS isn't the platform for Unix devs that it used to be. Well, what we call "bad tongues" in French could say that MacOS never was a platform for Unix development ;) >Here's the list: I'll know where to point the next person who wonders why disabling SIP will be the 1st thing I'll do after I've finally upgraded from 10.9 But I see quite a few items on there that are unlikely to call other applications. If this is only about preserving DYLD_LIBRARY_PRELOAD, why do you need to treat utilities like cat or rm? And what happens when you (re)set one of the tainted env. variables in a shell or interpreter with the SIP bit set? Is it unset or filtered out when you call another executable, even if that exec doesn't have the SIP bit set? R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: Fakeroot destrooting [Was: Re: lldb ...]
Hi, On Sat, Sep 10, 2016 at 02:42:36PM +0200, René J.V. Bertin wrote: > >Except that fakeroot uses library preloading, a technique that's more > >or less dead on modern OS X thanks to Apple's changes related to SIP: > >DYLD_INSERT_LIBRARIES is stripped for all binaries with the SIP bit > >set. > > Fakeroot uses library preloading on Linux, but that doesn't mean other > solutions aren't possible. Evidently that would require an official > Apple fakeroot implementation. There are no other solutions that I'm aware of on OS X. We discussed using the sandboxing mechanism, but it doesn't support our use case. Short of writing a kernel module or writing your own loader, there are no other options. > > It can still be done with utter hacks (copying the binary into a > > file without the SIP bit and executing it from there, which we do > > for trace mode), > > Do you mean something along the lines of `cat /bin/sh > /tmp/sh`? If > so, why not simply use port:bash for trace mode? Doing this only for the Shell isn't enough. Next affected tool would be make(1), then install(1), cp(1), sandbox-exec(1), gzip(1), etc. pp. Just FYI, I'm including a list of the files that are currently affected on my system by the last 200-or-so runs of MacPorts at the end of this mail. > BTW, how do you get calls like system() to use something other than > /bin/sh ? Hook execve(2) and posix_spawn(2) to transparently run a different binary. That requires parsing the shebang line, too. > > but I have neither seen any other library preloading utility that > > used to work on OS X implement these changes nor convinced any of > > their developers to do so. > > They simpy advise to disable SIP, then? They simply stopped caring. For most of them, it has always been additional effort to support macOS for little gain. It's no secret that macOS isn't the platform for Unix devs that it used to be. Here's the list: . ├── System │ └── Library │ └── Frameworks │ ├── Python.framework │ │ └── Versions │ │ ├── 2.6 │ │ │ └── Resources │ │ │ └── Python.app │ │ │ └── Contents │ │ │ └── MacOS │ │ │ └── Python │ │ └── 2.7 │ │ └── Resources │ │ └── Python.app │ │ └── Contents │ │ └── MacOS │ │ └── Python │ └── Ruby.framework │ └── Versions │ └── Current │ └── usr │ └── bin │ └── ruby ├── bin │ ├── bash │ ├── cat │ ├── chmod │ ├── cp │ ├── csh │ ├── date │ ├── dd │ ├── echo │ ├── ed │ ├── expr │ ├── hostname │ ├── ksh │ ├── launchctl │ ├── ln │ ├── ls │ ├── mkdir │ ├── mv │ ├── ps │ ├── pwd │ ├── rm │ ├── rmdir │ ├── sh │ ├── sleep │ └── test ├── sbin │ └── md5 └── usr ├── bin │ ├── Rez │ ├── SetFile │ ├── ar │ ├── arch │ ├── awk │ ├── basename │ ├── bc │ ├── bison │ ├── bzip2 │ ├── c++ │ ├── c++filt │ ├── cc │ ├── clang │ ├── clang++ │ ├── cmp │ ├── codesign │ ├── codesign_allocate │ ├── comm │ ├── cpio │ ├── cpp │ ├── ctags │ ├── curl │ ├── cut │ ├── diff │ ├── dirname │ ├── dsymutil │ ├── egrep │ ├── env │ ├── etags │ ├── expand │ ├── false │ ├── fgrep │ ├── file │ ├── find │ ├── flex │ ├── g++ │ ├── gcc │ ├── getconf │ ├── git │ ├── gm4 │ ├── gnumake │ ├── gperf │ ├── grep │ ├── groff │ ├── grotty │ ├── gzip │ ├── head │ ├── hiutil │ ├── hostinfo │ ├── ibtool │ ├── id │ ├── indent │ ├── install │ ├── install_name_tool │ ├── jar │ ├── java │ ├── javac │ ├── javadoc │ ├── join │ ├── ld │ ├── less │ ├── libtool │ ├── lipo │ ├── llvm-gcc │ ├── locale │ ├── logname │ ├── m4 │ ├── machine │ ├── make │ ├── makeinfo │ ├── man │ ├── mig │ ├── mkfifo │ ├── mktemp │ ├── nm │ ├── nmedit │ ├── od │ ├── otool │ ├── patch │ ├── perl │ ├── perl5.18 │ ├── pr │ ├── python │ ├── python2.6 │ ├── python2.7 │ ├── ranlib │ ├── readlink │ ├── rpcgen │ ├── rsync │ ├── ruby │ ├── sandbox-exec │ ├── sed │ ├── size │ ├── sort │ ├── split │ ├── stat │ ├── strings │ ├── strip │ ├── svn │ ├── svnversion │ ├── sw_vers │ ├── tail │ ├── tar │ ├── tbl │ ├── tclsh
Re: Fakeroot destrooting [Was: Re: lldb ...]
On Saturday September 10 2016 14:22:16 Clemens Lang wrote: Hi, >Except that fakeroot uses library preloading, a technique that's more or >less dead on modern OS X thanks to Apple's changes related to SIP: >DYLD_INSERT_LIBRARIES is stripped for all binaries with the SIP bit set. Fakeroot uses library preloading on Linux, but that doesn't mean other solutions aren't possible. Evidently that would require an official Apple fakeroot implementation. >It can still be done with utter hacks (copying the binary into a file >without the SIP bit and executing it from there, which we do for trace >mode), Do you mean something along the lines of `cat /bin/sh > /tmp/sh`? If so, why not simply use port:bash for trace mode? BTW, how do you get calls like system() to use something other than /bin/sh ? >but I have neither seen any other library preloading utility that >used to work on OS X implement these changes nor convinced any of their >developers to do so. They simpy advise to disable SIP, then? R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Fakeroot destrooting [Was: Re: lldb ...]
Hi,
On Fri, Sep 09, 2016 at 01:59:50PM -0700, Jeremy Huddleston Sequoia wrote:
> > As an aside, I'd be in favour of setting up MacPorts such that
> > ${prefix} is owned by a ${macports_operator} who's got admin rights
> > (= myself) and reserve use of actual root privilege to those few
> > ports that require setting up SETUID/GETUID executables or that need
> > to create users or groups.
>
> YES! We should not be needing to do such things as root. That is
> 100% true, and I am in full support of moving away from that and only
> using root for activate. We should be able to use fakeroot
> (https://wiki.debian.org/FakeRoot) for destdir.
Except that fakeroot uses library preloading, a technique that's more or
less dead on modern OS X thanks to Apple's changes related to SIP:
DYLD_INSERT_LIBRARIES is stripped for all binaries with the SIP bit set.
Combine that with every binary in /usr/bin and /bin having the bit, and
you'll end up the variable being removed on the first invocation of a
shell (which is basically everywhere in the build systems of our ports).
It can still be done with utter hacks (copying the binary into a file
without the SIP bit and executing it from there, which we do for trace
mode), but I have neither seen any other library preloading utility that
used to work on OS X implement these changes nor convinced any of their
developers to do so.
Other approaches that would allow simulating permissions, such as Linux'
user namespaces don't exist on OS X at all. I think it's pretty obvious
that implementing new code that relies on library preloading is riding a
dead horse on macOS.
--
Clemens
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-09 22:59, Jeremy Huddleston Sequoia wrote: > >> On Sep 9, 2016, at 04:38, René J.V. Bertin wrote: >> >> On Friday September 09 2016 12:10:05 Rainer Müller wrote: >> >> different than your case either. Either way, the debugger and all its dependencies need to be signed by a valid certificate. >>> >>> That does not seem to be the case. In my testing on OS X 10.10 Yosemite, >>> it is enough to sign /opt/local/bin/ggdb with a trusted certificate to >>> get it working. > > It requires the ggdb executable and all libraries it links against to be > signed. The port is written such that it only links against Apple-provided > executables, so that solves that dependency. No? $ otool -L /opt/local/bin/ggdb /opt/local/bin/ggdb: /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.5.0) /opt/local/lib/libncurses.6.dylib (compatibility version 6.0.0, current version 6.0.0) /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0) /opt/local/lib/libiconv.2.dylib (compatibility version 8.0.0, current version 8.1.0) /opt/local/lib/libexpat.1.dylib (compatibility version 8.0.0, current version 8.2.0) On OS X 10.10 Yosemite, signing only the ggdb binary was certainly enough. I cannot reproduce this on macOS 10.12 Sierra, so the requirements might have changed. Also on Sierra it looks like I can no longer give codesign a certificate, which is not known and trusted to the system. Both of these facts would destroy my idea of signing with a self-signed certificate, but requiring the user to add trust on the certificate. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-10 00:23, Jeremy Sequoia wrote: >>> YES! We should not be needing to do such things as root. That is >>> 100% true, and I am in full support of moving away from that and only >>> using root for activate. We should be able to use fakeroot >>> (https://wiki.debian.org/FakeRoot) for destdir. +1 > It isn't that complicated. It should be fairly easy for someone to > write the moral equivalent of it under a virus-free license. I'd also > be happy to sponsor/advise anyone interested in doing it for GSoC or > self-ed. I added this to the list of ideas, in case we want to do GSoC again next year. https://trac.macports.org/wiki/SummerOfCode#fakeroot Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Friday September 09 2016 13:59:50 Jeremy Huddleston Sequoia wrote:
>> As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
>> owned by a ${macports_operator} who's got admin rights (= myself) and
>> reserve use of actual root privilege to those few ports that require setting
>> up SETUID/GETUID executables or that need to create users or groups.
>
>YES! We should not be needing to do such things as root. That is 100% true,
>and I am in full support of moving away from that and only using root for
>activate. We should be able to use fakeroot
>(https://wiki.debian.org/FakeRoot) for destdir.
Why would we even require root for activation, except for the few exceptions
that install items outside of ${prefix} or that install SETUID/GUID to another
user/group? It's already an implicit requirement that the MacPorts operator
(the user who installs it and ports) be an admin user, and once ${prefix} is
created there's no need for it and anything below it to require root access.
I've run MacPorts for years like that, and only moved away from it very
recently because I made an error (forgot a ${destroot}) that caused pollution
of my ${prefix}. Of course the protection I gained with that move is very
relative, and depends on the destroot step NOT being run as root.
Would fakeroot work on OS X, including on versions that predate SIP/rootless?
Funny btw, I trust Debian to have written a safe fakeroot implementation, but
if you read the wiki you get the impression it's a dangerous little hacking
tool, which could be misused easily e.g. to make any executable setuid root...
>It's quite a bit more complicated than that. First off, these settings are on
>by default but can be configured through SIP flags, boot args, etc. There
>>are also many types of restrictions that have different effects.
Hah, TMI :)
>Because of the CS_HARD restriction, all libraries that are linked against
>require a valid code signature.
Out of curiosity, if an IDE were to use a proper lldb debugger implementation
that uses liblldb rather than an existing external driver (lldb-mi, python,
...), will all of the IDE have to be signed or is that still a requirement only
for the debugserver utility?
>This is because you likely already launched the executable, so the old
>signature for that particular inode was already cached. If you copied
>debugserver somewhere else and then copied it back, it would have addressed
>the problem for you.
Presumable, but that's the point, it didn't. I tried that manually, but
shouldn't reinstalling via MacPorts have taken care of that too? Because that
does
- delete the previous copy
- install a new, unsigned copy
- sign the new copy
I understand that the certificate catching is coupled to the file's inode
(whatever that is under HFS+), and the new copy indeed had a new inode. And yet
another inode after signing it.
>> I'm concerned about every step that takes OS X away from a regular Unix
>> (underneath a nice and truly integrated desktop) and towards a locked-in OS
>> like iOS.'
>
>Well macOS is still UNIX. We continue to verify that through continual
>conformance testing. I don't expect that to ever change.
iOS used to be a Unix too. I don't know whether one can still think of it as
that, though. But anyway that wasn't my real point. For me, "regular Unix"
carries connotations and associations of an open developers' OS of choice that
date back to the late 80s.
Something "breaks" in that perception of things when we get to the point where
you have to ask (or pay) for a certificate to install and run even your own
code even if it stays away from system areas. That's exactly why I never got
into iOS development, too.
>FWIW, I really love my 2015 rMBP. I was a holdout staying on the pre-retina
>ones so I could continue to have a DVD drive and a 2.5" drive bay, but I
>finally gave that up and am really glad that I did because the newer SSDs are
>blazing fast.
Going completely OT here :)
I guess that if I had the money I might be less sceptic, but the fact is that I
simply cannot afford to dump the amount required to replace my current "mobile
workstation" mid 2011 13" MBP with something comparable from Apple (or anyone
else if I want to stick to an i7). It's got a 1Tb Hitachi HDD which is plenty
fast for me (and cost me all of 80€), evolved from 4 to 8 to 12Gb RAM as prices
dropped and lacks an expensive LCD screen that won't outlive the computer
itself. I've got a 1080p 21" external screen that is largely sufficient. When I
go mobile I either can make do with the low internal resolution, or I have
another external screen.
That large disk allowed me to make 4 partitions and keep large working
directories like Qt5's source+build+destroot around without running out of
space. I wouldn't feel comfortable at all with a non-replaceable SSD of a
comparable size knowing its cost and that rapidly evolving tech is almost by
definitio
Re: lldb ...
Sent from my iPhone...
On Sep 9, 2016, at 14:46, Lawrence Velázquez wrote:
>> On Sep 9, 2016, at 4:59 PM, Jeremy Huddleston Sequoia
>> wrote:
>>
>>> On Sep 9, 2016, at 04:38, René J.V. Bertin wrote:
>>>
>>> As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
>>> owned by a ${macports_operator} who's got admin rights (= myself) and
>>> reserve use of actual root privilege to those few ports that require
>>> setting up SETUID/GETUID executables or that need to create users or groups.
>>
>> YES! We should not be needing to do such things as root. That is 100%
>> true, and I am in full support of moving away from that and only using root
>> for activate. We should be able to use fakeroot
>> (https://wiki.debian.org/FakeRoot) for destdir.
>
> Isn't fakeroot GPL-licensed? We wouldn't be able to distribute it with base.
It isn't that complicated. It should be fairly easy for someone to write the
moral equivalent of it under a virus-free license. I'd also be happy to
sponsor/advise anyone interested in doing it for GSoC or self-ed.
--Jeremy
>
> vq
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 4:59 PM, Jeremy Huddleston Sequoia
> wrote:
>
>> On Sep 9, 2016, at 04:38, René J.V. Bertin wrote:
>>
>> As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
>> owned by a ${macports_operator} who's got admin rights (= myself) and
>> reserve use of actual root privilege to those few ports that require setting
>> up SETUID/GETUID executables or that need to create users or groups.
>
> YES! We should not be needing to do such things as root. That is 100% true,
> and I am in full support of moving away from that and only using root for
> activate. We should be able to use fakeroot
> (https://wiki.debian.org/FakeRoot) for destdir.
Isn't fakeroot GPL-licensed? We wouldn't be able to distribute it with base.
vq
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 04:38, René J.V. Bertin wrote:
>
> On Friday September 09 2016 12:10:05 Rainer Müller wrote:
>
>
>>> different than your case either. Either way, the debugger and all
>>> its dependencies need to be signed by a valid certificate.
>>
>> That does not seem to be the case. In my testing on OS X 10.10 Yosemite,
>> it is enough to sign /opt/local/bin/ggdb with a trusted certificate to
>> get it working.
It requires the ggdb executable and all libraries it links against to be
signed. The port is written such that it only links against Apple-provided
executables, so that solves that dependency.
> And lldb requires only the debugserver executable to be signed.
Again, because debugserver uses entitlements, all its dependencies need to be
signed as well.
$ otool -L /opt/local/libexec/llvm-devel/bin/debugserver
/opt/local/libexec/llvm-devel/bin/debugserver:
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
(compatibility version 1.0.0, current version 22.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
(compatibility version 150.0.0, current version 1348.1.0)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
(compatibility version 300.0.0, current version 1349.0.0)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version
228.0.0)
As you can see, all of its dependencies are system libraries, so they're all
valid.
>> Did this change with El Capitan or Sierra?
>
> I'm guessing that would somehow be evident from the lldb-3.9 CMake files and
> build instructions...
>> At least on OS X 10.10 Yosemite, I can use any path to a keychain with
>> `codesign --keychain`. This keychain does not have to be listed in
>> `security list-keychains`.
>
> Does `man codesign` still mention the search list requirement in the
> documentation for --keychain?
Yes, that is still a requirement.
>
> On Friday September 09 2016 02:26:19 Jeremy Huddleston Sequoia wrote:
>
I'd hate that because I've set things up with macports_user=myself and
everything under workpath owned by myself, to avoid constant need for
UID changing. I kind of doubt I'm the only one maintaining lots of
ports who does that.> >
>>> I don't understand. Can you elaborate? Doesn't post_destroot{} run as
>>> root in order to setup desired permissions in the destroot? That's a
>>> good place to handle the resigning.
>
> I do a lot of development on projects for which I also maintain ports; in
> fact, development that's tightly coupled to the fact the code is available
> via a port. Doing all that work "as myself" only to transfer it to the port
> afterwards is a hassle and waste of time as far as I'm concerned. It's not so
> much that I want to avoid having to put sudo in front of each and every port
> command (I'd write a `sport` wrapper/alias). I don't want to grow the habit
> of putting sudo in front of everything I'm doing around those ports' build
> directories and that I'd also be doing in development that's got nothing to
> do with MacPorts. I also want to be able to open a port's source tree in an
> IDE, run incremental builds in subdirs for ${build.dir}, etc. So I've set my
> macportsuser to my own username, and most of the time I run everything up to
> and including `port destroot` as myself (and set up symlinks to `port work`
> for easy access). Typically, for -devel ports that use fetch.git I also
> replace ${worksrcdir} with a symlink to the working copy under my own home
> directory (that way I also won't lose local changes if I ever forget to use
> -o or -k).
> Many of the contributions I've made to KDE over the past few years were
> developed with this approach. I've used the standard approach (macportsuser =
> macports) on a VM Bradley gave me access to, and it's definitely a lot less
> productive.
> Evidently this is not the kind of advanced use we need to consider for
> regular users, but port developers/maintainers are users too, and I think we
> *could* consider their needs too.
>
> As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
> owned by a ${macports_operator} who's got admin rights (= myself) and reserve
> use of actual root privilege to those few ports that require setting up
> SETUID/GETUID executables or that need to create users or groups.
YES! We should not be needing to do such things as root. That is 100% true,
and I am in full support of moving away from that and only using root for
activate. We should be able to use fakeroot (https://wiki.debian.org/FakeRoot)
for destdir.
> MacPorts installs into a parallel prefix, and there are only few ports that
> require access to system directories.
>
>>> That's because you're using su
Re: lldb ...
On Friday September 09 2016 12:12:34 Lawrence Velázquez wrote: > I don't know why that would happen, unless drastic changes are released > (which does happen sometimes). It's not inherent to base; I update from trunk > all the time with no problems. Yeah, well. I like stability in things that form the basis for stuff that might be in flux all the time (or not). And base is named appropriately :) Also explains why I still haven't gotten myself to upgrade from 10.9 ... > Any code pushed to the repository goes out to all users immediately. This is > nice for certain applications but not others. The whole point of trunk base > is to hit the middle ground of getting certain changes field-tested by other > devs quickly without affecting users. > > If you're suggesting that we test the code by sending around patches outside > source control... well, that sounds like an antipattern to me. The point remains that we may want to push this functionality out as soon as it's ready to be tested in the wild, at least as far as selective signing of specific applications is concerned. Otherwise a port like lldb will have to wait until base is updated, or it'd have to include a "preview copy" of the signing code. R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 10:28 AM, René J.V. Bertin wrote: > >> On Friday September 09 2016 09:27:31 Lawrence Velázquez wrote: >> >> If our release process is too cumbersome and infrequent, we should change >> that. I don't see reason to divide base's functionality more than it already >> is. > > In my perception, "base" updates usually come with a lot of updating > installed ports and quite the share of stress if everything is going to work > like before. There should indeed be no need for that; it should usually be > possible just to update a file or two, at least as far as the Tcl library is > concerned. I don't know why that would happen, unless drastic changes are released (which does happen sometimes). It's not inherent to base; I update from trunk all the time with no problems. >> And I would definitely not want any security-related functionality to be >> implemented in a portgroup, which is immediately pushed to all users. > > Who said everything about immediately, before it's well tested? The only > thing that I'm taking into consideration is the fact that we may not hit the > ideal implementation immediately that has all the required functionality and > flexibility. Any code pushed to the repository goes out to all users immediately. This is nice for certain applications but not others. The whole point of trunk base is to hit the middle ground of getting certain changes field-tested by other devs quickly without affecting users. If you're suggesting that we test the code by sending around patches outside source control... well, that sounds like an antipattern to me. >> Apple provides developer certificates that can be used for signatures. > > Yes they do. But usually that's between Apple and the developer who pays to > sign his/her software. I do wonder whether there would be legal considerations. > In MacPorts it means that basically anyone with commit access can start using > that certificate for free. Sort of. I expect portmgr would be responsible for it. They already delegate responsibility for the integrity of our software via commit privileges, and committers already piggyback on their reputation when archives get signed and distributed from Buildbot. vq ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Friday September 09 2016 09:27:31 Lawrence Velázquez wrote: >> I know there's been talk about making at least parts of "base" updateable as >> a port; this could be an easy alternative. > >If our release process is too cumbersome and infrequent, we should change >that. I don't see reason to divide base's functionality more than it already >is. In my perception, "base" updates usually come with a lot of updating installed ports and quite the share of stress if everything is going to work like before. There should indeed be no need for that; it should usually be possible just to update a file or two, at least as far as the Tcl library is concerned. >And I would definitely not want any security-related functionality to be >implemented in a portgroup, which is immediately pushed to all users. Who said everything about immediately, before it's well tested? The only thing that I'm taking into consideration is the fact that we may not hit the ideal implementation immediately that has all the required functionality and flexibility. > We have no evidence either way, so appealing to an invisible mass of > developers is not a convincing line of argument. It would just be a pity if that evidence starts coming in when people start complaining about things that are no longer as feasible as before. But I know I've probably raised an issue that's going to oblige me to start jumping through all kinds of additional hoops . > Apple provides developer certificates that can be used for signatures. Yes they do. But usually that's between Apple and the developer who pays to sign his/her software. In MacPorts it means that basically anyone with commit access can start using that certificate for free. And yes, revoking was exactly the risk I was thinking about. Either way, I'm just thinking aloud. If it's all FUD, so much the better. R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 6:00 AM, Rainer Müller wrote: > >> On 2016-09-09 03:38, Jeremy Sequoia wrote: >> You are describing a system to automatically create and automatically >> trust a code signing certificate that contains a private key in a file >> on disk that is not encrypted with a passphrase and only depends on file >> system ACLs to protect it. >> >> That is trivially insecure and attack-prone. > > Adding trust for an additional certificate also only relies on the > filesystem permissions. If I get root write access to System.keychain, I > can add my own certificate. > > If we add a trusted certificate to System.keychain and the corresponding > private key is only accessible by root, that would still be the same > level of security in my opinion. If the private key isn't encrypted, doesn't that basically eliminate the cryptographic security? At that point we're just relying on the OS to maintain access control. vq Sent from my iPhone ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 7:38 AM, René J.V. Bertin wrote:
>
> I do a lot of development on projects for which I also maintain ports; in
> fact, development that's tightly coupled to the fact the code is available
> via a port. Doing all that work "as myself" only to transfer it to the port
> afterwards is a hassle and waste of time as far as I'm concerned. It's not so
> much that I want to avoid having to put sudo in front of each and every port
> command (I'd write a `sport` wrapper/alias). I don't want to grow the habit
> of putting sudo in front of everything I'm doing around those ports' build
> directories and that I'd also be doing in development that's got nothing to
> do with MacPorts. I also want to be able to open a port's source tree in an
> IDE, run incremental builds in subdirs for ${build.dir}, etc. So I've set my
> macportsuser to my own username, and most of the time I run everything up to
> and including `port destroot` as myself (and set up symlinks to `port work`
> for easy access). Typically, for -devel ports that use fetch.git I also
> replace ${worksrcdir} with a symlink to the working copy under my own home
> directory (that way I also won't lose local changes if I ever forget to use
> -o or -k).
> Many of the contributions I've made to KDE over the past few years were
> developed with this approach. I've used the standard approach (macportsuser =
> macports) on a VM Bradley gave me access to, and it's definitely a lot less
> productive.
> Evidently this is not the kind of advanced use we need to consider for
> regular users, but port developers/maintainers are users too, and I think we
> *could* consider their needs too.
Meh. There really aren't that many port developers, and I bet a lot fewer of
them set macportsuser than you think.
We have no evidence either way, so appealing to an invisible mass of developers
is not a convincing line of argument.
> As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
> owned by a ${macports_operator} who's got admin rights (= myself) and reserve
> use of actual root privilege to those few ports that require setting up
> SETUID/GETUID executables or that need to create users or groups. MacPorts
> installs into a parallel prefix, and there are only few ports that require
> access to system directories.
We consider the fact that we install as root to be a security feature. One can
already install MacPorts as a non-root user if they want to.
Isn't that a bit of a special case, with you certainly having Apple's
benediction to work on that particular product?> >
>>> XQuartz isn't any more blessed in that respect than MacPorts.
>
> And the fact you're an Apple's payroll doesn't have anything to do with it
> either? Currently Apple doesn't have a say in the admission and upgrade of
> each and every port in MacPorts, do they? In a way I'd even hope they cannot
> force the exclusion of a port (other than ones clearly doing unlawful
> things); signing everything with an official key seems like a way to give
> them (more) leverage to control over the ports tree.
What? Apple provides developer certificates that can be used for signatures.
Jeremy is suggesting we take advantage of this. Third-party developers do this
all the time. Using a certificate doesn't give Apple review power over anybody
(other than revocation in case of abuse).
vq
Sent from my iPhone
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 4:17 AM, René J.V. Bertin wrote: > > As a side-thought: it shouldn't be particularly difficult to implement a > "base" feature which defines a set of PortGroups to be included by default by > every port, if such a thing doesn't already exist. As Josh has said previously, this is called "base". > I know there's been talk about making at least parts of "base" updateable as > a port; this could be an easy alternative. If our release process is too cumbersome and infrequent, we should change that. I don't see reason to divide base's functionality more than it already is. And I would definitely not want any security-related functionality to be implemented in a portgroup, which is immediately pushed to all users. vq Sent from my iPhone ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Friday September 09 2016 14:25:53 Rainer Müller wrote: >I think you misunderstand this sentence. codesign will search in this >keychain for the certificate itself, but the keychain will not be used >to find other certificates in the certificate chain (all intermediates >up to a root CA). With a self-signed certificate, there is no additional >certificate in the chain. In my experience the security framework simply doesn't find keychains that haven't been added to the search list, not even if you just created that keychain via the Security framework itself (as opposed to installing a pre-existing keychain file from some source). R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-09 13:38, René J.V. Bertin wrote: >> At least on OS X 10.10 Yosemite, I can use any path to a keychain with >> `codesign --keychain`. This keychain does not have to be listed in >> `security list-keychains`. > > Does `man codesign` still mention the search list requirement in the > documentation for --keychain? >From the man page: """ --keychain filename ... Note that _filename_ will not be searched to resolve the signing identity's certificate chain unless it is also on the user's keychain search list. """ I think you misunderstand this sentence. codesign will search in this keychain for the certificate itself, but the keychain will not be used to find other certificates in the certificate chain (all intermediates up to a root CA). With a self-signed certificate, there is no additional certificate in the chain. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Friday September 09 2016 12:10:05 Rainer Müller wrote:
> > different than your case either. Either way, the debugger and all
> > its dependencies need to be signed by a valid certificate.
>
> That does not seem to be the case. In my testing on OS X 10.10 Yosemite,
> it is enough to sign /opt/local/bin/ggdb with a trusted certificate to
> get it working.
And lldb requires only the debugserver executable to be signed.
> Did this change with El Capitan or Sierra?
I'm guessing that would somehow be evident from the lldb-3.9 CMake files and
build instructions...
> At least on OS X 10.10 Yosemite, I can use any path to a keychain with
> `codesign --keychain`. This keychain does not have to be listed in
> `security list-keychains`.
Does `man codesign` still mention the search list requirement in the
documentation for --keychain?
On Friday September 09 2016 02:26:19 Jeremy Huddleston Sequoia wrote:
> > > I'd hate that because I've set things up with macports_user=myself and
> > > everything under workpath owned by myself, to avoid constant need for
> > > UID changing. I kind of doubt I'm the only one maintaining lots of
> > > ports who does that.> >
> > I don't understand. Can you elaborate? Doesn't post_destroot{} run as
> > root in order to setup desired permissions in the destroot? That's a
> > good place to handle the resigning.
I do a lot of development on projects for which I also maintain ports; in fact,
development that's tightly coupled to the fact the code is available via a
port. Doing all that work "as myself" only to transfer it to the port
afterwards is a hassle and waste of time as far as I'm concerned. It's not so
much that I want to avoid having to put sudo in front of each and every port
command (I'd write a `sport` wrapper/alias). I don't want to grow the habit of
putting sudo in front of everything I'm doing around those ports' build
directories and that I'd also be doing in development that's got nothing to do
with MacPorts. I also want to be able to open a port's source tree in an IDE,
run incremental builds in subdirs for ${build.dir}, etc. So I've set my
macportsuser to my own username, and most of the time I run everything up to
and including `port destroot` as myself (and set up symlinks to `port work` for
easy access). Typically, for -devel ports that use fetch.git I also replace
${worksrcdir} with a symlink to the working copy under my own home directory
(that way I also won't lose local changes if I ever forget to use -o or -k).
Many of the contributions I've made to KDE over the past few years were
developed with this approach. I've used the standard approach (macportsuser =
macports) on a VM Bradley gave me access to, and it's definitely a lot less
productive.
Evidently this is not the kind of advanced use we need to consider for regular
users, but port developers/maintainers are users too, and I think we *could*
consider their needs too.
As an aside, I'd be in favour of setting up MacPorts such that ${prefix} is
owned by a ${macports_operator} who's got admin rights (= myself) and reserve
use of actual root privilege to those few ports that require setting up
SETUID/GETUID executables or that need to create users or groups. MacPorts
installs into a parallel prefix, and there are only few ports that require
access to system directories.
> > That's because you're using sudo, which doesn't change the bootstrap
> > session. You likely want to do something like:
> > sudo launchctl asuser `id -u joebob` sudo -u joebob codesign ...
> > which requires launchd2 (OS X 10.10+)
>From what I can tell the Security framework (and the security and codesign
>execs) work from a remote (SSH) connection too. Indeed, sudo doesn't by
>default change HOME (there's the -H option for that) ... but that's basically
>what I'm arguing.
> > > Isn't that a bit of a special case, with you certainly having Apple's
> > > benediction to work on that particular product?> >
> > XQuartz isn't any more blessed in that respect than MacPorts.
And the fact you're an Apple's payroll doesn't have anything to do with it
either? Currently Apple doesn't have a say in the admission and upgrade of each
and every port in MacPorts, do they? In a way I'd even hope they cannot force
the exclusion of a port (other than ones clearly doing unlawful things);
signing everything with an official key seems like a way to give them (more)
leverage to control over the ports tree.
> > > So even you don't know what the ad hoc identity can and won't allow?
> >
> > I'm not sure I understand your question.
Sorry, I understood that you hadn't realised that signing with the ad hoc
identity prevents the persistent firewall popup for internet-enabled
applications.
> > > because everything from before that magic moment will have been signed
> > > with a different key that *might* break unpredictable things?> >
> > Can you elaborate here? What do you mean by "a different key"? Adhoc
>
Re: lldb ...
On 2016-09-09 10:17, René J.V. Bertin wrote: > On Thursday September 08 2016 16:03:21 Jeremy Huddleston Sequoia wrote: > >> That's not really necessary. All that is relevant is that the macports user >> has read access to the file. > > The fact that codesign only accepts keychain file arguments that are also in > the user's keychain search list may have something to do with that. At least on OS X 10.10 Yosemite, I can use any path to a keychain with `codesign --keychain`. This keychain does not have to be listed in `security list-keychains`. >>> Technically it doesn't really matter if it's implemented in "base" or in a >>> PortGroup, right? >> >> In order for *every* port to benefit, it needs to be in base. > > I don't see this argument. Are you considering codesigning each and every > binary automatically, without any need for requesting that from the Portfile? > What's the point in that? > OTOH, if portfile devs have to indicate which binary is to be signed they can > just as well add a PortGroup to be able to access that functionality. I also have the impression we are talking about different things here. I only want to add code-signing to the binaries that require it to work, such as gdb and lldb. I see no reason to and no benefits in adding a (ad-hoc) signature to every binary MacPorts creates. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-09 11:26, Jeremy Huddleston Sequoia wrote: > Yes. The fact that we aren't doing that for the binary packages that > we ship is quite embarrassing. We should solve this problem more > generally such that we can ship properly signed binaries for every > port. Users installing the binary packages that we ship right now > are running unsigned code, and that is quite frightening. There's > nothing guaranteeing that the package hasn't been MITMd. There's no > way for us to revoke a certificate if it turns out that our build > servers had been compromised, etc. This is just not true. All of our binary archives are in fact signed with a detached .rmd160 signature that is verified before installation when downloading from a mirror. This signature is for all files in the tarball and not just for the binaries. This is already more than codesigning would provide. If your machine is compromised in a way that the binaries can be replaced, this is out of the scope of MacPorts and a signature on the binary will not help in any way. The key can be revoked by releasing a new MacPorts version, or you can just remove it from /opt/local/etc/macports/pubkeys.conf. >> OTOH, if portfile devs have to indicate which binary is to be >> signed they can just as well add a PortGroup to be able to access >> that functionality. > > Yeah, it would be much better if we just signed every Mach-O in the > destroot of every port. What do we gain from that? Everything else would still be unsigned. >> So in your approach users who want to install a debugger port will >> become power users, change their configuration and then what? >> Rebuild everything if they've been building from source, > > No, they just need everything that the debugger executable links > against to be signed with a trusted certificate. That is no > different than your case either. Either way, the debugger and all > its dependencies need to be signed by a valid certificate. That does not seem to be the case. In my testing on OS X 10.10 Yosemite, it is enough to sign /opt/local/bin/ggdb with a trusted certificate to get it working. Did this change with El Capitan or Sierra? Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-09 03:38, Jeremy Sequoia wrote: > You are describing a system to automatically create and automatically > trust a code signing certificate that contains a private key in a file > on disk that is not encrypted with a passphrase and only depends on file > system ACLs to protect it. > > That is trivially insecure and attack-prone. Adding trust for an additional certificate also only relies on the filesystem permissions. If I get root write access to System.keychain, I can add my own certificate. If we add a trusted certificate to System.keychain and the corresponding private key is only accessible by root, that would still be the same level of security in my opinion. >> Consider that any software can already use your developer certificate >> from your user keychain to sign whatever it wants. > > No, it cannot. It requires my explicit authorization to do so. I must > unlock the keychain for it to gain access. > >> You will not even be >> asked when that happens. > > You should be if you set it up to do so. If you put your private keys > in your login keychain, then the act of logging in with your password is > what unlocks them. I don't consider that secure enough for my needs, so > I place mine in another keychain which requires me to explicitly unlock > it before use. Fair enough. However, I guess most users only have a login.keychain. >> I propose we add an additional keychain, readable by root only that is >> used to sign MacPorts binaries. As root is required to access it, your >> security would be defeated anyway if anyone gets to it. > > It's great that only root can read it, but it should not be created > automatically, added to any trusted store automatically, or stored > unencrypted automatically. We should instead give good instructions for > setting it up with either a self-signed or ADC-Provided code signing > certificate. Requiring manual setup defeats my goal of just making it work by installing the port. The status quo is already that everything needs to be done manually. > The user should just specify the path in macports.conf. If it is > locked, we should prompt for the passphrase to unlock it before use. Would need both the path to the .keychain and the CN of the certificate, right? As a compromise, I would propose we generate an identity in a separate keychain and sign the binary with it. However, adding the certificate to the trust store in System.keychain needs to be done manually by the user. Would you be comfortable with that? Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
One more thought: if indeed we're going to try to codesign everything at some
point, in the post-destroot, we'll probably want to ignore errors during that
particular step, i.e. catch the codesign return code and simply raise a warning
instead of bailing out. Ports that *require* signing (and only those) could set
a flag to make that warning an actual error.
I guess `codesign ${destroot.dir}` won't work, right?
BTW, what about the reproducible build principle and the idea of code-signing
everything on the buildbots using an official Apple key? Those seem to be
clearly incompatible...
R.
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 9, 2016, at 01:17, René J.V. Bertin wrote:
>
> On Thursday September 08 2016 16:03:21 Jeremy Huddleston Sequoia wrote:
>
>> That's not really necessary. All that is relevant is that the macports user
>> has read access to the file.
>
> If memory serves me well I had to set read access on ~macports,
> ~macports/Library and ~macports/LibraryKeychains too in order to be able to
> read the test keychain in there as myself.
> The fact that codesign only accepts keychain file arguments that are also in
> the user's keychain search list may have something to do with that. Anyway,
> yes, a user specification may not be really necessary, but giving
> ${macports_user} read access to your own keychain directory (for instance)
> doesn't sound like such a good idea.
You would have needed to set executable access on those directories, not read
access. And yes, the file would need to be in the user's keychain search list.
Also, yes, giving ${macports_user} read access to your user's login keychain is
not a good idea. It should be locked down to just the users that need it and
still guarded by a passphrase.
>>> Technically it doesn't really matter if it's implemented in "base" or in a
>>> PortGroup, right?
>>
>> In order for *every* port to benefit, it needs to be in base.
> I don't see this argument. Are you considering codesigning each and every
> binary automatically, without any need for requesting that from the Portfile?
> What's the point in that?
Yes. The fact that we aren't doing that for the binary packages that we ship
is quite embarrassing. We should solve this problem more generally such that
we can ship properly signed binaries for every port. Users installing the
binary packages that we ship right now are running unsigned code, and that is
quite frightening. There's nothing guaranteeing that the package hasn't been
MITMd. There's no way for us to revoke a certificate if it turns out that our
build servers had been compromised, etc.
> OTOH, if portfile devs have to indicate which binary is to be signed they can
> just as well add a PortGroup to be able to access that functionality.
Yeah, it would be much better if we just signed every Mach-O in the destroot of
every port.
>> If it's in a PortGroup, it is opt-in and doesn't solve the "let's sign
>> everything" case.
>
> Ah, the problem with inline replying :) I don't think anyone of us has even
> thought of a "let's sign everything case". There's clearly never been a need
> for that, and even if at some point it's going to be a necessity there will
> 1) be enough time to have integrated the then mature feature into "base" and
> 2) it will become a forced necessicity.
>
> As a side-thought: it shouldn't be particularly difficult to implement a
> "base" feature which defines a set of PortGroups to be included by default by
> every port, if such a thing doesn't already exist. I know there's been talk
> about making at least parts of "base" updateable as a port; this could be an
> easy alternative.
Yeah, I could see this being a PortGroup that is opt-in now and then becomes
always-on or opt-out in a future version of base.
>> post_destroot{} would resign it as specified in the macports.conf
>
> I'd hate that because I've set things up with macports_user=myself and
> everything under workpath owned by myself, to avoid constant need for UID
> changing. I kind of doubt I'm the only one maintaining lots of ports who does
> that.
I don't understand. Can you elaborate? Doesn't post_destroot{} run as root in
order to setup desired permissions in the destroot? That's a good place to
handle the resigning.
>>> There are 2 details to work out:
>>> - HOME is set to ${prefix}/var/macports/home (~macports)
>>
>> HOME is irrelevant.
>
> No, it isn't. It's what the Security framework uses to find keychains. Try it
> out for yourself and then correct me if I'm wrong ;)
> I was very surprised to discover that, but then I realised that this is why
> `sudo codesign ...` still uses the calling user's keychains. Of course I
> don't know if there have been changes in this aspect in 10.10 or later, but
> it's true for 10.9 .
That's because you're using sudo, which doesn't change the bootstrap session.
You likely want to do something like:
sudo launchctl asuser `id -u joebob` sudo -u joebob codesign ...
which requires launchd2 (OS X 10.10+)
>> No, the user doing the signing in post_destroot{} would be root.
>
> Codesign will still need to find the keychain, be able to read it, and the
> file will need to be in the searchlist of the user in who's homedir the file
> lives.
Oh, right... because it needs to find the user's keychain search list to
validate that the given keychain is in it. crap. yeah.
>>> (except for things like kexts in ports like the one I once proposed for
>>> ZFS) and is that something that Apple would accept
>>
>> I don't see why not. I have such a key for XQuartz.
>
>
Re: lldb ...
On Thursday September 08 2016 16:03:21 Jeremy Huddleston Sequoia wrote:
>That's not really necessary. All that is relevant is that the macports user
>has read access to the file.
If memory serves me well I had to set read access on ~macports,
~macports/Library and ~macports/LibraryKeychains too in order to be able to
read the test keychain in there as myself. The fact that codesign only accepts
keychain file arguments that are also in the user's keychain search list may
have something to do with that. Anyway, yes, a user specification may not be
really necessary, but giving ${macports_user} read access to your own keychain
directory (for instance) doesn't sound like such a good idea.
>> Technically it doesn't really matter if it's implemented in "base" or in a
>> PortGroup, right?
>
>In order for *every* port to benefit, it needs to be in base.
I don't see this argument. Are you considering codesigning each and every
binary automatically, without any need for requesting that from the Portfile?
What's the point in that?
OTOH, if portfile devs have to indicate which binary is to be signed they can
just as well add a PortGroup to be able to access that functionality.
>
>If it's in a PortGroup, it is opt-in and doesn't solve the "let's sign
>everything" case.
Ah, the problem with inline replying :) I don't think anyone of us has even
thought of a "let's sign everything case". There's clearly never been a need
for that, and even if at some point it's going to be a necessity there will 1)
be enough time to have integrated the then mature feature into "base" and 2) it
will become a forced necessicity.
As a side-thought: it shouldn't be particularly difficult to implement a "base"
feature which defines a set of PortGroups to be included by default by every
port, if such a thing doesn't already exist. I know there's been talk about
making at least parts of "base" updateable as a port; this could be an easy
alternative.
>post_destroot{} would resign it as specified in the macports.conf
I'd hate that because I've set things up with macports_user=myself and
everything under workpath owned by myself, to avoid constant need for UID
changing. I kind of doubt I'm the only one maintaining lots of ports who does
that.
>> There are 2 details to work out:
>> - HOME is set to ${prefix}/var/macports/home (~macports)
>
>HOME is irrelevant.
No, it isn't. It's what the Security framework uses to find keychains. Try it
out for yourself and then correct me if I'm wrong ;)
I was very surprised to discover that, but then I realised that this is why
`sudo codesign ...` still uses the calling user's keychains. Of course I don't
know if there have been changes in this aspect in 10.10 or later, but it's true
for 10.9 .
>
>No, the user doing the signing in post_destroot{} would be root.
Codesign will still need to find the keychain, be able to read it, and the file
will need to be in the searchlist of the user in who's homedir the file lives.
>> (except for things like kexts in ports like the one I once proposed for ZFS)
>> and is that something that Apple would accept
>
>I don't see why not. I have such a key for XQuartz.
Isn't that a bit of a special case, with you certainly having Apple's
benediction to work on that particular product?
>That should be controlled at the base layer. We shouldn't have a mixmatch of
>signing identities for different files as that will break library validation
>for any ports that might need LV.
Are you telling us that there's work in progress that will make it near
impossible to do traditional Unix development on OS X without jumping through
hoops to sign everything?
>> All the others I can think of are apps like kmail and family (port:kdepim)
>> which *benefit* from signing to stop the nagging about "do you want this
>> application to accept internet connections". And that can be handled just
>> fine with the ad-hoc identify.
>
>If that's the case, then my proposal seems perfect. It results in everything
>being adhoc signed by default and lets power users (and the buildbots)
>configure their system to resign executables with a configurable key.
So even you don't know what the ad hoc identity can and won't allow?
The ad hoc key is *not* useable for signing a debugger to make it functional.
So in your approach users who want to install a debugger port will become power
users, change their configuration and then what? Rebuild everything if they've
been building from source, because everything from before that magic moment
will have been signed with a different key that *might* break unpredictable
things?
In my book that would be a clear argument *against* signing everything, and for
signing only those things that really have to be signed.
>If the user can't be bothered to create a codesigning keychain and edit
>macports.conf, they probably aren't going to be trying to setup their own
>custom lldb installation in the first place and a
Re: lldb ...
Sent from my iPhone... > On Sep 8, 2016, at 17:32, Rainer Müller wrote: > >> On 2016-09-08 22:03, Jeremy Huddleston Sequoia wrote: >> >>> On Sep 2, 2016, at 05:19, Rainer Müller wrote: >>> >>> In my opinion, the actual implementation of codesigning should be in base: >>> >>> 1) Create a self-signed certificate/identity for code-signing >>> 2) Import certificate/identity into keychain >>> 3) Add it to trust store (`security add-trusted-cert ...`) >>> 4) In activate phase, if requested in Portfile, codesign the binary >> >> No, that is incredibly dangerous. code should not be signed during >> activate, it should be signed at build time. The packages that a user >> downloads from MacPorts should not be altered at activation time. Ideally, >> the packages they download should be codesigned by a MacPorts developer >> codesigning certificate. > > I do not want to bless binary archives produced on the buildbot or make > them special. MacPorts is not a binary-only distribution, we allow and > support local compilation of ports. Yes, and users that build on their own can sign them however they want. Binaries that we produce should be signed by MacPorts as a generally good security practice. The same system should be used for both purposes. >>> Unsolved problems: >>> For step 1), how to to automate certificate creation. >> >> Don't. This should be a manual process. The user should create a keychain >> and specify the path to the keychain in macports.conf. If that's not set, >> we should just adhoc sign. > > Adhoc code-signing does not solve the problem at hand with gdb/lldb as > that requires a trusted certificate in keychain as far as I understand it. Correct. Users that want to create their own debugger will need to sign the executable with a trusted certificate. > For what use case would ad-hoc code-signing be useful on macOS? Consistency and correctness. > What would be the benefit of adding an ad-hoc signature? It's more of a correctness check at that point than a security check. > >>> We cannot click >>> that in Keychain Assistent. That means finding a way to do it >>> programmatically with other tools. As far as I see, this is just a >>> standard x509 certificate which could be done with openssl, but which >>> extensions need to be enabled? >>> >>> For step 4), how to give user 'macports' access to the System.keychain >>> without entering a password? >> >> Don't use System.keychain. A separate keychain should be used just for the >> codesigning. It should be password protected, and the user should be >> prompted for the password to unlock the keychain as part of 'port build'. > > Users just want to get a working gdb/lldb, they don't want to learn > stuff about keychain or code-signing. They would need to create an > identity with a specific name, know where to place it, and set the > correct trust on the certificate. All these factors will lead to user > errors and that makes the manual process not feasible in my opinion. Automating it leads to attack vectors that are trivially exploitable. If you go down this road, you will introduce a very juicy exploit, and this project will loose trust of many users. Yes, users make mistakes. Let such error-prone users just use our binaries and let experts do it themselves. Let's make it secure by default. > After some testing, I would no longer want to add the identity (private > key) to System.keychain, but instead keep it in a separate keychain > accessible by root only. Yes. And also support keeping it encrypted with a passphrase that the user would be prompted for on use. > However, as far as I see, for trusting the certificate (public key) it > needs to be in System.keychain or taskgated will not accept it. Yes, the certificate and public key needs to be there if self signed, but it's equally possible to use a certificate that has an existing chain of trust (e.g.: ADC-provided developer codesigning key) >>> As an alternative, how to import the >>> identity (both private/public key) into a different keychain with >>> unlocked access? >> >> It should be unlocked by the user providing the password to unlock the >> keychain. > > I definitely do not want my upgrades to get stalled by password dialogs. You would just need to provide it once (at the start of running port). I already enter my password for sudo access when upgrading. This is just one more prompt. > That will only annoy users as when they have to type their password more > often. Only the ones that opt into it, and they will understand the need and likely desire this functionality. > > Rainer > ___ > macports-dev mailing list > macports-dev@lists.macosforge.org > https://lists.macosforge.org/mailman/listinfo/macports-dev ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
Sent from my iPhone... > On Sep 8, 2016, at 17:30, Rainer Müller wrote: > > On 2016-09-08 22:09, Jeremy Huddleston Sequoia wrote: >>> On Sep 5, 2016, at 03:49, Rainer Müller wrote: >>> My intention here is to describe a way how the code-signing can be >>> automated. We do not gain much by providing a solution that still >>> requires manual interaction by the user. Generating a certificate and >>> signing the binary should be completely transparent to the user. >> >> That obfuscation is very bad for security purposes. We should not hide this >> detail from users. It needs to be very explicit. > > At the moment it is very explicit. We have no automation at all and you > need to do all of the code-signing yourself or gdb/lldb will not work as > intended. > > The alternative way, recommended in the notes of the gdb port, requires > disabling SIP to edit /System/Library/com.apple.taskgated.plist, which I > would consider even worse for security. See [1]. > > Where do you see a security risk in adding a new trusted cert? You are describing a system to automatically create and automatically trust a code signing certificate that contains a private key in a file on disk that is not encrypted with a passphrase and only depends on file system ACLs to protect it. That is trivially insecure and attack-prone. > Consider that any software can already use your developer certificate > from your user keychain to sign whatever it wants. No, it cannot. It requires my explicit authorization to do so. I must unlock the keychain for it to gain access. > You will not even be > asked when that happens. You should be if you set it up to do so. If you put your private keys in your login keychain, then the act of logging in with your password is what unlocks them. I don't consider that secure enough for my needs, so I place mine in another keychain which requires me to explicitly unlock it before use. > I propose we add an additional keychain, readable by root only that is > used to sign MacPorts binaries. As root is required to access it, your > security would be defeated anyway if anyone gets to it. It's great that only root can read it, but it should not be created automatically, added to any trusted store automatically, or stored unencrypted automatically. We should instead give good instructions for setting it up with either a self-signed or ADC-Provided code signing certificate. The user should just specify the path in macports.conf. If it is locked, we should prompt for the passphrase to unlock it before use. > > Rainer > > [1] https://trac.macports.org/ticket/49815 > ___ > macports-dev mailing list > macports-dev@lists.macosforge.org > https://lists.macosforge.org/mailman/listinfo/macports-dev ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-08 22:03, Jeremy Huddleston Sequoia wrote: > >> On Sep 2, 2016, at 05:19, Rainer Müller wrote: >> >> In my opinion, the actual implementation of codesigning should be in base: >> >> 1) Create a self-signed certificate/identity for code-signing >> 2) Import certificate/identity into keychain >> 3) Add it to trust store (`security add-trusted-cert ...`) >> 4) In activate phase, if requested in Portfile, codesign the binary > > No, that is incredibly dangerous. code should not be signed during activate, > it should be signed at build time. The packages that a user downloads from > MacPorts should not be altered at activation time. Ideally, the packages > they download should be codesigned by a MacPorts developer codesigning > certificate. I do not want to bless binary archives produced on the buildbot or make them special. MacPorts is not a binary-only distribution, we allow and support local compilation of ports. >> Unsolved problems: >> For step 1), how to to automate certificate creation. > > Don't. This should be a manual process. The user should create a keychain > and specify the path to the keychain in macports.conf. If that's not set, we > should just adhoc sign. Adhoc code-signing does not solve the problem at hand with gdb/lldb as that requires a trusted certificate in keychain as far as I understand it. For what use case would ad-hoc code-signing be useful on macOS? What would be the benefit of adding an ad-hoc signature? >> We cannot click >> that in Keychain Assistent. That means finding a way to do it >> programmatically with other tools. As far as I see, this is just a >> standard x509 certificate which could be done with openssl, but which >> extensions need to be enabled? >> >> For step 4), how to give user 'macports' access to the System.keychain >> without entering a password? > > Don't use System.keychain. A separate keychain should be used just for the > codesigning. It should be password protected, and the user should be > prompted for the password to unlock the keychain as part of 'port build'. Users just want to get a working gdb/lldb, they don't want to learn stuff about keychain or code-signing. They would need to create an identity with a specific name, know where to place it, and set the correct trust on the certificate. All these factors will lead to user errors and that makes the manual process not feasible in my opinion. After some testing, I would no longer want to add the identity (private key) to System.keychain, but instead keep it in a separate keychain accessible by root only. However, as far as I see, for trusting the certificate (public key) it needs to be in System.keychain or taskgated will not accept it. >> As an alternative, how to import the >> identity (both private/public key) into a different keychain with >> unlocked access? > > It should be unlocked by the user providing the password to unlock the > keychain. I definitely do not want my upgrades to get stalled by password dialogs. That will only annoy users as when they have to type their password more often. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-08 22:09, Jeremy Huddleston Sequoia wrote: >> On Sep 5, 2016, at 03:49, Rainer Müller wrote: >> My intention here is to describe a way how the code-signing can be >> automated. We do not gain much by providing a solution that still >> requires manual interaction by the user. Generating a certificate and >> signing the binary should be completely transparent to the user. > > That obfuscation is very bad for security purposes. We should not hide this > detail from users. It needs to be very explicit. At the moment it is very explicit. We have no automation at all and you need to do all of the code-signing yourself or gdb/lldb will not work as intended. The alternative way, recommended in the notes of the gdb port, requires disabling SIP to edit /System/Library/com.apple.taskgated.plist, which I would consider even worse for security. See [1]. Where do you see a security risk in adding a new trusted cert? Consider that any software can already use your developer certificate from your user keychain to sign whatever it wants. You will not even be asked when that happens. I propose we add an additional keychain, readable by root only that is used to sign MacPorts binaries. As root is required to access it, your security would be defeated anyway if anyone gets to it. Rainer [1] https://trac.macports.org/ticket/49815 ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 8, 2016, at 15:13, René J.V. Bertin wrote:
>
> On Thursday September 08 2016 13:09:33 Jeremy Huddleston Sequoia wrote:
>>> codesign.files ${prefix}/bin/binary1 developer \
>>>${prefix}/bin/binary2 firewall
>>> codesign.identifier org.macports.${name}
>>
>>
>> I don't like this approach.
>>
>> We should solve this type of problem at the base layer. Everything should
>> be codesigned as configured in macports.conf (using a specified identity in
>> a specified keychain, or defaulting to adhoc).
>
> I think you'll also need to specify the user who owns the keychain, let's
> call him/her the macports_operator.
That's not really necessary. All that is relevant is that the macports user
has read access to the file.
> Technically it doesn't really matter if it's implemented in "base" or in a
> PortGroup, right?
In order for *every* port to benefit, it needs to be in base.
> I like the idea of a PortGroup because it allows to deploy the new feature
> without waiting for a new base release, and also makes pushing updates so
> much easier.
If it's in a PortGroup, it is opt-in and doesn't solve the "let's sign
everything" case.
> Similarly, it shouldn't really make a difference whether the information is
> read from macports.conf or somewhere else - but I suppose a PortGroup can
> read from that file too.
>
>> The port itself should build() using adhoc signing, and base's post-build
>> should resign everything (preserving all attributes) using the settings
>> specified in macports.conf.
>
> build{} and post-build{} are executed as ${macports_user}, right? That should
> be compatible with my observation that codesign will change ownership of the
> target file (unless it's executed as root).
Ah, right, sorry. Mixing up build system paradigms.
build{} would sign it adhoc.
post_destroot{} would resign it as specified in the macports.conf
> There are 2 details to work out:
> - HOME is set to ${prefix}/var/macports/home (~macports)
HOME is irrelevant.
> , not to ~${macports_user} i.e. not the home directory of the user specified
> via the macports_user variable. It will need to point to the home directory
> of the user who owns the keychain (${macports_operator})
No, it doesn't.
> - evidently ${macports_user} will need to be able to read the required
> keychain, possibly every directory in the path to that keychain.
No, the user doing the signing in post_destroot{} would be root.
> The first detail is trivial, the 2nd a bit less. I get around it because the
> post-activate{} step executes as root, so read access is automatic. The only
> easy way to solve it for the macports_user is to let the keychain be owned by
> that user.
If you want the macports user to do the signing, that's fine too. It doesn't
need to own the keychain, just have read access to it. It could be group read
access or other read access.
> I've tried to set things up that way by preparing a keychain with a signing
> certificate and its 2 keys. Once that's done you can automate the process of
> installing that keychain in ~macports/Library/Keychains and adding it to the
> search list via the security command. However, codesign refused to find that
> identity, when called by macports user. There's a transcript of my attempts
> in an earlier of my messages in this thread.
> The base layer could of course do a dedicated post-build that is executed as
> root. I presume...
Yes.
>> This approach allows us to specify an identity on the builders that is an
>> apple developer signing identity such that all binaries coming out of
>> MacPorts have a chain of trust back to Apple.
>
> Is that something we want
I'd say most definitely.
> (except for things like kexts in ports like the one I once proposed for ZFS)
> and is that something that Apple would accept
I don't see why not. I have such a key for XQuartz.
> ... without requiring a vote in when updates can be published, what can be
> accepted etc?
> I also don't really see why it would be bad to allow ports to let users sign
> or resign binaries with an identity of their own.
That should be controlled at the base layer. We shouldn't have a mixmatch of
signing identities for different files as that will break library validation
for any ports that might need LV.
> If you allow that for building from source, why not allow it for prebuilt
> packages too? Can you give an example what's so incredibly dangerous about
> resigning code that was signed on the buildbots?
You could certainly resign it after first validating the original signature.
Signing unsigned code without first validating it is a clear security concern.
> I can see that for things like kexts (which will be rejected anyway if you
> resign them with an inappropriate key), but for most other stuff in MacPorts?
> I know of only 1 port that actually requires signing the binary after
> installing - gdb. All the others I can think of are apps
Re: lldb ...
On Thursday September 08 2016 13:09:33 Jeremy Huddleston Sequoia wrote:
>> codesign.files ${prefix}/bin/binary1 developer \
>> ${prefix}/bin/binary2 firewall
>> codesign.identifier org.macports.${name}
>
>
>I don't like this approach.
>
>We should solve this type of problem at the base layer. Everything should be
>codesigned as configured in macports.conf (using a specified identity in a
>specified keychain, or defaulting to adhoc).
I think you'll also need to specify the user who owns the keychain, let's call
him/her the macports_operator.
Technically it doesn't really matter if it's implemented in "base" or in a
PortGroup, right? I like the idea of a PortGroup because it allows to deploy
the new feature without waiting for a new base release, and also makes pushing
updates so much easier.
Similarly, it shouldn't really make a difference whether the information is
read from macports.conf or somewhere else - but I suppose a PortGroup can read
from that file too.
>The port itself should build() using adhoc signing, and base's post-build
>should resign everything (preserving all attributes) using the settings
>specified in macports.conf.
build{} and post-build{} are executed as ${macports_user}, right? That should
be compatible with my observation that codesign will change ownership of the
target file (unless it's executed as root). There are 2 details to work out:
- HOME is set to ${prefix}/var/macports/home (~macports), not to
~${macports_user} i.e. not the home directory of the user specified via the
macports_user variable. It will need to point to the home directory of the user
who owns the keychain (${macports_operator})
- evidently ${macports_user} will need to be able to read the required
keychain, possibly every directory in the path to that keychain.
The first detail is trivial, the 2nd a bit less. I get around it because the
post-activate{} step executes as root, so read access is automatic. The only
easy way to solve it for the macports_user is to let the keychain be owned by
that user. I've tried to set things up that way by preparing a keychain with a
signing certificate and its 2 keys. Once that's done you can automate the
process of installing that keychain in ~macports/Library/Keychains and adding
it to the search list via the security command. However, codesign refused to
find that identity, when called by macports user. There's a transcript of my
attempts in an earlier of my messages in this thread.
The base layer could of course do a dedicated post-build that is executed as
root. I presume...
>This approach allows us to specify an identity on the builders that is an
>apple developer signing identity such that all binaries coming out of MacPorts
>have a chain of trust back to Apple.
Is that something we want (except for things like kexts in ports like the one I
once proposed for ZFS) and is that something that Apple would accept ...
without requiring a vote in when updates can be published, what can be accepted
etc?
I also don't really see why it would be bad to allow ports to let users sign or
resign binaries with an identity of their own. If you allow that for building
from source, why not allow it for prebuilt packages too? Can you give an
example what's so incredibly dangerous about resigning code that was signed on
the buildbots? I can see that for things like kexts (which will be rejected
anyway if you resign them with an inappropriate key), but for most other stuff
in MacPorts? I know of only 1 port that actually requires signing the binary
after installing - gdb. All the others I can think of are apps like kmail and
family (port:kdepim) which *benefit* from signing to stop the nagging about "do
you want this application to accept internet connections". And that can be
handled just fine with the ad-hoc identify.
>That obfuscation is very bad for security purposes. We should not hide this
>detail from users. It needs to be very explicit.
A middle ground can probably be found, one that makes it easier for users
(and/= increases the likelihood that every user gets it right the 1st time) but
that still requires some form of explicit action from the user.
If you look at lldb's instructions for setting up a signing key it isn't very
explicit at all what's going on for someone who (honestly?) doesn't really
care. It's just tedious. I think that a good compromise would be if an
implementation of a codesigning procedure prints the command to execute (a
script provided by macports) as a function of the signing requirements, a
command that the user can then paste in a CLI . Probably a set of commands; one
to create and populate a keychain in the own keychain directory, and a sudo'ed
one to install that keychain where it should go, if needed.
> The user should create a keychain and specify the path to the keychain in
> macports.conf. If that's not set, we should just adhoc sign.
What I don't like about a
Re: lldb ...
> On Sep 5, 2016, at 03:49, Rainer Müller wrote:
>
> On 2016-09-05 00:57, René J.V. Bertin wrote:
>> In a first draft the command accepted a list of binaries to sign, and read
>> the identify from a configuration file. I suppose that corresponds to what
>> you mean with a declarative syntax?
>> I then went to a simpler, single-binary interface because I realised that it
>> would sometimes be necessary to specify a different identity. It shouldn't
>> be hard to change that into
>>
>> ```
>> codesign identity binary1 [binary2 [...]]
>> ```
>
> In a Portfile you would not be interested in the name of the identity,
> the but trust policy it grants (firewall bypass, task_for_pid, etc.).
>
> Otherwise you would expect users to set up different identities and
> assign the correct trust policy, which is just not feasible.
>
> I am thinking of a list of files and policies such as:
>
> codesign.files ${prefix}/bin/binary1 developer \
> ${prefix}/bin/binary2 firewall
> codesign.identifier org.macports.${name}
I don't like this approach.
We should solve this type of problem at the base layer. Everything should be
codesigned as configured in macports.conf (using a specified identity in a
specified keychain, or defaulting to adhoc).
The port itself should build() using adhoc signing, and base's post-build
should resign everything (preserving all attributes) using the settings
specified in macports.conf.
This approach allows us to specify an identity on the builders that is an apple
developer signing identity such that all binaries coming out of MacPorts have a
chain of trust back to Apple.
> Of course this identifier would usually just be the default value, so it
> does not need to be specified explicitly.
>
Alternatively, MacPorts could create the certificate and then export it in
such a way that users only have to import it. There might be other
benefits to that (signing kexts provided by ports ...).
>>>
>>> I don't think I understand your idea... Why would we need to export it
>>> for users to import it again? The generated identity is meant only to be
>>> used by MacPorts and nothing else.
>>
>> I don't know how easy it is to automate the certificate creation, if at all
>> possible. Creating one that users only have to import removes a step or two
>> that could go wrong (if it cannot be automated), importing may be easier to
>> automate, and there may be an advantage if everyone signs using the same
>> certificate.
>
> If users need a identity, they can generate one with Keychain Access.
> The identity generated by MacPorts is only meant to be used by MacPorts.
>
>>> This approach cannot be used to sign kext, as that requires a
>>> certificate signed by Apple.
>>
>> I was thinking about a hypothetical possibility to redistribute such a
>> signed certificate. I know, probably extremely unlikely, but one can dream :)
>
> No way to do that. As we are code-signing on the local machine, we would
> need to distribute the private key. Everyone would be allowed to sign
> their kext with this, which defeats the whole purpose of signed kexts.
>
>>> This is not fully automatic and requires user interaction. This makes it
>>> fragile as it requires users to follow manual steps very closely to
>>> generate a certificate and set the required trust. That needs to be
>>> automated.
>>
>> I wouldn't be completely surprised if it turns out to be impossible to
>> automated the whole process; Apple could well have seen to that as a
>> protection against unexpected certificates popping up, with unlimited trust
>> etc. It's been a while since I used the API, and I only used the parts
>> related to traditional credentials, and keychain creation/management.
>
> My intention here is to describe a way how the code-signing can be
> automated. We do not gain much by providing a solution that still
> requires manual interaction by the user. Generating a certificate and
> signing the binary should be completely transparent to the user.
That obfuscation is very bad for security purposes. We should not hide this
detail from users. It needs to be very explicit.
smime.p7s
Description: S/MIME cryptographic signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Sep 2, 2016, at 05:19, Rainer Müller wrote:
>
> On 2016-08-31 23:25, Lawrence Velázquez wrote:
>>> On Aug 31, 2016, at 4:57 PM, René J.V. Bertin wrote:
>>>
>>> I noticed that Apple don't ship an lldb-mi executable (at least they don't
>>> for OS X 10.9).
>>
>> Xcode 7.3 includes it (`xcrun --run lldb-mi`) but Xcode 4.6.3 does not.
>> Someone else is welcome to bisect on that.
>>
>>> Has anyone looked at building an lldb port before?
>>
>>> The real problem is going to be with the code-signing. This is done
>>> automagically by lldb's Xcode projects so it's not entirely clear which
>>> files have to be signed ... nor if an "ad-hoc" signing identify ("-") will
>>> lead to a usable debugger.
>>
>> You opened a ticket about this a while ago. One of Eric's comments hints at
>> what a pain it might be to get it working with code signing.
>>
>> https://trac.macports.org/ticket/45251
>
> That would equivalent to what gdb recommends for codesigning.
>
> https://sourceware.org/gdb/wiki/BuildingOnDarwin#Giving_gdb_permission_to_control_other_processes
>
>
> In my opinion, the actual implementation of codesigning should be in base:
>
> 1) Create a self-signed certificate/identity for code-signing
> 2) Import certificate/identity into keychain
> 3) Add it to trust store (`security add-trusted-cert ...`)
> 4) In activate phase, if requested in Portfile, codesign the binary
No, that is incredibly dangerous. code should not be signed during activate,
it should be signed at build time. The packages that a user downloads from
MacPorts should not be altered at activation time. Ideally, the packages they
download should be codesigned by a MacPorts developer codesigning certificate.
> Unsolved problems:
> For step 1), how to to automate certificate creation.
Don't. This should be a manual process. The user should create a keychain and
specify the path to the keychain in macports.conf. If that's not set, we
should just adhoc sign.
> We cannot click
> that in Keychain Assistent. That means finding a way to do it
> programmatically with other tools. As far as I see, this is just a
> standard x509 certificate which could be done with openssl, but which
> extensions need to be enabled?
>
> For step 4), how to give user 'macports' access to the System.keychain
> without entering a password?
Don't use System.keychain. A separate keychain should be used just for the
codesigning. It should be password protected, and the user should be prompted
for the password to unlock the keychain as part of 'port build'.
> As an alternative, how to import the
> identity (both private/public key) into a different keychain with
> unlocked access?
It should be unlocked by the user providing the password to unlock the keychain.
>
> Rainer
> ___
> macports-dev mailing list
> macports-dev@lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/macports-dev
smime.p7s
Description: S/MIME cryptographic signature
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: gdb (Re: lldb ...)
On Thu, 1 Sep 2016, [ISO-8859-1] René J.V. Bertin wrote: > On Wednesday August 31 2016 17:29:54 Lawrence Velázquez wrote: > > >The gdb port instructs the user to modify a system LaunchDaemon to > >effectively circumvent taskgated. I frankly think that's an awful idea, > >and future ports should not follow its example. > > They certainly shouldn't if there are other ways to get the software to work. > > As to gdb, is it even possible to make it work normally? Last times I > tried gdb it was at least crippled for code written in C++. > > I'm tempted to say that as long as gdb cannot be a true alternative to > lldb there's little point in maintaining the port if in addition it > requires users to do all kinds of unhealthy things. It's not like it's > particularly difficult to build gdb (for the port's target audience at > least). It's still quite useful to have gdb for cross-debugging in cases where the relevant stub/server is for gdb rather than lldb. Although lldb in principle works with the gdb stub, it expects to get a machine description from the stub that's not present in the gdb case. There's an alternative way to supply the MD to lldb as a Python module (if it exists for the relevant architecture), but my experience with that approach has been less than satisfactory, and I didn't take the time to track down why. I don't think the cross-debugging case should require any interactions with taskgated, so any mention of taskgated tweaking should probably say that it's only needed for *local* debugging. At least the taskgated tweak doesn't open anywhere near as big a security hole as what you need to do to run your own kexts on >=10.10. :-) Fred Wright ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Monday September 05 2016 16:23:23 Rainer Müller wrote:
> I see no problem in generating certificates automatically. We also add
> users to the system or create launchd configurations without forcing the
> user to do it manually.
Yeah, exactly, I'm not really comfortable with adding users to the system
either. Adding launchd configurations is a bit different, those are just files
that can simply be suppressed/edited, and they're unambiguously part of the
content of the port that installed them.
Hmm. About that ... what if a port requiring certain binaries to be signed
simply installed its own keychain file into
${prefix}/var/macports/home/Library/Keychains (or
${portdbpath}/home/Library/Keychains ?) ? I don't think anything is stopping
them from doing this right now, and I also fail to see what kind of attack
vectors that could open up. Advantages:
- port maintainers can set up the keychain exactly as works best, tested
locally. I'm assuming that trust settings are stored in the keychain here.
- no need to design a possibly complex system for generating certificates with
the appropriate policies and trust settings
- signing is guaranteed to use the same data for everyone, making builds and
installs more reproducible
> > So, why would you force an expiration date on users? Say they install
> > lldb-3.x and stay at that version for whatever reason. That port isn't
> > going to change anymore at some point, so why force people to go through
> > the hassle even of de/re-activating. That seems rather unheard of, and a
> > first step on planned and forced obsolescence (where software stops
> > working after a given date).>
> I am not forcing it on them. x509 certificates have an expiry date. That
> is just a fact.
Doh, indeed. Hadn't noticed that that's set to 1 year if you just use the
certificate assistant. Cannot even recall seeing an option to set it to
something else.
> That's expected behavior. The permissions on the file itself are not
> relevant for unlinking, only permissions of the directory matter.
Ah, indeed. I hadn't noticed the inode thing when I was surprised by the owner
and group changes.
Still, it is kind of surprising that the utility accepts to sign a file that
you don't own and don't have write-access to. For one thing, signing of app
bundles is *not* possible when you don't have write access to the directories
inside them that need to be modified (and that behaviour is just as expected).
> But yes, that means we have to take care of preserving the permissions
> when codesigning.
If we stick with the idea of specifying a signing user for the moment, this
works:
{{{
set home [glob "~${user}"]
ui_info "Signing ${app} with ${identity} from ${user}'s
keychains under HOME=${home}"
if {[catch {system "env HOME=${home} codesign -s
${identity} --preserve-metadata -f -vvv --deep ${app}"} err]} {
ui_error "Signing ${app} with ${identity} from
${user}'s keychains under HOME=${home}: ${err}"
} else {
return 0
}
}}}
> hy set HOME when you can use --keychain?
Maybe that will work just as well, but we might still need to set HOME (from
`man codesign`):
"Note that the standard keychain search path is still consulted while
constructing the certificate chain being embedded in the signature.
Note that filename will not be searched to resolve the signing identity's
certificate chain unless it is also on the user's keychain search list."
That 2nd note may mean that my idea of ports installing a keychain into
~macports will only work if there's also a possibility to add them to the
user's keychain search list...
`security list-keychains` only allows to set the search list, not to
add/delete items (that would require some Tcl coding)
`security add-trusted-cert` seems to allow everything we (or a port) might
need.
However, a keychain created with `security create-keychain` is no longer added
to the keychain search list, and doesn't show up in the Keychain utility. It
has to be added by hand, or by adding it to `security list-keychains`
programmatically (and that, btw, has to filter out the System keychain lest it
appear twice in the final search list). That limitation of programmatic
keychain creation isn't documented anywhere; the API (and docs) haven't changed
since 10.6 where new keychains were still added to the search list. Maybe this
has been corrected in newer OS X versions?
One final thing: if you set `macportsuser` to something other than "macports"
the HOME variable will still be set to ~macports during the activate phase ...
but apparently ~macports belongs to the actual macportsuser, not user macports.
Something that will need to be documented somewhere, if confirmed.
Here's a breakdown of what I've been trying while composing this message:
%> sudo -u macports -H security create-keychain blabla
Re: lldb ...
On 2016-09-05 12:56, René J.V. Bertin wrote:
> On Monday September 05 2016 12:21:08 Rainer Müller wrote:
>
> Do the steps in that TN include the trust setting and the magic command in
> lldb's code-signing document?
I was talking about generating self-signed identities/certificates and
that is covered in TN 2206.
>> However, another thing I did not think of before, what should be used as
>> expiry date on the certificate? According to the TN, OS X does not care
>> about the expiration date for both verification or signing for "simple
>> applications", whatever that means.
>
> So, why would you force an expiration date on users? Say they install
> lldb-3.x and stay at that version for whatever reason. That port isn't going
> to change anymore at some point, so why force people to go through the hassle
> even of de/re-activating. That seems rather unheard of, and a first step on
> planned and forced obsolescence (where software stops working after a given
> date).
I am not forcing it on them. x509 certificates have an expiry date. That
is just a fact.
> I still think the easiest would be if that certificate and the corresponding
> keys would be stored in a keychain belonging to the MacPorts operator, not
> the macports. That's going to make maintenance and support so much easier.
> However, there's one surprising thing one should be aware of:
>
> {{{
> %> ll /opt/local/bin/411toppm
> 288233 0 -rwxrwxr-x 1 root wheel 33904 Dec 7 2014 /opt/local/bin/411toppm*
> %> id | fgrep wheel
> Exit 1
> %> codesign -s lldb_codesign -vvv /opt/local/bin/411toppm
> /opt/local/bin/411toppm: signed Mach-O universal (i386 x86_64) [411toppm]
> %> ll /opt/local/bin/411toppm
> 57101195 52 -rwxrwxr-x 1 bertin admin 51488 Sep 5 12:41
> /opt/local/bin/411toppm*
> }}}
>
> IOW: I'm not a member of the wheel group, so I do not normally have write
> access to 411toppm even if I own its parent directory. Yet I could run
> codesign on it without having to authenticate. It's probably better that the
> root ownership wasn't preserved even if that shouldn't make a difference for
> execs that aren't SETUID or SETGUID, but still. FWIW, running `sudo codesign`
> doesn't alter the file's ownership:
That's expected behavior. The permissions on the file itself are not
relevant for unlinking, only permissions of the directory matter. Your
user has write access to /opt/local/bin or this would have failed with
an error message.
But yes, that means we have to take care of preserving the permissions
when codesigning.
> {{{
> %> sudo codesign -s lldb_codesign -vvv -f /opt/local/bin/asciitopgm
> /opt/local/bin/asciitopgm: replacing existing signature
> /opt/local/bin/asciitopgm: signed Mach-O universal (i386 x86_64) [asciitopgm]
> %> ll /opt/local/bin/asciitopgm
> 57101222 52 -rwxrwxr-x 1 bertin admin 51456 Sep 5 12:53
> /opt/local/bin/asciitopgm*
> }}}
>
> So the current solution in my code-signing PortGroup isn't good, probably;
> only the HOME directory should be set to point to where the keychain holding
> the signing identify is to be found.
Why set HOME when you can use --keychain?
Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-05 13:29, René J.V. Bertin wrote: > Reminds me: I'm not really comfortable with the idea that PortFiles > would be able to create and store certificates. I'd rather see a form > where they can check for the availability of the identity, and then > inform the user exactly what command to run. More tedious, but at > least it forces you to see what's being done. I see no problem in generating certificates automatically. We also add users to the system or create launchd configurations without forcing the user to do it manually. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Monday September 05 2016 12:49:22 Rainer Müller wrote:
> In a Portfile you would not be interested in the name of the identity,
> the but trust policy it grants (firewall bypass, task_for_pid, etc.).
That depends, I'd say. Is it even possible to uncouple identifier and trust
policy to the extent that a single identifier can sign for either of the set of
trust policies available for that identity, instead of all of them?
Because if it can't, you'll still be looking at a certificate per policy, each
with an identity (presumably based on the policy name) ... and someone will
have to create them.
> Otherwise you would expect users to set up different identities and
> assign the correct trust policy, which is just not feasible.
That problem (if it is one) goes away when certificates (identities) can be
created automatically, no?
Either way, it really seems that signing is a requirement that's rarely
necessary. And for "basic" things like preventing recurrent requests to allow
an app to accept internet connections can be done with the ad-hoc identity, at
least on 10.9 . Not that we shouldn't strive to make this all as streamlined as
possible, but are the steps to get lldb (or gdb) to work really too complicated
to follow for the intended audience of those ports?
>
> I am thinking of a list of files and policies such as:
>
> codesign.files ${prefix}/bin/binary1 developer \
> ${prefix}/bin/binary2 firewall
> codesign.identifier org.macports.${name}
That looks like just a small variation, icing on the cake that can be
formalised once the basic functionality is there (and we know what's feasible
in practice, etc).
> >> I don't think I understand your idea... Why would we need to export it
> >> for users to import it again? The generated identity is meant only to be
> >> used by MacPorts and nothing else.
> >
> > I don't know how easy it is to automate the certificate creation, if at all
> > possible. Creating one that users only have to import removes a step or two
> > that could go wrong (if it cannot be automated), importing may be easier to
> > automate, and there may be an advantage if everyone signs using the same
> > certificate.
>
> If users need a identity, they can generate one with Keychain Access.
> The identity generated by MacPorts is only meant to be used by MacPorts.
I wasn't hinting at it being used for other purposes.
> My intention here is to describe a way how the code-signing can be
> automated. We do not gain much by providing a solution that still
> requires manual interaction by the user. Generating a certificate and
> signing the binary should be completely transparent to the user.
Ideally, yes. But having a formal framework to codesign binaries is a gain even
if the process cannot be completely automated, and whether the functionality is
implemented in a PortGroup (my preference at least until everything is mature)
or in "base".
Reminds me: I'm not really comfortable with the idea that PortFiles would be
able to create and store certificates. I'd rather see a form where they can
check for the availability of the identity, and then inform the user exactly
what command to run. More tedious, but at least it forces you to see what's
being done.
R
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Monday September 05 2016 12:21:08 Rainer Müller wrote:
Do the steps in that TN include the trust setting and the magic command in
lldb's code-signing document?
> However, another thing I did not think of before, what should be used as
> expiry date on the certificate? According to the TN, OS X does not care
> about the expiration date for both verification or signing for "simple
> applications", whatever that means.
So, why would you force an expiration date on users? Say they install lldb-3.x
and stay at that version for whatever reason. That port isn't going to change
anymore at some point, so why force people to go through the hassle even of
de/re-activating. That seems rather unheard of, and a first step on planned and
forced obsolescence (where software stops working after a given date).
I still think the easiest would be if that certificate and the corresponding
keys would be stored in a keychain belonging to the MacPorts operator, not the
macports. That's going to make maintenance and support so much easier. However,
there's one surprising thing one should be aware of:
{{{
%> ll /opt/local/bin/411toppm
288233 0 -rwxrwxr-x 1 root wheel 33904 Dec 7 2014 /opt/local/bin/411toppm*
%> id | fgrep wheel
Exit 1
%> codesign -s lldb_codesign -vvv /opt/local/bin/411toppm
/opt/local/bin/411toppm: signed Mach-O universal (i386 x86_64) [411toppm]
%> ll /opt/local/bin/411toppm
57101195 52 -rwxrwxr-x 1 bertin admin 51488 Sep 5 12:41
/opt/local/bin/411toppm*
}}}
IOW: I'm not a member of the wheel group, so I do not normally have write
access to 411toppm even if I own its parent directory. Yet I could run codesign
on it without having to authenticate. It's probably better that the root
ownership wasn't preserved even if that shouldn't make a difference for execs
that aren't SETUID or SETGUID, but still. FWIW, running `sudo codesign` doesn't
alter the file's ownership:
{{{
%> sudo codesign -s lldb_codesign -vvv -f /opt/local/bin/asciitopgm
/opt/local/bin/asciitopgm: replacing existing signature
/opt/local/bin/asciitopgm: signed Mach-O universal (i386 x86_64) [asciitopgm]
%> ll /opt/local/bin/asciitopgm
57101222 52 -rwxrwxr-x 1 bertin admin 51456 Sep 5 12:53
/opt/local/bin/asciitopgm*
}}}
So the current solution in my code-signing PortGroup isn't good, probably; only
the HOME directory should be set to point to where the keychain holding the
signing identify is to be found.
R.
PS: the curious and observative will have noticed that signing changes the
file's inode. I'm guessing that means it'll break hard links.
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-05 00:57, René J.V. Bertin wrote:
> In a first draft the command accepted a list of binaries to sign, and read
> the identify from a configuration file. I suppose that corresponds to what
> you mean with a declarative syntax?
> I then went to a simpler, single-binary interface because I realised that it
> would sometimes be necessary to specify a different identity. It shouldn't be
> hard to change that into
>
> ```
> codesign identity binary1 [binary2 [...]]
> ```
In a Portfile you would not be interested in the name of the identity,
the but trust policy it grants (firewall bypass, task_for_pid, etc.).
Otherwise you would expect users to set up different identities and
assign the correct trust policy, which is just not feasible.
I am thinking of a list of files and policies such as:
codesign.files ${prefix}/bin/binary1 developer \
${prefix}/bin/binary2 firewall
codesign.identifier org.macports.${name}
Of course this identifier would usually just be the default value, so it
does not need to be specified explicitly.
>>> Alternatively, MacPorts could create the certificate and then export it in
>>> such a way that users only have to import it. There might be other benefits
>>> to that (signing kexts provided by ports ...).
>>
>> I don't think I understand your idea... Why would we need to export it
>> for users to import it again? The generated identity is meant only to be
>> used by MacPorts and nothing else.
>
> I don't know how easy it is to automate the certificate creation, if at all
> possible. Creating one that users only have to import removes a step or two
> that could go wrong (if it cannot be automated), importing may be easier to
> automate, and there may be an advantage if everyone signs using the same
> certificate.
If users need a identity, they can generate one with Keychain Access.
The identity generated by MacPorts is only meant to be used by MacPorts.
>> This approach cannot be used to sign kext, as that requires a
>> certificate signed by Apple.
>
> I was thinking about a hypothetical possibility to redistribute such a signed
> certificate. I know, probably extremely unlikely, but one can dream :)
No way to do that. As we are code-signing on the local machine, we would
need to distribute the private key. Everyone would be allowed to sign
their kext with this, which defeats the whole purpose of signed kexts.
>> This is not fully automatic and requires user interaction. This makes it
>> fragile as it requires users to follow manual steps very closely to
>> generate a certificate and set the required trust. That needs to be
>> automated.
>
> I wouldn't be completely surprised if it turns out to be impossible to
> automated the whole process; Apple could well have seen to that as a
> protection against unexpected certificates popping up, with unlimited trust
> etc. It's been a while since I used the API, and I only used the parts
> related to traditional credentials, and keychain creation/management.
My intention here is to describe a way how the code-signing can be
automated. We do not gain much by providing a solution that still
requires manual interaction by the user. Generating a certificate and
signing the binary should be completely transparent to the user.
Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-02 14:19, Rainer Müller wrote: > For step 4), how to give user 'macports' access to the System.keychain > without entering a password? As an alternative, how to import the > identity (both private/public key) into a different keychain with > unlocked access? I found the missing information in this technical note: https://developer.apple.com/library/mac/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG10 However, another thing I did not think of before, what should be used as expiry date on the certificate? According to the TN, OS X does not care about the expiration date for both verification or signing for "simple applications", whatever that means. Would it be enough to just use 365 days as every port is expected to be updated/reinstalled during this time period? And after that we just add another certificate valid for the next 365 days. Rainer ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Monday September 05 2016 00:13:04 Rainer Müller wrote:
> >> In my opinion, the actual implementation of codesigning should be in base:
> >>
> >> 1) Create a self-signed certificate/identity for code-signing
> >> 2) Import certificate/identity into keychain
> >> 3) Add it to trust store (`security add-trusted-cert ...`)
> >> 4) In activate phase, if requested in Portfile, codesign the binary
> >
> > That's what's done in my PortGroup, which could of course be a vector to
> > test out the functionality before integrating it into "base" (and make it
> > available before the next new base release).
>
> For reference, this would be about #51504:
> https://trac.macports.org/ticket/51504
>
> As far as I see it, this only implements step 4), it already expects an
Erm, indeed, yes.
> identity generated by the user before. Also, this is implement as a
> command, I would prefer a declarative syntax (list of binaries for
> codesigning).
In a first draft the command accepted a list of binaries to sign, and read the
identify from a configuration file. I suppose that corresponds to what you mean
with a declarative syntax?
I then went to a simpler, single-binary interface because I realised that it
would sometimes be necessary to specify a different identity. It shouldn't be
hard to change that into
```
codesign identity binary1 [binary2 [...]]
```
> > Alternatively, MacPorts could create the certificate and then export it in
> > such a way that users only have to import it. There might be other benefits
> > to that (signing kexts provided by ports ...).
>
> I don't think I understand your idea... Why would we need to export it
> for users to import it again? The generated identity is meant only to be
> used by MacPorts and nothing else.
I don't know how easy it is to automate the certificate creation, if at all
possible. Creating one that users only have to import removes a step or two
that could go wrong (if it cannot be automated), importing may be easier to
automate, and there may be an advantage if everyone signs using the same
certificate.
> This approach cannot be used to sign kext, as that requires a
> certificate signed by Apple.
I was thinking about a hypothetical possibility to redistribute such a signed
certificate. I know, probably extremely unlikely, but one can dream :)
> > If the security utility doesn't already allow it, it shouldn't be
> > particularly hard to write some code that will import a certificate into a
> > possibly newly created keychain that is then set default. When executed as
> > the macports user this should result in a keychain that's accessible to ...
> > the macports user.
>
> codesign accepts a --keychain option, so there is no need to modify the
> default keychain.
The macports user doesn't currently have a keychain, so what would be wrong
with modifying a dedicated default keychain? (see also below).
I just checked: the Security framework indeed uses $HOME to determine which
keychains are available. On OS X 10.9, that is :
```
sudo $SHELL
env HOME=~theOtherUser /Applications/Utilities/Keychain\
Access.app/Contents/MacOS/Keychain\ Access
```
> You are correct that only root would need access. That could mean that
> the certificate could just be in the System.keychain. In my testing that
> always seems to require a password, though...
If we manage to create a keychain for the macports user that will probably also
be the case, unless that user corresponds to the actual user. Because otherwise
the signing user won't be signed/logged in, and its keychain(s) will thus be
locked. That's actually an argument to store the certificate in a default
keychain, as that one will be open after logging in for most users.
> This is not fully automatic and requires user interaction. This makes it
> fragile as it requires users to follow manual steps very closely to
> generate a certificate and set the required trust. That needs to be
> automated.
I wouldn't be completely surprised if it turns out to be impossible to
automated the whole process; Apple could well have seen to that as a protection
against unexpected certificates popping up, with unlimited trust etc. It's been
a while since I used the API, and I only used the parts related to traditional
credentials, and keychain creation/management.
> In your testing, did you move the private/public key ("identity") or
> maybe just the public key ("certificate")?
I followed the lldb document to the letter so yes, only the certificate.
R.
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-09-02 14:45, René J.V. Bertin wrote:
> On Friday September 02 2016 14:19:02 Rainer Müller wrote:
>
>>> https://trac.macports.org/ticket/45251
>>
>> That would equivalent to what gdb recommends for codesigning.
>
> Yes, minus the part related to taskgated. And plus a few extra steps that
> probably serve to add the certificate to the trust store.
>
>> In my opinion, the actual implementation of codesigning should be in base:
>>
>> 1) Create a self-signed certificate/identity for code-signing
>> 2) Import certificate/identity into keychain
>> 3) Add it to trust store (`security add-trusted-cert ...`)
>> 4) In activate phase, if requested in Portfile, codesign the binary
>
> That's what's done in my PortGroup, which could of course be a vector to test
> out the functionality before integrating it into "base" (and make it
> available before the next new base release).
For reference, this would be about #51504:
https://trac.macports.org/ticket/51504
As far as I see it, this only implements step 4), it already expects an
identity generated by the user before. Also, this is implement as a
command, I would prefer a declarative syntax (list of binaries for
codesigning).
>> Unsolved problems:
>> For step 1), how to to automate certificate creation. We cannot click
>> that in Keychain Assistent. That means finding a way to do it
>> programmatically with other tools. As far as I see, this is just a
>> standard x509 certificate which could be done with openssl, but which
>> extensions need to be enabled?
>
> Alternatively, MacPorts could create the certificate and then export it in
> such a way that users only have to import it. There might be other benefits
> to that (signing kexts provided by ports ...).
I don't think I understand your idea... Why would we need to export it
for users to import it again? The generated identity is meant only to be
used by MacPorts and nothing else.
This approach cannot be used to sign kext, as that requires a
certificate signed by Apple. There is no (documented) way to trust
another certificate in the kernel.
> If the security utility doesn't already allow it, it shouldn't be
> particularly hard to write some code that will import a certificate into a
> possibly newly created keychain that is then set default. When executed as
> the macports user this should result in a keychain that's accessible to ...
> the macports user.
codesign accepts a --keychain option, so there is no need to modify the
default keychain.
> But are you sure that the activate and/or post-activate phases are executed
> as the macports user? I don't have the impression that's the case.
You are correct that only root would need access. That could mean that
the certificate could just be in the System.keychain. In my testing that
always seems to require a password, though...
>> For step 4), how to give user 'macports' access to the System.keychain
>> without entering a password? As an alternative, how to import the
>> identity (both private/public key) into a different keychain with
>> unlocked access?
>
> Is there anything wrong with my approach of specifying a username (typically
> your own) that's used only for the signing operation?
>
> BTW, I'm pretty sure that I tried putting the certificate back into the
> System keychain and attempt to sign without specifying a user. That still
> gave the error that the certificate wasn't found, which is why I kept
> searching and found the solution currently implemented in the PortGroup.
This is not fully automatic and requires user interaction. This makes it
fragile as it requires users to follow manual steps very closely to
generate a certificate and set the required trust. That needs to be
automated.
In your testing, did you move the private/public key ("identity") or
maybe just the public key ("certificate")?
Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Friday September 02 2016 14:19:02 Rainer Müller wrote: >> https://trac.macports.org/ticket/45251 > >That would equivalent to what gdb recommends for codesigning. Yes, minus the part related to taskgated. And plus a few extra steps that probably serve to add the certificate to the trust store. >In my opinion, the actual implementation of codesigning should be in base: > >1) Create a self-signed certificate/identity for code-signing >2) Import certificate/identity into keychain >3) Add it to trust store (`security add-trusted-cert ...`) >4) In activate phase, if requested in Portfile, codesign the binary That's what's done in my PortGroup, which could of course be a vector to test out the functionality before integrating it into "base" (and make it available before the next new base release). >Unsolved problems: >For step 1), how to to automate certificate creation. We cannot click >that in Keychain Assistent. That means finding a way to do it >programmatically with other tools. As far as I see, this is just a >standard x509 certificate which could be done with openssl, but which >extensions need to be enabled? Alternatively, MacPorts could create the certificate and then export it in such a way that users only have to import it. There might be other benefits to that (signing kexts provided by ports ...). If the security utility doesn't already allow it, it shouldn't be particularly hard to write some code that will import a certificate into a possibly newly created keychain that is then set default. When executed as the macports user this should result in a keychain that's accessible to ... the macports user. But are you sure that the activate and/or post-activate phases are executed as the macports user? I don't have the impression that's the case. >For step 4), how to give user 'macports' access to the System.keychain >without entering a password? As an alternative, how to import the >identity (both private/public key) into a different keychain with >unlocked access? Is there anything wrong with my approach of specifying a username (typically your own) that's used only for the signing operation? BTW, I'm pretty sure that I tried putting the certificate back into the System keychain and attempt to sign without specifying a user. That still gave the error that the certificate wasn't found, which is why I kept searching and found the solution currently implemented in the PortGroup. R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On 2016-08-31 23:25, Lawrence Velázquez wrote:
>> On Aug 31, 2016, at 4:57 PM, René J.V. Bertin wrote:
>>
>> I noticed that Apple don't ship an lldb-mi executable (at least they don't
>> for OS X 10.9).
>
> Xcode 7.3 includes it (`xcrun --run lldb-mi`) but Xcode 4.6.3 does not.
> Someone else is welcome to bisect on that.
>
>> Has anyone looked at building an lldb port before?
>
>> The real problem is going to be with the code-signing. This is done
>> automagically by lldb's Xcode projects so it's not entirely clear which
>> files have to be signed ... nor if an "ad-hoc" signing identify ("-") will
>> lead to a usable debugger.
>
> You opened a ticket about this a while ago. One of Eric's comments hints at
> what a pain it might be to get it working with code signing.
>
> https://trac.macports.org/ticket/45251
That would equivalent to what gdb recommends for codesigning.
https://sourceware.org/gdb/wiki/BuildingOnDarwin#Giving_gdb_permission_to_control_other_processes
In my opinion, the actual implementation of codesigning should be in base:
1) Create a self-signed certificate/identity for code-signing
2) Import certificate/identity into keychain
3) Add it to trust store (`security add-trusted-cert ...`)
4) In activate phase, if requested in Portfile, codesign the binary
Unsolved problems:
For step 1), how to to automate certificate creation. We cannot click
that in Keychain Assistent. That means finding a way to do it
programmatically with other tools. As far as I see, this is just a
standard x509 certificate which could be done with openssl, but which
extensions need to be enabled?
For step 4), how to give user 'macports' access to the System.keychain
without entering a password? As an alternative, how to import the
identity (both private/public key) into a different keychain with
unlocked access?
Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Wednesday August 31 2016 23:51:46 René J.V. Bertin wrote: > The lldb build system is apparently less well prepared to set the appropriate > Python include path; the build just failed because pyconfig.h couldn't be > found. Or I have something I shouldn't have: a directory > /opt/local/include/python2.7 which does not belong to port:python27.. That's done. I have a first draft for an lldb-3.8 subport: https://github.com/RJVB/macstrop/commit/4608ad87a12075e63125e5ee2ffc4c3fc2bfb668#diff-6409887f54daa21d3d10e36ddda20f05 This appears to work, except for the whole signing business. Running the debugger via sudo allows it to attach but is of course no viable solution. I wonder if there is no more elegant method to change an executables signing identity that doesn't involve rebooting as lldb's code-signing document suggests? R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
gdb (Re: lldb ...)
On Wednesday August 31 2016 17:29:54 Lawrence Velázquez wrote: >The gdb port instructs the user to modify a system LaunchDaemon to effectively >circumvent taskgated. I frankly think that's an awful idea, and future ports >should not follow its example. They certainly shouldn't if there are other ways to get the software to work. As to gdb, is it even possible to make it work normally? Last times I tried gdb it was at least crippled for code written in C++. I'm tempted to say that as long as gdb cannot be a true alternative to lldb there's little point in maintaining the port if in addition it requires users to do all kinds of unhealthy things. It's not like it's particularly difficult to build gdb (for the port's target audience at least). R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Wednesday August 31 2016 17:25:18 Lawrence Velázquez wrote: > You opened a ticket about this a while ago. One of Eric's comments hints at > what a pain it might be to get it working with code signing. > https://trac.macports.org/ticket/45251 I remembered, and apparently it was (over) 2y ago, time enough for someone to have looked into this without necessarily updating or knowing of that ticket. The document Eric cites hasn't changed (much), and isn't that difficult to follow (presuming the instructions to reboot aren't actually required). I have now scanned the lldb sources, and it appears that the cmake system is also designed to do the code-signing on OS X. So yeah, the port should find some way to tell the user to create the lldb_codesign certificate, or else we might try patching the build system to use the ad-hoc identity. Checking for the certificate can be done via `security find-certificate -c lldb_codesign` The lldb build system is apparently less well prepared to set the appropriate Python include path; the build just failed because pyconfig.h couldn't be found. Or I have something I shouldn't have: a directory /opt/local/include/python2.7 which does not belong to port:python27.. R. ___ macports-dev mailing list macports-dev@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Aug 31, 2016, at 5:06 PM, Brandon Allbery wrote:
>
>> On Wed, Aug 31, 2016 at 4:57 PM, René J.V. Bertin
>> wrote:
>>
>> The real problem is going to be with the code-signing. This is done
>> automagically by lldb's Xcode projects so it's not entirely clear which
>> files have to be signed ... nor if an "ad-hoc" signing identify ("-") will
>> lead to a usable debugger.
>
> You might look at what the gdb port does. IIRC there's a flag you need to add
> to taskgated (the ui_notes will tell you what needs to be done) to use it.
The gdb port instructs the user to modify a system LaunchDaemon to effectively
circumvent taskgated. I frankly think that's an awful idea, and future ports
should not follow its example.
vq
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
> On Aug 31, 2016, at 4:57 PM, René J.V. Bertin wrote:
>
> I noticed that Apple don't ship an lldb-mi executable (at least they don't
> for OS X 10.9).
Xcode 7.3 includes it (`xcrun --run lldb-mi`) but Xcode 4.6.3 does not. Someone
else is welcome to bisect on that.
> Has anyone looked at building an lldb port before?
> The real problem is going to be with the code-signing. This is done
> automagically by lldb's Xcode projects so it's not entirely clear which files
> have to be signed ... nor if an "ad-hoc" signing identify ("-") will lead to
> a usable debugger.
You opened a ticket about this a while ago. One of Eric's comments hints at
what a pain it might be to get it working with code signing.
https://trac.macports.org/ticket/45251
vq
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
Re: lldb ...
On Wed, Aug 31, 2016 at 4:57 PM, René J.V. Bertin
wrote:
> The real problem is going to be with the code-signing. This is done
> automagically by lldb's Xcode projects so it's not entirely clear which
> files have to be signed ... nor if an "ad-hoc" signing identify ("-") will
> lead to a usable debugger.
You might look at what the gdb port does. IIRC there's a flag you need to
add to taskgated (the ui_notes will tell you what needs to be done) to use
it.
--
brandon s allbery kf8nh sine nomine associates
allber...@gmail.com ballb...@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev
