Re: Restoring from Time Machine backup relocates home directories

[db]

On 17 Sep 2017, at 00:15, Ryan Schmidt  wrote:
> File ownership can be preserved on disk images, but that ownership is 
> preserved by uid and gid, not by user name and group name.

I suspected this but didn't know how it actually works on OS X.

>> MP could also checked if the UIDs/GIDs are already in use, before a port 
>> creates a user/group.
> MacPorts already does this, in that the MacPorts command to create a new user 
> will use the first uid numbered 500 or greater that does not already exist in 
> the directory service. Or did you mean something different by "already in 
> use"?

I meant not already in use in the file system, but that would introduce the 
delay upon uninstallation and eventually wouldn't avoid the pitfalls of file 
ownership preservation.

Couldn't MP use a higher numbered block of UID/GIDs instead of availability for 
user creation?

Re: Restoring from Time Machine backup relocates home directories

[Ryan Schmidt]

On Sep 16, 2017, at 04:57, db wrote:
> On 15 Sep 2017, at 23:47, Ryan Schmidt wrote:
>> MacPorts and ports can add users, but MacPorts doesn't remove them. I'm not 
>> sure how we could change that. Suppose you install a port that has a user 
>> that needs to own some data, such as postresql96-server. You then uninstall 
>> the port, which then hypothetically removes the user. Now the databases you 
>> created with that port are owned by a userid that doesn't exist anymore. Now 
>> you reinstall the port, which creates a postgres user but with a new userid. 
>> This creates more work for the user: needing to reassociate files the port 
>> needs with the new userid.
> 
> Shouldn't MP delete it if no files are owned by that user? In any case, it 
> could notify during installation that a user was created and upon 
> uninstallation that a user still owns files (and which) or not, and if not, 
> it could give the option to delete it.

Maybe. But how would MacPorts know that no files are owned by that user? I 
wouldn't want a port uninstallation to introduce a several-minute delay while 
MacPorts checks the ownership of every file on disk. And as you found below, 
you may have unexpected files owned by such a user.


> I searched for files owned by users created by two ports installed not long 
> ago (quagga is still installed, dbus uninstalled) to find the following (just 
> an example of a mess I now realised it's in place and I don't have a clue how 
> it happened).
> 
> $ ls -l /Library/Screen\ Savers/Substrate.saver/Contents
> total 16
> -rw-r--r--@ 1 rootmessagebus   692 Apr 12  2010 CodeResources
> -rw-r--r--@ 1 rootmessagebus  1491 Apr 12  2010 Info.plist
> drwxr-xr-x@ 3 quagga  messagebus   102 Apr 12  2010 MacOS
> drwxr-xr-x@ 3 quagga  messagebus   102 Apr 12  2010 Resources
> drwxr-xr-x@ 3 quagga  messagebus   102 Apr 12  2010 _CodeSignature
> 
> AFAIR, I installed that screen saver by dragging and dropping, no installer.

This is probably normal.

When a user is created on Mac OS X 10.5 Leopard and later, its group is set to 
"staff". But on Mac OS X Tiger 10.4 and earlier, each user got a group of its 
own, named the same as the user.

File ownership can be preserved on disk images, but that ownership is preserved 
by uid and gid, not by user name and group name.

Suppose that the person who packed up Substrate.saver was on a Mac running 
Tiger with two user accounts, one with uid 501 (alice) and gid 501 (alice) and 
another with uid 502 (bob) and gid 502 (bob), and that the person was logged in 
as bob. This person compiles Substrate.saver, which causes it to be owned by 
uid/gid 502, then creates a disk image with ownership preserved, and copies the 
screensaver bundle to the disk image and uploads the disk image to the 
Internet, where you download it.

You're on a Mac with Leopard or later and you only have one user account, with 
uid 501 and gid 20 (staff). You copy the screensaver from the disk image to 
your disk, and ownership is preserved. If you look at the ownership of the 
screensaver now with ls -l, you'll see it's owned by uid 502 and gid 502, even 
though you don't have a user nor a group with those ids.

Later you install the dbus and quagga ports. They each create their own user 
accounts, using whatever uids and gids are unused on your system. They may not 
be the same. For example on my system my messagebus user has uid 500 but my 
messagebus group has gid 504. On your system, the quagga user happened to get 
uid 502 and the messagebus group happened to get gid 502. When you now look 
with ls -l, you see that the screensaver is owned by user quagga and group 
messagebus.

It wouldn't matter if the order was reversed--if you installed the ports and 
MacPorts created the users and groups first, and then you copied the 
screensaver--the end result would be the same.

The problem should not have occurred if the disk image was set not to preserve 
ownership. In that case, when you copied the file from the disk image to your 
disk, it would have received the user and group of your user.

You can also run into this type of problem if you have multiple Macs that you 
copy files between while preserving ownership, such as using a USB or FireWire 
disk with owners turned on, or by using Apple file sharing, and the user 
accounts on the different Macs are not all using the same uid/gid.


> MP could also checked if the UIDs/GIDs are already in use, before a port 
> creates a user/group.

MacPorts already does this, in that the MacPorts command to create a new user 
will use the first uid numbered 500 or greater that does not already exist in 
the directory service. Or did you mean something different by "already in use"?

Re: Restoring from Time Machine backup relocates home directories

[db]

On 16 Sep 2017, at 16:03, Rainer Müller  wrote:
> As you just run it on your system, you know how long that will take.
> Clearly not an option to do this before/after every install/uninstall.

Only those that create users.

> And what would you gain from that? A free uid. They are not a scarce
> resource, so I do not think the effort would be justified.

A cleaner system. Users created by MP won't end up owning files from some other 
remaining user, as in the case mentioned.

Re: Restoring from Time Machine backup relocates home directories

[Rainer Müller]

On 2017-09-16 11:57, db wrote:
> Shouldn't MP delete it if no files are owned by that user? In any case, it 
> could notify during installation that a user was created and upon 
> uninstallation that a user still owns files (and which) or not, and if not, 
> it could give the option to delete it.

> MP could also checked if the UIDs/GIDs are already in use, before a port 
> creates a user/group.

I am not sure if you are serious here. You would need to recursively
walk all filesystems to find files owned by a particular user.

As you just run it on your system, you know how long that will take.
Clearly not an option to do this before/after every install/uninstall.

And what would you gain from that? A free uid. They are not a scarce
resource, so I do not think the effort would be justified.

Rainer

Re: Restoring from Time Machine backup relocates home directories

[db]

On 15 Sep 2017, at 23:47, Ryan Schmidt  wrote:
> MacPorts and ports can add users, but MacPorts doesn't remove them. I'm not 
> sure how we could change that. Suppose you install a port that has a user 
> that needs to own some data, such as postresql96-server. You then uninstall 
> the port, which then hypothetically removes the user. Now the databases you 
> created with that port are owned by a userid that doesn't exist anymore. Now 
> you reinstall the port, which creates a postgres user but with a new userid. 
> This creates more work for the user: needing to reassociate files the port 
> needs with the new userid.

Shouldn't MP delete it if no files are owned by that user? In any case, it 
could notify during installation that a user was created and upon 
uninstallation that a user still owns files (and which) or not, and if not, it 
could give the option to delete it.

I searched for files owned by users created by two ports installed not long ago 
(quagga is still installed, dbus uninstalled) to find the following (just an 
example of a mess I now realised it's in place and I don't have a clue how it 
happened).

$ ls -l /Library/Screen\ Savers/Substrate.saver/Contents
total 16
-rw-r--r--@ 1 rootmessagebus   692 Apr 12  2010 CodeResources
-rw-r--r--@ 1 rootmessagebus  1491 Apr 12  2010 Info.plist
drwxr-xr-x@ 3 quagga  messagebus   102 Apr 12  2010 MacOS
drwxr-xr-x@ 3 quagga  messagebus   102 Apr 12  2010 Resources
drwxr-xr-x@ 3 quagga  messagebus   102 Apr 12  2010 _CodeSignature

AFAIR, I installed that screen saver by dragging and dropping, no installer.

MP could also checked if the UIDs/GIDs are already in use, before a port 
creates a user/group.

Re: Restoring from Time Machine backup relocates home directories

[Rainer Müller]

On 2017-09-15 20:56, db wrote:
> On 15 Sep 2017, at 18:06, Ryan Schmidt  wrote:
>> Not always, because many ports use local variables when setting the home 
>> directory, so you then have to find the place in the Portfile where that 
>> local variable is set. And also, many ports split the add_users invocation 
>> over multiple lines, such that the line that sets the home directory might 
>> not be on the same line that includes the string "add_users".
> 
> I ran
> 
> dscl . list /Users | grep -v ^_.*
> 
> and noticed the user messagebus, so I filtered it from
> 
> port cat all
> 
> and realised that dbus set it, but MP didn't remove it when uninstalled.
> 
> MP should also take care of that.

That would first require to track which ports actually created and need
that user, which we do not do at the moment.

Anyway, when you uninstall a port, there might still be files on the
filesystem owned by this uid. If the user is deleted, the file remain
owned by that uid. Later the uid will be reused when the next user is
created as it will be assigned the first unallocated uid. The new user
suddenly owns files it was never supposed to own.

This might not only lead to confusion, but if the original port is
installed again later, it could also cause permission problems that are
hard to debug. Leaving the uid allocated indefinitely avoids that.

Rainer

Re: Restoring from Time Machine backup relocates home directories

[db]

On 15 Sep 2017, at 18:06, Ryan Schmidt  wrote:
> Not always, because many ports use local variables when setting the home 
> directory, so you then have to find the place in the Portfile where that 
> local variable is set. And also, many ports split the add_users invocation 
> over multiple lines, such that the line that sets the home directory might 
> not be on the same line that includes the string "add_users".

I ran

dscl . list /Users | grep -v ^_.*

and noticed the user messagebus, so I filtered it from

port cat all

and realised that dbus set it, but MP didn't remove it when uninstalled.

MP should also take care of that.

Re: Restoring from Time Machine backup relocates home directories

[Ryan Schmidt]

> On Sep 15, 2017, at 07:09, db  wrote:
> 
> On 15 Sep 2017, at 04:32, Ryan Schmidt  wrote:
>> I recently reinstalled macOS and, during the setup process, restored from a 
>> Time Machine backup. One would think that after restoration, the machine 
>> would be in a state sufficiently similar to the one it was in at backup 
>> time, but for our purposes, it is not, specifically with regard to user 
>> accounts.
> 
> May I assume that by macOS you mean 10.12?
> 
> Hasn't this problem already been reported in previous versions of OS X?

Right, I'm using 10.12, but I assume the problem exists in earlier versions as 
well. I may have written about the problem before, but if so, I couldn't find 
it, and don't remember what conclusions we may have reached.


> To prepare for the manual fix, would it be enough to run `port cat installed 
> | grep add_user`?

Not always, because many ports use local variables when setting the home 
directory, so you then have to find the place in the Portfile where that local 
variable is set. And also, many ports split the add_users invocation over 
multiple lines, such that the line that sets the home directory might not be on 
the same line that includes the string "add_users".

Re: Restoring from Time Machine backup relocates home directories

[Daniel J. Luke]

On Sep 14, 2017, at 10:32 PM, Ryan Schmidt  wrote:
> I had numerous ports installed that create their own user accounts with 
> add_users or add_user, and of course MacPorts has its macports user. For each 
> of these users, I could select if I wanted the user to be restored, but if 
> so, it indicated that it would relocate that user's home directory to /Users; 
> if a user is to be restored, there is no option presented not to move its 
> home directory. I chose to accept the defaults and restore all users.

Do we know what happens (or anyone want to try) if you don't choose to 
'restore' the user?

Might be good to include instructions for both outcomes and/or include both in 
any plans for how we make MacPorts deal with this automatically. 

-- 
Daniel J. Luke

Re: Restoring from Time Machine backup relocates home directories

[db]

On 15 Sep 2017, at 04:32, Ryan Schmidt  wrote:
> I recently reinstalled macOS and, during the setup process, restored from a 
> Time Machine backup. One would think that after restoration, the machine 
> would be in a state sufficiently similar to the one it was in at backup time, 
> but for our purposes, it is not, specifically with regard to user accounts.

May I assume that by macOS you mean 10.12?

Hasn't this problem already been reported in previous versions of OS X?

To prepare for the manual fix, would it be enough to run `port cat installed | 
grep add_user`?