Re: copy directory tree, mapping to new owners
On 14/09/21 6:50 pm, to...@tuxteam.de wrote: On Tue, Sep 14, 2021 at 12:17:05PM +1200, Richard Hector wrote: On 13/09/21 7:04 pm, to...@tuxteam.de wrote: >On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: >>On 12/09/21 6:52 pm, john doe wrote: > >[...] > >>>If you are doing this in a script, I would use a temporary directory. >>>That way, in case of failure the destination directory is not rongly >>>modified. >>> >>>EG: >>> >>>$ rsync >>> >>>Make the way you want it to be. >>> >>>$ rsync >> >>That is true, but firstly it would require more available space [...] > >This isn't necessary, as you could replace the second `rsync' by a `mv' >(provided your temp tree is on the same storage volume as your target >dir, that is). I was assuming the suggestion was to rsync the source to the temp while the destination still exists, before rsyncing or mv'ing over the top of it. Total of 3 copies (temporarily) rather than 2. Then, it's different. But in your scenario you would probably want to take down whatever "service" relies on the destination dir while the copy is in progress. This is all academic, since rsync with --usermap and --groupmap does what I want, in place. But john doe's proposal had the rationale of "That way, in case of failure the destination directory is not rongly modified." That implies the destination is staying put. Well, I guess deleting it entirely avoids "wrongly modifying" it too :-) In any case, if you haven't the space, you haven't it. Sysadmin's life ain't always nice :) It's available if I want it; everything is on LVM. It's easy to grow. It's also all on a VPS, so expanding the total is a matter of tweaking the settings in a control panel. But it's harder to shrink (especially since I use xfs), so I prefer not to grow it if it's not necessary. Cheers :-) Richard
Re: copy directory tree, mapping to new owners
On Tue 14 Sep 2021 at 08:50:34 (+0200), to...@tuxteam.de wrote: > On Tue, Sep 14, 2021 at 12:17:05PM +1200, Richard Hector wrote: > > On 13/09/21 7:04 pm, to...@tuxteam.de wrote: > > >On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: > > >>On 12/09/21 6:52 pm, john doe wrote: > > > > > >[...] > > > > > >>>If you are doing this in a script, I would use a temporary directory. > > >>>That way, in case of failure the destination directory is not rongly > > >>>modified. > > >>> > > >>>EG: > > >>> > > >>>$ rsync > > >>> > > >>>Make the way you want it to be. > > >>> > > >>>$ rsync > > >> > > >>That is true, but firstly it would require more available space [...] > > > > > >This isn't necessary, as you could replace the second `rsync' by a `mv' > > >(provided your temp tree is on the same storage volume as your target > > >dir, that is). > > > > I was assuming the suggestion was to rsync the source to the temp > > while the destination still exists, before rsyncing or mv'ing over > > the top of it. Total of 3 copies (temporarily) rather than 2. > > Then, it's different. But in your scenario you would probably want > to take down whatever "service" relies on the destination dir while > the copy is in progress. As I read the first paragraph of the OP, the service relies on the *source* files, and the destination is just the copy, being created and/or updated for testing. I presume that is why the OP is not too worried by failure. > In any case, if you haven't the space, you haven't it. Sysadmin's > life ain't always nice :) Cheers, David.
Re: copy directory tree, mapping to new owners
On Tue, Sep 14, 2021 at 12:17:05PM +1200, Richard Hector wrote: > On 13/09/21 7:04 pm, to...@tuxteam.de wrote: > >On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: > >>On 12/09/21 6:52 pm, john doe wrote: > > > >[...] > > > >>>If you are doing this in a script, I would use a temporary directory. > >>>That way, in case of failure the destination directory is not rongly > >>>modified. > >>> > >>>EG: > >>> > >>>$ rsync > >>> > >>>Make the way you want it to be. > >>> > >>>$ rsync > >> > >>That is true, but firstly it would require more available space [...] > > > >This isn't necessary, as you could replace the second `rsync' by a `mv' > >(provided your temp tree is on the same storage volume as your target > >dir, that is). > > I was assuming the suggestion was to rsync the source to the temp > while the destination still exists, before rsyncing or mv'ing over > the top of it. Total of 3 copies (temporarily) rather than 2. Then, it's different. But in your scenario you would probably want to take down whatever "service" relies on the destination dir while the copy is in progress. In any case, if you haven't the space, you haven't it. Sysadmin's life ain't always nice :) Cheers - t signature.asc Description: Digital signature
Re: copy directory tree, mapping to new owners
On 13/09/21 7:04 pm, to...@tuxteam.de wrote: On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: On 12/09/21 6:52 pm, john doe wrote: [...] >If you are doing this in a script, I would use a temporary directory. >That way, in case of failure the destination directory is not rongly >modified. > >EG: > >$ rsync > >Make the way you want it to be. > >$ rsync That is true, but firstly it would require more available space [...] This isn't necessary, as you could replace the second `rsync' by a `mv' (provided your temp tree is on the same storage volume as your target dir, that is). I was assuming the suggestion was to rsync the source to the temp while the destination still exists, before rsyncing or mv'ing over the top of it. Total of 3 copies (temporarily) rather than 2. Cheers, Richard
Re: copy directory tree, mapping to new owners
On Mon, Sep 13, 2021 at 11:45:02AM +1200, Richard Hector wrote: > On 12/09/21 6:52 pm, john doe wrote: [...] > >If you are doing this in a script, I would use a temporary directory. > >That way, in case of failure the destination directory is not rongly > >modified. > > > >EG: > > > >$ rsync > > > >Make the way you want it to be. > > > >$ rsync > > That is true, but firstly it would require more available space [...] This isn't necessary, as you could replace the second `rsync' by a `mv' (provided your temp tree is on the same storage volume as your target dir, that is). Cheers - t signature.asc Description: Digital signature
Re: copy directory tree, mapping to new owners
On 12/09/21 7:46 pm, Teemu Likonen wrote: * 2021-09-12 12:43:29+1200, Richard Hector wrote: The context of my question is that I'm creating (or updating) a test copy of a website. The files are owned by one of two owners, depending on whether they were written by the server (actually php-fpm). To do that, I want all the permissions to remain the same, but the ownership should be changed according to a provided map. Looks exactly like what "rsync --usermap=FROM:TO" can do. There is also "--groupmap" option for mapping groups. Aha - thanks a lot :-) I guess I've been caught not reading the man page thoroughly enough. The pitfalls of thinking that you know everything that a tool does, just because you use it often ... The way the docs are written seems to imply a sender and receiver; I'll have to check it works for a local copy ... it does. Now I need to rewrite my script somewhat, since to do this it's going to have to run as root ... Thanks, Richard
Re: copy directory tree, mapping to new owners
On 12/09/21 6:53 pm, l0f...@tuta.io wrote: # actually not necessary? rsync will create it mkdir -p mysite_test/doc_root You can make a simple test to know that but I would say that rsync doesn't create your destination "root" directory (the one you specify on the command line) unless `--mkpath` is used. I was fairly confident of that, from my regular usage. # The trailing / matters. Does it matter on the source as well? # I generally include it. rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters. Actually, I'm not sure to understand Greg's remark here. In my opinion, trailing slash doesn't matter for destination folder on the contrary of *source* folder. In other words, for me, the following are equal: rsync -a mysite/doc_root/ mysite_test/doc_root/ rsync -a mysite/doc_root/ mysite_test/doc_root But not the following: rsync -a mysite/doc_root mysite_test/doc_root => you will get an extra "doc_root" folder (the source one) in your dest, i.e. : mysite_test/doc_root/doc_root and then the content of doc_root source rsync -a mysite/doc_root/ mysite_test/doc_root => your doc_root (destination) folder will get doc_root content (source) directly Yep. As I've mentioned elsewhere, I habitually include trailing slashes for both source and destination when I'm replicating a whole tree. I couldn't remember the details of what happens when you don't, but I know what happens when I do :-) Cheers, Richard
Re: copy directory tree, mapping to new owners
On 12/09/21 6:52 pm, john doe wrote: On 9/12/2021 3:45 AM, Richard Hector wrote: Thanks, that looks reasonable. It does mean, though, that the files exist for a while with the wrong ownership. That probably doesn't matter, but somehow 'feels wrong' to me. If you are doing this in a script, I would use a temporary directory. That way, in case of failure the destination directory is not rongly modified. EG: $ rsync Make the way you want it to be. $ rsync That is true, but firstly it would require more available space, and secondly, as long as I know about the failure, it doesn't worry me too much. This is a script I will call manually. Given that you want to change the ownership, you may want to emulate the options implied by '-a' but without the ownership option ('-o'). I do that in my existing script (also not -g (group) or -D (specials/devices (which don't exist there anyway)) - and sometimes -p (permissions), if I've pre-created the tree with the permissions I want. The last one is a slightly different use case, but I should make sure I know which use case I'm trying for ... My habits would also lead me to do all of the above from above the two directories in question (in my case, /srv - for /srv/mysite/doc_root and /srv/mysite_test/doc_root) So: cd /srv # actually not necessary? rsync will create it mkdir -p mysite_test/doc_root # The trailing / matters. Does it matter on the source as well? According to the man page it does (1): "CWrsync -avz foo:src/bar/ /data/tmp A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning lqcopy the contents" # I generally include it. In a script, you need to be sure what the cmd does do or not do!!! :) True. But many options I've learned from my everyday usage, and sometimes forgotten the rationale - I just know it does what I want :-) rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters. find mysite_dest -user mysite -exec chown mysite_test {} + # I prefer mysite_test-run; it keeps consistency with # the ${sitename}-run pattern used elsewhere find mysite_dest -user mysite-run -exec chown mysite_test-run {} + Have I broken anything there? Not that I can see but testing will tell! This is what I would do. And I would do it *interactively*. If you insist on making a script, then it will be slightly more complicated, because you'll need to add error checking. The trouble with doing it interactively, when it needs to be done many times (and on several sites), is that each time there's opportunity to make a mistake. And it means the process needs to be documented separately from the script. In fact, I'd incorporate the above in a larger script, which does things like copying the database (and changing the db name in the site config). Error checking in shell scripts is something I certainly need to learn At the very least, ' ['&&'|'||'] handle the error>. If you use Bash and don't need to be POSIX compliant and /or portable you can be a bit more creative. Heh - you've cut my sentence in two, which appears to change the meaning slightly :-) " ... is something I need to learn and practice more". You make it look like I need to learn error checking from scratch :-) I am using bash. And I am currently using 'set -e' to handle that kind of error globally. As I say, as long as I can see that it broke, it doesn't matter too much. and practice more - I tend to rely somewhat on 'knowing' what I'm working with, which is probably not a good idea. > Yes in a script it is like shooting yourself in the foot. I do test it. Cheers, Richard
Re: copy directory tree, mapping to new owners
Hi Curt, 12 sept. 2021, 16:23 de cu...@free.fr: > On 2021-09-12, l0f...@tuta.io wrote: > >> In my opinion, trailing slash doesn't matter for destination folder on >> the contrary of *source* folder. >> > It makes a difference when the source is a file and the destination > directory is inexistant, because in that case a copy of $file named > $destination will be created, which is not what you want here. > I think it is probably not applicable here for the OP (it's a directory) but thanks for this special use case :) This explanation is given in the aforementioned `--mkpath` switch by the way. The moral is: Pay attention to the single one character you type on the command line! Best regards, l0f4r0
Re: copy directory tree, mapping to new owners
On 2021-09-12, l0f...@tuta.io wrote: >> # The trailing / matters. Does it matter on the source as well? >> # I generally include it. >> rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / >> matters. >> > Actually, I'm not sure to understand Greg's remark here. > > In my opinion, trailing slash doesn't matter for destination folder on > the contrary of *source* folder. > It makes a difference when the source is a file and the destination directory is inexistant, because in that case a copy of $file named $destination will be created, which is not what you want here.
Re: copy directory tree, mapping to new owners
On Du, 12 sep 21, 08:52:55, john doe wrote: > On 9/12/2021 3:45 AM, Richard Hector wrote: > > On 12/09/21 12:52 pm, Greg Wooledge wrote: > > > > > > cd /src > > > mkdir -p /dest > > > rsync -a . /dest/ # The trailing / matters. > > > cd /dest > > > find . -user mysite -exec chown mysite_test {} + > > > find . -user mysite-run -exec chown mysite-run_test {} + > > Given that you want to change the ownership, you may want to emulate the > options implied by '-a' but without the ownership option ('-o'). It might not matter for a site[1], but do note that -a doesn't include -H (preserve hardlinks), -A (preserve ACLs) and -X (preserve extended attributes). [1] It definitely matters for system installations, as I found out the hard way. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: copy directory tree, mapping to new owners
On Sun, Sep 12, 2021 at 10:46:01AM +0300, Teemu Likonen wrote: [...] > Looks exactly like what "rsync --usermap=FROM:TO" can do. There is also > "--groupmap" option for mapping groups. Never underestimate rsync :-) Actually, I do read its manpage from time to time. Should do it more often, perhaps as (yet another) after-breakfast routine. Thanks, Teemu, this one wasn't aware to me! Cheers - t signature.asc Description: Digital signature
Re: copy directory tree, mapping to new owners
* 2021-09-12 12:43:29+1200, Richard Hector wrote: > The context of my question is that I'm creating (or updating) a test > copy of a website. The files are owned by one of two owners, depending > on whether they were written by the server (actually php-fpm). > > To do that, I want all the permissions to remain the same, but the > ownership should be changed according to a provided map. Looks exactly like what "rsync --usermap=FROM:TO" can do. There is also "--groupmap" option for mapping groups. -- /// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/ // OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450 signature.asc Description: PGP signature
Re: copy directory tree, mapping to new owners
12 sept. 2021, 08:53 de l0f...@tuta.io: > 12 sept. 2021, 03:45 de rich...@walnut.gen.nz: > >> # actually not necessary? rsync will create it >> mkdir -p mysite_test/doc_root >> > You can make a simple test to know that but I would say that rsync doesn't > create your destination "root" directory (the one you specify on the command > line) unless `--mkpath` is used. > Actually, I've just did some tests and it appears that rsync creates the destination folder if its parent already exists and if you have the right permissions. For example ("my" and "dest" don't exist yet): rsync -a src/ dest => "dest" is created rsync -a src/ my/dest => error, "my" and "dest" don't exist (and are not created then) rsync --mkpath -a src/ my/dest => "my" and "dest" are created l0f4r0
Re: copy directory tree, mapping to new owners
Hi, 12 sept. 2021, 03:45 de rich...@walnut.gen.nz: > On 12/09/21 12:52 pm, Greg Wooledge wrote: > >> cd /src >> mkdir -p /dest >> rsync -a . /dest/ # The trailing / matters. >> cd /dest >> find . -user mysite -exec chown mysite_test {} + >> find . -user mysite-run -exec chown mysite-run_test {} + >> > > [...] > # actually not necessary? rsync will create it > mkdir -p mysite_test/doc_root > You can make a simple test to know that but I would say that rsync doesn't create your destination "root" directory (the one you specify on the command line) unless `--mkpath` is used. > # The trailing / matters. Does it matter on the source as well? > # I generally include it. > rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters. > Actually, I'm not sure to understand Greg's remark here. In my opinion, trailing slash doesn't matter for destination folder on the contrary of *source* folder. In other words, for me, the following are equal: rsync -a mysite/doc_root/ mysite_test/doc_root/ rsync -a mysite/doc_root/ mysite_test/doc_root But not the following: rsync -a mysite/doc_root mysite_test/doc_root => you will get an extra "doc_root" folder (the source one) in your dest, i.e. : mysite_test/doc_root/doc_root and then the content of doc_root source rsync -a mysite/doc_root/ mysite_test/doc_root => your doc_root (destination) folder will get doc_root content (source) directly Best regards, l0f4r0
Re: copy directory tree, mapping to new owners
On 9/12/2021 3:45 AM, Richard Hector wrote: On 12/09/21 12:52 pm, Greg Wooledge wrote: On Sun, Sep 12, 2021 at 12:43:29PM +1200, Richard Hector wrote: The context of my question is that I'm creating (or updating) a test copy of a website. The files are owned by one of two owners, depending on whether they were written by the server (actually php-fpm). To do that, I want all the permissions to remain the same, but the ownership should be changed according to a provided map. For example, if the old file was owned by 'mysite', the copy should be owned by 'mysite_test'. If the old file was owned by 'mysite-run' (the user php runs as), the copy should be owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make things easier, I can live with that). cd /src mkdir -p /dest rsync -a . /dest/ # The trailing / matters. cd /dest find . -user mysite -exec chown mysite_test {} + find . -user mysite-run -exec chown mysite-run_test {} + Thanks, that looks reasonable. It does mean, though, that the files exist for a while with the wrong ownership. That probably doesn't matter, but somehow 'feels wrong' to me. If you are doing this in a script, I would use a temporary directory. That way, in case of failure the destination directory is not rongly modified. EG: $ rsync Make the way you want it to be. $ rsync Given that you want to change the ownership, you may want to emulate the options implied by '-a' but without the ownership option ('-o'). My habits would also lead me to do all of the above from above the two directories in question (in my case, /srv - for /srv/mysite/doc_root and /srv/mysite_test/doc_root) So: cd /srv # actually not necessary? rsync will create it mkdir -p mysite_test/doc_root # The trailing / matters. Does it matter on the source as well? According to the man page it does (1): "CWrsync -avz foo:src/bar/ /data/tmp A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning lqcopy the contents" # I generally include it. In a script, you need to be sure what the cmd does do or not do!!! :) rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters. find mysite_dest -user mysite -exec chown mysite_test {} + # I prefer mysite_test-run; it keeps consistency with # the ${sitename}-run pattern used elsewhere find mysite_dest -user mysite-run -exec chown mysite_test-run {} + Have I broken anything there? Not that I can see but testing will tell! This is what I would do. And I would do it *interactively*. If you insist on making a script, then it will be slightly more complicated, because you'll need to add error checking. The trouble with doing it interactively, when it needs to be done many times (and on several sites), is that each time there's opportunity to make a mistake. And it means the process needs to be documented separately from the script. In fact, I'd incorporate the above in a larger script, which does things like copying the database (and changing the db name in the site config). Error checking in shell scripts is something I certainly need to learn At the very least, ' ['&&'|'||'] handle the error>. If you use Bash and don't need to be POSIX compliant and /or portable you can be a bit more creative. and practice more - I tend to rely somewhat on 'knowing' what I'm working with, which is probably not a good idea. > Yes in a script it is like shooting yourself in the foot. 1) https://linux.die.net/man/1/rsync -- John Doe
Re: copy directory tree, mapping to new owners
On 12/09/21 12:52 pm, Greg Wooledge wrote: On Sun, Sep 12, 2021 at 12:43:29PM +1200, Richard Hector wrote: The context of my question is that I'm creating (or updating) a test copy of a website. The files are owned by one of two owners, depending on whether they were written by the server (actually php-fpm). To do that, I want all the permissions to remain the same, but the ownership should be changed according to a provided map. For example, if the old file was owned by 'mysite', the copy should be owned by 'mysite_test'. If the old file was owned by 'mysite-run' (the user php runs as), the copy should be owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make things easier, I can live with that). cd /src mkdir -p /dest rsync -a . /dest/ # The trailing / matters. cd /dest find . -user mysite -exec chown mysite_test {} + find . -user mysite-run -exec chown mysite-run_test {} + Thanks, that looks reasonable. It does mean, though, that the files exist for a while with the wrong ownership. That probably doesn't matter, but somehow 'feels wrong' to me. My habits would also lead me to do all of the above from above the two directories in question (in my case, /srv - for /srv/mysite/doc_root and /srv/mysite_test/doc_root) So: cd /srv # actually not necessary? rsync will create it mkdir -p mysite_test/doc_root # The trailing / matters. Does it matter on the source as well? # I generally include it. rsync -a mysite/doc_root/ mysite_test/doc_root/ # The trailing / matters. find mysite_dest -user mysite -exec chown mysite_test {} + # I prefer mysite_test-run; it keeps consistency with # the ${sitename}-run pattern used elsewhere find mysite_dest -user mysite-run -exec chown mysite_test-run {} + Have I broken anything there? This is what I would do. And I would do it *interactively*. If you insist on making a script, then it will be slightly more complicated, because you'll need to add error checking. The trouble with doing it interactively, when it needs to be done many times (and on several sites), is that each time there's opportunity to make a mistake. And it means the process needs to be documented separately from the script. In fact, I'd incorporate the above in a larger script, which does things like copying the database (and changing the db name in the site config). Error checking in shell scripts is something I certainly need to learn and practice more - I tend to rely somewhat on 'knowing' what I'm working with, which is probably not a good idea. Thanks, Richard
Re: copy directory tree, mapping to new owners
On Sun, Sep 12, 2021 at 12:43:29PM +1200, Richard Hector wrote: > The context of my question is that I'm creating (or updating) a test copy of > a website. The files are owned by one of two owners, depending on whether > they were written by the server (actually php-fpm). > > To do that, I want all the permissions to remain the same, but the ownership > should be changed according to a provided map. For example, if the old file > was owned by 'mysite', the copy should be owned by 'mysite_test'. If the old > file was owned by 'mysite-run' (the user php runs as), the copy should be > owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make > things easier, I can live with that). cd /src mkdir -p /dest rsync -a . /dest/ # The trailing / matters. cd /dest find . -user mysite -exec chown mysite_test {} + find . -user mysite-run -exec chown mysite-run_test {} + This is what I would do. And I would do it *interactively*. If you insist on making a script, then it will be slightly more complicated, because you'll need to add error checking.
copy directory tree, mapping to new owners
Hi all, The context of my question is that I'm creating (or updating) a test copy of a website. The files are owned by one of two owners, depending on whether they were written by the server (actually php-fpm). To do that, I want all the permissions to remain the same, but the ownership should be changed according to a provided map. For example, if the old file was owned by 'mysite', the copy should be owned by 'mysite_test'. If the old file was owned by 'mysite-run' (the user php runs as), the copy should be owned by 'mysite_test-run' (if that has to be 'mysite-run_test' to make things easier, I can live with that). Group ownership is or would be the same, but in fact it's simpler because both users are members of teh same group - all files are or should be group-owned by the same group (mysite, mapping to mysite_test). Is there any pre-existing tool that will do this? Or will I need to write a perl script or similar? What I've done in the past is use the same users for both production and testing, and do the copy by running rsync as the mysite user, but firstly I'd rather have more isolation between the two, secondly the mysite user might not be able to read all the mysite-run files, and thirdly the ownership of those (mysite-run) files gets changed, making it an imperfect copy. Thanks in advance, Richard