Re: Question on folder sync with directory name translation
Yeah, I don't have other symlink. But I'm thinking of changing my folder structure to reflect the data I really need on the NAS. So, as a side effect, the special rsync is not needed any more :) Anyway, thanks for the answers! Bye *___Gionata Boccalini* 2015-06-09 13:25 GMT+02:00 Michael Johnson - MJ m...@revmj.com: Should be as long as you don't have other symlinks in the tree. On Mon, Jun 8, 2015, 15:14 Gionata Boccalini gionata.boccal...@gmail.com wrote: OK , but then the solution with symlinks is equivalent, just with the right options for rsync. Make the link. Sync + exclude. Remove the link. Don't have to live with the folder on the source. *___* *Gionata Boccalini* 2015-06-08 22:49 GMT+02:00 Michael Johnson - MJ m...@revmj.com: Oh, actually, I just thought of a couple other another options that don't require any multiplexing or ssh keys, but it would require that your source machine is linux. The first option would be: mkdir /A/FolderB mount --bind /A/FolderA /A/FolderB Then just exclude /A/FolderA from the rsync and you are done. This does mean that you have to be ok with /A/FolderB existing on the source. The second option would be to use somthing like aufs or overlayfs to create a new mountpoint that contains everything you want and perhaps with a little bit of mount --bind thrown in. I just saw your response, and and what you describe makes sense. Sounds like the mount bind option + exclude might be the most elegant option for this case. Just make sure to add the bind mount into your fstab so it comes back after a reboot. :) On Mon, Jun 8, 2015 at 1:18 PM, Michael Johnson - MJ m...@revmj.com wrote: Thought I would chime in here. To the best of my knowledge what you are trying to do cannot be done in a single run. I supposed --fuzzy might work for you but I've never used that option and it sounds scary to me. Perhaps if I spent some time and learned the methodology it uses I would be less concerned. If a feature like this were to be added, it seems like it would make sense to add it as a new filter type. But mainly you've piqued my curiosity. The requirement that it be able to happen in a single rsync run seems very odd. Is this just a desire, or is there really something that bad that would happen if you did one pass syncing A to B excluding FolderA and FolderB and then a second pass syncing FolderA to FolderB? The most likely scenario I imagine is that you are running this by hand and manually enter the SSH password. Given the process take a long time, you don't want to enter the password again mid stream. If this is the case, you could set up SSH keys to allow this to happen without a manually typed password. You can find how to set that up here: http://www.chainsawonatireswing.com/2012/01/15/ssh-into-your-synology-diskstation-with-ssh-keys/ If you don't want to do ssh keys w/o a password, you could use ssh agent with keys. Finally you could also utilize ssh multiplexing (it looks like that should work with the synology nass). You can find information about this option here: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing The other (very unlikely) scenario I can imagine is that there is something that prevents you from logging in more than once every X hours/days like a time lock safe. multiplexing would help here as well. But this scenario seems unlikely, it was probably silly to even mention it. If there is another case I have not considered I would be very interested to know where this requirement comes from. Thanks! Hope there was something useful for you in all this. :) On Sun, Jun 7, 2015 at 9:38 AM, Gionata Boccalini gionata.boccal...@gmail.com wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. *___Gionata Boccalini* -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read:
Re: Question on folder sync with directory name translation
Should be as long as you don't have other symlinks in the tree. On Mon, Jun 8, 2015, 15:14 Gionata Boccalini gionata.boccal...@gmail.com wrote: OK , but then the solution with symlinks is equivalent, just with the right options for rsync. Make the link. Sync + exclude. Remove the link. Don't have to live with the folder on the source. *___* *Gionata Boccalini* 2015-06-08 22:49 GMT+02:00 Michael Johnson - MJ m...@revmj.com: Oh, actually, I just thought of a couple other another options that don't require any multiplexing or ssh keys, but it would require that your source machine is linux. The first option would be: mkdir /A/FolderB mount --bind /A/FolderA /A/FolderB Then just exclude /A/FolderA from the rsync and you are done. This does mean that you have to be ok with /A/FolderB existing on the source. The second option would be to use somthing like aufs or overlayfs to create a new mountpoint that contains everything you want and perhaps with a little bit of mount --bind thrown in. I just saw your response, and and what you describe makes sense. Sounds like the mount bind option + exclude might be the most elegant option for this case. Just make sure to add the bind mount into your fstab so it comes back after a reboot. :) On Mon, Jun 8, 2015 at 1:18 PM, Michael Johnson - MJ m...@revmj.com wrote: Thought I would chime in here. To the best of my knowledge what you are trying to do cannot be done in a single run. I supposed --fuzzy might work for you but I've never used that option and it sounds scary to me. Perhaps if I spent some time and learned the methodology it uses I would be less concerned. If a feature like this were to be added, it seems like it would make sense to add it as a new filter type. But mainly you've piqued my curiosity. The requirement that it be able to happen in a single rsync run seems very odd. Is this just a desire, or is there really something that bad that would happen if you did one pass syncing A to B excluding FolderA and FolderB and then a second pass syncing FolderA to FolderB? The most likely scenario I imagine is that you are running this by hand and manually enter the SSH password. Given the process take a long time, you don't want to enter the password again mid stream. If this is the case, you could set up SSH keys to allow this to happen without a manually typed password. You can find how to set that up here: http://www.chainsawonatireswing.com/2012/01/15/ssh-into-your-synology-diskstation-with-ssh-keys/ If you don't want to do ssh keys w/o a password, you could use ssh agent with keys. Finally you could also utilize ssh multiplexing (it looks like that should work with the synology nass). You can find information about this option here: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing The other (very unlikely) scenario I can imagine is that there is something that prevents you from logging in more than once every X hours/days like a time lock safe. multiplexing would help here as well. But this scenario seems unlikely, it was probably silly to even mention it. If there is another case I have not considered I would be very interested to know where this requirement comes from. Thanks! Hope there was something useful for you in all this. :) On Sun, Jun 7, 2015 at 9:38 AM, Gionata Boccalini gionata.boccal...@gmail.com wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. *___Gionata Boccalini* -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Michael Johnson - MJ -- Michael Johnson - MJ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on folder sync with directory name translation
Hello, I've been tasked with migrating a smallish (@90 mailboxes) company from a linux/dovecot mail server to Office 365, and after experiencing a ton of issues with Microsoft's native Imap syncing tool, I decided to use Imapsync, and it is working perfectly. It has the ability to add a simple regex translation for mapping folders of different names - mainly used for the special use folders that can have different names depending on the server/client involved. Ie, dovecot/Thunderbird use 'Sent', 'Trash' and 'Junk', while Office65/Outlook use 'Sent Items', 'Deleted Items' and 'Junk Emails'. So, a few simple regex lines in the ImapSync commandline options and problem solved: --regextrans2 's/Sent$/Sent Items/' \ --regextrans2 's/Trash$/Deleted Items/' \ --regextrans2 's/Junk$/Junk Email/' \ I do use a few more to catch a few other cases, but you get the idea. I'd be surprised if rsync can't do something similar? Charles On 6/7/2015 12:38 PM, Gionata Boccalini gionata.boccal...@gmail.com wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. /___ Gionata Boccalini/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on folder sync with directory name translation
Thanks Joe for the reply: 1) why do you say to use fuzzy twice? Do you mean in both directions? 2) I have to mention that the remote system is a Synology NAS, which for whatever reason (I can't think about), doesn't support symlinks, even in the same disk volume or share! But I could make some symlinks in the local system (archlinux) and user exclude filters to remove the directories I don't want to synchronize. But still there is no way of doing this with rsync options alone? *___Gionata Boccalini* 2015-06-07 19:28 GMT+02:00 Joe jose...@main.nc.us: I'm sure one of the experts will have a better answer, but two things come to mind as options to explore: 1) Use --fuzzy twice so files which are the same but possibly with different names and locations are synced 2) Use some sort of symlinks on the destination so the names actually match (these could be added and removed right before and after the rsync respectively if you don't want them around later.) Symlinks are tricky with rsync. You have to set the options right, so I'm not sure of the exact details. On 06/07/2015 12:38 PM, Gionata Boccalini wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. /___ Gionata Boccalini/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on folder sync with directory name translation
OK , but then the solution with symlinks is equivalent, just with the right options for rsync. Make the link. Sync + exclude. Remove the link. Don't have to live with the folder on the source. *___Gionata Boccalini* 2015-06-08 22:49 GMT+02:00 Michael Johnson - MJ m...@revmj.com: Oh, actually, I just thought of a couple other another options that don't require any multiplexing or ssh keys, but it would require that your source machine is linux. The first option would be: mkdir /A/FolderB mount --bind /A/FolderA /A/FolderB Then just exclude /A/FolderA from the rsync and you are done. This does mean that you have to be ok with /A/FolderB existing on the source. The second option would be to use somthing like aufs or overlayfs to create a new mountpoint that contains everything you want and perhaps with a little bit of mount --bind thrown in. I just saw your response, and and what you describe makes sense. Sounds like the mount bind option + exclude might be the most elegant option for this case. Just make sure to add the bind mount into your fstab so it comes back after a reboot. :) On Mon, Jun 8, 2015 at 1:18 PM, Michael Johnson - MJ m...@revmj.com wrote: Thought I would chime in here. To the best of my knowledge what you are trying to do cannot be done in a single run. I supposed --fuzzy might work for you but I've never used that option and it sounds scary to me. Perhaps if I spent some time and learned the methodology it uses I would be less concerned. If a feature like this were to be added, it seems like it would make sense to add it as a new filter type. But mainly you've piqued my curiosity. The requirement that it be able to happen in a single rsync run seems very odd. Is this just a desire, or is there really something that bad that would happen if you did one pass syncing A to B excluding FolderA and FolderB and then a second pass syncing FolderA to FolderB? The most likely scenario I imagine is that you are running this by hand and manually enter the SSH password. Given the process take a long time, you don't want to enter the password again mid stream. If this is the case, you could set up SSH keys to allow this to happen without a manually typed password. You can find how to set that up here: http://www.chainsawonatireswing.com/2012/01/15/ssh-into-your-synology-diskstation-with-ssh-keys/ If you don't want to do ssh keys w/o a password, you could use ssh agent with keys. Finally you could also utilize ssh multiplexing (it looks like that should work with the synology nass). You can find information about this option here: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing The other (very unlikely) scenario I can imagine is that there is something that prevents you from logging in more than once every X hours/days like a time lock safe. multiplexing would help here as well. But this scenario seems unlikely, it was probably silly to even mention it. If there is another case I have not considered I would be very interested to know where this requirement comes from. Thanks! Hope there was something useful for you in all this. :) On Sun, Jun 7, 2015 at 9:38 AM, Gionata Boccalini gionata.boccal...@gmail.com wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. *___Gionata Boccalini* -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Michael Johnson - MJ -- Michael Johnson - MJ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on folder sync with directory name translation
I should describe the problem more in details, but I believe this is off topic for this list. The FolderA is named Musica (in Italian) because.. I like it that way.. and is in my home folder. PC # /home/gionata/Musica FolderB MUST be named music, in my home folder on the NFS filesystem, NAS # /volume1/homes/gionata/music *because the synology media parse service only indexes the directory name music in the user home dirs!* (Damn Syno...) So I have different name on different system for the same folder! I could change the name on my home, do it in two runs or do some symlinks.. but I was just wondering if a more elegant solution could be found within rsync. Anyway I can login with ssh as many time I wantm with RSA key pair. Even multiplexing is not necessary, but thanks anyway for the info. I hope now the problem is more clear. Is not even a real problem.. :) *___Gionata Boccalini* 2015-06-08 22:18 GMT+02:00 Michael Johnson - MJ m...@revmj.com: Thought I would chime in here. To the best of my knowledge what you are trying to do cannot be done in a single run. I supposed --fuzzy might work for you but I've never used that option and it sounds scary to me. Perhaps if I spent some time and learned the methodology it uses I would be less concerned. If a feature like this were to be added, it seems like it would make sense to add it as a new filter type. But mainly you've piqued my curiosity. The requirement that it be able to happen in a single rsync run seems very odd. Is this just a desire, or is there really something that bad that would happen if you did one pass syncing A to B excluding FolderA and FolderB and then a second pass syncing FolderA to FolderB? The most likely scenario I imagine is that you are running this by hand and manually enter the SSH password. Given the process take a long time, you don't want to enter the password again mid stream. If this is the case, you could set up SSH keys to allow this to happen without a manually typed password. You can find how to set that up here: http://www.chainsawonatireswing.com/2012/01/15/ssh-into-your-synology-diskstation-with-sshRSA-keys/ http://www.chainsawonatireswing.com/2012/01/15/ssh-into-your-synology-diskstation-with-ssh-keys/ If you don't want to do ssh keys w/o a password, you could use ssh agent with keys. Finally you could also utilize ssh multiplexing (it looks like that should work with the synology nass). You can find information about this option here: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing The other (very unlikely) scenario I can imagine is that there is something that prevents you from logging in more than once every X hours/days like a time lock safe. multiplexing would help here as well. But this scenario seems unlikely, it was probably silly to even mention it. If there is another case I have not considered I would be very interested to know where this requirement comes from. Thanks! Hope there was something useful for you in all this. :) On Sun, Jun 7, 2015 at 9:38 AM, Gionata Boccalini gionata.boccal...@gmail.com wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. *___Gionata Boccalini* -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Michael Johnson - MJ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on folder sync with directory name translation
The symlinks was mostly a shot in the dark. They're often useful when you need synonyms. The --fuzzy: I believe once handles different names and the second one adds different locations. I have thought about using it for issues I have reorganizing collections of media files, but never got around to actually testing it. Joe On 06/08/2015 03:40 PM, Gionata Boccalini wrote: Thanks Joe for the reply: 1) why do you say to use fuzzy twice? Do you mean in both directions? 2) I have to mention that the remote system is a Synology NAS, which for whatever reason (I can't think about), doesn't support symlinks, even in the same disk volume or share! But I could make some symlinks in the local system (archlinux) and user exclude filters to remove the directories I don't want to synchronize. But still there is no way of doing this with rsync options alone? /___ Gionata Boccalini/ 2015-06-07 19:28 GMT+02:00 Joe jose...@main.nc.us mailto:jose...@main.nc.us: I'm sure one of the experts will have a better answer, but two things come to mind as options to explore: 1) Use --fuzzy twice so files which are the same but possibly with different names and locations are synced 2) Use some sort of symlinks on the destination so the names actually match (these could be added and removed right before and after the rsync respectively if you don't want them around later.) Symlinks are tricky with rsync. You have to set the options right, so I'm not sure of the exact details. On 06/07/2015 12:38 PM, Gionata Boccalini wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. /___ Gionata Boccalini/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Question on folder sync with directory name translation
I'm sure one of the experts will have a better answer, but two things come to mind as options to explore: 1) Use --fuzzy twice so files which are the same but possibly with different names and locations are synced 2) Use some sort of symlinks on the destination so the names actually match (these could be added and removed right before and after the rsync respectively if you don't want them around later.) Symlinks are tricky with rsync. You have to set the options right, so I'm not sure of the exact details. On 06/07/2015 12:38 PM, Gionata Boccalini wrote: Hello everyone, I'm new to this mailing list but I have been using rsync for some years up to now. I'm trying to synchronize two directory trees, but I want a special behavior that I didn't find on the net nor in the manual (or maybe there is a combination of options to get what I want but I couldn't find it). Tree A is like: A * * FolderA * * And tree B (on a remote filesystem) is like: B * * * FolderB * * I have to synchronize everything in one rsync run, like rysnc -arv A/ B/ but I want FolderA to be synchronized with FolderB. They must contain the same files but have a different name! I want something like a directory name translation in the rsync run... Is it possible? Do you see any another way of doing this? (A part of using two rsync runs) Please let me know if I didn't explain the problem correctly or you need further information. Thank you for your attention and time. Best regards. /___ Gionata Boccalini/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
