Re: Rsync between OpenVMS OpenVMS
On Thu, Aug 16, 2012 at 09:24:56PM +, Horn, James wrote: I'm looking at need to duplicate some drives data from one OpenVMS site to another OpenVMS site. Clustering is not an option, so was wondering if Rsync could be used between two OpenVMS sites and if there would be any issues with the files. I've never installed rsync on VMS, but don't forget that you can use $ backup. It will work fine and will reproduce the tree structure, but will only allow a single thread across the network. That is, $ backup on the local system and output a save_set on the remote end, you can then unpack the save_set on the remote system. James Horn SHSU 2449 Computer Services Sam Houston State University Huntsville TX 77340 Phone: (936) 294-1042 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Brian R Cuttler brian.cutt...@wadsworth.org Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 IMPORTANT NOTICE: This e-mail and any attachments may contain confidential or sensitive information which is, or may be, legally privileged or otherwise protected by law from further disclosure. It is intended only for the addressee. If you received this in error or from someone who was not authorized to send it to you, please do not distribute, copy or use it or any attachments. Please notify the sender immediately by reply e-mail and delete this from your system. Thank you for your cooperation. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
jw schultz wrote: On Tue, Oct 21, 2003 at 11:28:11PM -0400, John E. Malmberg wrote: There are approximately 235 global and static variables in rsync, and of those 105 are obviously never modified after the fork() takes place. That may be the case for several of the others, like the 10 in batch.c As the use of the remaining ones are identified, that issue is easily fixed. What I need to do is map the routines called by the receive process/thread to see what static/global variables that they modify. If you put the ones that get modified post-fork into a thread-local structure there is a reasonable chance of that getting into mainline. Now what would be more acceptable: Adding a context parameter to all the routines that reference the global/static variables that are currently shared? or Having each routine that uses those type of variables to go through the work of determining which thread that they are in? It seems to me that adding the context parameter would make it easier to maintain, and have a lower overhead. It also saves having thread lookup calls added to the routines. In some other cases, it looks like the sharing should not hurt, but their should be some synchronization. Since there are only two processes or threads that need to be tracked, while the parameter could be a pointer to global structure, it can be disruptive to put all the local thread specific static values in that structure. Mainly because a lot of them have the same symbol names, just different scope. So there could be a global variable that is void * recv_context, and a void * send_context for the sender for completeness. A routine that needs separate local static variables would check the context parameter passed to it against one of those pointers, and then use the appropriate set of local variables. With this, the static variables would become an array, and the context parameter would determine if index 0 or 1 was used. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
On Wed, Oct 22, 2003 at 11:59:48PM -0400, John E. Malmberg wrote: jw schultz wrote: On Tue, Oct 21, 2003 at 11:28:11PM -0400, John E. Malmberg wrote: There are approximately 235 global and static variables in rsync, and of those 105 are obviously never modified after the fork() takes place. That may be the case for several of the others, like the 10 in batch.c As the use of the remaining ones are identified, that issue is easily fixed. What I need to do is map the routines called by the receive process/thread to see what static/global variables that they modify. If you put the ones that get modified post-fork into a thread-local structure there is a reasonable chance of that getting into mainline. Now what would be more acceptable: Adding a context parameter to all the routines that reference the global/static variables that are currently shared? or Having each routine that uses those type of variables to go through the work of determining which thread that they are in? It seems to me that adding the context parameter would make it easier to maintain, and have a lower overhead. It also saves having thread lookup calls added to the routines. In some other cases, it looks like the sharing should not hurt, but their should be some synchronization. Long term thread support would be to use a pointer in thread local storage or to pass it to any functions that use it. For now just define the structure and treat it as global in mainline. Since there are only two processes or threads that need to be tracked, while the parameter could be a pointer to global structure, it can be disruptive to put all the local thread specific static values in that structure. Mainly because a lot of them have the same symbol names, just different scope. So there could be a global variable that is void * recv_context, and a void * send_context for the sender for completeness. That sounds like it might be OK to me but will require much more thought and discussion. A routine that needs separate local static variables would check the context parameter passed to it against one of those pointers, and then use the appropriate set of local variables. That sort of indirection just complicates things. If you are going to pass a context parameter just pass the pointer to the structure itself. With this, the static variables would become an array, and the context parameter would determine if index 0 or 1 was used. There are up to three processes. Local has sender, generator and receiver. Daemon has the daemon and either the sender, or the generator and receiver. Client has either sender or generator and receiver. Getting it threadable will have to be done by inches. The structure is a single step that should help to reduce the size of your patches while enhancing the documentation of some of the globals. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
jw schultz wrote: Sounds promising. The pitfall you with rsync in threads is that rsync forks with a COW expectation using a great deal of data set prior to the fork. Some of that data is altered. In particular a slew of global variables that must become thread unique when modified or things will break in subtle ways. Yes, I can see a few cases of this. There are approximately 235 global and static variables in rsync, and of those 105 are obviously never modified after the fork() takes place. That may be the case for several of the others, like the 10 in batch.c As the use of the remaining ones are identified, that issue is easily fixed. What I need to do is map the routines called by the receive process/thread to see what static/global variables that they modify. All the more reason to get more people working on this that want it to work on the OpenVMS operating system. Several people have contacted me that they are willing to assist. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
On Tue, Oct 21, 2003 at 11:28:11PM -0400, John E. Malmberg wrote: jw schultz wrote: Sounds promising. The pitfall you with rsync in threads is that rsync forks with a COW expectation using a great deal of data set prior to the fork. Some of that data is altered. In particular a slew of global variables that must become thread unique when modified or things will break in subtle ways. Yes, I can see a few cases of this. There are approximately 235 global and static variables in rsync, and of those 105 are obviously never modified after the fork() takes place. That may be the case for several of the others, like the 10 in batch.c As the use of the remaining ones are identified, that issue is easily fixed. What I need to do is map the routines called by the receive process/thread to see what static/global variables that they modify. All the more reason to get more people working on this that want it to work on the OpenVMS operating system. Several people have contacted me that they are willing to assist. If you put the ones that get modified post-fork into a thread-local structure there is a reasonable chance of that getting into mainline. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
jw schultz wrote: On Sun, Oct 12, 2003 at 12:38:40AM -0400, John E. Malmberg wrote: I am trying to restart getting rsync to run on OpenVMS, and find a way around the fork() issue, posibly using POSIX threads. It occurs to me that i may have been overly encouraging in my last followup. Getting rsync to work using threads, whether pthreads or a superior form, will require enormous intrusive changes to the code. While my testing is far from complete, and I have a few OpenVMS specific things to work out, it appears that the changes to make the client use pthreads were minor, and as of yesterday a client on OpenVMS is able to properly replicate a sample directory of plain text files from a local server running rsync on Windows 2000. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
On Mon, Oct 20, 2003 at 12:20:27AM -0400, John E. Malmberg wrote: John E. Malmberg wrote: While my testing is far from complete, and I have a few OpenVMS specific things to work out, it appears that the changes to make the client use pthreads were minor, and as of yesterday a client on OpenVMS is able to properly replicate a sample directory of plain text files from a local server running rsync on Windows 2000. I just successfully used rsync running on OpenVMS to download the current rsync development tree. I deleted a few files, and reran rsync which repopulated just the missing files. It looks like I have a working client. :-) My next step is to make sure that I can build the current CVS build and post a beta kit for ftp download with build instructions for the other OpenVMS developers that have expressed an interest in working on this. Sounds promising. The pitfall you with rsync in threads is that rsync forks with a COW expectation using a great deal of data set prior to the fork. Some of that data is altered. In particular a slew of global variables that must become thread unique when modified or things will break in subtle ways. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Pysync 2.24 release, was Re: rsync on OpenVMS
On Tue, 2003-10-14 at 11:01, Donovan Baarda wrote: On Mon, 2003-10-13 at 13:00, John E. Malmberg wrote: jw schultz wrote: On Sun, Oct 12, 2003 at 12:38:40AM -0400, John E. Malmberg wrote: [...] I have not heard of unison. I have heard that pysync was successful in a limited test on OpenVMS. As near as I can tell though the librsync it is based on is a bit out of date. [...] Something possibly worth trying on it is psyco... it compiles python to native code on the fly using a simple import psyco. Pure python is a bit slow compared to native C implementations, but psyco could help close the gap a bit. Following up on this... I tried using psyco with python2.2 and it cut the pysync tests on my machine from 21secs down to 14secs... that's a 33% speedup. In the past I'd tried using pyrex to speed things up with no success. psyco not only gives a better boost, but is much easier to use. I haven't touched pysync for a while, but it should still work with the latest librsync as the API hasn't changed. If there are any problems, please let me know. I believe rdiff-backup also has a python wrapper for librsync that might be more advanced than the one in pysync. I have plans for both pysync and librsync, but I haven't worked on them much lately. I find I am mostly motivated by feedback from others when funding is not available :-) This little bit of interest motivated me to have a look at it again, and I've just released version 2.24. From it's NEWS: Updates between release 2.16 and 2.24 - * Added TODO and NEWS files. * Changed to use psyco if available, giving a 33% speedup. * Updated to use librsync 0.9.6. * Changed to using a faster md4sum implementation based on the librsync implementation, modified to use the RSA API. * Added rollin/rollout support to historical adler32.py. * Minor cleanups to rollsum code. * Minor tweaks to handling of block fragment matching. -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
On Mon, 2003-10-13 at 13:00, John E. Malmberg wrote: jw schultz wrote: On Sun, Oct 12, 2003 at 12:38:40AM -0400, John E. Malmberg wrote: [...] I do not know but if OpenVMS support is a problem for rsync proper you might wish to look at pysync or unison which might meet your immediate needs. I have not heard of unison. I have heard that pysync was successful in a limited test on OpenVMS. As near as I can tell though the librsync it cool... someone playing with pysync :-) is based on is a bit out of date. pysync includes a pure-python implementation that will run on anything that can run python, as well as a basic wrapper for librsync. It also includes extension modules for md4sum and rollsum that speed it up a fair bit. Something possibly worth trying on it is psyco... it compiles python to native code on the fly using a simple import psyco. Pure python is a bit slow compared to native C implementations, but psyco could help close the gap a bit. librsync is still under active development by myself and others, and recently had a new release. I haven't touched pysync for a while, but it should still work with the latest librsync as the API hasn't changed. If there are any problems, please let me know. I believe rdiff-backup also has a python wrapper for librsync that might be more advanced than the one in pysync. I have plans for both pysync and librsync, but I haven't worked on them much lately. I find I am mostly motivated by feedback from others when funding is not available :-) -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync on OpenVMS
jw schultz wrote: On Sun, Oct 12, 2003 at 12:38:40AM -0400, John E. Malmberg wrote: I am trying to restart getting rsync to run on OpenVMS, and find a way around the fork() issue, posibly using POSIX threads. It occurs to me that i may have been overly encouraging in my last followup. Getting rsync to work using threads, whether pthreads or a superior form, will require enormous intrusive changes to the code. I do not know but if OpenVMS support is a problem for rsync proper you might wish to look at pysync or unison which might meet your immediate needs. I have not heard of unison. I have heard that pysync was successful in a limited test on OpenVMS. As near as I can tell though the librsync it is based on is a bit out of date. For a native rsync, my options are, in order of performance are: 1. Use native asynchronous I/O. This would likely require a complete change for a large section of the code, and would require that the protocol state machine be completely described. The resulting code would probably be of most use to OpenVMS, but it could be adapted to threads. 2. Use pthreads. As I have limited experience with pthreads, I do not yet know the size of this task. The result though would be of benefit to all platforms if successful. 3. Use vfork() instead of fork(). An OpenVMS limitation is that the child process starts from the beginning and must initialize everything. 4. Use the OpenVMS LIB$SPAWN to start the receiving subprocess and link pipes to it. Most of the same difficulties as above, except that it is easier to use asynchronous I/O between the processes. All have challenges. -John [EMAIL PROTECTED] Personal Opinion Only -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
