Re: Ideas for creating processes
J dreadpiratej...@gmail.com (J) wrote:
J And now I'm looking at subprocess and I can set shell=True and it will
J intrepret special characters like
J So could I do something like this:
J for item in pathlist:
J subprocess.Popen('rsync command ', shell=True)
J and simply wait unti they are all done?
Using shell=True is often the wrong thing to do. And certainly just to
get a process to run in the background. subprocess will run them in the
background by default. Besides if you do it in the way you propose you
can't wait for them. You can only wait for the shell that starts the
rsync, but that will be finished almost immediately.
import subprocess
p = subprocess.Popen('sleep 1000 ', shell=True)
p.wait()
0
The wait() returns immediately.
--
Piet van Oostrum p...@vanoostrum.org
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]
Nu Fair Trade woonaccessoires op http://www.zylja.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: Ideas for creating processes
On Mar 10, 4:52 pm, J dreadpiratej...@gmail.com wrote: I'm working on a project and thought I'd ask for a suggestion on how to proceed (I've got my own ideas, but I wanted to see if I was on the right track) For now, I've got this: def main(): ## get our list of directories to refresh releases=sys.argv[1:] if len(releases) 1: print You need to provide at least one dir to update sys.exit() ## Lets figure out what there is to update updateDirs = [] for rel in releases: currentDir = os.path.join(homedir, rel) for item in os.listdir(currentDir): updateDirs += [os.path.join(homedir, rel, item)] which returns a list of full pathnames to directories that need to be updated (updates will be carried out by calling rsync or zsync eventually) The directory hierarchy looks like this: /home/user/files /home/user/files/version1 /home/user/files/version1/type1 /home/user/files/version1/type2 /home/user/files/version2 /home/user/files/version2/type1 and the list ends up looking like this: ['/home/user/files/version1/type1','/home/user/files/version1/type2','/home/user/files/version2/type1','/home/user/files/version2/type2'] the next thing I need to do is figure out how to update those. the quick and dirty would be (as I'm imagining it at the moment): for path in pathlist: chdir into path execute rsync or zsync but that gets me moving into one dir, updating, then moving into another. What I was wondering about though, is spawning off separate rsync processes to run concurrently (each rsync will hit a different remote dir for each local dir) so I'm trying to figure out a good way to do this: for path in pathlist: kick off an individual rsync|zsync process to update path wait for all child processes to end exit program. I've been looking at subprocess because at the moment, that's all I really know... But is there a better way of kicking off multiple simultaneous processes so I can update all dirs at once instead of one at a time? No, this isn't homework, it's something I'm working on to sync a couple directories of ISO images to grab nightly builds Yes, there are plenty of pre-made scripts out there, but I don't want to even look at those because I figured this would be a good learning experience, and I want to try to solve this as much on my own as I can without just cut and pasting from someone elses program. So, with that said, any ideas on the best way to proceed? I'm going to start looking at ways to use subprocess to do this, or would there be a better way (multi-threading maybe?) Or am I even in the right ballpark? Cheers Jeff You might be able to use the SIMPL toolkit for this one. (http://www.icanprogram.com/06py/lesson1/lesson1.html) You could wrap the rsync executable as a SIMPL receiver module and then message to that from inside your Python script to kick it off and synchronize actions. bob -- http://mail.python.org/mailman/listinfo/python-list
Ideas for creating processes
I'm working on a project and thought I'd ask for a suggestion on how to proceed (I've got my own ideas, but I wanted to see if I was on the right track) For now, I've got this: def main(): ## get our list of directories to refresh releases=sys.argv[1:] if len(releases) 1: print You need to provide at least one dir to update sys.exit() ## Lets figure out what there is to update updateDirs = [] for rel in releases: currentDir = os.path.join(homedir, rel) for item in os.listdir(currentDir): updateDirs += [os.path.join(homedir, rel, item)] which returns a list of full pathnames to directories that need to be updated (updates will be carried out by calling rsync or zsync eventually) The directory hierarchy looks like this: /home/user/files /home/user/files/version1 /home/user/files/version1/type1 /home/user/files/version1/type2 /home/user/files/version2 /home/user/files/version2/type1 and the list ends up looking like this: ['/home/user/files/version1/type1','/home/user/files/version1/type2','/home/user/files/version2/type1','/home/user/files/version2/type2'] the next thing I need to do is figure out how to update those. the quick and dirty would be (as I'm imagining it at the moment): for path in pathlist: chdir into path execute rsync or zsync but that gets me moving into one dir, updating, then moving into another. What I was wondering about though, is spawning off separate rsync processes to run concurrently (each rsync will hit a different remote dir for each local dir) so I'm trying to figure out a good way to do this: for path in pathlist: kick off an individual rsync|zsync process to update path wait for all child processes to end exit program. I've been looking at subprocess because at the moment, that's all I really know... But is there a better way of kicking off multiple simultaneous processes so I can update all dirs at once instead of one at a time? No, this isn't homework, it's something I'm working on to sync a couple directories of ISO images to grab nightly builds Yes, there are plenty of pre-made scripts out there, but I don't want to even look at those because I figured this would be a good learning experience, and I want to try to solve this as much on my own as I can without just cut and pasting from someone elses program. So, with that said, any ideas on the best way to proceed? I'm going to start looking at ways to use subprocess to do this, or would there be a better way (multi-threading maybe?) Or am I even in the right ballpark? Cheers Jeff -- http://mail.python.org/mailman/listinfo/python-list
Re: Ideas for creating processes
On Wed, Mar 10, 2010 at 16:52, J dreadpiratej...@gmail.com wrote:
the quick and dirty would be (as I'm imagining it at the moment):
for path in pathlist:
chdir into path
execute rsync or zsync
but that gets me moving into one dir, updating, then moving into another.
What I was wondering about though, is spawning off separate rsync
processes to run concurrently (each rsync will hit a different remote
dir for each local dir)
so I'm trying to figure out a good way to do this:
for path in pathlist:
kick off an individual rsync|zsync process to update path
And now I'm looking at subprocess and I can set shell=True and it will
intrepret special characters like
So could I do something like this:
for item in pathlist:
subprocess.Popen('rsync command ', shell=True)
and simply wait unti they are all done?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Ideas for creating processes
On 03/10/10 21:52, J wrote: I'm working on a project and thought I'd ask for a suggestion on how to proceed (I've got my own ideas, but I wanted to see if I was on the right track) cut subprocess idea Well I can't speak with authority but I would go into similar lines, especially since you want to call an external program in the end anyway. Otherwise threading might have been an option. Just make sure that all subprocesses are exited nicely before you are on the end of your script ;-) -- mph -- http://mail.python.org/mailman/listinfo/python-list
Re: Ideas for creating processes
J wrote: I'm working on a project and thought I'd ask for a suggestion on how to proceed (I've got my own ideas, but I wanted to see if I was on the right track) For now, I've got this: def main(): ## get our list of directories to refresh releases=sys.argv[1:] if len(releases) 1: print You need to provide at least one dir to update sys.exit() ## Lets figure out what there is to update updateDirs = [] for rel in releases: currentDir = os.path.join(homedir, rel) for item in os.listdir(currentDir): updateDirs += [os.path.join(homedir, rel, item)] which returns a list of full pathnames to directories that need to be updated (updates will be carried out by calling rsync or zsync eventually) The directory hierarchy looks like this: /home/user/files /home/user/files/version1 /home/user/files/version1/type1 /home/user/files/version1/type2 /home/user/files/version2 /home/user/files/version2/type1 and the list ends up looking like this: ['/home/user/files/version1/type1','/home/user/files/version1/type2','/home/user/files/version2/type1','/home/user/files/version2/type2'] the next thing I need to do is figure out how to update those. the quick and dirty would be (as I'm imagining it at the moment): for path in pathlist: chdir into path execute rsync or zsync but that gets me moving into one dir, updating, then moving into another. What I was wondering about though, is spawning off separate rsync processes to run concurrently (each rsync will hit a different remote dir for each local dir) so I'm trying to figure out a good way to do this: for path in pathlist: kick off an individual rsync|zsync process to update path wait for all child processes to end exit program. I've been looking at subprocess because at the moment, that's all I really know... But is there a better way of kicking off multiple simultaneous processes so I can update all dirs at once instead of one at a time? No, this isn't homework, it's something I'm working on to sync a couple directories of ISO images to grab nightly builds Yes, there are plenty of pre-made scripts out there, but I don't want to even look at those because I figured this would be a good learning experience, and I want to try to solve this as much on my own as I can without just cut and pasting from someone elses program. So, with that said, any ideas on the best way to proceed? I'm going to start looking at ways to use subprocess to do this, or would there be a better way (multi-threading maybe?) Or am I even in the right ballpark? Are you sure that you would gain from doing more than one at a time? The bottleneck will probably be the speed of your network connection, and if that's working at its maximum speed with one sync then doing several concurrently won't save any time. (The syncs will also be completing for disk I/O.) It might, however, save time if the remote server takes time to respond to a sync and/or doesn't use your network connection at its maximum speed (the server's connection will have its own limits, of course, and might be talking to other computers at the same time). You could increase the number of concurrent syncs until you find the number that gives the best results. -- http://mail.python.org/mailman/listinfo/python-list
Re: Ideas for creating processes
On Wed, Mar 10, 2010 at 18:03, MRAB pyt...@mrabarnett.plus.com wrote: Are you sure that you would gain from doing more than one at a time? The bottleneck will probably be the speed of your network connection, and if that's working at its maximum speed with one sync then doing several concurrently won't save any time. (The syncs will also be completing for disk I/O.) Good point MRAB... thanks for making it. I started to wonder the same thing, but it took me a couple hours away from working on it and a good dinner before I saw that... Yeah, looking at it freshly now, I agree, I think I probably would be better off running them one at a time. The goal is to update several directories of ISO images with nightly builds. I can use rsync and only download images that are changed, or I can use zsync and only download diffs (I believe that's how it works) which is faster anyway. The only reason I was considering doing them all simultaneously, or in batches at least, is that I have to update 10 - 12 ISOs daily... BUT, now that I look at it from your perspective, yeah, it would probably be faster overall to do them one at a time instead because of the bottleneck. Thanks! That helped a lot (and probably saved me a lot of headache later on) :-) Cheers Jeff -- http://mail.python.org/mailman/listinfo/python-list
