Re: [Tutor] file move with wait period
Thanks Cameron (I hope I address your first name correctly). Get Outlook for iOS<https://aka.ms/o0ukef> From: Cameron Simpson Sent: Friday, August 4, 2017 5:18:58 AM To: banda gunda Cc: tutor@python.org Subject: Re: [Tutor] file move with wait period On 03Aug2017 11:22, banda gunda wrote: >Dear tutors, > >I am interested to move a set of files, periodically, from source_folder to >dest_folder. >source_folder receives processed files from a ‘decompressing program’. The >files are deposited from the ‘decompressing program’ at periodic random >intervals. Furthermore some of the files dropped in the source_folder are >large, of typical size 800MB. These files take time before it is completely >deposited in the source_folder. > >I could use shutil.move (source_folder, dest_folder). But then how? You'd normally use shutil.move(source_file, dest_file) i.e. move individual files. Or os.rename, but that requires that the source and dest are the same filesystem. >The frequency of moving files from source_folder to dest_folder is flexible. >It could be once in 15 minutes (or even lengthier). >Challenge for me: How could the code include the wait statement to make sure >that the file in the source_folder is completely formed before attempting to >move to dest_folder? >Thanks in advance for your comments/guidance. That is the tricky thing; this shows up a lot, also with FTP upload locations and so forth. The problem, as you state, is knowing when the file is complete. There are 2 basic approaches: do not put a fil into the transfer directory before it is complete, or to monitor the files for changes. On the premise that the file will be being actively written until it is complete you could keep a list if the files in the directory. For each name, record the file's size and modification time. Wait for that to be unchanged "long enough"; you might then decide it is ready to move. Cheers, Cameron Simpson (formerly c...@zip.com.au) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] file move with wait period
On 08/03/2017 05:22 AM, banda gunda wrote: > Dear tutors, > > I am interested to move a set of files, periodically, from source_folder to > dest_folder. > source_folder receives processed files from a ‘decompressing program’. The > files are deposited from the ‘decompressing program’ at periodic random > intervals. Furthermore some of the files dropped in the source_folder are > large, of typical size 800MB. These files take time before it is completely > deposited in the source_folder. > > I could use shutil.move (source_folder, dest_folder). But then how? > > The frequency of moving files from source_folder to dest_folder is flexible. > It could be once in 15 minutes (or even lengthier). > Challenge for me: How could the code include the wait statement to make sure > that the file in the source_folder is completely formed before attempting to > move to dest_folder? Well, we cannot answer that question with the information you have given. "How can you know" is actually the key question to explore. You need to answer it before you worry about any Pythonic implementation details of solving your problem. - is the decompressor initiated by your program? In that case there will be techniques to maintain communication with it to find out when it is done (perhaps as simple as waiting for it to quit-with-success). - if it launches completely independently, how do you find out that it has deposited files? Do you intend to just can periodically? Or is there a way that the system causing the files to be generated can trigger something that your program could be asked to be notified about when a file is available? - is it possible to convice the compressor to give files a certain suffix or other recognizable pattern, but only when they are complete? (downloader systems often work like this... while they're writing the file, it has some temporary name, then when it finishes it is renamed to the intended target name). and so on. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] file move with wait period
On 03Aug2017 11:22, banda gunda wrote: Dear tutors, I am interested to move a set of files, periodically, from source_folder to dest_folder. source_folder receives processed files from a ‘decompressing program’. The files are deposited from the ‘decompressing program’ at periodic random intervals. Furthermore some of the files dropped in the source_folder are large, of typical size 800MB. These files take time before it is completely deposited in the source_folder. I could use shutil.move (source_folder, dest_folder). But then how? You'd normally use shutil.move(source_file, dest_file) i.e. move individual files. Or os.rename, but that requires that the source and dest are the same filesystem. The frequency of moving files from source_folder to dest_folder is flexible. It could be once in 15 minutes (or even lengthier). Challenge for me: How could the code include the wait statement to make sure that the file in the source_folder is completely formed before attempting to move to dest_folder? Thanks in advance for your comments/guidance. That is the tricky thing; this shows up a lot, also with FTP upload locations and so forth. The problem, as you state, is knowing when the file is complete. There are 2 basic approaches: do not put a fil into the transfer directory before it is complete, or to monitor the files for changes. On the premise that the file will be being actively written until it is complete you could keep a list if the files in the directory. For each name, record the file's size and modification time. Wait for that to be unchanged "long enough"; you might then decide it is ready to move. Cheers, Cameron Simpson (formerly c...@zip.com.au) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor