Re: [PHP] Check for open file
On Fri, Mar 4, 2011 at 7:09 AM, Steve Staples wrote: > > Depending on the size of the file, wouldn't this fall under the 2gb > limitation on windows 32bit OS? I ran into this problem on a project I > was working on, and ended up switching to Python (but that is a WHOLE > other conversation) > > just food for thought, since I am not sure of the size of files they are > dealing with. > > Steve > > Jumping in late. I've been too busy. Steve, I think you're probably referring to FAT32 file system. IIRC, NTFS doesn't have that low limit. [1] Regards, Tommy [1] http://technet.microsoft.com/en-us/library/cc938937.aspx -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Check for open file
On Fri, 2011-03-04 at 09:55 -0500, Daniel Brown wrote: > On Thu, Mar 3, 2011 at 14:59, Ashley M. Kirchner wrote: > > > >Can PHP detect this, or should I look into some delayed > > process of checking the file's modified time stamp versus current time and > > not touch the file till a certain threshold has been reached (say 30 seconds > > difference?). > > Give the native stat() function a spin: > > http://php.net/stat > > -- > > Network Infrastructure Manager > http://www.php.net/ > Depending on the size of the file, wouldn't this fall under the 2gb limitation on windows 32bit OS? I ran into this problem on a project I was working on, and ended up switching to Python (but that is a WHOLE other conversation) just food for thought, since I am not sure of the size of files they are dealing with. Steve -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Check for open file
On Thu, Mar 3, 2011 at 14:59, Ashley M. Kirchner wrote: > > Can PHP detect this, or should I look into some delayed > process of checking the file's modified time stamp versus current time and > not touch the file till a certain threshold has been reached (say 30 seconds > difference?). Give the native stat() function a spin: http://php.net/stat -- Network Infrastructure Manager http://www.php.net/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Check for open file
On 3 March 2011 19:59, Ashley M. Kirchner wrote: > > > Is there a clean or reliable way of checking to see if a > file is still being written to before doing anything with it? > > > > Here's the scenario: we have a Samba share that we can copy > files to (from within Windows or Macs). The server picks up the file and > does some processing of said file. At the moment this is all a manual > process: we copy a file into the Samba share, wait for that to finish then > go to a web page (on the server) and tell it to process the file that was > just copied into the Samba share. I'm trying to see if there is a way to > automate this where a file, or multiple files, get copied into the share and > the server picks them up and process automatically without needing any > interaction. > > > > Can PHP detect this, or should I look into some delayed > process of checking the file's modified time stamp versus current time and > not touch the file till a certain threshold has been reached (say 30 seconds > difference?). > > > > Ideas, suggestions, comments . > > I have a similar issue with files coming from a scanner connected to our network over a VPN. There end is only 10mb and the scans are significant. The one thing I found that I can rely on is the date the file was last modified. I have a process to copy the files if the saved file is more than 10 minutes old. Locking didn't happen (the scanning didn't create the file locked in any way). It was quite easy to amend the file whilst it was being written to by the scanner. Nothing I could do about that. The 10 minute delay on reading the file was fine for us. -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Check for open file
> -Original Message- > From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] > Sent: Thursday, March 03, 2011 2:03 PM > To: sstap...@mnsi.net > Cc: php-general@lists.php.net > Subject: RE: [PHP] Check for open file > > > As far as I was aware, if you're in the middle of writing to a file > and another script was attempting to write to it, the OS would prevent > that as you had an open lock on it. > > -- > Thanks, > Ash > http://www.ashleysheridan.co.uk > And as it turned out, that was not the case. I started copying a 200MiB file from my desktop machine onto the Samba share. While that was copying, I got onto the server and tried to copy the file to /tmp and it happily did that for me without any indication that it was still being written to (which is was by the way.) So, that's not a good way of dealing with this. I think I'm going to have to go with a time delay function here ... -- There are potential problems using timing delays, and networking compounds them. It might be more robust to use some simple method outside of php to indicate a file is busy being copied, like using a reserved extension for the copy and then renaming to the proper filename after the copy is complete, or copying to a temp directory and then moving the file to the destination directory once copied (renames and moves take milliseconds). Assuming you're able to automate the copy process. Cheers Arno -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: RE: [PHP] Check for open file
On Mar 4, 2011 4:53 AM, "Ashley M. Kirchner" wrote: > > > -Original Message- > > From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] > > Sent: Thursday, March 03, 2011 2:03 PM > > To: sstap...@mnsi.net > > Cc: php-general@lists.php.net > > Subject: RE: [PHP] Check for open file > > > > > > As far as I was aware, if you're in the middle of writing to a file and another > > script was attempting to write to it, the OS would prevent that as you had an > > open lock on it. > > > > -- > > Thanks, > > Ash > > http://www.ashleysheridan.co.uk > > > >And as it turned out, that was not the case. I started copying a 200MiB file from my desktop machine onto the Samba share. While that was copying, I got onto the server and tried to copy the file to /tmp and it happily did that for me without any indication that it was still being written to (which is was by the way.) So, that's not a good way of dealing with this. > >I think I'm going to have to go with a time delay function here ... > If on Linux, you can check for open file with lsof, I think the command is called. Regards Peter > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php >
RE: [PHP] Check for open file
> -Original Message- > From: Ashley Sheridan [mailto:a...@ashleysheridan.co.uk] > Sent: Thursday, March 03, 2011 2:03 PM > To: sstap...@mnsi.net > Cc: php-general@lists.php.net > Subject: RE: [PHP] Check for open file > > > As far as I was aware, if you're in the middle of writing to a file and > another > script was attempting to write to it, the OS would prevent that as you had an > open lock on it. > > -- > Thanks, > Ash > http://www.ashleysheridan.co.uk > And as it turned out, that was not the case. I started copying a 200MiB file from my desktop machine onto the Samba share. While that was copying, I got onto the server and tried to copy the file to /tmp and it happily did that for me without any indication that it was still being written to (which is was by the way.) So, that's not a good way of dealing with this. I think I'm going to have to go with a time delay function here ... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Check for open file
> As far as I was aware, if you're in the middle of writing to a file and > another > script was attempting to write to it, the OS would prevent that as you had an > open lock on it. > > -- > Thanks, > Ash > http://www.ashleysheridan.co.uk I guess I would have to test that. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Check for open file
On Thu, 2011-03-03 at 15:29 -0500, Steve Staples wrote: > On Thu, 2011-03-03 at 13:23 -0700, Ashley M. Kirchner wrote: > > > Write the file with a temporary name and extension. Once the file is > > closed, > > > change the name to the pattern your server is looking for. Once you finish > > > processing it, either change the name again, or move it to a different > > > directory. Don't reuse the same file name, but add a numeric value which > > > increases every time you create it. Keep a log of which files have been > > > processed and any errors each one produced. > > > > > > Bob McConnell > > > > I can't require nor expect those copying the files into the share folder to > > do this. No, they will simply be grabbing a set of image files from one > > network share and drag them into this Samba share, as is. I'm not worried > > with what happens when PHP picks it up (name changes, moving to a diff > > folder, etc., etc.) I'm only concerned with the first step ... picking up > > the file only *after* it's done copying. > > > > I can run PHP as a timed crontask, but I need to figure out a safe way for > > it to either grab a file or leave it alone because it's not done yet. > > > > > > If i recall correctly, with FTP, the file is copied into the directory, > but it is not "ready" for use... I have an application that reads the > contents of a FTP directory, looking for files there... i've never had > any issues where it only got a part of the file (that i know of > anyways)... maybe the samba does the same thing? > > how large are these files (or how large would the largest file typically > be) ? would it be worth looking at the "time" of the file, and waiting > until that file is at least (say) 5 mintues old, then do something with > it? if it only takes < 1 minute to put these files on teh share, then > maybe you can reduce that time... > > Just thinking of alternatives... > > Steve > > As far as I was aware, if you're in the middle of writing to a file and another script was attempting to write to it, the OS would prevent that as you had an open lock on it. -- Thanks, Ash http://www.ashleysheridan.co.uk
RE: [PHP] Check for open file
On Thu, 2011-03-03 at 13:23 -0700, Ashley M. Kirchner wrote: > > Write the file with a temporary name and extension. Once the file is > closed, > > change the name to the pattern your server is looking for. Once you finish > > processing it, either change the name again, or move it to a different > > directory. Don't reuse the same file name, but add a numeric value which > > increases every time you create it. Keep a log of which files have been > > processed and any errors each one produced. > > > > Bob McConnell > > I can't require nor expect those copying the files into the share folder to > do this. No, they will simply be grabbing a set of image files from one > network share and drag them into this Samba share, as is. I'm not worried > with what happens when PHP picks it up (name changes, moving to a diff > folder, etc., etc.) I'm only concerned with the first step ... picking up > the file only *after* it's done copying. > > I can run PHP as a timed crontask, but I need to figure out a safe way for > it to either grab a file or leave it alone because it's not done yet. > > If i recall correctly, with FTP, the file is copied into the directory, but it is not "ready" for use... I have an application that reads the contents of a FTP directory, looking for files there... i've never had any issues where it only got a part of the file (that i know of anyways)... maybe the samba does the same thing? how large are these files (or how large would the largest file typically be) ? would it be worth looking at the "time" of the file, and waiting until that file is at least (say) 5 mintues old, then do something with it? if it only takes < 1 minute to put these files on teh share, then maybe you can reduce that time... Just thinking of alternatives... Steve -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Check for open file
> Write the file with a temporary name and extension. Once the file is closed, > change the name to the pattern your server is looking for. Once you finish > processing it, either change the name again, or move it to a different > directory. Don't reuse the same file name, but add a numeric value which > increases every time you create it. Keep a log of which files have been > processed and any errors each one produced. > > Bob McConnell I can't require nor expect those copying the files into the share folder to do this. No, they will simply be grabbing a set of image files from one network share and drag them into this Samba share, as is. I'm not worried with what happens when PHP picks it up (name changes, moving to a diff folder, etc., etc.) I'm only concerned with the first step ... picking up the file only *after* it's done copying. I can run PHP as a timed crontask, but I need to figure out a safe way for it to either grab a file or leave it alone because it's not done yet. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Check for open file
From: Ashley M. Kirchner > Is there a clean or reliable way of checking to see if a > file is still being written to before doing anything with it? > > Here's the scenario: we have a Samba share that we can copy > files to (from within Windows or Macs). The server picks up > the file and does some processing of said file. At the moment > this is all a manual process: we copy a file into the Samba > share, wait for that to finish then go to a web page (on the > server) and tell it to process the file that was just copied > into the Samba share. I'm trying to see if there is a way to > automate this where a file, or multiple files, get copied into > the share and the server picks them up and process automatically > without needing any interaction. > > Can PHP detect this, or should I look into some delayed > process of checking the file's modified time stamp versus current > time and not touch the file till a certain threshold has been > reached (say 30 seconds difference?). Write the file with a temporary name and extension. Once the file is closed, change the name to the pattern your server is looking for. Once you finish processing it, either change the name again, or move it to a different directory. Don't reuse the same file name, but add a numeric value which increases every time you create it. Keep a log of which files have been processed and any errors each one produced. Bob McConnell -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
