Re: [PHP] Large/unreliable file uploading over HTTP
Wouldn't you know - the latest Google Gears has the capability to upload files - I haven't delved deeper to understand how it works on the server end of it, but it allows for a standard file upload box and has capabilities built in for progress bars, etc. The "blob" interface allows you to use a "slice" of the file too. Sounds to me like it can split files up? I'm trying to hack up my first run of a Google Gears app right now, but it looks like it will be a little bit more time consuming than I thought. Using this page for reference: http://www.youtube.com/my_videos_multiupload -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Large/unreliable file uploading over HTTP
On Mon, Sep 8, 2008 at 2:00 PM, Boyd, Todd M. <[EMAIL PROTECTED]> wrote: > I know that Base64 encoding adds something like 30% to the overall volume of > the object once it has been encoded. For huge files, that might be > unacceptable. I'm not sure how UUEncode handles it. yeah, I found an email thread from 2001 saying base64 is the way to go. however for every 6 bytes it creates 8 bytes or something (hence the 33% bloat) there's also options like http://www.yenc.org/ - but this would require java support (or appropriate extensions written) and server-end support (PHP, or in-webserver code) base64 is -probably- the easiest. PHP has both uuencode and base64 standard. However, need to look at it from the client angle - Java probably has base64 easily available, not sure about uudecode. Not sure about flash, or any other client programs; or web browsers themselves too... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Large/unreliable file uploading over HTTP
> -Original Message- > From: mike [mailto:[EMAIL PROTECTED] > Sent: Monday, September 08, 2008 1:38 PM > To: Boyd, Todd M. > Cc: php-general@lists.php.net > Subject: Re: [PHP] Large/unreliable file uploading over HTTP > > On Mon, Sep 8, 2008 at 11:35 AM, Boyd, Todd M. <[EMAIL PROTECTED]> > wrote: > > > The checksum is not being performed in the last version I worked on. > That was going to be my next step--verifying each chunk as it came down > the tube. As far as encoding, I believe I was UUEncoding it. I don't > have the source here with me at work... it may have been something less > efficient (but easier to implement for beta), like Base64. I can't > remember off-hand. > > > > I'm pretty sure I've still got decent free storage floating around on > the net somewhere (maybe qsh.eu or freehyperspace.com) so that I can > post a link to the source for those that are interested. > > > > More news in a few hours. ;) > > I can host it for you if you need it. Contact me off list if you want. > > Also - I figured uuencoding is best, as usenet uses that and has no > issues. > > Base64 was another option but I decided (not sure how) that uuencoding > probably makes the most sense. I know that Base64 encoding adds something like 30% to the overall volume of the object once it has been encoded. For huge files, that might be unacceptable. I'm not sure how UUEncode handles it. Todd Boyd Web Programmer
Re: [PHP] Large/unreliable file uploading over HTTP
On Mon, Sep 8, 2008 at 11:35 AM, Boyd, Todd M. <[EMAIL PROTECTED]> wrote: > The checksum is not being performed in the last version I worked on. That was > going to be my next step--verifying each chunk as it came down the tube. As > far as encoding, I believe I was UUEncoding it. I don't have the source here > with me at work... it may have been something less efficient (but easier to > implement for beta), like Base64. I can't remember off-hand. > > I'm pretty sure I've still got decent free storage floating around on the net > somewhere (maybe qsh.eu or freehyperspace.com) so that I can post a link to > the source for those that are interested. > > More news in a few hours. ;) I can host it for you if you need it. Contact me off list if you want. Also - I figured uuencoding is best, as usenet uses that and has no issues. Base64 was another option but I decided (not sure how) that uuencoding probably makes the most sense. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Large/unreliable file uploading over HTTP
> -Original Message- > From: mike [mailto:[EMAIL PROTECTED] > Sent: Monday, September 08, 2008 1:19 PM > To: Boyd, Todd M. > Cc: Jason Pruim; php-general@lists.php.net > Subject: Re: [PHP] Large/unreliable file uploading over HTTP > > Yes, share it please! > > We also have a Java guy who was going to look into creating this. > > Do you checksum each segment and validate it? Do you do any encoding > on the wire to keep the bytes safe? I figure uuencoding or something > would help with transferring over the wire (remember to checksum the > uudecoded content, not the uuencoded) ---8<--- snip! The checksum is not being performed in the last version I worked on. That was going to be my next step--verifying each chunk as it came down the tube. As far as encoding, I believe I was UUEncoding it. I don't have the source here with me at work... it may have been something less efficient (but easier to implement for beta), like Base64. I can't remember off-hand. I'm pretty sure I've still got decent free storage floating around on the net somewhere (maybe qsh.eu or freehyperspace.com) so that I can post a link to the source for those that are interested. More news in a few hours. ;) Todd Boyd Web Programmer
Re: [PHP] Large/unreliable file uploading over HTTP
On Mon, Sep 8, 2008 at 6:49 AM, Craige Leeder <[EMAIL PROTECTED]> wrote: > The only concern I would have is that you are using a third party > software/applet to do these uploads. I'm not a fan of MAKING users have a > piece of software enabled to allow them basic web-standard functionality on > a site. Well, that's the problem. There is no web standard for intelligently uploading large files or fixing unreliable/slow connectivity. If this can be adopted into some sort of pseudo-standard perhaps people can make some browser addons/plugins to take advantage, and who knows, maybe we'll be able to get it into a browser or two natively. Today an applet has to be used no matter what, and any site that accepts large files or has any sort of "advanced" uploading is using applets. HTTP uploading doesn't have any resuming or anything, period. If people use normal multipart POST forms and they get large files, they're just lucky :) Note: I actually am talking to an nginx module developer about creating the server component as an nginx module, which would reduce any PHP complexities; ideally it would be a drop-in replacement for uploading and would be transparent to PHP... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Large/unreliable file uploading over HTTP
Yes, share it please! We also have a Java guy who was going to look into creating this. Do you checksum each segment and validate it? Do you do any encoding on the wire to keep the bytes safe? I figure uuencoding or something would help with transferring over the wire (remember to checksum the uudecoded content, not the uuencoded) On Mon, Sep 8, 2008 at 9:11 AM, Boyd, Todd M. <[EMAIL PROTECTED]> wrote: >> -Original Message- >> From: Jason Pruim [mailto:[EMAIL PROTECTED] >> Sent: Monday, September 08, 2008 10:48 AM >> To: Boyd, Todd M. >> Cc: php-general@lists.php.net; Craige Leeder >> Subject: Re: [PHP] Large/unreliable file uploading over HTTP >> >> >> On Sep 8, 2008, at 11:25 AM, Boyd, Todd M. wrote: >> >> >> -Original Message- >> >> From: Craige Leeder [mailto:[EMAIL PROTECTED] >> >> Sent: Monday, September 08, 2008 8:49 AM >> >> To: mike >> >> Cc: PHP General list >> >> Subject: Re: [PHP] Large/unreliable file uploading over HTTP >> >> >> >> The only concern I would have is that you are using a third party >> >> software/applet to do these uploads. I'm not a fan of MAKING users >> >> have >> >> a piece of software enabled to allow them basic web-standard >> >> functionality on a site. >> >> >> >> It is however, an interesting concept. It would really come in > handy >> >> for >> >> extremely large files. >> >> >> >> - Craige >> >> >> >> mike wrote: >> >>> Let's face it - HTTP is not very good for file uploads. It's >> >> stateless >> >>> nature, slow connections, inability to resume (technically), etc, >> >> etc. >> >>> >> >>> What I've been thinking about is a way to skip all the normal >> >>> annoyances with file uploading - multipart form encodings, file >> >> upload >> >>> tools with specific needs, PUT vs POST, connection resets, ... the >> >>> list goes on and on. >> > >> > ---8<--- snip! >> > >> >>> I can't think of any method to do it in-browser, but doing it this >> >> way >> >>> could open the gates for things like Google Gears to possibly work >> >>> too... >> > >> > I have an "almost-fully-functional" Java Applet file uploader. >> > Progress bar, unlimited* file size, blah blah blah... can hook into >> > nearly any language on the other end (though I'm currently working >> > with PHP). Some bytes are missing from the resulting file when I try >> > to upload an MP3, but it's a work in progress. >> > >> > I don't see the problem with using a Java Applet or something that >> > is easily installed on-demand. Java doesn't quite have the >> > widespread adoption I would hope for, but it definitely makes its' >> > way around. >> >> Hey Todd, >> >> This sounds exactly like what my day job needs... I just took over the >> website, and currently they have it where people have to ftp the file >> into their server to get large files, and with them being a print >> shop... A couple gigs per job is not unheard of. >> >> If there is anything that I can do to help get that finished up let me >> know! :) > > Jason, > > I'd be more than happy to spread the source around. Files of any size > are not a problem... the Applet breaks a file into (I believe) 512kb > chunks for transfer. The first POST to PHP tells PHP how many chunks and > some other identifying information. Each POST after that is a 512k chunk > of the file. These files are then strung back together after the last > chunk is uploaded. > > I probably need to do some hash checking to ensure that files have, in > fact, uploaded (and uploaded completely)... but for testing on my local > machine, I didn't bother (yet). The problem is that a few bytes > (depending on the size of the file--and therefore, the number of chunks) > are missing from the uploaded file. This makes MP3s sound like they were > recorded underwater, and text files miss a letter every now and again. > > I'll dig through the code I have for it when I get home from work this > evening. To be honest, I haven't poked around it for some time now--a > month, perhaps?--and it might take me a day or two to make sense of it > all again. :) > > Anyway, I'll e-mail you what I've got so far and see what you can make > of it. I had begun to find a pattern in the missing information with a > "Diff"-like text editor and a notepad (yes, the physical kind made of > paper :D), but I was forced to abandon it for "real" work as the > uploader is a side project of mine. > > If anybody else is interested in the PHP/Java Applet source, I am glad > to share it. I started working on it myself for a friend's website when > I discovered that people are actually charging money for software > packages that accomplish this (seemingly) simple process. > > > Todd Boyd > Web Programmer > > > > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Large/unreliable file uploading over HTTP
> -Original Message- > From: Jason Pruim [mailto:[EMAIL PROTECTED] > Sent: Monday, September 08, 2008 10:48 AM > To: Boyd, Todd M. > Cc: php-general@lists.php.net; Craige Leeder > Subject: Re: [PHP] Large/unreliable file uploading over HTTP > > > On Sep 8, 2008, at 11:25 AM, Boyd, Todd M. wrote: > > >> -Original Message- > >> From: Craige Leeder [mailto:[EMAIL PROTECTED] > >> Sent: Monday, September 08, 2008 8:49 AM > >> To: mike > >> Cc: PHP General list > >> Subject: Re: [PHP] Large/unreliable file uploading over HTTP > >> > >> The only concern I would have is that you are using a third party > >> software/applet to do these uploads. I'm not a fan of MAKING users > >> have > >> a piece of software enabled to allow them basic web-standard > >> functionality on a site. > >> > >> It is however, an interesting concept. It would really come in handy > >> for > >> extremely large files. > >> > >> - Craige > >> > >> mike wrote: > >>> Let's face it - HTTP is not very good for file uploads. It's > >> stateless > >>> nature, slow connections, inability to resume (technically), etc, > >> etc. > >>> > >>> What I've been thinking about is a way to skip all the normal > >>> annoyances with file uploading - multipart form encodings, file > >> upload > >>> tools with specific needs, PUT vs POST, connection resets, ... the > >>> list goes on and on. > > > > ---8<--- snip! > > > >>> I can't think of any method to do it in-browser, but doing it this > >> way > >>> could open the gates for things like Google Gears to possibly work > >>> too... > > > > I have an "almost-fully-functional" Java Applet file uploader. > > Progress bar, unlimited* file size, blah blah blah... can hook into > > nearly any language on the other end (though I'm currently working > > with PHP). Some bytes are missing from the resulting file when I try > > to upload an MP3, but it's a work in progress. > > > > I don't see the problem with using a Java Applet or something that > > is easily installed on-demand. Java doesn't quite have the > > widespread adoption I would hope for, but it definitely makes its' > > way around. > > Hey Todd, > > This sounds exactly like what my day job needs... I just took over the > website, and currently they have it where people have to ftp the file > into their server to get large files, and with them being a print > shop... A couple gigs per job is not unheard of. > > If there is anything that I can do to help get that finished up let me > know! :) Jason, I'd be more than happy to spread the source around. Files of any size are not a problem... the Applet breaks a file into (I believe) 512kb chunks for transfer. The first POST to PHP tells PHP how many chunks and some other identifying information. Each POST after that is a 512k chunk of the file. These files are then strung back together after the last chunk is uploaded. I probably need to do some hash checking to ensure that files have, in fact, uploaded (and uploaded completely)... but for testing on my local machine, I didn't bother (yet). The problem is that a few bytes (depending on the size of the file--and therefore, the number of chunks) are missing from the uploaded file. This makes MP3s sound like they were recorded underwater, and text files miss a letter every now and again. I'll dig through the code I have for it when I get home from work this evening. To be honest, I haven't poked around it for some time now--a month, perhaps?--and it might take me a day or two to make sense of it all again. :) Anyway, I'll e-mail you what I've got so far and see what you can make of it. I had begun to find a pattern in the missing information with a "Diff"-like text editor and a notepad (yes, the physical kind made of paper :D), but I was forced to abandon it for "real" work as the uploader is a side project of mine. If anybody else is interested in the PHP/Java Applet source, I am glad to share it. I started working on it myself for a friend's website when I discovered that people are actually charging money for software packages that accomplish this (seemingly) simple process. Todd Boyd Web Programmer -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Large/unreliable file uploading over HTTP
On Sep 8, 2008, at 11:25 AM, Boyd, Todd M. wrote: -Original Message- From: Craige Leeder [mailto:[EMAIL PROTECTED] Sent: Monday, September 08, 2008 8:49 AM To: mike Cc: PHP General list Subject: Re: [PHP] Large/unreliable file uploading over HTTP The only concern I would have is that you are using a third party software/applet to do these uploads. I'm not a fan of MAKING users have a piece of software enabled to allow them basic web-standard functionality on a site. It is however, an interesting concept. It would really come in handy for extremely large files. - Craige mike wrote: Let's face it - HTTP is not very good for file uploads. It's stateless nature, slow connections, inability to resume (technically), etc, etc. What I've been thinking about is a way to skip all the normal annoyances with file uploading - multipart form encodings, file upload tools with specific needs, PUT vs POST, connection resets, ... the list goes on and on. ---8<--- snip! I can't think of any method to do it in-browser, but doing it this way could open the gates for things like Google Gears to possibly work too... I have an "almost-fully-functional" Java Applet file uploader. Progress bar, unlimited* file size, blah blah blah... can hook into nearly any language on the other end (though I'm currently working with PHP). Some bytes are missing from the resulting file when I try to upload an MP3, but it's a work in progress. I don't see the problem with using a Java Applet or something that is easily installed on-demand. Java doesn't quite have the widespread adoption I would hope for, but it definitely makes its' way around. Hey Todd, This sounds exactly like what my day job needs... I just took over the website, and currently they have it where people have to ftp the file into their server to get large files, and with them being a print shop... A couple gigs per job is not unheard of. If there is anything that I can do to help get that finished up let me know! :) -- Jason Pruim Raoset Inc. Technology Manager MQC Specialist 11287 James St Holland, MI 49424 www.raoset.com [EMAIL PROTECTED] -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] Large/unreliable file uploading over HTTP
> -Original Message- > From: Craige Leeder [mailto:[EMAIL PROTECTED] > Sent: Monday, September 08, 2008 8:49 AM > To: mike > Cc: PHP General list > Subject: Re: [PHP] Large/unreliable file uploading over HTTP > > The only concern I would have is that you are using a third party > software/applet to do these uploads. I'm not a fan of MAKING users have > a piece of software enabled to allow them basic web-standard > functionality on a site. > > It is however, an interesting concept. It would really come in handy > for > extremely large files. > > - Craige > > mike wrote: > > Let's face it - HTTP is not very good for file uploads. It's > stateless > > nature, slow connections, inability to resume (technically), etc, > etc. > > > > What I've been thinking about is a way to skip all the normal > > annoyances with file uploading - multipart form encodings, file > upload > > tools with specific needs, PUT vs POST, connection resets, ... the > > list goes on and on. ---8<--- snip! > > I can't think of any method to do it in-browser, but doing it this > way > > could open the gates for things like Google Gears to possibly work > > too... I have an "almost-fully-functional" Java Applet file uploader. Progress bar, unlimited* file size, blah blah blah... can hook into nearly any language on the other end (though I'm currently working with PHP). Some bytes are missing from the resulting file when I try to upload an MP3, but it's a work in progress. I don't see the problem with using a Java Applet or something that is easily installed on-demand. Java doesn't quite have the widespread adoption I would hope for, but it definitely makes its' way around. Todd Boyd Web Programmer
Re: [PHP] Large/unreliable file uploading over HTTP
The only concern I would have is that you are using a third party software/applet to do these uploads. I'm not a fan of MAKING users have a piece of software enabled to allow them basic web-standard functionality on a site. It is however, an interesting concept. It would really come in handy for extremely large files. - Craige mike wrote: Let's face it - HTTP is not very good for file uploads. It's stateless nature, slow connections, inability to resume (technically), etc, etc. What I've been thinking about is a way to skip all the normal annoyances with file uploading - multipart form encodings, file upload tools with specific needs, PUT vs POST, connection resets, ... the list goes on and on. Usenet and BitTorrent and other protocols have the right idea - split up the workload into smaller sets of data. It's easier to operate on. Usenet has NZB files. BitTorrent splits everything up into chunks. Not only would it make working with the data more portable (no need to set your PHP memory limit or POST limits to insane amounts to support large files) but it could also support multiple segments of the file being transferred at once... Here's somewhat of a process braindump of what I'm thinking. It still requires a 'smart' applet (Flash, Java, anything that can split a file up and send data over HTTP/HTTPS) - no special socket needs, no PUT support needed, don't even need to use multipart POST encoding (as far as I know) - just send the data in chunks over the wire and have a PHP script on the other side collect the data and reassemble it. Does this sound insane? I think this is a pretty good approach - no PUT needed, no large POST configuration required, anything could upload to it as long as it sends the information properly (I'm thinking HTTP POST for the header info, and the data could be sent as another POST field maybe base64 encoded or something that will stay safe during transit...) - take input file, checksum it (md5 / sha1) - calculate number of chunks to split it up based on $chunk configured size (for example 128k chunks) - split the file into chunks of $chunk size and create checksums for all (could use sfv?) - send request to the server - with the info - use JSON? action=begin filename=$filename filesize=$filesize checksum=$checksum segments=list of segments (unique segment id, checksum, bytes) - server sends back a "server ready" and unique $transaction_id - start sending to the server, send header with transaction key and unique chunk identifier in it action=process transaction=$transaction_id segment=$unique_segment_id data=base64_encode($segment_data) - when done, server sends back $transaction_id, $segment_id, $status (okay, fail) - client compares checksum for identifier, if okay, move to next chunk - if it does not match, retry uploading to server again - when all chunks are done, send request with transaction key and action=finish transaction=$transaction_id - when the server receives this, it assembles the file from the segments and does a final checksum, and reports the checksum back to the client (warning: on a large file this could take a bit?) and sends back $transaction_id, $checksum - client does one last check against the file's original checksum, if it matches report success, otherwise report failure (would need to determine why though - if all segments match this should not be able to happen...) I'd appreciate everyone's thoughts. This would also allow for file upload progress, more or less, as the server and client are constantly communicating when chunks are done and in progress (but again, that has to be done with an applet) I can't think of any method to do it in-browser, but doing it this way could open the gates for things like Google Gears to possibly work too... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Large/unreliable file uploading over HTTP
Thanks - there's a lot of uploaders out there... I'll look at it more, but I'd really like a solution which allows for us to use any applet language we want. We'd prefer not to use java too :) I don't see anything specific right off the bat that says how it uploads (POST vs PUT) or splitting up of files. I would like to design something that can work without any changes to server settings - so anyone can use it... On 8/26/08, Simcha Younger <[EMAIL PROTECTED]> wrote: > > Is this what you are looking for -- It's a java applet that has most of the > features you mentioned. I have used it for very large files with no problem. > > http://www.javazoom.net/applets/jclientupload/jclientupload.html > > (Sorry, its not free software.) > > -Original Message- > From: mike [mailto:[EMAIL PROTECTED] > Sent: Tuesday, August 26, 2008 10:07 AM > To: PHP General list > Subject: [PHP] Large/unreliable file uploading over HTTP > > Let's face it - HTTP is not very good for file uploads. It's stateless > nature, slow connections, inability to resume (technically), etc, etc. > > What I've been thinking about is a way to skip all the normal > annoyances with file uploading - multipart form encodings, file upload > tools with specific needs, PUT vs POST, connection resets, ... the > list goes on and on. > > Usenet and BitTorrent and other protocols have the right idea - split > up the workload into smaller sets of data. It's easier to operate on. > Usenet has NZB files. BitTorrent splits everything up into chunks. Not > only would it make working with the data more portable (no need to set > your PHP memory limit or POST limits to insane amounts to support > large files) but it could also support multiple segments of the file > being transferred at once... > > Here's somewhat of a process braindump of what I'm thinking. It still > requires a 'smart' applet (Flash, Java, anything that can split a file > up and send data over HTTP/HTTPS) - no special socket needs, no PUT > support needed, don't even need to use multipart POST encoding (as far > as I know) - just send the data in chunks over the wire and have a PHP > script on the other side collect the data and reassemble it. > > Does this sound insane? I think this is a pretty good approach - no > PUT needed, no large POST configuration required, anything could > upload to it as long as it sends the information properly (I'm > thinking HTTP POST for the header info, and the data could be sent as > another POST field maybe base64 encoded or something that will stay > safe during transit...) > > > > - take input file, checksum it (md5 / sha1) > - calculate number of chunks to split it up based on $chunk configured > size (for example 128k chunks) > - split the file into chunks of $chunk size and create checksums for > all (could use sfv?) > - send request to the server - with the info - use JSON? >action=begin >filename=$filename >filesize=$filesize >checksum=$checksum >segments=list of segments (unique segment id, checksum, bytes) >- server sends back a "server ready" and unique $transaction_id > - start sending to the server, send header with transaction key and > unique chunk identifier in it >action=process >transaction=$transaction_id >segment=$unique_segment_id > data=base64_encode($segment_data) >- when done, server sends back $transaction_id, $segment_id, $status > (okay, fail) > - client compares checksum for identifier, if okay, move to next chunk >- if it does not match, retry uploading to server again > - when all chunks are done, send request with transaction key and >action=finish >transaction=$transaction_id > - when the server receives this, it assembles the file > from the segments and does a final checksum, and reports the checksum > back to the client (warning: on a large file this could take a bit?) > and sends back $transaction_id, $checksum > - client does one last check against the file's original checksum, if > it matches report success, otherwise report failure (would need to > determine why though - if all segments match this should not be able > to happen...) > > I'd appreciate everyone's thoughts. This would also allow for file > upload progress, more or less, as the server and client are constantly > communicating when chunks are done and in progress (but again, that > has to be done with an applet) > > I can't think of any method to do it in-browser, but doing it this way > could open the gates for things like Google Gears to possibl
RE: [PHP] Large/unreliable file uploading over HTTP
Is this what you are looking for -- It's a java applet that has most of the features you mentioned. I have used it for very large files with no problem. http://www.javazoom.net/applets/jclientupload/jclientupload.html (Sorry, its not free software.) -Original Message- From: mike [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 26, 2008 10:07 AM To: PHP General list Subject: [PHP] Large/unreliable file uploading over HTTP Let's face it - HTTP is not very good for file uploads. It's stateless nature, slow connections, inability to resume (technically), etc, etc. What I've been thinking about is a way to skip all the normal annoyances with file uploading - multipart form encodings, file upload tools with specific needs, PUT vs POST, connection resets, ... the list goes on and on. Usenet and BitTorrent and other protocols have the right idea - split up the workload into smaller sets of data. It's easier to operate on. Usenet has NZB files. BitTorrent splits everything up into chunks. Not only would it make working with the data more portable (no need to set your PHP memory limit or POST limits to insane amounts to support large files) but it could also support multiple segments of the file being transferred at once... Here's somewhat of a process braindump of what I'm thinking. It still requires a 'smart' applet (Flash, Java, anything that can split a file up and send data over HTTP/HTTPS) - no special socket needs, no PUT support needed, don't even need to use multipart POST encoding (as far as I know) - just send the data in chunks over the wire and have a PHP script on the other side collect the data and reassemble it. Does this sound insane? I think this is a pretty good approach - no PUT needed, no large POST configuration required, anything could upload to it as long as it sends the information properly (I'm thinking HTTP POST for the header info, and the data could be sent as another POST field maybe base64 encoded or something that will stay safe during transit...) - take input file, checksum it (md5 / sha1) - calculate number of chunks to split it up based on $chunk configured size (for example 128k chunks) - split the file into chunks of $chunk size and create checksums for all (could use sfv?) - send request to the server - with the info - use JSON? action=begin filename=$filename filesize=$filesize checksum=$checksum segments=list of segments (unique segment id, checksum, bytes) - server sends back a "server ready" and unique $transaction_id - start sending to the server, send header with transaction key and unique chunk identifier in it action=process transaction=$transaction_id segment=$unique_segment_id data=base64_encode($segment_data) - when done, server sends back $transaction_id, $segment_id, $status (okay, fail) - client compares checksum for identifier, if okay, move to next chunk - if it does not match, retry uploading to server again - when all chunks are done, send request with transaction key and action=finish transaction=$transaction_id - when the server receives this, it assembles the file from the segments and does a final checksum, and reports the checksum back to the client (warning: on a large file this could take a bit?) and sends back $transaction_id, $checksum - client does one last check against the file's original checksum, if it matches report success, otherwise report failure (would need to determine why though - if all segments match this should not be able to happen...) I'd appreciate everyone's thoughts. This would also allow for file upload progress, more or less, as the server and client are constantly communicating when chunks are done and in progress (but again, that has to be done with an applet) I can't think of any method to do it in-browser, but doing it this way could open the gates for things like Google Gears to possibly work too... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php No virus found in this incoming message. Checked by AVG - http://www.avg.com Version: 8.0.138 / Virus Database: 270.6.7/1632 - Release Date: 25/08/2008 07:05 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Large/unreliable file uploading over HTTP
Let's face it - HTTP is not very good for file uploads. It's stateless nature, slow connections, inability to resume (technically), etc, etc. What I've been thinking about is a way to skip all the normal annoyances with file uploading - multipart form encodings, file upload tools with specific needs, PUT vs POST, connection resets, ... the list goes on and on. Usenet and BitTorrent and other protocols have the right idea - split up the workload into smaller sets of data. It's easier to operate on. Usenet has NZB files. BitTorrent splits everything up into chunks. Not only would it make working with the data more portable (no need to set your PHP memory limit or POST limits to insane amounts to support large files) but it could also support multiple segments of the file being transferred at once... Here's somewhat of a process braindump of what I'm thinking. It still requires a 'smart' applet (Flash, Java, anything that can split a file up and send data over HTTP/HTTPS) - no special socket needs, no PUT support needed, don't even need to use multipart POST encoding (as far as I know) - just send the data in chunks over the wire and have a PHP script on the other side collect the data and reassemble it. Does this sound insane? I think this is a pretty good approach - no PUT needed, no large POST configuration required, anything could upload to it as long as it sends the information properly (I'm thinking HTTP POST for the header info, and the data could be sent as another POST field maybe base64 encoded or something that will stay safe during transit...) - take input file, checksum it (md5 / sha1) - calculate number of chunks to split it up based on $chunk configured size (for example 128k chunks) - split the file into chunks of $chunk size and create checksums for all (could use sfv?) - send request to the server - with the info - use JSON? action=begin filename=$filename filesize=$filesize checksum=$checksum segments=list of segments (unique segment id, checksum, bytes) - server sends back a "server ready" and unique $transaction_id - start sending to the server, send header with transaction key and unique chunk identifier in it action=process transaction=$transaction_id segment=$unique_segment_id data=base64_encode($segment_data) - when done, server sends back $transaction_id, $segment_id, $status (okay, fail) - client compares checksum for identifier, if okay, move to next chunk - if it does not match, retry uploading to server again - when all chunks are done, send request with transaction key and action=finish transaction=$transaction_id - when the server receives this, it assembles the file from the segments and does a final checksum, and reports the checksum back to the client (warning: on a large file this could take a bit?) and sends back $transaction_id, $checksum - client does one last check against the file's original checksum, if it matches report success, otherwise report failure (would need to determine why though - if all segments match this should not be able to happen...) I'd appreciate everyone's thoughts. This would also allow for file upload progress, more or less, as the server and client are constantly communicating when chunks are done and in progress (but again, that has to be done with an applet) I can't think of any method to do it in-browser, but doing it this way could open the gates for things like Google Gears to possibly work too... -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
