Hey Hector, Ah, yes, I was operating with the assumption that the "client" machine was a server machine itself, although if it's a user's machine and a browser, the ability to download files in a chunked manner is somewhat different. The PHP scheme envisioned would work if the PHP machine were a Compute Engine instance, although be aware that the PHP App Engine runtime, due to the App Engine serving infrastructure, buffers whole responses until the request-handler terminates, so a chunked download from a PHP App Engine module wouldn't function here.
As for network costs, I think it's impossible to avoid network transfer of data from the upload machine to the machine that the user's browser will connect to for a chunked download. One stream of data is going to Cloud Storage, and another is going to a location that it will be possible to forward the chunks to the user from. As to the question of filesystem writing, really the only cross-browser way to do this is with file downloads. At this point, if you don't want to stream to both Cloud Storage and the machine the client connects to before the upload is completed, it seems possible that you could just cut losses and wait for the upload to complete before initiating a download from the client? Is there any particular reason that's undesirable, other than the time offset? Perhaps the uploads and downloads will be very large files? Maybe there's a way to structure the program so that the upload-wait period isn't damaging to the user experience / system behaviour? If you'd like, you could make a Feature Request in the Public Issue Tracker <http://code.google.com/p/google-cloud-platform/issues/list> for the Cloud Platform to have some means of chunked-downloading an object while it's being chunked-uploaded. We always encourage people to make Feature Requests, it helps improve the platform! I hope these thoughts have been helpful; let me know your thoughts. Sincerely, Nick Cloud Platform Community Support On Tuesday, April 26, 2016 at 9:01:31 AM UTC-4, Hector Montaner wrote: > > Hi Nick, > > Thanks for answering! Your solution has one inconvenient for me: the > downloading client needs also a bit of sophistication, not only the > uploading client. > > If I understood correctly, the downloading client will download one chunk > after another. If this is the case, the browser would have to download all > chunks, keep them in memory, assemble them into one blob, and then trigger > the "download" from memory to the user file system. This is a problem if > the file size is several GBs. > > I know there are some browsers like Chrome that allow writing files to > internal filesystem so that the assemble process can be done in disk. > However, I need this process specially for those browsers that do not > support FileSystem writing :) > > The best solution I can think of is to have an instance machine with for > example Apache+PHP that accepts an HTTP Get request for the download. The > PHP would iterate over the chunks and output them to the client in the body > of the only one HTTP Get request, and wait when the rest of chunks have not > yet been uploaded. I think this will work, but it would not be cheap, as I > would have to pay for the instance hours and for the bandwidth consumed to > send chunks from GAE to the client. > > What is your opinion? > > On Thursday, April 21, 2016 at 11:53:24 PM UTC+2, Nick (Cloud Platform > Support) wrote: >> >> Hey Hector, >> >> While this isn't possible in the way you described exactly, since you can >> only begin fetching an object once it's been uploaded successfully, there >> is a solution which accomplishes essentially the same thing, although it >> will require a slight bit of sophistication in the uploading client to >> ensure things go well: >> >> Have the uploading client send chunks of the file to the Cloud Storage >> API as a multipart upload >> <https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload#multipart> >> while, >> at the same time, sending these chunks to the other client (or sends them >> to a location where it will be available to the other client upon request). >> >> The complexity arises when you need to perform 2 multi-part upload >> processes, with retry / back-off logic for the two streams being >> independent. >> >> I hope this is clear enough in my explanation, let me know if you have >> any questions. >> >> Cheers, >> >> Nick >> Cloud Platform Community Support >> >> On Tuesday, April 19, 2016 at 1:11:41 PM UTC-4, Hector Montaner wrote: >>> >>> What I want to achieve is one client uploads a file to Google Cloud >>> Storage (several GB) and another client starts to download it before the >>> upload is finished, in a pipelined way. >>> Is it possible? If not, what alternatives do I have? >>> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/247c0f86-4448-4ade-81c0-323ffa496f48%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
