[Lift] Re: Thoughts on file streaming
Awesome! That's a nice way to bypass the issues I was running into. I suppose that if someone wanted to control it on a per-form basis then they could do it themselves via their own field mappings. Derek On Wed, Jul 1, 2009 at 12:20 PM, David Pollak feeder.of.the.be...@gmail.com wrote: Okay... I've checked in new code. In LiftRules, there's: /** * The function that converts a fieldName, contentType, fileName and an InputStream into * a FileParamHolder. By default, create an in-memory instance. Use OnDiskFileParamHolder * to create an on-disk version */ var handleMimeFile: (String, String, String, InputStream) = FileParamHolder = (fieldName, contentType, fileName, inputStream) = new InMemFileParamHolder(fieldName, contentType, fileName, Helpers.readWholeStream(inputStream)) You can change handleMimeFile to use OnDiskFileParamHolder. There's a companion object that has a helper that'll allow you to create an instance out of the parameters. By default, the OnDiskFileParamHolder deletes the temporary file when it is finalized. If you want to monitor the progress of an upload, use LiftRules: /** * The global multipart progress listener: * pBytesRead - The total number of bytes, which have been read so far. *pContentLength - The total number of bytes, which are being read. May be -1, if this number is unknown. *pItems - The number of the field, which is currently being read. (0 = no item so far, 1 = first item is being read, ...) */ var progessListener: (Long, Long, Int) = Unit = (_, _, _) = () By default, it does nothing, but you can put in a function that'll look for a session-specific CometActor and send it messages. If the above does not satisfy all your needs, you can subclass FileParamHolder and do whatever you want. Thanks, David On Wed, Jul 1, 2009 at 10:31 AM, David Pollak feeder.of.the.be...@gmail.com wrote: K... I'll check some code in later today. On Wed, Jul 1, 2009 at 9:05 AM, Andrew Scherpbier and...@scherpbier.orgwrote: It would also be nice to be able to provide progress feedback. The page returned after the form submission can then use comet to display a progress bar or something like that. (The media upload at Vimeo comes to mind as a good example of this!) Also, I would suggest some smarts when creating temporary files. It might be good to abstract the cache so that small files don't get put on disk. --Andrew Vlad Seryakov wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp -- Lift, the simply functional web framework http://liftweb.net Beginning Scala
[Lift] Re: Thoughts on file streaming
It is not about dropping Lift, it is about starting to use it. I want it very much but i need to consider all pluses and minuses before we jump in. External server for uploading could be a solution but it will not be in Scala and will require some other tools to be involved. I am still looking for solutions how i can do it with Lift. Frankly out there i do not see anything to replace Scala/Lift. On Jul 1, 1:08 am, Naftoli Gugenhem naftoli...@gmail.com wrote: I'm not too familiar with the topic but instead of dropping lift, as a temporary workaround can't you have the upload be handled by another servlet which will save it and then redirect? - Vlad Seryakovvserya...@gmail.com wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
On Tue, Jun 30, 2009 at 11:32 PM, David Pollak feeder.of.the.be...@gmail.com wrote: On Tue, Jun 30, 2009 at 2:22 PM, Viktor Klang viktor.kl...@gmail.comwrote: Create a tmp-file, store the localpath in S.param and register a delete on request done will be more streamlined than potentially monkeypatching it in Thoughts? Huge hack Yes, and I apologize for suggesting it, it was inappropriate. As we all know, the problem is that if you want to have access to the _full_ current state of the client (S) at the time of parsing the request (the parsing of the file likely not the last item in the request) in order to be able to process the file in a stream, unless some other solution is found, the stream parsing of the file has to rely on not having current client side state (S) available. Now, I've made such solutions in the past, and the impression I've had from API users have been that they actually expected client state to be available at file parse time. So in order to solve this you'd have to either guarantee that the file(s) present in the multipart request is always located _after_ any and all other form fields/parameters and the issue the callback at the time of each file item being encountered (at the time of calling openStream()) Also, so far in the thread different behaviors of file uploads are mentioned: 1) Process one or many files in a streaming fashion 2) Multiple file upload (store on server and process when user says he's done) As we've all said in this thread: The problem is not trivial to solve in a satisfactory way, but I think we all can agree that you, David, will most likely be able to solve it, as usual, very cleanly. One possible solution that I can imagine is to have a multiple request solution: On the client, before every file upload, send the current client state as a separate request and wait for a 200/OK before sending a request containing one or more files. This way you can rely on the client state being fresh when processing the files, and lift would invoke the input stream processor associated with the file field in question. -- Viktor On Jun 30, 2009 11:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array[Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, before LiftSession has been set up, and the act of parsing the request for form data, particularly for large upload streams (the target of these changes) precludes holding on to any data for later processing (the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that their form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp -- Viktor Klang Scala Loudmouth --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
It would also be nice to be able to provide progress feedback. The page returned after the form submission can then use comet to display a progress bar or something like that. (The media upload at Vimeo comes to mind as a good example of this!) Also, I would suggest some smarts when creating temporary files. It might be good to abstract the cache so that small files don't get put on disk. --Andrew Vlad Seryakov wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
K... I'll check some code in later today. On Wed, Jul 1, 2009 at 9:05 AM, Andrew Scherpbier and...@scherpbier.orgwrote: It would also be nice to be able to provide progress feedback. The page returned after the form submission can then use comet to display a progress bar or something like that. (The media upload at Vimeo comes to mind as a good example of this!) Also, I would suggest some smarts when creating temporary files. It might be good to abstract the cache so that small files don't get put on disk. --Andrew Vlad Seryakov wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
Okay... I've checked in new code. In LiftRules, there's: /** * The function that converts a fieldName, contentType, fileName and an InputStream into * a FileParamHolder. By default, create an in-memory instance. Use OnDiskFileParamHolder * to create an on-disk version */ var handleMimeFile: (String, String, String, InputStream) = FileParamHolder = (fieldName, contentType, fileName, inputStream) = new InMemFileParamHolder(fieldName, contentType, fileName, Helpers.readWholeStream(inputStream)) You can change handleMimeFile to use OnDiskFileParamHolder. There's a companion object that has a helper that'll allow you to create an instance out of the parameters. By default, the OnDiskFileParamHolder deletes the temporary file when it is finalized. If you want to monitor the progress of an upload, use LiftRules: /** * The global multipart progress listener: * pBytesRead - The total number of bytes, which have been read so far. *pContentLength - The total number of bytes, which are being read. May be -1, if this number is unknown. *pItems - The number of the field, which is currently being read. (0 = no item so far, 1 = first item is being read, ...) */ var progessListener: (Long, Long, Int) = Unit = (_, _, _) = () By default, it does nothing, but you can put in a function that'll look for a session-specific CometActor and send it messages. If the above does not satisfy all your needs, you can subclass FileParamHolder and do whatever you want. Thanks, David On Wed, Jul 1, 2009 at 10:31 AM, David Pollak feeder.of.the.be...@gmail.com wrote: K... I'll check some code in later today. On Wed, Jul 1, 2009 at 9:05 AM, Andrew Scherpbier and...@scherpbier.orgwrote: It would also be nice to be able to provide progress feedback. The page returned after the form submission can then use comet to display a progress bar or something like that. (The media upload at Vimeo comes to mind as a good example of this!) Also, I would suggest some smarts when creating temporary files. It might be good to abstract the cache so that small files don't get put on disk. --Andrew Vlad Seryakov wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp
[Lift] Re: Thoughts on file streaming
Wow! This is very cool. Can't wait to try it out. --Andrew David Pollak wrote: Okay... I've checked in new code. In LiftRules, there's: /** * The function that converts a fieldName, contentType, fileName and an InputStream into * a FileParamHolder. By default, create an in-memory instance. Use OnDiskFileParamHolder * to create an on-disk version */ var handleMimeFile: (String, String, String, InputStream) = FileParamHolder = (fieldName, contentType, fileName, inputStream) = new InMemFileParamHolder(fieldName, contentType, fileName, Helpers.readWholeStream(inputStream)) You can change handleMimeFile to use OnDiskFileParamHolder. There's a companion object that has a helper that'll allow you to create an instance out of the parameters. By default, the OnDiskFileParamHolder deletes the temporary file when it is finalized. If you want to monitor the progress of an upload, use LiftRules: /** * The global multipart progress listener: * pBytesRead - The total number of bytes, which have been read so far. * pContentLength - The total number of bytes, which are being read. May be -1, if this number is unknown. * pItems - The number of the field, which is currently being read. (0 = no item so far, 1 = first item is being read, ...) */ var progessListener: (Long, Long, Int) = Unit = (_, _, _) = () By default, it does nothing, but you can put in a function that'll look for a session-specific CometActor and send it messages. If the above does not satisfy all your needs, you can subclass FileParamHolder and do whatever you want. Thanks, David On Wed, Jul 1, 2009 at 10:31 AM, David Pollak feeder.of.the.be...@gmail.com wrote: K... I'll check some code in later today. On Wed, Jul 1, 2009 at 9:05 AM, Andrew Scherpbier and...@scherpbier.org wrote: It would also be nice to be able to provide progress feedback. The page returned after the form submission can then use comet to display a progress bar or something like that. (The media upload at Vimeo comes to mind as a good example of this!) Also, I would suggest some smarts when creating temporary files. It might be good to abstract the cache so that small files don't get put on disk. --Andrew Vlad Seryakov wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp -- Lift, the simply functional web framework
[Lift] Re: Thoughts on file streaming
Create a tmp-file, store the localpath in S.param and register a delete on request done will be more streamlined than potentially monkeypatching it in Thoughts? -- Viktor On Jun 30, 2009 11:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array[Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, before LiftSession has been set up, and the act of parsing the request for form data, particularly for large upload streams (the target of these changes) precludes holding on to any data for later processing (the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that their form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
On Tue, Jun 30, 2009 at 2:22 PM, Viktor Klang viktor.kl...@gmail.comwrote: Create a tmp-file, store the localpath in S.param and register a delete on request done will be more streamlined than potentially monkeypatching it in Thoughts? Huge hack -- Viktor On Jun 30, 2009 11:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array[Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, before LiftSession has been set up, and the act of parsing the request for form data, particularly for large upload streams (the target of these changes) precludes holding on to any data for later processing (the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that their form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.comwrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array[Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, before LiftSession has been set up, and the act of parsing the request for form data, particularly for large upload streams (the target of these changes) precludes holding on to any data for later processing (the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that their form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
It's just an open item, and one that I had hoped I could contribute on. If you have some ideas on restructuring the param handling I can take it from there. No rush. Derek On Tue, Jun 30, 2009 at 3:33 PM, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.comwrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array[Byte]) ⇒ Any or (String, String, InputStream) ⇒ Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, before LiftSession has been set up, and the act of parsing the request for form data, particularly for large upload streams (the target of these changes) precludes holding on to any data for later processing (the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that their form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) ⇒ Any or (String, String, InputStream) ⇒ Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Git some: http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
I'm not too familiar with the topic but instead of dropping lift, as a temporary workaround can't you have the upload be handled by another servlet which will save it and then redirect? - Vlad Seryakovvserya...@gmail.com wrote: I am also contemplating to use Lift but lack of big file upload is the showstopper. We need to upload images, and big video files and currently there i snot way to do it in Lift, i need something else to handle that which makes the whole stuff more complex than needed. Spooling into temp file and doing async upload of multiple files at the same time is what needed. Nowadays media uploads is normal and those files are getting bigger and bigger. On Jun 30, 5:58 pm, Timothy Perrett timo...@getintheloop.eu wrote: This has been hurting me for quite a while now (raised it on list about 2 months ago) and could really do with getting it fixed. As derek points out, it's not a small change which is why I've done nothing about it to date - a little too much core hacking to feel happy If you think your able to do something about it that would be absolutly brilliant! Cheers Tim Sent from my iPhone On 30 Jun 2009, at 22:33, David Pollak feeder.of.the.be...@gmail.com wrote: What kind of priority is this issue? I think I can abstract things in such a way that it works correctly, but it'll take a couple of days. On Tue, Jun 30, 2009 at 2:08 PM, Derek Chen-Becker dchenbec...@gmail.com wrote: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array [Byte]) = Any or (String, String, InputStream) = Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, befor e LiftSession has been set up, and the act of parsing the request fo r form data, particularly for large upload streams (the target of th ese changes) precludes holding on to any data for later processing ( the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that t heir form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Derek -- Lift, the simply functional web frameworkhttp://liftweb.net Beginning Scalahttp://www.apress.com/book/view/1430219890 Follow me:http://twitter.com/dpp Git some:http://github.com/dpp --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
[Lift] Re: Thoughts on file streaming
Derek Chen-Becker dchenbec...@gmail.com writes: Well, as usual something that seemed simple at first glance is now looking somewhat complex. I'm thinking of reworking the fileUpload handling to allow a user to register either a (String, String, Array[Byte]) ⇒ Any or (String, String, InputStream) ⇒ Any function, which would then be executed during request processing. The issue is that form field processing (ParamHolders) takes place in Req, before LiftSession has been set up, and the act of parsing the request for form data, particularly for large upload streams (the target of these changes) precludes holding on to any data for later processing (the servlet container cannot be expected to hold the entire request in memory). On the other hand, users should reasonably expect that their form handling functions are stateful, so I'm trying to think of some way to meet in the middle on form processing. Ideas? Sounds nice. Will this in some way be able to display any progress of the upload (which, imho, is needed for large uploads). E.g. something like this http://wiki.nginx.org//NginxHttpUploadProgressModule /Jeppe --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Lift group. To post to this group, send email to liftweb@googlegroups.com To unsubscribe from this group, send email to liftweb+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/liftweb?hl=en -~--~~~~--~~--~--~---
