Re: [openstack-dev] [horizon] [javascript] Async file uploads in a new JS-centric UI

[Timur Sufiev]

Since 10 lines of code > 1000 words, here are references to 2 patch chains:

* New Swift UI file upload https://review.openstack.org/#/c/316143/
* New Angular Create Image file upload
https://review.openstack.org/#/c/317456/

I like Create Image solution more because it doesn't use Django csrf_exempt
and single FileField form hacks just to accept a binary stream on Django
side. So CORS offers us a shorter and more elegant solution to the task of
file uploads.

I got an off-ML feedback that the question / intention of original mail is
not clear. My intention / proposal is to adopt the approach used for
uploading files in Create Image workflow as the standard way for other
interactions (which include file uploads) between Web Clients and OpenStack
services.

On Sat, May 14, 2016 at 2:48 AM Timur Sufiev  wrote:

> Hello, fellow web developers!
>
> I'd like to share some of my thoughts and findings that I made during
> playing with ng-file-upload [1] for file uploads in Swift UI and Create
> Image wizard in Horizon (both Angular). Sorry for a long letter, couldn't
> put it shorter (TL;DR => go to CONCLUSION section).
>
> As a foreword: it's a really useful library, both for customizing stubborn
>  widget appearance (hello, themability!) and behavior
> and for the actual file transfer. More on the file transfer below, since
> it's the topic that really interests me.
>
> MAIN PART
>
> First, in modern browsers (by modern I mean HTML5 support and particularly
> FileReader API) you don't need a single-purposed library to upload file
> asynchronously, both jQuery $.ajax() and vanilla Angular $http calls
> support it - just pass File()/Blob() object as data (no wrapping {} please)
> and it works - browser transparently reads data chunk by chunk  from a
> local file system and sends it to the server. There is even a working
> solution for Django and jQuery-based 'Create Image' form [2]. There are a
> few subtleties though. Most importantly, there should be no other data
> (like other key-value pairs from form fields), just the file blob - and
> then the server endpoint must support raw byte stream as well. This rules
> out Django views which expect certain headers and body structure.
>
> (Second,) What ng-file-upload offers us to solve the challenge of file
> transfers? There are 2 methods in Upload service: .http() and .upload().
> First one is a very thin wrapper around Angular $http, with one difference
> that it allows to notify() browser of file upload progress (when just a
> single file blob is passed in .http(), as in case of $http() as well). The
> second method offers more features, like abortable/resumable uploads and
> transparent handling of data like {key1: value1, key2: value2, file:
> FileBlob}. Uploading such data is implemented using standard
> multipart/form-data content type, so actually, it's just a convenient
> wrapper around facilities we've already seen. Anyways it's better to just
> feed the data into Upload.upload() than to construct FormData() on your own
> (still the same is happening under the bonnet).
>
> Third, and most important point, we still have to couple Upload.http() /
> Upload.upload() with a server-side machinery. If it's a single file upload
> with Upload.http(), then the server must be able to work with raw binary
> stream (I'm repeating myself here). If it's a form data including file
> blob, it's easily handled on front-end with Upload.upload(), then the
> server must be able to parse multipart/form-data (Django perfectly does
> that). What's bad in this situation is that it also needs to store any
> sufficiently sized file in a web server's file system - which is both
> bug-prone [4] and suboptimal from performance POV. First we need to send a
> file (possibly GB-sized) from browser to web server, then from web server
> to the Glance/Swift/any other service host. So, blindly using
> Upload.upload() won't solve our _real_ problems with file uploads.
>
> CONCLUSION
>
> What can be done here to help JS UI to handle really large uploads? Split
> any API calls / views / whatever server things we have into 2 parts:
> lightweight JSON metadata + heavyweight binary stream. Moreover, use CORS
> for the second part to send binary streams directly to that data consumers
> (I know of 2 services atm - Glance & Swift, maybe there are more?). That
> will reduce the processing time, increasing the possibility that an
> operation will complete successfully before Keystone token expires :). IMO
> any new Angular wizard in Horizon should be designed with this thing in
> mind: a separate API call for binary stream transfer.
>
> Thoughts, suggestions?
>
> P.S. None of above means that we shouldn't use ng-file-upload, it's still
> a very convenient tool.
>
> [1] https://github.com/danialfarid/ng-file-upload
> [2] https://review.openstack.org/#/c/230434/
> [3]
> https://github.com/openstack/horizon/blob/master/horizon/static/horizon/js/horizon.modals.js#L216
> [4] 

[openstack-dev] [horizon] [javascript] Async file uploads in a new JS-centric UI

[Timur Sufiev]

Hello, fellow web developers!

I'd like to share some of my thoughts and findings that I made during
playing with ng-file-upload [1] for file uploads in Swift UI and Create
Image wizard in Horizon (both Angular). Sorry for a long letter, couldn't
put it shorter (TL;DR => go to CONCLUSION section).

As a foreword: it's a really useful library, both for customizing stubborn
 widget appearance (hello, themability!) and behavior
and for the actual file transfer. More on the file transfer below, since
it's the topic that really interests me.

MAIN PART

First, in modern browsers (by modern I mean HTML5 support and particularly
FileReader API) you don't need a single-purposed library to upload file
asynchronously, both jQuery $.ajax() and vanilla Angular $http calls
support it - just pass File()/Blob() object as data (no wrapping {} please)
and it works - browser transparently reads data chunk by chunk  from a
local file system and sends it to the server. There is even a working
solution for Django and jQuery-based 'Create Image' form [2]. There are a
few subtleties though. Most importantly, there should be no other data
(like other key-value pairs from form fields), just the file blob - and
then the server endpoint must support raw byte stream as well. This rules
out Django views which expect certain headers and body structure.

(Second,) What ng-file-upload offers us to solve the challenge of file
transfers? There are 2 methods in Upload service: .http() and .upload().
First one is a very thin wrapper around Angular $http, with one difference
that it allows to notify() browser of file upload progress (when just a
single file blob is passed in .http(), as in case of $http() as well). The
second method offers more features, like abortable/resumable uploads and
transparent handling of data like {key1: value1, key2: value2, file:
FileBlob}. Uploading such data is implemented using standard
multipart/form-data content type, so actually, it's just a convenient
wrapper around facilities we've already seen. Anyways it's better to just
feed the data into Upload.upload() than to construct FormData() on your own
(still the same is happening under the bonnet).

Third, and most important point, we still have to couple Upload.http() /
Upload.upload() with a server-side machinery. If it's a single file upload
with Upload.http(), then the server must be able to work with raw binary
stream (I'm repeating myself here). If it's a form data including file
blob, it's easily handled on front-end with Upload.upload(), then the
server must be able to parse multipart/form-data (Django perfectly does
that). What's bad in this situation is that it also needs to store any
sufficiently sized file in a web server's file system - which is both
bug-prone [4] and suboptimal from performance POV. First we need to send a
file (possibly GB-sized) from browser to web server, then from web server
to the Glance/Swift/any other service host. So, blindly using
Upload.upload() won't solve our _real_ problems with file uploads.

CONCLUSION

What can be done here to help JS UI to handle really large uploads? Split
any API calls / views / whatever server things we have into 2 parts:
lightweight JSON metadata + heavyweight binary stream. Moreover, use CORS
for the second part to send binary streams directly to that data consumers
(I know of 2 services atm - Glance & Swift, maybe there are more?). That
will reduce the processing time, increasing the possibility that an
operation will complete successfully before Keystone token expires :). IMO
any new Angular wizard in Horizon should be designed with this thing in
mind: a separate API call for binary stream transfer.

Thoughts, suggestions?

P.S. None of above means that we shouldn't use ng-file-upload, it's still a
very convenient tool.

[1] https://github.com/danialfarid/ng-file-upload
[2] https://review.openstack.org/#/c/230434/
[3]
https://github.com/openstack/horizon/blob/master/horizon/static/horizon/js/horizon.modals.js#L216
[4] https://bugs.launchpad.net/horizon/+bug/1403129
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev