On 08/04/2013 01:48 AM, Jonas Sicking wrote:
Wow, lots of different subjects here. And I don't fully see how they all tie together. I'll try to answer the parts that I understand, but let me know if I missed anything.
Thanks for the very thorough follow-up!
OTOH, if you are saving the draft in IMAP, then why ever store the data in IndexedDB?
The draft currently is only saved locally. It's in the backlog to also save the draft on the server, but there are real-world complexities that mean there are 4 distinct cases, most of which (by volume) we can't do this for:
- on IMAP servers with the CATENATE and URLAUTH extension accompanied by an SMTP server with BURL extension support we can both (CATENATE) update/replace a draft without re-uploading the attachments and send the message from the IMAP server (we use BURL to tell the SMTP server to suck the message or pieces of it out of the IMAP server via URLAUTH). In the future, IMAP SUBMIT will become the way to do this, but it's still an in-progress draft (http://datatracker.ietf.org/doc/draft-kundrat-imap-submit/). I am not aware of any servers supporting this, though I'm sure they exist somewhere. (Dovecot 2.2+ supports URLAUTH, but postfix, a popular choice, does not support BURL).
- on IMAP servers with only the CATENATE extension or no BURL support on the SMTP server despite URLAUTH IMAP support, we can keep the draft up-to-date with the attachments, but sending the message is still a completely separate upload so we have to keep the attachments around locally (unless we re-download them right before sending, which seems bad). We would probably upload the attachments to the server for those servers for which the act of sending the message via SMTP does not automatically place the message into the sent folder since we would end up doing a duplicate APPEND into Sent anyways. This works on modern releases of IMAP servers like Dovecot (2.2+), Zimbra (used on mozilla.com) and Cyrus (used on fastmail.fm).
- on IMAP servers without CATENATE we have to APPEND the message from scratch to the server every time the draft get edited. For these servers I don't think we will ever upload the attachments by default, at least not until we have the ability to detect we are on an unmetered high-bandwidth connection. This is the case for GMail, Yahoo, all domains hosted by dreamhost.com, all domains hosted by godaddy.com, etc.
- on ActiveSync servers the protocol has no support for sticking stuff in a drafts folder. We can never upload drafts ever.
However I don't understand why you simply couldn't create a big Blob which represents the whole thing that you want to "stream". You can do stuff like x = new Blob([encodedPrefix, attachment1asBlob, encodedSeparator, attachment2asBlob]);
Yeah, that's the super-blob I was talking about. Sorry for making up arbitrary terms instead of just demonstrating what I meant!
I think we might have an implementation bug right now so that when you use XHR to submit a Blob, we load that whole Blob into memory. That is something we should be able to fix though. No new APIs needed, someone just needs to flag this as an important issue.
Are you sure about this? I had trouble finding any bugs and it appears that all Blob/File implementations inherit from nsDOMFileBase which implements nsIXHRSendable. XHR from a worker marshals the nsIDOMFile across to the main thread where a fresh mozilla::dom::workers::file::Blob/File is created to wrap it. That gets turned into an XPCVariant with type VTYPE_INTERFACE_IS. nsXMLHttpRequest then keeps trying interfaces until it gets to nsIXHRSendable and the file's GetInternalStream is called to produce an nsIInputStream. All of the implementations look pretty straightforward; nsDOMMultipartFile creates a multiplex stream, nsDOMFileFile creates a local stream, nsDOMMemoryFile creates a byte input stream. The IPC RemoteBlob stuff also seems okay, but it shouldn't be involved as far as I can tell.
If you think there's a subtle error where something breaks, I can do a run with gdb and check what's actually happening. Or if I'm completely wrong about the path the data is taking, please let me know!
It sounds like you are asking if you should save the base64-encoded Blobs to DeviceStorage or IndexedDB. Since you are creating base64-encoded data you are by necessity creating a copy of the data. So I don't really see a reason to write the copy to DeviceStorage. That just introduces all the same problems as discussed above. I.e. what happens if the user suddenly removes the SD card? And it means that other apps can see email drafts that are being created, which would be a privacy problem. Remember that DeviceStorage can be read by other apps. It is intended to store user data like camera pictures and downloaded documents. Not application specific data that other apps shouldn't load.
I think we're in vigorous agreement then that the downsides of storing potentially large things in IndexedDB are outweighed by the downsides of storing the data in device storage. This is also handy because the IndexedDB refcounting is awesome.
The only tricky part with writing base64-encoded Blobs is that you probably don't want to base64 encode the whole Blob in one go. The FileHandle API would let you handle this, but it doesn't work on B2G yet.
Yeah, this is the other sadness I've had to work around. See my other post, "[b2g] Writing large Blobs to DeviceStorage without getting OOM-killed without getEditable, IndexedDB's mozCreateFileHandle being main-proc only?" (https://groups.google.com/forum/#!topic/mozilla.dev.b2g/N8aHf8AN1mk) request for confirmation on that. I think you're confirming it here, though!
What you can do is that you can encode the data in small chunks and store those chunks as Blobs in IndexedDB. Once you have encoded all parts, you can load back all the chunks (which will now be stored on disk rather than in memory) and do new fullBlob = new Blob([part1, part2, part3...]); and then write fullBlob to IndexedDB. That will cause the data to be merged into a new file while never having all the data in memory. Though temporarily the base64 encoded data lives in two copies on disk. I.e. you'll have both the various parts, and the merged file. Alternatively you can use the parts directly when sending the data to the server.
My patch keeps the encoded chunks in their separate files since it seems like merging the file doesn't gain us anything but will result in additional disk I/O.
Thanks again! Andrew _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
