On Fri, Jun 28, 2013 at 7:37 AM, Andrew Sutherland <[email protected]> wrote: > Currently, when you save a draft with attachments in the e-mail app, the > attached files will be duplicated from wherever they came from (probably > DeviceStorage) into the IndexedDB blob store. It's my understanding that we > model the internal storage where IndexedDB lives as very limited, so this is > a bad idea. > > The nice things are that: > - IndexedDB does all the book-keeping so that when the Blobs are no longer > referenced by the DB or the app the Blobs go away. > - We don't run into the horrible problem where the user swaps their SD card > out with another one because they want to listen to a different set of music > and now the e-mail app is missing all kinds of important data it needs. > > There is the begged question of whether the e-mail app should try and avoid > creating duplicate copies of blobs when the Blob is actually a File from > DeviceStorage and we are able to detect this. (Do Files express the > DeviceStorage storage areas they came from? Would we just try and reverse > map based on mime-type or hope sdcard and the storage name line up exactly?) > If we did this, there is the potential UX problem of the user deleting the > attachment after attaching it to a message, but either way, the issues below > still hold. > > > I am currently overhauling some aspects of mail sending so that we can > stream the contents of the composed message over the network rather than > generating a single in-memory string that potentially kills the app without > the message getting sent, so I would like to choose the correct/best option. > > My planned changes call for storing the attachment portions of messages as > pre-base64-encoded rfc2822 MIME body (sub-)parts. This is primarily driven > by the fact that ActiveSync uses XHR to communicate with the server and so > we need to provide it with a Blob since we can't generate the data on demand > as our network buffers empty out. If we were chrome-space, we could > apparently implement nsIInputStream and pass that in, but we aren't. For > IMAP and SMTP we use mozTCPSocket which does allow us to generate data on > demand by virtue of providing flow control signalling via return values and > 'drain' events. We could, of course, re-implement XHR on top of > mozTCPSocket to try and get around that problem, but that's pretty > undesirable. By storing the attachments as pre-encoded on-disk blobs, this > lets us stitch together a single super-Blob from the less memory-intensive > MIME parts and the living-on-disk attachments when we go to SMTP send, IMAP > APPEND, or ActiveSync send. It's notable that the SMTP and IMAP operations > may potentially vary in representation if there are any bcc recipients. > > So the question is then whether to use IndexedDB or DeviceStorage. For > DeviceStorage, we presumably would use 'sdcard' under a directory we create > for e-mail, using an "x-" mime-type we make up. We would manually handle > Blob life-cycle issues by deleting the Blobs when the draft is either > completely purged or the message is completely sent. When initially > generating the mail database, we would trigger a scan of DeviceStorage to > purge any attachments left over from a previous install of the app. And I > guess if the user uninstalls the e-mail app, we just leave our detritus > lying around. > > Since the number of drafts is currently unbounded and we will soon support > background mail-sending for e-mail which will then also allow for unbounded > growth in data, DeviceStorage seems like the right place. Or at least the > least-bad place. > > Does this sound right?
Hi Andrew, Wow, lots of different subjects here. And I don't fully see how they all tie together. I'll try to answer the parts that I understand, but let me know if I missed anything. Regarding data getting duplicated in IndexedDB: Yes, when you use a WebActivity to get a File, and the returned File comes from DeviceStorage, if you store that file in IndexedDB the data will get duplicated. We could make it so that when the gallery app pass a File back from DeviceStorage, it also passes the filename and DeviceStorage area that was used. That way the email app could choose to drop the returned File on the floor and instead just remember the storage-area and filename. This would avoid the data duplication, but like you point out, if the user removes the SD card the file would get lost. This is especially problematic if we allow long lived drafts to be stored locally. But it may not be as big of a problem if you are planning on saving the draft in IMAP right away. OTOH, if you are saving the draft in IMAP, then why ever store the data in IndexedDB? Regarding streaming data to the server: You are correct that you can't really stream data using XHR right now. It's something that is being looked at, but we don't have the capability yet. However I don't understand why you simply couldn't create a big Blob which represents the whole thing that you want to "stream". You can do stuff like x = new Blob([encodedPrefix, attachment1asBlob, encodedSeparator, attachment2asBlob]); When this is done, any Blob (or File) objects that are passed to the Blob constructor are *not* copied. Instead the newly created Blob simply grabs a reference to the passed in Blob sections. When someone, including the XHR code, reads from the newly created Blob we internally read from the referenced blobs. I think we might have an implementation bug right now so that when you use XHR to submit a Blob, we load that whole Blob into memory. That is something we should be able to fix though. No new APIs needed, someone just needs to flag this as an important issue. Regarding base64-encoded Blobs: If you want to base64-encode a Blob and save it to disk so that you can use the trick above to stream directly from disk, then that should be quite doable. It sounds like you are asking if you should save the base64-encoded Blobs to DeviceStorage or IndexedDB. Since you are creating base64-encoded data you are by necessity creating a copy of the data. So I don't really see a reason to write the copy to DeviceStorage. That just introduces all the same problems as discussed above. I.e. what happens if the user suddenly removes the SD card? And it means that other apps can see email drafts that are being created, which would be a privacy problem. Remember that DeviceStorage can be read by other apps. It is intended to store user data like camera pictures and downloaded documents. Not application specific data that other apps shouldn't load. The only tricky part with writing base64-encoded Blobs is that you probably don't want to base64 encode the whole Blob in one go. The FileHandle API would let you handle this, but it doesn't work on B2G yet. What you can do is that you can encode the data in small chunks and store those chunks as Blobs in IndexedDB. Once you have encoded all parts, you can load back all the chunks (which will now be stored on disk rather than in memory) and do new fullBlob = new Blob([part1, part2, part3...]); and then write fullBlob to IndexedDB. That will cause the data to be merged into a new file while never having all the data in memory. Though temporarily the base64 encoded data lives in two copies on disk. I.e. you'll have both the various parts, and the merged file. Alternatively you can use the parts directly when sending the data to the server. Ok, I think that covers your questions. But let me know if I missed anything. / Jonas _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
