Hi Alex, Sorry for the delayed answer, I've been swamped with work and haven't had the time to deal with this for a while. Looks like I will have quite some time this week though, and I hope to commit by the end of the week. My suggestion is to commit on a separate branch and we can then build it out further from there.
Anyhow, here's an outline of what we've done so far; In order to be able to deal with larger files we now work extensively with blobs and typed arrays rather than strings. Blob's are a good way to handle large chunks of data since they are stored on the file system and not just kept in memory. Rather than dealing with blob objects right away we created two new buffer objects that basically reads in a small part of the blob to avoid keeping it all in memory. It reads in a small chunk and returns a typed array. By doing this we can perform normal array operations on the data which is more convenient than dealing with a blob. Currently we have on buffer for uint8array (used for encryption/decryption) and one for int32Arrays (for SHA generation). I'd also like to point out that we've only changed the parts that are critical for us. This would mainly be AES encryption/decryption and generating SHA-1 hashes. Below is a more detailed description of the code changes we made. Encryption: We've created a new function in openpgp.packet.literaldata named write_packet_large. It takes a buffer object as input and returns a new buffer containing the data + header. We've also created a new function in openpgp_packet_encrypteddata again called write_packet_large. This function is not very different from the normal one besides that it of course deals with the new buffer object rather than strings. From there openpgp_crypto_symmetricEncrypt is called. We always use AES when doing symmetric encryption so we haven't modified any of the other ones. Last but not least we've added a new openpgp_cfb_encrypt_large function which takes in a buffer object. We've modified the original function so it will now read in chunks from the blob and encrypt them. When done it returns a new buffer containing the cipher text. Besides that it is also possible to pass in a callback object to which progress is reported back. One note here is that we always use the resync option when encrypting which means we have so far just converted that. I could possible convert the non resync case before committing as well. Decryption The changes made to the decryption flow is similar to the ones we did for encryption. Our typical flow looks like this: We added a new read_packet_large to openpgp_packet. In here we made one major change. Previously there was a loop to determine the length for a packet of indeterminate length (https://tools.ietf.org/html/rfc4880#section-4.2.1, https://tools.ietf.org/html/rfc4880#section-4.2.2). We found that this loop was delaying the decryption process quite a bit for larger files so it was removed. I don't believe there is really no need to know the length in advance, correct me if I'm wrong? Instead we're calculating the length while decrypting. Just as previously, the functions returns a package. The difference is that the encrypted data is stored as a buffer rather than a string. Again, only the methods we are using have been converted. The only packages using large amounts of data that we use is the type 9 packages. The decryption is then done very similar to how the encryption was accomplished. SHA-1 Generation Similarly we've modified the SHA generation to accept a buffer as input. Again, We've appended _large to the modified functions to ensure that regular strings works as before. At last Since encryption/decryption can be a time consuming task we decided to do it from web workers. There is currently a JQuery reference when encoding a html string which won't work when used in a Web Worker context. We never used it so we just ended up commenting it out although that is of course not a good solution. Another problem we ran into is that window.crypto.getRandomValues() is not defined for Firefox (and not defined in a web worker context for any other browser either). Currently we will instead use SJCL (the crypto lib from Stanford) to get random numbers for Firefox. This is however handled outside of OpenPGP since we need to generate the numbers from the main thread and pass them to the worker. Currently all of this work with Firefox, Chrome and IE10. It is possible to add support for Safari as well and I hope to be able to do this this week, although that's nothing I can promise. Other browsers are not tested (except for lower versions of IE that will not work). Anyhow, Sorry for the much delayed answer, let me know if there's any questions.. On Jan 9, 2013, at 2:56 PM, Alex (via OpenPGP.js) <[email protected]> wrote: > Hi Erik, > > Thank you for the answer. > >> I work in a project where we are using OpenPGP.js in a production >> environment. We however use it to encrypt larger files which forced us to >> make a few changes. > > Would love to read more about this. > >> If you guys want to use our logo on the OpenPGP.js site we can probably >> arrange that as well. > > Yes! > >> If you guys are interested we would like to commit this back to the project >> for others to use. > ... >> Would these changes be of interest to the project? > > Definitely! I think it would be extremely helpful to have a short benchmark > before and after your changes that we can publish. > > Best regards, Alex > > On 08.01.2013, at 18:33, Erik Larsson <[email protected]> wrote: > >> Hi, >> >> I've been on this mailing list for a while without writing. I work in a >> project where we are using OpenPGP.js in a production environment. We >> however use it to encrypt larger files which forced us to make a few >> changes. The major one being to support blobs when decrypting/encrypting >> data. We had to do this since we are likely to operate on files > 100 mb and >> it was just not possible to pass around a string with that size. >> >> If you guys are interested we would like to commit this back to the project >> for others to use. Today I'm mostly reaching out to get the conversation >> started. There's some work that has to be done on our part before merging so >> I just wanted to touch base on what we've done. Right now we have added >> prefixed functions when we deal with large files so rather than calling >> write_packet we call write_packet_large and so forth. We did this mostly to >> keep our changes separate from the original source. We are willing to change >> this in however way fits the project guidelines better. There are some >> additional changes that comes to this but this would be the major one. Would >> these changes be of interest to the project? >> >> If you guys want to use our logo on the OpenPGP.js site we can probably >> arrange that as well. >> >> Best Regards >> Erik >> >> On Jan 8, 2013, at 3:16 AM, Alex (via OpenPGP.js) <[email protected]> wrote: >> >>> Dear all, >>> >>> Happy New Year. I think it's time to give the OpenPGP.js project a bit more >>> "love" in 2013. There are many items on the todo list - so let us address >>> the first ones: >>> >>> 1. Marketing: Which projects are currently using OpenPGP.js? I would like >>> to add links and logos to our web page. Also I just restarted to use the >>> Twitter account http://twitter.com/openpgpjs to retweet and answer related >>> posts. Anyone is welcome to join. Also I've created a new simple logo (see >>> attached). >>> >>> 2. Developing: It should be very easy for users to integrate the library >>> into their web pages and for developers to enhance the current version. I >>> think we can improve the current situation. So we also might want to move >>> this mailing list to another one with archive support (btw: is a mailing >>> list still an adequate perfect medium?) >>> >>> 3. Security: There are a lot of discussions about the advantages and >>> drawbacks of using a JavaScript based OpenPGP library (within browsers or >>> not). We should write some sort of "summarized and syntetic" (@naif: >>> thanks.) >>> >>> What should we address in 2013 from your point of view? >>> >>> Best regards, Alex >>> >>> -- >>> http://openpgpjs.org >>> >>> >>> <icon_openpgpjs.png> >>> _______________________________________________ >>> >>> http://openpgpjs.org >> >> _______________________________________________ >> >> http://openpgpjs.org > > _______________________________________________ > > http://openpgpjs.org
_______________________________________________ http://openpgpjs.org
