Hi Jarrod, > I need to encrypt and decrypt large files (up to a few GB) so I need > to do the encryption/decryption one chunk at a time (128KB) to keep > memory utilization low. CTR mode will probably be easier to use for throttling. Otherwise, I *think* you need to call GetNextIV() from the previous block of 128 KB so that you can feed it to your next block of 128KB.
> byte* plainbuffer = NULL; > byte* cypherbuffer = NULL; > ... > plainbuffer = new byte[CBufferSize]; A c++ string may simplify this for you. No buffer management, and Crypto++ works with it natively. Remember to call clear() on the string between invocations on the 128KB chunk. Jeff On 5/28/09, wiired <[email protected]> wrote: > > I need to encrypt and decrypt large files (up to a few GB) so I need > to do the encryption/decryption one chunk at a time (128KB) to keep > memory utilization low. > > Processing in chunks _and_ using padding appears to be more complex > than I thought. Here is the (slightly simplified) technique I'm using. > LSource and LDest are the input and output files read from and written > to directly. > > Is there a more simple solution that I'm missing? > > Also, is there much benefit in using a random iv? Doesn't that mean > that it would need to be stored with the file and therefore little > more secure than using a fixed value? > > const int CBufferSize = Blowfish::BLOCKSIZE * 16384; // 128KB > > Encryption: > > byte* plainbuffer = NULL; > byte* cypherbuffer = NULL; > try { > plainbuffer = new byte[CBufferSize]; > // Allow for padding in encrypted buffer - up to an extra block > size is added. > cypherbuffer = new byte[CBufferSize + Blowfish::BLOCKSIZE]; > > // Initialise > byte iv[Blowfish::BLOCKSIZE] = { ... }; // 8 bytes > Blowfish::Encryption blowfishEncryption(AKey, AKeyLen); > CBC_Mode_ExternalCipher::Encryption cbcEncryption > (blowfishEncryption, iv); > > int LBytesLeft; > int LBytesProcessed; > while ((LBytesLeft = LSource->Size - LSource->Position) > 0) { > LBytesProcessed = LSource->Read(plainbuffer, CBufferSize); > > // If at end use a StreamTransformationFilter to handle > padding > if (LBytesLeft <= CBufferSize) { > StreamTransformationFilter cbcEncryptor(cbcEncryption); > cbcEncryptor.Put(plainbuffer, LBytesProcessed); > cbcEncryptor.MessageEnd(); > LBytesProcessed = cbcEncryptor.MaxRetrievable(); > LBytesProcessed = cbcEncryptor.Get(cypherbuffer, > LBytesProcessed); > } else { > cbcEncryption.ProcessData(cypherbuffer, plainbuffer, > LBytesProcessed); > } > > LDest->Write(cypherbuffer, LBytesProcessed); > } > } > __finally { > delete[] cypherbuffer; > delete[] plainbuffer; > } > > Decryption: > > byte* plainbuffer = NULL; > byte* cypherbuffer = NULL; > try { > plainbuffer = new byte[CBufferSize]; > // Allow read of padding in encrypted buffer - up to an extra > block size is added. > cypherbuffer = new byte[CBufferSize + Blowfish::BLOCKSIZE]; > > // Initialise > byte iv[Blowfish::BLOCKSIZE] = { ... }; // 8 bytes > Blowfish::Decryption blowfishDecryption(AKey, AKeyLen); > CBC_Mode_ExternalCipher::Decryption cbcDecryption > (blowfishDecryption, iv); > > int LBytesLeft; > int LBytesProcessed; > while ((LBytesLeft = LSource->Size - LSource->Position) > 0) { > // If at end use a StreamTransformationFilter to handle > padding. Not sure > // if StreamTransformationFilter handles reading just the > padding block at the end > // so group it with the last data blocks. > if (LBytesLeft <= CBufferSize + Blowfish::BLOCKSIZE) { > LBytesProcessed = LSource->Read(cypherbuffer, CBufferSize + > Blowfish::BLOCKSIZE); > StreamTransformationFilter cbcDecryptor(cbcDecryption); > cbcDecryptor.Put(cypherbuffer, LBytesProcessed); > cbcDecryptor.MessageEnd(); > LBytesProcessed = cbcDecryptor.MaxRetrievable(); > LBytesProcessed = cbcDecryptor.Get(plainbuffer, > LBytesProcessed); > } else { > LBytesProcessed = LSource->Read(cypherbuffer, CBufferSize); > cbcDecryption.ProcessData(plainbuffer, cypherbuffer, > LBytesProcessed); > } > > LDest->Write(plainbuffer, LBytesProcessed); > } > } > __finally { > delete[] cypherbuffer; > delete[] plainbuffer; > } > > Now this seems to work ok but the conditional code to detect the end > of the input buffer and then use a StreamTransformationFilter seems > like a hack. It would be nice to be able to reuse a single instance of > StreamTransformationFilter and reuse it to process every chunk (Put > followed by a Get) but with padding only at the end of the entire data > (not at the end of each chunk), and to be able to reset its internal > buffers for each chunk so that it doesn't build up the entire data in > its buffers. > > Regards, > > Jarrod Hollingworth > Complete Time Tracking > http://www.complete-time-tracking.com/ > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the "Crypto++ Users" Google Group. To unsubscribe, send an email to [email protected]. More information about Crypto++ and this group is available at http://www.cryptopp.com. -~----------~----~----~----~------~----~------~--~---
