Perhaps if I explained the problem: Currently (as of 1.39.27)
No filters = Works fine Sparse = Works fine Compression = Works fine Encryption = Works fine Sparse + Compression = Restore broken Sparse + Encryption = Restore broken Sparse + Compression + Encryption = Restore broken Compression + Encryption = Restore broken With my fix (> 1.39.27): All combinations work fine. But restoration of old backups (1.39.x) using encryption alone or in combination with other filters is broken. Since all cases of encryption in combination with anything else were already broken, that just leaves encryption alone. I could special case the handling of just encryption so that it would be stored differently than when used in combination with other filters. This would allow old encrypted backups to be read at the expense of having two versions of the encryption and decryption code. Since this is a brand new feature I don't think changing the archive format at this point would affect that many people. Particularly since the archives are unreadable if they've used it in combination with compression or sparse file handling. I think it is safer to just say that previously written archives using encryption aren't readable and should be recreated. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kern Sibbald Sent: Friday, November 03, 2006 4:40 PM To: Robert Nelson Cc: [EMAIL PROTECTED]; 'Landon Fuller'; bacula-users@lists.sourceforge.net Subject: Re: [Bacula-devel] [Bacula-users] Encryption/Compression Conflict in CVS > This code is backwards compatible for everything except encrypted data. > Previously compressed backups will still work fine. I'm not 100% sure what you mean, but here are my thoughts: If it breaks something that previously worked, then it is does not fit with the Bacula philosophy of always being able to read Volumes written by prior versions. If something was previously broken -- i.e. could not be read -- then we should attempt to fix it, if at all possible. Maintaining backward Volume compatibility and fixing the problem is the best solution. Hopefully this is what you can do ... If we *must* create an incompatibility with Volumes written by prior versions of Bacula, then we need to think really hard about how to handle it because to the best of my knowledge this has never happened. If certain combinations of options created data that cannot be read under any conditions, then we need to carefully document it and inform the users. > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Kern > Sibbald > Sent: Friday, November 03, 2006 4:15 PM > To: Robert Nelson > Cc: [EMAIL PROTECTED]; 'Landon Fuller'; > bacula-users@lists.sourceforge.net > Subject: Re: [Bacula-users] Encryption/Compression Conflict in CVS > > >> Landon, >> >> I've changed the code so that the encryption code prefixes the data >> block >> with a block length prior to encryption. >> >> The decryption code accumulates data until a full data block is >> decrypted >> before passing it along to the decompression code. >> >> The code now works for all four scenarios with encryption and >> compression: >> none, encryption, compression, and encryption + compression. >> Unfortunately >> the code is no longer compatible for previously encrypted backups. >> >> I could add some more code to make the encryption only case work like >> before. However, since this is a new feature in 1.39 and there >> shouldn't >> be >> a lot of existing backups, I would prefer to invalidate the previous >> backups >> and keep the code simpler. >> >> Also I think we should have a design rule that says any data filters >> like >> encryption, compression, etc must maintain the original buffer >> boundaries. >> >> This will allow us to define arbitrary, dynamically extensible filter >> stacks >> in the future. >> >> What do you think? > > I'm unfortuntely not in a good position to examine this problem in detail, > but I suggest that we should do our best to keep the old data readable by > any kludge necessary. > > One possible solution for the new code that you have implemented is to put > the new compressed data in a new stream -- i.e. a different one from the > old compressed data -- this could possibly allow old Volumes to be read > and any new data written to Volumes will be written correctly. > > One thing to be very careful about is to make sure the length that you > store is bigendian-littlendian independent. Probably you have already done > this, but if not you need to use the serialization code that is also used > for sparse file length. > > >> >> -----Original Message----- >> From: [EMAIL PROTECTED] >> [mailto:[EMAIL PROTECTED] On Behalf Of Landon >> Fuller >> Sent: Wednesday, November 01, 2006 7:08 PM >> To: Michael Brennen >> Cc: bacula-users@lists.sourceforge.net >> Subject: Re: [Bacula-users] Encryption/Compression Conflict in CVS >> >> >> On Nov 1, 2006, at 2:20 PM, Michael Brennen wrote: >> >>> On Wednesday 01 November 2006 15:33, Arno Lehmann wrote: >>> >>>>>> This sounds like compression should be automatically disabled when >>>>>> encrypton is enabled. Should be useless anyway as encrypted data >>>>>> should >>>>>> no longer be compressible. >>>>> >>>>> Not if compression happens prior to encryption. :) >>>> >>>> Theoretically - yes, but I'm quite sure that encryption usually also >>>> compresses data. This is completely unverified and refers to >>>> encryption >>>> programs that are rather outdated by now, though... >>>> >>>> But I suppose you could inform us if encryption in Bacula also >>>> compresses :-) >>> >>> Landon, what is your take on this? Since you wrote the code you >>> seem to be >>> the best source on whether the openssl functions you are using >>> compress data. >> >> Howdy, >> >> The encryption does not include compression -- It made more sense to >> piggyback on the existing compression code. >> Also, thanks for catching this! I'm embarrassed that I forgot to test >> backup+restore with both compression and encryption enabled. >> >> -landonf >> >> >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your job >> easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users >> > > > Best regards, Kern > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Bacula-devel mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/bacula-devel > Best regards, Kern ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Bacula-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/bacula-devel ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
