Before I cause a panic, there is a cut/paste error in my description below. There is nothing wrong with Sparse + Compression, it should say "Works fine" not "Restore broken".
-----Original Message----- From: Kern Sibbald [mailto:[EMAIL PROTECTED] Sent: Friday, November 03, 2006 5:05 PM To: Robert Nelson Cc: [EMAIL PROTECTED]; 'Landon Fuller'; bacula-users@lists.sourceforge.net Subject: RE: [Bacula-devel] [Bacula-users] Encryption/Compression Conflict in CVS I've got to go now. Thanks for the explaination. I'll respond as soon as I can, but I would also like to see Landon's response. > Perhaps if I explained the problem: > > Currently (as of 1.39.27) > > No filters = Works fine > Sparse = Works fine > Compression = Works fine > Encryption = Works fine > > Sparse + Compression = Restore broken > Sparse + Encryption = Restore broken > Sparse + Compression + Encryption = Restore broken > Compression + Encryption = Restore broken > > With my fix (> 1.39.27): > > All combinations work fine. > > But restoration of old backups (1.39.x) using encryption alone or in > combination with other filters is broken. Since all cases of encryption > in > combination with anything else were already broken, that just leaves > encryption alone. I could special case the handling of just encryption so > that it would be stored differently than when used in combination with > other > filters. This would allow old encrypted backups to be read at the expense > of having two versions of the encryption and decryption code. > > Since this is a brand new feature I don't think changing the archive > format > at this point would affect that many people. Particularly since the > archives are unreadable if they've used it in combination with compression > or sparse file handling. > > I think it is safer to just say that previously written archives using > encryption aren't readable and should be recreated. > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Kern > Sibbald > Sent: Friday, November 03, 2006 4:40 PM > To: Robert Nelson > Cc: [EMAIL PROTECTED]; 'Landon Fuller'; > bacula-users@lists.sourceforge.net > Subject: Re: [Bacula-devel] [Bacula-users] Encryption/Compression Conflict > in CVS > > >> This code is backwards compatible for everything except encrypted data. >> Previously compressed backups will still work fine. > > I'm not 100% sure what you mean, but here are my thoughts: > > If it breaks something that previously worked, then it is does not fit > with the Bacula philosophy of always being able to read Volumes written by > prior versions. > > If something was previously broken -- i.e. could not be read -- then we > should attempt to fix it, if at all possible. > > Maintaining backward Volume compatibility and fixing the problem is the > best solution. Hopefully this is what you can do ... > > If we *must* create an incompatibility with Volumes written by prior > versions of Bacula, then we need to think really hard about how to handle > it because to the best of my knowledge this has never happened. > > If certain combinations of options created data that cannot be read under > any conditions, then we need to carefully document it and inform the > users. > >> >> -----Original Message----- >> From: [EMAIL PROTECTED] >> [mailto:[EMAIL PROTECTED] On Behalf Of Kern >> Sibbald >> Sent: Friday, November 03, 2006 4:15 PM >> To: Robert Nelson >> Cc: [EMAIL PROTECTED]; 'Landon Fuller'; >> bacula-users@lists.sourceforge.net >> Subject: Re: [Bacula-users] Encryption/Compression Conflict in CVS >> >> >>> Landon, >>> >>> I've changed the code so that the encryption code prefixes the data >>> block >>> with a block length prior to encryption. >>> >>> The decryption code accumulates data until a full data block is >>> decrypted >>> before passing it along to the decompression code. >>> >>> The code now works for all four scenarios with encryption and >>> compression: >>> none, encryption, compression, and encryption + compression. >>> Unfortunately >>> the code is no longer compatible for previously encrypted backups. >>> >>> I could add some more code to make the encryption only case work like >>> before. However, since this is a new feature in 1.39 and there >>> shouldn't >>> be >>> a lot of existing backups, I would prefer to invalidate the previous >>> backups >>> and keep the code simpler. >>> >>> Also I think we should have a design rule that says any data filters >>> like >>> encryption, compression, etc must maintain the original buffer >>> boundaries. >>> >>> This will allow us to define arbitrary, dynamically extensible filter >>> stacks >>> in the future. >>> >>> What do you think? >> >> I'm unfortuntely not in a good position to examine this problem in >> detail, >> but I suggest that we should do our best to keep the old data readable >> by >> any kludge necessary. >> >> One possible solution for the new code that you have implemented is to >> put >> the new compressed data in a new stream -- i.e. a different one from the >> old compressed data -- this could possibly allow old Volumes to be read >> and any new data written to Volumes will be written correctly. >> >> One thing to be very careful about is to make sure the length that you >> store is bigendian-littlendian independent. Probably you have already >> done >> this, but if not you need to use the serialization code that is also >> used >> for sparse file length. >> >> >>> >>> -----Original Message----- >>> From: [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] On Behalf Of Landon >>> Fuller >>> Sent: Wednesday, November 01, 2006 7:08 PM >>> To: Michael Brennen >>> Cc: bacula-users@lists.sourceforge.net >>> Subject: Re: [Bacula-users] Encryption/Compression Conflict in CVS >>> >>> >>> On Nov 1, 2006, at 2:20 PM, Michael Brennen wrote: >>> >>>> On Wednesday 01 November 2006 15:33, Arno Lehmann wrote: >>>> >>>>>>> This sounds like compression should be automatically disabled when >>>>>>> encrypton is enabled. Should be useless anyway as encrypted data >>>>>>> should >>>>>>> no longer be compressible. >>>>>> >>>>>> Not if compression happens prior to encryption. :) >>>>> >>>>> Theoretically - yes, but I'm quite sure that encryption usually also >>>>> compresses data. This is completely unverified and refers to >>>>> encryption >>>>> programs that are rather outdated by now, though... >>>>> >>>>> But I suppose you could inform us if encryption in Bacula also >>>>> compresses :-) >>>> >>>> Landon, what is your take on this? Since you wrote the code you >>>> seem to be >>>> the best source on whether the openssl functions you are using >>>> compress data. >>> >>> Howdy, >>> >>> The encryption does not include compression -- It made more sense to >>> piggyback on the existing compression code. >>> Also, thanks for catching this! I'm embarrassed that I forgot to test >>> backup+restore with both compression and encryption enabled. >>> >>> -landonf >>> >>> >>> >>> ------------------------------------------------------------------------- >>> Using Tomcat but need to do more? Need to support web services, >>> security? >>> Get stuff done quickly with pre-integrated technology to make your job >>> easier >>> Download IBM WebSphere Application Server v.1.0.1 based on Apache >>> Geronimo >>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >>> _______________________________________________ >>> Bacula-users mailing list >>> Bacula-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/bacula-users >>> >> >> >> Best regards, Kern >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your job >> easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users >> >> >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your job >> easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Bacula-devel mailing list >> [EMAIL PROTECTED] >> https://lists.sourceforge.net/lists/listinfo/bacula-devel >> > > > Best regards, Kern > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Bacula-devel mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/bacula-devel > > Best regards, Kern ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
