The basic intention of my patch would be to ensure that unencrypted documents and attachments are not on disk. You're right that there are many other questions to answer for a general encryption feature.
For my exact case, which might always remain a local patch, I wouldn't mind the data passing unencrypted to and from the boxes, but should be encrypted while "at rest" on disk. B. On Tue, Jan 26, 2010 at 2:02 PM, Filipe David Manana <[email protected]> wrote: > Robert, > > I think your plans are very interesting, as they present not only > interesting challenges but the feature itself I find useful also. > Questions such as: > - what kind of encryption? (symmetric, asymmetric, or both) > - where are the keys stored? Or for the symmetric case, we would use a > diffie-helman protocol for e.g.? > - is the objective to have privacy at the DB storage level or also at the > network level? (and force the decryption on the client side only) > there are many more details of course. > I do have the same opinion as you, the code would affect many of the parts > regarding the compression (specially couch_stream). For doc compression, I > imagine it would touch more places, and also present some difficulties to > assure compatibility with the previous DB file formats. > Let me know if somehow I can help you. > cheers > > On Tue, Jan 26, 2010 at 2:51 PM, Robert Newson <[email protected]> > wrote: >> >> that was my intention, but the option to send the encrypted bytes (for >> decryption at the client end) is intriguing and also echoes the choice >> to send compressed vs uncompressed responses. >> >> I don't mean to hold up this work and I doubt I'll have a patch any >> time soon, it just seems that these two features have significant >> overlap (you can send data in with a transformation applied or not, >> and request it with or without that transformation). >> >> My brief look at the related code led me to believe that adding >> encryption support would touch several places, and I would think that >> most, perhaps all, of those places would also be touched by >> compression support. >> >> Sorry to be vague, I only intended to add another perspective to the >> discussion. >> >> On Tue, Jan 26, 2010 at 11:01 AM, Filipe David Manana >> <[email protected]> wrote: >> > Hi Robert, >> > >> > That's interesting. >> > I think that abstraction is doable, but maybe not trivial. >> > >> > In your idea, you plan to always decrypt the docs/attachments before >> > sending >> > them to the client? >> > >> > >> > On Tue, Jan 26, 2010 at 11:34 AM, Robert Newson >> > <[email protected]>wrote: >> > >> >> fyi: I have a (much delayed) plan to work up an encryption patch for >> >> documents and attachments. Since encryption and compression both apply >> >> at the same level (and the order of the two is important), I wonder if >> >> that would change the approach taken here? That is, would an >> >> abstraction that allowed a chain of transformations when storing (and >> >> the reverse chain when retrieving) be in order? >> >> >> >> B. >> >> >> >> On Tue, Jan 26, 2010 at 8:02 AM, Filipe Manana (JIRA) <[email protected]> >> >> wrote: >> >> > >> >> > [ >> >> >> >> https://issues.apache.org/jira/browse/COUCHDB-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804946#action_12804946] >> >> > >> >> > Filipe Manana commented on COUCHDB-583: >> >> > --------------------------------------- >> >> > >> >> > @Chris >> >> > >> >> > Good point, I totally agree. >> >> > It would be interesting to test with real couchapps, real data and >> >> > see >> >> how worth it really is. >> >> > >> >> > A 10Mb text file, for instance, was compressed to about 100Kb in one >> >> > of >> >> my tests. >> >> > >> >> > Also, as for the minified JavaScript files for example, it's still >> >> > worth >> >> compressing them. For example, the minified Ext JS lib file ( >> >> http://www.extjs.com, ext-all.js) is about 630Kb big. Compressed with >> >> gzip stays at about 170Kb, therefore a reasonably good size reduction. >> >> > >> >> > As Damien said in a previous comment, not only saves disk space but >> >> > also >> >> reduces disk IO (attachment download requests, compaction). >> >> > >> >> > I also look forward to see the impact on real, production level, >> >> applications. >> >> > >> >> >> storing attachments in compressed form and serving them in >> >> >> compressed >> >> form if accepted by the client >> >> >> >> >> >> >> ---------------------------------------------------------------------------------------------------- >> >> >> >> >> >> Key: COUCHDB-583 >> >> >> URL: >> >> >> https://issues.apache.org/jira/browse/COUCHDB-583 >> >> >> Project: CouchDB >> >> >> Issue Type: New Feature >> >> >> Components: Database Core, HTTP Interface >> >> >> Environment: CouchDB trunk >> >> >> Reporter: Filipe Manana >> >> >> Attachments: couchdb-583-trunk-10th-try.patch, >> >> couchdb-583-trunk-11th-try.patch, couchdb-583-trunk-12th-try.patch, >> >> couchdb-583-trunk-13th-try.patch, couchdb-583-trunk-14th-try-git.patch, >> >> couchdb-583-trunk-15th-try-git.patch, couchdb-583-trunk-3rd-try.patch, >> >> couchdb-583-trunk-4th-try-trunk.patch, couchdb-583-trunk-5th-try.patch, >> >> couchdb-583-trunk-6th-try.patch, couchdb-583-trunk-7th-try.patch, >> >> couchdb-583-trunk-8th-try.patch, couchdb-583-trunk-9th-try.patch, >> >> jira-couchdb-583-1st-try-trunk.patch, >> >> jira-couchdb-583-2nd-try-trunk.patch >> >> >> >> >> >> >> >> >> This feature allows Couch to gzip compress attachments as they are >> >> >> being >> >> received and store them in compressed form. >> >> >> When a client asks for downloading an attachment (e.g. GET >> >> somedb/somedoc/attachment.txt), the attachment is sent in compressed >> >> form if >> >> the client's http request has gzip specified as a valid transfer >> >> encoding >> >> for the response (using the http header "Accept-Encoding"). Otherwise >> >> couch >> >> decompresses the attachment before sending it back to the client. >> >> >> Attachments are compressed only if their MIME type matches one of >> >> >> those >> >> listed in a separate config file. Compression level is also >> >> configurable in >> >> the default.ini file. >> >> >> This follows Damien's suggestion from 30 November: >> >> >> "Perhaps we need a separate user editable ini file to specify >> >> compressable or non-compressable files (would probably be too big for >> >> the >> >> regular ini file). What do other web servers do? >> >> >> Also, a potential optimization is to compress the file while writing >> >> >> to >> >> disk, and serve the compressed bytes directly to clients that can >> >> handle it, >> >> and decompressed for those that can't. For compressable types, it's a >> >> win >> >> for both disk IO for reads and writes, and CPU on read." >> >> >> Patch attached. >> >> > >> >> > -- >> >> > This message is automatically generated by JIRA. >> >> > - >> >> > You can reply to this email to add a comment to the issue online. >> >> > >> >> > >> >> >> > >> > >> > >> > -- >> > Filipe David Manana, >> > [email protected] >> > PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B >> > >> > "Reasonable men adapt themselves to the world. >> > Unreasonable men adapt the world to themselves. >> > That's why all progress depends on unreasonable men." >> > > > > > -- > Filipe David Manana, > [email protected] > PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B > > "Reasonable men adapt themselves to the world. > Unreasonable men adapt the world to themselves. > That's why all progress depends on unreasonable men." > >
