Re: pre-screening new package: SQLCipher, based on SQLite3
On Oct 12, 2012, at 9:03 PM, Hans-Christoph Steiner wrote: On Oct 1, 2012, at 7:36 PM, Hans-Christoph Steiner wrote: On 10/01/2012 06:32 PM, Stephen Lombardo wrote: Hello Florian, On Mon, Oct 1, 2012 at 1:57 PM, Florian Weimer f...@deneb.enyo.de wrote: Okay. Can your fork open unencrypted databases? Are there any symbol collisions with vanilla SQLite? Yes, SQLCipher can open standard, unencrypted SQLite databases without a problem and it has the same public API and symbols as vanilla SQLite. Many users take advantage of this today to allow substitution in languages like python and ruby. For example they may build SQLCipher in a separate directory then set LD_LIBRARY_PATH to load the SQLCipher-enhanced library for use in their programs. I believe this is is the reason Hans opted to alter the library name to libsqlcipher, to ensure there wouldn't be any confusion between the two, but I'll let him comment on that further. Cheers, Stephen SQLCipher shares all of the public symbols that SQLite3 has, plus a few more related to the encryption. It is a lightly modified version of SQLite3. Like Stephen said, its possible to dynamically load the SQLCipher library for an app that was compiled against SQLite3 and it'll work. Also, when not using encryption, SQLCipher works the same as SQLite3 in terms of SQL commands, C API, etc. To make it friendly to packaging, I've made the library called libsqlcipher.so with its own ABI version related to the SQLCipher version, then the headers are in /usr/include/sqlcipher so they don't get easily confused for the SQLite3 headers. Are you thinking that this package should replace the SQLite3 package in Debian? I suppose that is a possibility. I'm guessing that the headers would need to be split up. Right now the SQLCipher symbols are just in sqlite3.h, and that header is in /usr/include/sqlcipher. I suppose sqlite3.h could be untouched, then the SQLCipher symbols could go into /usr/include/sqlcipher.h or something like that. The question remaining there would be how to represent the ABI versions. So far, there is no established ABI versioning scheme since SQLCipher has mostly been used on iOS and Android. So that's the good news. The bad news is that SQLCipher 1.19, 2.0 thru 2.0.3 are all based on SQLite 3.7.9. In other words, SQLCipher's release cycle is not necessarily in sync with SQLite3's release cycle. We discussed this a bunch in early October and then the discussion died. I just wanted to check in to see if anyone has any time to green light this packaging approach, or nail down a different way. The discussion ended with the above: me asking for more information on an idea of whether the SQLCipher package could provide SQLite. .hc -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4a222c87-cfab-4be9-8e23-e2cdb2c60...@eds.org
Re: pre-screening new package: SQLCipher, based on SQLite3
On Oct 1, 2012, at 7:36 PM, Hans-Christoph Steiner wrote: On 10/01/2012 06:32 PM, Stephen Lombardo wrote: Hello Florian, On Mon, Oct 1, 2012 at 1:57 PM, Florian Weimer f...@deneb.enyo.de wrote: Okay. Can your fork open unencrypted databases? Are there any symbol collisions with vanilla SQLite? Yes, SQLCipher can open standard, unencrypted SQLite databases without a problem and it has the same public API and symbols as vanilla SQLite. Many users take advantage of this today to allow substitution in languages like python and ruby. For example they may build SQLCipher in a separate directory then set LD_LIBRARY_PATH to load the SQLCipher-enhanced library for use in their programs. I believe this is is the reason Hans opted to alter the library name to libsqlcipher, to ensure there wouldn't be any confusion between the two, but I'll let him comment on that further. Cheers, Stephen SQLCipher shares all of the public symbols that SQLite3 has, plus a few more related to the encryption. It is a lightly modified version of SQLite3. Like Stephen said, its possible to dynamically load the SQLCipher library for an app that was compiled against SQLite3 and it'll work. Also, when not using encryption, SQLCipher works the same as SQLite3 in terms of SQL commands, C API, etc. To make it friendly to packaging, I've made the library called libsqlcipher.so with its own ABI version related to the SQLCipher version, then the headers are in /usr/include/sqlcipher so they don't get easily confused for the SQLite3 headers. Are you thinking that this package should replace the SQLite3 package in Debian? I suppose that is a possibility. I'm guessing that the headers would need to be split up. Right now the SQLCipher symbols are just in sqlite3.h, and that header is in /usr/include/sqlcipher. I suppose sqlite3.h could be untouched, then the SQLCipher symbols could go into /usr/include/sqlcipher.h or something like that. The question remaining there would be how to represent the ABI versions. So far, there is no established ABI versioning scheme since SQLCipher has mostly been used on iOS and Android. So that's the good news. The bad news is that SQLCipher 1.19, 2.0 thru 2.0.3 are all based on SQLite 3.7.9. In other words, SQLCipher's release cycle is not necessarily in sync with SQLite3's release cycle. Since it been 11 days without a reply since Stephen and I replied, I just wanted to check in on this. We're both currently available to work directly on further concerns, Stephen as upstream and me as packager, so it would be a great time for us to nail this stuff down. .hc -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/92d0a24d-1de1-4ddb-9b01-10048c811...@at.or.at
Re: pre-screening new package: SQLCipher, based on SQLite3
On 09/28/2012 04:23 PM, Florian Weimer wrote: * Hans-Christoph Steiner: The tricky part is that it is a modified version of SQLite3, and lintian properly gives an error about that. But because of the features that SQLCipher provides, it must modify the core of SQLite to work, therefore it cannot be made into a plugin. Why isn't it implemented as a VFS plugin? I don't know of any other precedent for a VFS plugin for adding encryption support, do you? Maybe Stephen can speak more to that. .hc -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/5069c924.3050...@eds.org
Re: pre-screening new package: SQLCipher, based on SQLite3
* Stephen Lombardo: I agree that implementing SQLCipher using a VFS plugin would work, and we've considered it in the past. However, we've decided to stick with the codec approach for now, given that some functionality could prove more complex to implement and a major shift / rewrite could introduce challenges for the project. Okay. Can your fork open unencrypted databases? Are there any symbol collisions with vanilla SQLite? -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87haqers77@mid.deneb.enyo.de
Re: pre-screening new package: SQLCipher, based on SQLite3
Hello Hans, On Mon, Oct 1, 2012 at 12:47 PM, Hans-Christoph Steiner h...@eds.orgwrote: On 09/28/2012 04:23 PM, Florian Weimer wrote: Why isn't it implemented as a VFS plugin? I don't know of any other precedent for a VFS plugin for adding encryption support, do you? Maybe Stephen can speak more to that. Yes, there have been a number of discussions on the SQLite list in the past few years about how to implement database encryption. Many posts indicate that people are building their own schemes using the VFS API. That said, in almost every case the advice to users who post these questions on the list is to buy the commercially licensed SQLite Encryption Extension (SEE) from the authors of SQLite. It uses the codec API rather than VFS, which is the same approach SQLCipher uses. Incidentally, there are also some potential implementation complexities involved with a VFS shim that may not affect trivial examples but that I suspect could become complex for a full-featured implementation (e.g. handling of attached databases, file conversion, safe rekey, offsets / paging to account for IV, MAC, database salt, etc). Also, after posting my previous reply, I realized that my message was blocked because I wasn't subscribed to the debian-security list. I've reproduced my original comments about SQLCipher's design here so it is maintained in the list archives of this conversation. My sincere apologies to those who receive this twice. SQLCipher was initially conceived as an open source alternative to the SEE. Since SQLite includes codec hooks and API functions (sqlite3_key, pragma key, etc) for SEE and CEROD support, we chose to use the same approach. The codec API is narrower in scope than VFS, which has the benefit of reducing OS-specific concerns. Finally, some of the original design and proof of concept code for SQLCipher pre-dated a stable VFS API. I agree that implementing SQLCipher using a VFS plugin would work, and we've considered it in the past. However, we've decided to stick with the codec approach for now, given that some functionality could prove more complex to implement and a major shift / rewrite could introduce challenges for the project. Cheers, Stephen
Re: pre-screening new package: SQLCipher, based on SQLite3
Hello Florian, On Mon, Oct 1, 2012 at 1:57 PM, Florian Weimer f...@deneb.enyo.de wrote: Okay. Can your fork open unencrypted databases? Are there any symbol collisions with vanilla SQLite? Yes, SQLCipher can open standard, unencrypted SQLite databases without a problem and it has the same public API and symbols as vanilla SQLite. Many users take advantage of this today to allow substitution in languages like python and ruby. For example they may build SQLCipher in a separate directory then set LD_LIBRARY_PATH to load the SQLCipher-enhanced library for use in their programs. I believe this is is the reason Hans opted to alter the library name to libsqlcipher, to ensure there wouldn't be any confusion between the two, but I'll let him comment on that further. Cheers, Stephen
Re: pre-screening new package: SQLCipher, based on SQLite3
On 10/01/2012 06:32 PM, Stephen Lombardo wrote: Hello Florian, On Mon, Oct 1, 2012 at 1:57 PM, Florian Weimer f...@deneb.enyo.de wrote: Okay. Can your fork open unencrypted databases? Are there any symbol collisions with vanilla SQLite? Yes, SQLCipher can open standard, unencrypted SQLite databases without a problem and it has the same public API and symbols as vanilla SQLite. Many users take advantage of this today to allow substitution in languages like python and ruby. For example they may build SQLCipher in a separate directory then set LD_LIBRARY_PATH to load the SQLCipher-enhanced library for use in their programs. I believe this is is the reason Hans opted to alter the library name to libsqlcipher, to ensure there wouldn't be any confusion between the two, but I'll let him comment on that further. Cheers, Stephen SQLCipher shares all of the public symbols that SQLite3 has, plus a few more related to the encryption. It is a lightly modified version of SQLite3. Like Stephen said, its possible to dynamically load the SQLCipher library for an app that was compiled against SQLite3 and it'll work. Also, when not using encryption, SQLCipher works the same as SQLite3 in terms of SQL commands, C API, etc. To make it friendly to packaging, I've made the library called libsqlcipher.so with its own ABI version related to the SQLCipher version, then the headers are in /usr/include/sqlcipher so they don't get easily confused for the SQLite3 headers. Are you thinking that this package should replace the SQLite3 package in Debian? I suppose that is a possibility. I'm guessing that the headers would need to be split up. Right now the SQLCipher symbols are just in sqlite3.h, and that header is in /usr/include/sqlcipher. I suppose sqlite3.h could be untouched, then the SQLCipher symbols could go into /usr/include/sqlcipher.h or something like that. The question remaining there would be how to represent the ABI versions. So far, there is no established ABI versioning scheme since SQLCipher has mostly been used on iOS and Android. So that's the good news. The bad news is that SQLCipher 1.19, 2.0 thru 2.0.3 are all based on SQLite 3.7.9. In other words, SQLCipher's release cycle is not necessarily in sync with SQLite3's release cycle. .hc -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/506a290e.2040...@eds.org
Re: pre-screening new package: SQLCipher, based on SQLite3
* Hans-Christoph Steiner: The tricky part is that it is a modified version of SQLite3, and lintian properly gives an error about that. But because of the features that SQLCipher provides, it must modify the core of SQLite to work, therefore it cannot be made into a plugin. Why isn't it implemented as a VFS plugin? -- To UNSUBSCRIBE, email to debian-security-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fw6128xx@mid.deneb.enyo.de
pre-screening new package: SQLCipher, based on SQLite3
Hey all, I'm reading to upload a new package called SQLCipher (http://sqlcipher.net/) and I want to run it by y'all first. The upside is that it provides AES256 encrypted SQLite databases in a DFSG-free package that has been pretty widely tested, deployed and audited. You can find out more here: http://sqlcipher.net/design The tricky part is that it is a modified version of SQLite3, and lintian properly gives an error about that. But because of the features that SQLCipher provides, it must modify the core of SQLite to work, therefore it cannot be made into a plugin. The sqlcipher package is based off of the sqlite3 package and includes most of its patches. Hurd and Tcl support are not there yet, so those are the omitted patches. SQLCipher is currrently based off of SQLite3 3.7.12, so its reasonably current (squeeze's SQLite3 is 3.7.3, wheezy's is 3.7.13). Here's the full reasoning from the original author of SQLCipher: On 09/27/2012 03:52 PM, Stephen Lombardo wrote: SQLCipher modifies SQLite itself, which is the reason we maintain a separate version of the full source tree. However, we do try very hard to minimize alterations to core SQLite code, especially to reduce the risk of things breaking when we merge in upstream changes. Unfortunately it would be exceedingly difficult / impossible to make SQLCipher a plugin to an unmodified SQLite for a number of reasons. 1. Enabling the codec system requires the compile-time definition of SQLITE_HAS_CODEC, which is not present on standard, unmodified SQLite builds. 2. Even when enabled, SQLite isn't setup to load codecs as plugins. While SQLite does have a plugin function for loadable modules, this doesn't extend to any system internals (it mainly used to allow custom user functions). 3. SQLCipher makes calls to internal functions that are not part of the public SQLite API. Sometimes these APIs change even in between minor SQLite versions, and thus require inspection, testing and verification on each update. Making SQLCipher portable across multiple versions of SQLite would not be feasible, nor could we easily force it to use only the public API (for instance, even the first critical step of attaching the codec callback to the pager uses an internal API). 4. SQLCipher modifies supporting functions to introduce special pragmas, built in functions, etc (e.g. PRAGMA cipher_*). Injecting this functionality in a plugin architecture wouldn't be possible, resulting in major API changes. 5. SQLCipher's test harness needs to be built into libsqlite3 testfixture to take advantage of the test API and various internal checks (memory reference counting, etc) 6. Even if it were possible to use a loadable plugin, dynamic libraries aren't available on all supported platforms (e.g. iOS) It is definitely undesirable to maintain a separate parallel codebase for SQLCipher, but unfortunately I don't think there is a better way at this time. Cheers, Stephen signature.asc Description: OpenPGP digital signature
