Re: [sqlite] Fill empty space with random
> > In the simple case, the VFS that the sqlite Db is mounted in is encrypted > with a long key. The key has cycles at 4096(A) and 16(B1-Bn) bytes > (4096/16 = 256 cycles of Bn); such that each sector is masked with > A^B1(256x), A^B2(256x), ... all together there is no repetition because the > change from Bn to B(n+1) at the 4096 boundary makes the stream overall > appear continuously random. > Only data that is written is actually masked… Apologies if I’m wrong or am missing something. This sounds like wrong usage of encryption (ECB). If the underlaying storage is encrypted properly ( maybe AES in counter mode with the counter being the block number or something), there is no way to recover a key from learning any other blocks. Thilo ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On 14-10-2018 17:07, J Decker wrote: > (sorry for the math err s/16/32/g and s/512/256/ - I double and halfed > the wrong directions.) > > On Sun, Oct 14, 2018 at 7:57 AM J Decker wrote: > >> >> On Sun, Oct 14, 2018 at 7:24 AM Luuk wrote: >> >>> On 14-10-2018 16:17, Simon Slavin wrote: On 14 Oct 2018, at 12:56pm, J Decker wrote: > Is there maybe a compile option for sqlite to fill empty space in a db >>> with random data rather than 0 ? There is not. But (A) It may be an easy change to the source code (B) Your operating system may have a setting to do this automatically >>> to freed blocks on a storage device. (C) Your device driver may have a setting to do this automatically to >>> freed blocks on the device. That type of security is normally done at OS or device level, not by >>> each individual app. Simon. >>> Can you give any hints on why it would be a security issue to fill >>> 'empty space' with 0, and why 'random data' should be used? >>> >>> ? >>> >> I hesitate to describe the real scenario; and want to instead manufacture >> one; but in either case I feel there will be more comments about the >> underlaying system than on Sqlite itself. >> >> In the simple case, the VFS that the sqlite Db is mounted in is encrypted >> with a long key. The key has cycles at 4096(A) and 16(B1-Bn) bytes >> (4096/16 = 256 cycles of Bn); such that each sector is masked with >> A^B1(256x), A^B2(256x), ... all together there is no repetition because the >> change from Bn to B(n+1) at the 4096 boundary makes the stream overall >> appear continuously random. >> Only data that is written is actually masked... >> >> Sqlite likes to write 0's in large splotches (in my usage); which leaks >> key information; (only slightly more than the data stored in tables >> typically, which is a lot of the same bytes (0, 1 for instance and A-Z, a-z >> less-so; but all of that has upper bit(s) that are 0... ) >> >> And even is a specific sector (or several) is 'cracked' it doesn't do any >> good for any other page... but if LOTS of pages are found, it becomes >> easier to find what the overall A key is, which makes finding sector keys >> that you only need a few 32-64 bytes of 0's to reveal the sector specific >> key (for later use?) >> >> The keys are a procedurally generated with a PRNG sha2 bit streams based; >> so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR >> numbers. which can be consumed as end-to-end bit streams. >> >> I might look into it; there are certainly a great test suite available to >> reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized >> (even when filled from disk) and that it will be a HUGE can of worms. >> >> >> Thanks for the explanation ... I never would have guessed that you "I double and halfedthe wrong directions. " 😊😊 ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
(sorry for the math err s/16/32/g and s/512/256/ - I double and halfed the wrong directions.) On Sun, Oct 14, 2018 at 7:57 AM J Decker wrote: > > > On Sun, Oct 14, 2018 at 7:24 AM Luuk wrote: > >> On 14-10-2018 16:17, Simon Slavin wrote: >> > On 14 Oct 2018, at 12:56pm, J Decker wrote: >> > >> >> Is there maybe a compile option for sqlite to fill empty space in a db >> with random data rather than 0 ? >> > There is not. But >> > >> > (A) It may be an easy change to the source code >> > (B) Your operating system may have a setting to do this automatically >> to freed blocks on a storage device. >> > (C) Your device driver may have a setting to do this automatically to >> freed blocks on the device. >> > >> > That type of security is normally done at OS or device level, not by >> each individual app. >> > >> > Simon. >> > >> Can you give any hints on why it would be a security issue to fill >> 'empty space' with 0, and why 'random data' should be used? >> >> ? >> > I hesitate to describe the real scenario; and want to instead manufacture > one; but in either case I feel there will be more comments about the > underlaying system than on Sqlite itself. > > In the simple case, the VFS that the sqlite Db is mounted in is encrypted > with a long key. The key has cycles at 4096(A) and 16(B1-Bn) bytes > (4096/16 = 256 cycles of Bn); such that each sector is masked with > A^B1(256x), A^B2(256x), ... all together there is no repetition because the > change from Bn to B(n+1) at the 4096 boundary makes the stream overall > appear continuously random. > Only data that is written is actually masked... > > Sqlite likes to write 0's in large splotches (in my usage); which leaks > key information; (only slightly more than the data stored in tables > typically, which is a lot of the same bytes (0, 1 for instance and A-Z, a-z > less-so; but all of that has upper bit(s) that are 0... ) > > And even is a specific sector (or several) is 'cracked' it doesn't do any > good for any other page... but if LOTS of pages are found, it becomes > easier to find what the overall A key is, which makes finding sector keys > that you only need a few 32-64 bytes of 0's to reveal the sector specific > key (for later use?) > > The keys are a procedurally generated with a PRNG sha2 bit streams based; > so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR > numbers. which can be consumed as end-to-end bit streams. > > I might look into it; there are certainly a great test suite available to > reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized > (even when filled from disk) and that it will be a HUGE can of worms. > > >> >> ___ >> sqlite-users mailing list >> sqlite-users@mailinglists.sqlite.org >> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >> > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On 14 Oct 2018, at 3:24pm, J Decker wrote: > B and C would apply if there was a vacuum also; adding data, and deleteting > data, the db ends up with lots of zeros > Also between non-integral pages; messages that are say 700 bytes; so 4096% > 700 is 596; which is all filled with zeros... True. And note the non-intuitive usage patterns of solid-state storage devices, where changing one byte of a 'sector' can result in the old 'sector' being freed and a new one used. Changes to SQLite (or anything else at app level) cannot help with this sort of thing. It comes down to why you're concerned about security, and whether you're concerned that someone may physically steal your hardware, or whether you're just sharing a virtual machine host with a possibly-hostile organisation. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On Sun, Oct 14, 2018 at 7:24 AM Luuk wrote: > On 14-10-2018 16:17, Simon Slavin wrote: > > On 14 Oct 2018, at 12:56pm, J Decker wrote: > > > >> Is there maybe a compile option for sqlite to fill empty space in a db > with random data rather than 0 ? > > There is not. But > > > > (A) It may be an easy change to the source code > > (B) Your operating system may have a setting to do this automatically to > freed blocks on a storage device. > > (C) Your device driver may have a setting to do this automatically to > freed blocks on the device. > > > > That type of security is normally done at OS or device level, not by > each individual app. > > > > Simon. > > > Can you give any hints on why it would be a security issue to fill > 'empty space' with 0, and why 'random data' should be used? > > ? > I hesitate to describe the real scenario; and want to instead manufacture one; but in either case I feel there will be more comments about the underlaying system than on Sqlite itself. In the simple case, the VFS that the sqlite Db is mounted in is encrypted with a long key. The key has cycles at 4096(A) and 16(B1-Bn) bytes (4096/16 = 256 cycles of Bn); such that each sector is masked with A^B1(256x), A^B2(256x), ... all together there is no repetition because the change from Bn to B(n+1) at the 4096 boundary makes the stream overall appear continuously random. Only data that is written is actually masked... Sqlite likes to write 0's in large splotches (in my usage); which leaks key information; (only slightly more than the data stored in tables typically, which is a lot of the same bytes (0, 1 for instance and A-Z, a-z less-so; but all of that has upper bit(s) that are 0... ) And even is a specific sector (or several) is 'cracked' it doesn't do any good for any other page... but if LOTS of pages are found, it becomes easier to find what the overall A key is, which makes finding sector keys that you only need a few 32-64 bytes of 0's to reveal the sector specific key (for later use?) The keys are a procedurally generated with a PRNG sha2 bit streams based; so 512 bits (16 bytes) at a time; and sha algorithms generates VERY good PR numbers. which can be consumed as end-to-end bit streams. I might look into it; there are certainly a great test suite available to reveal issues; but I expect Sqlite 'expects' memory to be 0 initialized (even when filled from disk) and that it will be a HUGE can of worms. > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On 14-10-2018 16:17, Simon Slavin wrote: > On 14 Oct 2018, at 12:56pm, J Decker wrote: > >> Is there maybe a compile option for sqlite to fill empty space in a db with >> random data rather than 0 ? > There is not. But > > (A) It may be an easy change to the source code > (B) Your operating system may have a setting to do this automatically to > freed blocks on a storage device. > (C) Your device driver may have a setting to do this automatically to freed > blocks on the device. > > That type of security is normally done at OS or device level, not by each > individual app. > > Simon. > Can you give any hints on why it would be a security issue to fill 'empty space' with 0, and why 'random data' should be used? ? ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On Sun, Oct 14, 2018 at 7:17 AM Simon Slavin wrote: > On 14 Oct 2018, at 12:56pm, J Decker wrote: > > > Is there maybe a compile option for sqlite to fill empty space in a db > with random data rather than 0 ? > > There is not. But > > (A) It may be an easy change to the source code > Sure; figured I'd toss out the question to see if there was at least a springboard > (B) Your operating system may have a setting to do this automatically to > freed blocks on a storage device. > (C) Your device driver may have a setting to do this automatically to > freed blocks on the device. > > B and C would apply if there was a vacuum also; adding data, and deleteting data, the db ends up with lots of zeros Also between non-integral pages; messages that are say 700 bytes; so 4096% 700 is 596; which is all filled with zeros... if I end up with messages that are say 2200 bytes; nearly half the page is blank for a while. > That type of security is normally done at OS or device level, not by each > individual app. > > Simon. > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On 14 Oct 2018, at 12:56pm, J Decker wrote: > Is there maybe a compile option for sqlite to fill empty space in a db with > random data rather than 0 ? There is not. But (A) It may be an easy change to the source code (B) Your operating system may have a setting to do this automatically to freed blocks on a storage device. (C) Your device driver may have a setting to do this automatically to freed blocks on the device. That type of security is normally done at OS or device level, not by each individual app. Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On Sun, Oct 14, 2018 at 5:48 AM Bob Gailer wrote: > On Oct 14, 2018 7:57 AM, "J Decker" wrote: > > > > Is there maybe a compile option for sqlite to fill empty space in a db > with > > random data rather than 0 ? > > I don't know about compile options. > > You could do a 1 time update to set all 0 columns to random(1,99), and > create a trigger to do the same on insert. > > This, of course is on a table-by-table basis. > Doesn't work for all the empty space outside of the table; between tables; not unused pages from deleted records, > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Fill empty space with random
On Oct 14, 2018 7:57 AM, "J Decker" wrote: > > Is there maybe a compile option for sqlite to fill empty space in a db with > random data rather than 0 ? I don't know about compile options. You could do a 1 time update to set all 0 columns to random(1,99), and create a trigger to do the same on insert. This, of course is on a table-by-table basis. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Fill empty space with random
Is there maybe a compile option for sqlite to fill empty space in a db with random data rather than 0 ? ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
