Re: [sqlite] compressed sqlite3 database file?

2019-04-13 Thread James K. Lowden
On Thu, 11 Apr 2019 18:45:01 -0600 Warren Young wrote: > Sure, but what *is* on the disk after a crash is always consistent > with ZFS, so any decent database engine can recover. It's been some years, but I saw a presentation about running Postgres on ZFS. Every "victory" was a way to

Re: [sqlite] compressed sqlite3 database file?

2019-04-11 Thread Simon Slavin
On 12 Apr 2019, at 1:45am, Warren Young wrote: > ZFS I've used ZFS. ZFS is a great file system for some purposes. Fully-fledged databases isn't one of them. Someone already mentioned the problem of a transaction-based DBMS running on a transaction-based filesystem. I can add the problems

Re: [sqlite] compressed sqlite3 database file?

2019-04-11 Thread Warren Young
On Apr 11, 2019, at 1:27 PM, James K. Lowden wrote: > > On Wed, 10 Apr 2019 15:14:59 -0600 > Warren Young wrote: > >> If you?re going to buy some more storage, you should put ZFS on it >> then, too. :) > > That's interesting advice for a DBMS mailing list. > > ZFS has built-in

Re: [sqlite] compressed sqlite3 database file?

2019-04-11 Thread James K. Lowden
On Wed, 10 Apr 2019 15:14:59 -0600 Warren Young wrote: > On Apr 10, 2019, at 2:12 PM, Keith Medcalf > wrote: > > > > It is far cheaper and much more reliable to just buy some file > > storage space. > > If you?re going to buy some more storage, you should put ZFS on it > then, too. :)

Re: [sqlite] compressed sqlite3 database file?

2019-04-11 Thread Peter da Silva
Oy. I've worked on safety-critical systems with hard real-time constraints too. For the most part they didn't *have* file systems or the file systems were basically read-only in production. Sticking a relational database any closer than the SCADA monitoring node would not be a thing that happens,

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Simon Slavin
On 10 Apr 2019, at 8:51pm, Peng Yu wrote: > What do you recommend for Mac? Thanks. I do not have a good answer ready for you. These days storage is so cheap that buying an external 2TB drive is cheaper than spending a lot of time doing clever programming. But if you want to investigate this

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Keith Medcalf
On Wednesday, 10 April, 2019 14:21, Peter da Silva wrote: >On Wed, Apr 10, 2019 at 3:12 PM Keith Medcalf wrote: >> Why would anyone fart about with added complication and the >> concomittant increased unreliability when storage is so damn cheap? >Embedded systems and mobile devices. You

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Wout Mertens
On Wed, Apr 10, 2019 at 9:51 PM Peng Yu wrote: > What do you recommend for Mac? Thanks. > Nothing. Apple doesn't want you to have compression, because then you would take longer to buy a new Mac. The afsctool compression is a laughable hack that only works on read-only data. Writing to the file

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Warren Young
On Apr 10, 2019, at 2:12 PM, Keith Medcalf wrote: > > It is far cheaper and much more reliable to just buy some file storage space. > If you’re going to buy some more storage, you should put ZFS on it then, too. :) You get a whole lot more from ZFS than just transparent compression. You

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread James K. Lowden
On Wed, 10 Apr 2019 15:20:32 -0500 Peter da Silva wrote: > > Why would anyone fart about with added complication and the > > concomittant increased unreliability when storage is so damn cheap? > > Embedded systems and mobile devices. > > But of course those probably don't apply here. :) Are

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Peter da Silva
On Wed, Apr 10, 2019 at 3:12 PM Keith Medcalf wrote: > Why would anyone fart about with added complication and the concomittant > increased unreliability when storage is so damn cheap? > Embedded systems and mobile devices. But of course those probably don't apply here. :)

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Keith Medcalf
ung >Sent: Wednesday, 10 April, 2019 09:05 >To: SQLite mailing list >Subject: Re: [sqlite] compressed sqlite3 database file? > >On Apr 9, 2019, at 11:39 PM, Peng Yu wrote: >> >> Is there a way to make the database file of a size comparable (at >least >> not over

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Peng Yu
What do you recommend for Mac? Thanks. On 4/10/19, Simon Slavin wrote: > On 10 Apr 2019, at 7:08pm, Peng Yu wrote: > >> https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos >> >> I work on Mac. Would this be worthwhile to try? > > I do not recommend

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Simon Slavin
On 10 Apr 2019, at 7:08pm, Peng Yu wrote: > https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos > > I work on Mac. Would this be worthwhile to try? I do not recommend it. Two reasons: A) If you copy the file you get an uncompressed result. This

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Warren Young
On Apr 10, 2019, at 12:08 PM, Peng Yu wrote: > > https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos > > I work on Mac. Would this be worthwhile to try? The first link didn’t work here because it didn’t like the APFS drive I tried it on.

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Peng Yu
https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos I work on Mac. Would this be worthwhile to try? Does the transparent compression work at the file system level or at the directory level? Would it have a slight chance to corrupt the existent files

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Peter da Silva
A database trades off space for performance and functionality. It is expected that a database with indexes that you can randomly access is going to take more space than the raw data, let alone a compressed version of the raw data. On Wed, Apr 10, 2019 at 12:39 AM Peng Yu wrote: > I have some

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Wout Mertens
As I said in my previous email, I have a 13GB database that transparently compresses to 800MB. Not sure if it got through, didn't get replies to my last two emails. Wout. On Wed., Apr. 10, 2019, 5:04 p.m. Warren Young wrote: > On Apr 9, 2019, at 11:39 PM, Peng Yu wrote: > > > > Is there a way

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Warren Young
On Apr 9, 2019, at 11:39 PM, Peng Yu wrote: > > Is there a way to make the database file of a size comparable (at least > not over 5 times) to the original TSV table in the .gz file? Transparent file compression is a feature of several filesystems: NTFS, ZFS, Btrfs, and more:

Re: [sqlite] compressed sqlite3 database file?

2019-04-10 Thread Keith Medcalf
The disassembled bird will always require more tar to coat than the assembled bird. This is because the disassembled bird will have a greater surface area to coat with tar than the assembled bird. This is a fact of physics which, although you may try as you might, you cannot change (unless

Re: [sqlite] compressed sqlite3 database file?

2019-04-09 Thread Wout Mertens
I know of two options: The proprietary https://sqlite.org/zipvfs/doc/trunk/www/readme.wiki and this extension that you have to call on strings yourself: https://github.com/siara-cc/Shox96_Sqlite_UDF Furthermore, some filesystems allow transparent compression, like ntfs, bcachefs, zfs and btrfs. I