Re: [HACKERS] Faster compression, again

2012-04-06 Thread Huchev
Well, the patent argument, used like this, looks like a wild card, which can be freely interpreted as a mortal danger for some, and a non-issue for others. A perfect scare-mongerer. Quite frankly, I don't buy that one implementation is safer because there is Google backing it. I can't think of any

Re: [HACKERS] Faster compression, again

2012-04-04 Thread Daniel Farina
On Tue, Apr 3, 2012 at 7:29 AM, Huchev hugochevr...@gmail.com wrote: For a C implementation, it could interesting to consider LZ4 algorithm, since it is written natively in this language. In contrast, Snappy has been ported to C by Andy from the original C++ Google code, which lso translate

Re: [HACKERS] Faster compression, again

2012-04-03 Thread Huchev
For a C implementation, it could interesting to consider LZ4 algorithm, since it is written natively in this language. In contrast, Snappy has been ported to C by Andy from the original C++ Google code, which lso translate into less extensive usage and tests. http://code.google.com/p/lz4/

Re: [HACKERS] Faster compression, again

2012-03-22 Thread Simon Riggs
On Thu, Mar 15, 2012 at 10:34 PM, Daniel Farina dan...@heroku.com wrote: I'd really like to find a way to layer both message-oblivious and message-aware transport under FEBE with both backend and frontend support without committing the project to new code for-ever-and-ever. I guess I could

Re: [HACKERS] Faster compression, again

2012-03-15 Thread Daniel Farina
On Wed, Mar 14, 2012 at 11:06 AM, Daniel Farina dan...@heroku.com wrote: ...and it has been ported to C (recently, and with some quirks, like no LICENSE file...yet, although it is linked from the original Snappy project). I poked the author about the license and he fixed it in a jiffy. Now

Re: [HACKERS] Faster compression, again

2012-03-15 Thread Simon Riggs
On Wed, Mar 14, 2012 at 6:06 PM, Daniel Farina dan...@heroku.com wrote: If we're curious how it affects replication traffic, I could probably gather statistics on LZO-compressed WAL traffic, of which we have a pretty huge amount captured. What's the compression like for shorter chunks of

Re: [HACKERS] Faster compression, again

2012-03-15 Thread Daniel Farina
On Thu, Mar 15, 2012 at 3:14 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Mar 14, 2012 at 6:06 PM, Daniel Farina dan...@heroku.com wrote: If we're curious how it affects replication traffic, I could probably gather statistics on LZO-compressed WAL traffic, of which we have a pretty

Re: [HACKERS] Faster compression, again

2012-03-15 Thread k...@rice.edu
On Thu, Mar 15, 2012 at 10:14:12PM +, Simon Riggs wrote: On Wed, Mar 14, 2012 at 6:06 PM, Daniel Farina dan...@heroku.com wrote: If we're curious how it affects replication traffic, I could probably gather statistics on LZO-compressed WAL traffic, of which we have a pretty huge amount

Re: [HACKERS] Faster compression, again

2012-03-14 Thread k...@rice.edu
On Wed, Mar 14, 2012 at 11:06:16AM -0700, Daniel Farina wrote: For 9.3 at a minimum. The topic of LZO became mired in doubts about: * Potential Patents * The author's intention for the implementation to be GPL Since then, Google released Snappy, also an LZ77-class implementation, and

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Merlin Moncure
On Wed, Mar 14, 2012 at 1:06 PM, Daniel Farina dan...@heroku.com wrote: For 9.3 at a minimum. The topic of LZO became mired in doubts about: * Potential Patents * The author's intention for the implementation to be GPL Since then, Google released Snappy, also an LZ77-class implementation,

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Andrew Dunstan
On 03/14/2012 04:10 PM, Merlin Moncure wrote: there are plenty of on gpl lz based libraries out there (for example: http://www.fastlz.org/) and always have been. they are all much faster than zlib. the main issue is patents...you have to be careful even though all the lz77/78 patents seem to

Re: [HACKERS] Faster compression, again

2012-03-14 Thread k...@rice.edu
On Wed, Mar 14, 2012 at 04:43:55PM -0400, Andrew Dunstan wrote: On 03/14/2012 04:10 PM, Merlin Moncure wrote: there are plenty of on gpl lz based libraries out there (for example: http://www.fastlz.org/) and always have been. they are all much faster than zlib. the main issue is

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Merlin Moncure
On Wed, Mar 14, 2012 at 3:43 PM, Andrew Dunstan and...@dunslane.net wrote: On 03/14/2012 04:10 PM, Merlin Moncure wrote: there are plenty of on gpl lz based libraries out there (for example: http://www.fastlz.org/) and always have been.  they are all much faster than zlib.  the main issue

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Daniel Farina
On Wed, Mar 14, 2012 at 2:03 PM, Merlin Moncure mmonc...@gmail.com wrote: er, typo: I meant to say: *non-gpl* lz based...  :-). Given that, few I would say have had the traction that LZO and Snappy have had, even though in many respects they are interchangeable in the general trade-off spectrum.

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Tom Lane
Daniel Farina dan...@heroku.com writes: Given that, few I would say have had the traction that LZO and Snappy have had, even though in many respects they are interchangeable in the general trade-off spectrum. The question is: what burden of proof is required to convince the project that Snappy

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Another not-exactly-trivial requirement is to figure out how to not break on-disk compatibility when installing an alternative compression scheme. In hindsight it might've been a good idea if pglz_compress had wasted a little bit of space on some sort of

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Daniel Farina
On Wed, Mar 14, 2012 at 2:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: Daniel Farina dan...@heroku.com writes: Given that, few I would say have had the traction that LZO and Snappy have had, even though in many respects they are interchangeable in the general trade-off spectrum. The question is:

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 6:08 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane t...@sss.pgh.pa.us wrote: Another not-exactly-trivial requirement is to figure out how to not break on-disk compatibility when installing an alternative compression scheme.  In hindsight it might've

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Mar 14, 2012 at 6:08 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Doesn't it always start with a header of two int32 values where the first must be smaller than the second? That seems like enough to get traction for an identifiably

Re: [HACKERS] Faster compression, again

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 9:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: Well, let's please not make the same mistake again of assuming that there will never again be any other ideas in this space.  IOW, let's find a way to shoehorn in an actual compression-method ID value of some sort.  I don't