Re: [HACKERS] pluggable compression support

2013-10-02 Thread Robert Haas
On Tue, Oct 1, 2013 at 9:56 PM, Daniel Farina dan...@heroku.com wrote: On Mon, Sep 30, 2013 at 1:49 PM, Huchev hugochevr...@gmail.com wrote: How come any compressor which could put some competition to pglz is systematically pushed out of the field on the ground of unverifiable legal risks ?

Re: [HACKERS] pluggable compression support

2013-10-01 Thread Daniel Farina
On Mon, Sep 30, 2013 at 1:49 PM, Huchev hugochevr...@gmail.com wrote: How come any compressor which could put some competition to pglz is systematically pushed out of the field on the ground of unverifiable legal risks ? Because pglz has been around for a while and has not caused patent

Re: [HACKERS] pluggable compression support

2013-09-30 Thread Huchev
I've been following this issue these last few months. Having the latest and best compressors built-in is a fashionable features these days. And for good reasons. I'm quite amazed that this issue is still considered a legal risk. To put this in perspective, the *whole world* is using LZ4 by now.

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Robert Haas
On Thu, Jun 20, 2013 at 8:09 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-15 12:20:28 +0200, Andres Freund wrote: On 2013-06-14 21:56:52 -0400, Robert Haas wrote: I don't think we need it. I think what we need is to decide is which algorithm is legally OK to use. And then put

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Andres Freund
On 2013-06-25 12:22:31 -0400, Robert Haas wrote: On Thu, Jun 20, 2013 at 8:09 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-15 12:20:28 +0200, Andres Freund wrote: On 2013-06-14 21:56:52 -0400, Robert Haas wrote: I don't think we need it. I think what we need is to decide is

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Josh Berkus
On 06/25/2013 11:42 AM, Andres Freund wrote: True. So, how do we proceed on that? The ASF decided it was safe to use lz4 in cassandra. Does anybody have contacts over there? Btw, I have the feeling we hold this topic to a higher standard wrt patent issues than other work in postgres...

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Andres Freund
On 2013-06-25 12:08:22 -0700, Josh Berkus wrote: On 06/25/2013 11:42 AM, Andres Freund wrote: True. So, how do we proceed on that? The ASF decided it was safe to use lz4 in cassandra. Does anybody have contacts over there? Btw, I have the feeling we hold this topic to a higher

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Claudio Freire
On Tue, Jun 25, 2013 at 4:15 PM, Andres Freund and...@2ndquadrant.com wrote: However, can you tell me what exactly you are concerned about? lz4 is under the BSD license, and released by Google. Snappy is released/copyrighted by google. lz4 by Yann Collet. Both are under BSD licenses (3 and 2

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: However, can you tell me what exactly you are concerned about? lz4 is under the BSD license, and released by Google. Why are we worried, exactly? Patents. The license on the code doesn't matter --- worst case, if someone objected, we could rewrite the

Re: [HACKERS] pluggable compression support

2013-06-25 Thread Josh Berkus
On 06/25/2013 12:23 PM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: However, can you tell me what exactly you are concerned about? lz4 is under the BSD license, and released by Google. Why are we worried, exactly? Patents. The license on the code doesn't matter --- worst case, if

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Hannu Krosing
On 06/16/2013 03:50 AM, Robert Haas wrote: On Sat, Jun 15, 2013 at 8:11 AM, Hannu Krosing ha...@2ndquadrant.com wrote: Claiming that the algorithm will be one of only two (current and whatever algorithm we come up with ) suggests that it is only one bit, which is undoubtedly too little for

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Simon Riggs
On 15 June 2013 13:02, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote: If it were truly pluggable - that is just load a .dll, set a GUG and start using it Ok. I officially rechristen the patchset to 'extensible compression support'. +1 (I

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Simon Riggs
On 15 June 2013 12:25, Hannu Krosing ha...@2ndquadrant.com wrote: additionally it *could* have the property Simon proposed earlier of *uncompressed* pages having some predetermined size, so we could retain optimisations of substring() even on compressed TOAST values. That wasn't about having

Re: [HACKERS] pluggable compression support

2013-06-16 Thread Alvaro Herrera
Simon Riggs wrote: On 15 June 2013 12:25, Hannu Krosing ha...@2ndquadrant.com wrote: additionally it *could* have the property Simon proposed earlier of *uncompressed* pages having some predetermined size, so we could retain optimisations of substring() even on compressed TOAST values.

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-14 21:56:52 -0400, Robert Haas wrote: I don't think we need it. I think what we need is to decide is which algorithm is legally OK to use. And then put it in. In the past, we've had a great deal of speculation about that legal question from people who are not lawyers. Maybe it

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 03:56 AM, Robert Haas wrote: On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 02:20 AM, Andres Freund wrote: On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: On 06/14/2013 04:01 PM, Andres Freund wrote: It still contains a guc as described in the above message to control the algorithm used for compressing new tuples but I think we should remove that guc

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-15 13:25:49 +0200, Hannu Krosing wrote: On 06/15/2013 02:20 AM, Andres Freund wrote: On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: On 06/14/2013 04:01 PM, Andres Freund wrote: It still contains a guc as described in the above message to control the algorithm used for

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote: If it were truly pluggable - that is just load a .dll, set a GUG and start using it Ok. I officially rechristen the patchset to 'extensible compression support'. - then the selection of algorithms would be much wider as several

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 01:56 PM, Andres Freund wrote: On 2013-06-15 13:25:49 +0200, Hannu Krosing wrote: On 06/15/2013 02:20 AM, Andres Freund wrote: On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: On 06/14/2013 04:01 PM, Andres Freund wrote: It still contains a guc as described in the above message

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Hannu Krosing
On 06/15/2013 02:02 PM, Andres Freund wrote: On 2013-06-15 13:11:47 +0200, Hannu Krosing wrote: If it were truly pluggable - that is just load a .dll, set a GUG and start using it Ok. I officially rechristen the patchset to 'extensible compression support'. - then the selection of

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Andres Freund
On 2013-06-15 14:11:54 +0200, Hannu Krosing wrote: Could you perhaps actually read the the description and the discussion before making wild suggestions? Possibly even the patch. Compressed toast datums now *do* have an identifier of the compression algorithm used. That's how we can

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Amit kapila
On Saturday, June 15, 2013 3:50 PM Andres Freund wrote: On 2013-06-14 21:56:52 -0400, Robert Haas wrote: I don't think we need it. I think what we need is to decide is which algorithm is legally OK to use. And then put it in. In the past, we've had a great deal of speculation about that

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Robert Haas
On Sat, Jun 15, 2013 at 8:11 AM, Hannu Krosing ha...@2ndquadrant.com wrote: Claiming that the algorithm will be one of only two (current and whatever algorithm we come up with ) suggests that it is only one bit, which is undoubtedly too little for having a pluggable compression API :) See

Re: [HACKERS] pluggable compression support

2013-06-15 Thread Robert Haas
On Sat, Jun 15, 2013 at 8:22 AM, Hannu Krosing ha...@2ndquadrant.com wrote: You suddently need to solve the question of how the identifiers for compression formats are allocated and preserved across pg_upgrade and across machines. This is something similar we already need to do for any

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Josh Berkus
On 06/14/2013 04:01 PM, Andres Freund wrote: It still contains a guc as described in the above message to control the algorithm used for compressing new tuples but I think we should remove that guc after testing. Did you add the storage attribute? -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Andres Freund
On 2013-06-14 17:12:01 -0700, Josh Berkus wrote: On 06/14/2013 04:01 PM, Andres Freund wrote: It still contains a guc as described in the above message to control the algorithm used for compressing new tuples but I think we should remove that guc after testing. Did you add the storage

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Josh Berkus
No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should just always use the new algorithm. Unless I missed it nobody seemed to have voiced a contrary position? For testing/evaluation the guc seems to be sufficient. Then

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Andres Freund
On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should just always use the new algorithm. Unless I missed it nobody seemed to have voiced a contrary position? For

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Robert Haas
On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we should just always use the new algorithm.

Re: [HACKERS] pluggable compression support

2013-06-14 Thread Joshua D. Drake
On 06/14/2013 06:56 PM, Robert Haas wrote: On Fri, Jun 14, 2013 at 8:45 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-06-14 17:35:02 -0700, Josh Berkus wrote: No. I think as long as we only have pglz and one new algorithm (even if that is lz4 instead of the current snappy) we