Re: [mail] Re: [HACKERS] 7.4 Wishlist
Would it be possible to make compression an optional thing, with the default being off? I'm in a position that many others may be in where the link between my app server and my database server isn't the bottleneck, and thus any time spent by the CPU performing compression and decompression tasks is CPU time that is in effect wasted. If a database is handling numerous small queries/updates and the request/response packets are compressed individually, then the overhead of compression and decompression may result in slower performance compared to leaving the request/response packets uncompressed. Al. - Original Message - From: Greg Copeland [EMAIL PROTECTED] To: Stephen L. [EMAIL PROTECTED] Cc: PostgresSQL Hackers Mailing List [EMAIL PROTECTED] Sent: Tuesday, December 10, 2002 4:56 PM Subject: [mail] Re: [HACKERS] 7.4 Wishlist On Tue, 2002-12-10 at 09:36, Stephen L. wrote: 6. Compression between client/server interface like in MySQL Mammoth is supposed to be donating their compression efforts back to this project, or so I've been told. I'm not exactly sure of their time-line as I've slept since my last conversation with them. The initial feedback that I've gotten back from them on this subject is that the compression has been working wonderfully for them with excellent results. IIRC, in their last official release, they announced their compression implementation. So, I'd think that it would be available for 7.4 of 7.5 time frame. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [mail] Re: [HACKERS] 7.4 Wishlist
On Tue, 2002-12-10 at 11:25, Al Sutton wrote: Would it be possible to make compression an optional thing, with the default being off? I'm not sure. You'd have to ask Command Prompt (Mammoth) or wait to see what appears. What I originally had envisioned was a per database and user permission model which would better control use. Since compression can be rather costly for some use cases, I also envisioned it being negotiated where only the user/database combo with permission would be able to turn it on. I do recall that compression negotiation is part of the Mammoth implementation but I don't know if it's a simple capability negotiation or part of a larger scheme. The reason I originally imagined a user/database type approach is because I would think only a subset of a typical installation would be needing compression. As such, this would help prevent users from arbitrarily chewing up database CPU compressing data because: o datasets are uncompressable or poorly compresses o environment cpu is at a premium o is in a bandwidth rich environment I'm in a position that many others may be in where the link between my app server and my database server isn't the bottleneck, and thus any time spent by the CPU performing compression and decompression tasks is CPU time that is in effect wasted. Agreed. This is why I'd *guess* that Mammoth's implementation does not force compression. If a database is handling numerous small queries/updates and the request/response packets are compressed individually, then the overhead of compression and decompression may result in slower performance compared to leaving the request/response packets uncompressed. Again, this is where I'm gray on their exact implementation. It's possible they implemented a compressed stream even though I'm hoping they implemented a per packet compression scheme (because adaptive compression becomes much more capable and powerful; in both algorithmically and logistical use). An example of this would be to avoid any compression on trivially sized result sets. Again, this is another area where I can imagine some tunable parameters. Just to be on the safe side, I'm cc'ing Josh Drake at Command Prompt (Mammoth) to see what they can offer up on it. Hope you guys don't mind. Greg - Original Message - From: Greg Copeland [EMAIL PROTECTED] To: Stephen L. [EMAIL PROTECTED] Cc: PostgresSQL Hackers Mailing List [EMAIL PROTECTED] Sent: Tuesday, December 10, 2002 4:56 PM Subject: [mail] Re: [HACKERS] 7.4 Wishlist On Tue, 2002-12-10 at 09:36, Stephen L. wrote: 6. Compression between client/server interface like in MySQL Mammoth is supposed to be donating their compression efforts back to this project, or so I've been told. I'm not exactly sure of their time-line as I've slept since my last conversation with them. The initial feedback that I've gotten back from them on this subject is that the compression has been working wonderfully for them with excellent results. IIRC, in their last official release, they announced their compression implementation. So, I'd think that it would be available for 7.4 of 7.5 time frame. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [mail] Re: [HACKERS] 7.4 Wishlist
Greg Copeland wrote: On Tue, 2002-12-10 at 11:25, Al Sutton wrote: Would it be possible to make compression an optional thing, with the default being off? I'm not sure. You'd have to ask Command Prompt (Mammoth) or wait to see what appears. What I originally had envisioned was a per database and user permission model which would better control use. Since compression can be rather costly for some use cases, I also envisioned it being negotiated where only the user/database combo with permission would be able to turn it on. I do recall that compression negotiation is part of the Mammoth implementation but I don't know if it's a simple capability negotiation or part of a larger scheme. I haven't heard anything about them contributing it. Doesn't mean it will not happen, just that I haven't heard it. I am not excited about per-db/user compression because of the added complexity of setting it up, and even set up, I can see cases where some queries would want it, and others not. I can see using GUC to control this. If you enable it and the client doesn't support it, it is a no-op. We have per-db and per-user settings, so GUC would allow such control if you wish. Ideally, it would be a tri-valued parameter, that is ON, OFF, or AUTO, meaning it would determine if there was value in the compression and do it only when it would help. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [mail] Re: [HACKERS] 7.4 Wishlist
On Tue, 2002-12-10 at 13:38, Bruce Momjian wrote: I haven't heard anything about them contributing it. Doesn't mean it will not happen, just that I haven't heard it. This was in non-mailing list emails that I was told this by Joshua Drake at Command Prompt. Of course, that doesn't have to mean it will be donated for sure but nonetheless, I was told it will be. Here's a quote from one of the emails. I don't think I'll be too far out of line posting this. On August 9, 2002, Joshua Drake said, One we plan on releasing this code to the developers after 7.3 comes out. We want to be good members of the community but we have to keep a slight commercial edge (wait to you see what we are going to do to vacuum). Obviously, I don't think that was official speak, so I'm not holding them to the fire, nonetheless, that's what was said. Additional follow ups did seem to imply that they were very serious about this and REALLY want to play nice as good shared source citizens. I am not excited about per-db/user compression because of the added complexity of setting it up, and even set up, I can see cases where some queries would want it, and others not. I can see using GUC to control this. If you enable it and the client doesn't support it, it is a no-op. We have per-db and per-user settings, so GUC would allow such control if you wish. I never thought beyond the need for what form an actual implementation of this aspect would look like. The reason for such a concept would be to simply limit the number of users that can be granted compression. If you have a large user base all using compression or even a small user base where very large result sets are common, I can imagine your database server becoming CPU bound. The database/user thinking was an effort to allow the DBA to better manage the CPU effect. Ideally, it would be a tri-valued parameter, that is ON, OFF, or AUTO, meaning it would determine if there was value in the compression and do it only when it would help. Yes, that makes sense and was something I had originally envisioned. Simply stated, some installations may never want compression while others may want it for every connection. Beyond that, I believe there needs to be something of a happy medium where a DBA can better control who and what is taking his CPU away (e.g. only that one remote location being fed via ISDN). If GUC can fully satisfy, I certainly won't argue against it. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [mail] Re: [HACKERS] 7.4 Wishlist
Yes, the issue was that give our TODO list, compressed transfer wasn't very high, and it was unknown how valuable it would be. However, if it were contributed, we could easily test its value with little work on our part and include the code if it were a win. --- Joshua D. Drake wrote: Hello, We would probably be open to contributing it if there was interest. There wasn't interest initially. Sincerely, Joshua Drake Bruce Momjian wrote: Greg Copeland wrote: On Tue, 2002-12-10 at 11:25, Al Sutton wrote: Would it be possible to make compression an optional thing, with the default being off? I'm not sure. You'd have to ask Command Prompt (Mammoth) or wait to see what appears. What I originally had envisioned was a per database and user permission model which would better control use. Since compression can be rather costly for some use cases, I also envisioned it being negotiated where only the user/database combo with permission would be able to turn it on. I do recall that compression negotiation is part of the Mammoth implementation but I don't know if it's a simple capability negotiation or part of a larger scheme. I haven't heard anything about them contributing it. Doesn't mean it will not happen, just that I haven't heard it. I am not excited about per-db/user compression because of the added complexity of setting it up, and even set up, I can see cases where some queries would want it, and others not. I can see using GUC to control this. If you enable it and the client doesn't support it, it is a no-op. We have per-db and per-user settings, so GUC would allow such control if you wish. Ideally, it would be a tri-valued parameter, that is ON, OFF, or AUTO, meaning it would determine if there was value in the compression and do it only when it would help. -- COMPANYCommandPrompt- http://www.commandprompt.com /COMPANY CONTACT PHONE+1.503.222-2783/PHONE /CONTACT -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [mail] Re: [HACKERS] 7.4 Wishlist
I'd like to show/register interest. I can see it being very useful when combined with replication for situations where the replicatiant databases are geographically seperated (i.e. Disaster Recover sites or systems maintaining replicants in order to reduce the distance from user to app to database). The bandwidth cost savings from compressing the replication information would be immensly useful. Al. - Original Message - From: Joshua D. Drake [EMAIL PROTECTED] To: Bruce Momjian [EMAIL PROTECTED] Cc: Greg Copeland [EMAIL PROTECTED]; Al Sutton [EMAIL PROTECTED]; Stephen L. [EMAIL PROTECTED]; PostgresSQL Hackers Mailing List [EMAIL PROTECTED] Sent: Tuesday, December 10, 2002 8:04 PM Subject: Re: [mail] Re: [HACKERS] 7.4 Wishlist Hello, We would probably be open to contributing it if there was interest. There wasn't interest initially. Sincerely, Joshua Drake Bruce Momjian wrote: Greg Copeland wrote: On Tue, 2002-12-10 at 11:25, Al Sutton wrote: Would it be possible to make compression an optional thing, with the default being off? I'm not sure. You'd have to ask Command Prompt (Mammoth) or wait to see what appears. What I originally had envisioned was a per database and user permission model which would better control use. Since compression can be rather costly for some use cases, I also envisioned it being negotiated where only the user/database combo with permission would be able to turn it on. I do recall that compression negotiation is part of the Mammoth implementation but I don't know if it's a simple capability negotiation or part of a larger scheme. I haven't heard anything about them contributing it. Doesn't mean it will not happen, just that I haven't heard it. I am not excited about per-db/user compression because of the added complexity of setting it up, and even set up, I can see cases where some queries would want it, and others not. I can see using GUC to control this. If you enable it and the client doesn't support it, it is a no-op. We have per-db and per-user settings, so GUC would allow such control if you wish. Ideally, it would be a tri-valued parameter, that is ON, OFF, or AUTO, meaning it would determine if there was value in the compression and do it only when it would help. -- COMPANYCommandPrompt - http://www.commandprompt.com /COMPANY CONTACT PHONE+1.503.222-2783/PHONE /CONTACT ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
