Re: [HACKERS] [PATCHES] updated hash functions for postgresql v1

2008-11-04 Thread Kenneth Marshall
lookup3 is 5n+20 where CRC is 9n+3. Regards, Ken On Tue, 4 Nov 2008, Kenneth Marshall wrote: Sorry about the delay for this update to the new hash index implementation. I was trying to get the WAL logging in place and forgot to post the actual patch. The WAL for hash indexes will need to wait

Re: [HACKERS] [PATCHES] updated hash functions for postgresql v1

2008-11-04 Thread Kenneth Marshall
Oleg, Here is a little more information on the use of CRC32 as a hash function, with some warning caveats: http://home.comcast.net/~bretm/hash/8.html Regards, Ken On Tue, Nov 04, 2008 at 03:15:44PM -0600, Kenneth Marshall wrote: On Tue, Nov 04, 2008 at 11:32:47PM +0300, Oleg Bartunov wrote

Re: [HACKERS] [PATCHES] updated hash functions for postgresql v1

2008-11-04 Thread Kenneth Marshall
Sorry about the delay for this update to the new hash index implementation. I was trying to get the WAL logging in place and forgot to post the actual patch. The WAL for hash indexes will need to wait for 8.5, but I did want to add back in the piece of the Bob Jenkins 2006 hash function that was

Re: [HACKERS] WIP patch: convert SQL-language functions to return tuplestores

2008-10-28 Thread Kenneth Marshall
On Tue, Oct 28, 2008 at 09:28:38AM -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Sun, 2008-10-26 at 21:49 -0400, Tom Lane wrote: So I'm concluding that we can easily afford to switch to tuplestore-always operation, especially if we are willing to put any effort into

Re: [HACKERS] minimal update

2008-10-22 Thread Kenneth Marshall
On Wed, Oct 22, 2008 at 06:05:26PM -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Wed, Oct 22, 2008 at 3:24 PM, Tom Lane [EMAIL PROTECTED] wrote: Minimal really fails to convey the point here IMHO. How about something like suppress_no_op_updates_trigger? I think it

Re: [HACKERS] CLUSTER, REINDEX, VACUUM in read only transaction?

2008-10-10 Thread Kenneth Marshall
On Fri, Oct 10, 2008 at 09:41:39AM -0400, Tom Lane wrote: So I was looking for other omissions in utility.c, and I noticed that check_xact_readonly() doesn't reject CLUSTER, REINDEX, or VACUUM. Now the notion of read only that we're trying to enforce is pretty weak (I think it's effectively no

Re: [HACKERS] 8.3 vs HEAD difference in Interval output?

2008-10-09 Thread Kenneth Marshall
On Thu, Oct 09, 2008 at 11:50:17AM -0700, Ron Mayer wrote: Kevin Grittner wrote: Kevin Grittner [EMAIL PROTECTED] wrote: Even more surprising is the behavior for interval(1) here: [ some context with nonsurprising examples removed ...] ccdev=# select '1 year 2 mons 3 days

Re: [HACKERS] 8.3 vs HEAD difference in Interval output?

2008-10-09 Thread Kenneth Marshall
On Thu, Oct 09, 2008 at 02:47:24PM -0500, Kevin Grittner wrote: Kenneth Marshall [EMAIL PROTECTED] wrote: Even more surprising is the behavior for interval(1) here: [ some context with nonsurprising examples removed ...] ccdev=# select '1 year 2 mons 3 days 04:05:06.64321

Re: [HACKERS] Need more reviewers!

2008-09-04 Thread Kenneth Marshall
On Thu, Sep 04, 2008 at 02:01:18PM -0400, Tom Lane wrote: Jonah H. Harris [EMAIL PROTECTED] writes: I'll push forward on reviewing and testing Xiao's hash index improvements for inclusion into core. Though, someone will still need to review my stuff. I think what the hash index patch

Re: [HACKERS] Patch: plan invalidation vs stored procedures

2008-08-20 Thread Kenneth Marshall
On Wed, Aug 20, 2008 at 09:16:56AM -0400, Andrew Sullivan wrote: On Wed, Aug 20, 2008 at 03:12:43PM +0300, Asko Oja wrote: - If there is nothing that can be done in 8.3 at least warning should be added into the documentation. It will be just one more don't in our long list don'ts for our

Re: [HACKERS] migrate data 6.5.3 - 8.3.1

2008-08-14 Thread Kenneth Marshall
When upgrading, you use the pg_dump from the new version to dump the old database. Then it can take care of incidental changes during the process. I think that the mailing list archives have articles on upgrading from v6.5. I do not think that you can go straight from v6.5 to v8.3. You will almost

Re: [HACKERS] [RFC] Unsigned integer support.

2008-07-26 Thread Kenneth Marshall
Hi Ryan, I agree, I have had applications use uint types to avoid using a larger data type. I have actually had to patch an application developed for MySQL uint8 to signed int8 on PostgreSQL. In that case, the only operations that were performed where assignment and lookup. If we need to use the

Re: [HACKERS] [PATCH]-hash index improving

2008-07-23 Thread Kenneth Marshall
On Tue, Jul 22, 2008 at 08:36:34PM -0700, Dann Corbit wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:pgsql-hackers- [EMAIL PROTECTED] On Behalf Of Xiao Meng Sent: Tuesday, July 22, 2008 7:57 PM To: Simon Riggs Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS]

Re: [HACKERS] [PATCH]-hash index improving

2008-07-18 Thread Kenneth Marshall
I just ran my original 16M word test case against the patched version, and like Tom noted below, the tuples per bucket calculation is wrong which results in identical index sizes for both the original version and the hash-value-only version. I suppose that the main point of #1 is to reduce index

Re: [HACKERS] [PATCH]-hash index improving

2008-07-18 Thread Kenneth Marshall
FYI, I just patched the fill-factor calculation and re-ran my test. The index size dropped from 513M to 43M which is the same disk footprint as the corresponding btree index. Have a nice weekend. Ken On Fri, Jul 18, 2008 at 12:23:14PM -0500, Kenneth Marshall wrote: I just ran my original 16M

Re: [HACKERS] [PATCH]-hash index improving

2008-07-17 Thread Kenneth Marshall
On Thu, Jul 17, 2008 at 12:42:39PM -0400, Alvaro Herrera wrote: Xiao Meng escribi?: The patch store hash code only in the index tuple. It based on Neil Conway's patch with an old version of PostgreSQL. It passes the regression test but I didn't test the performance yet. Anyone interested

Re: [HACKERS] [PATCH]-hash index improving

2008-07-17 Thread Kenneth Marshall
On Thu, Jul 17, 2008 at 02:00:07PM -0400, Jonah H. Harris wrote: On Thu, Jul 17, 2008 at 1:54 PM, Kenneth Marshall [EMAIL PROTECTED] wrote: I think having the HASHVALUE_ONLY define is not a good idea -- it just makes the patch harder to read. I suggest just removing the old code

Re: [HACKERS] [PATCH]-hash index improving

2008-07-17 Thread Kenneth Marshall
On Thu, Jul 17, 2008 at 04:24:28PM -0400, Jonah H. Harris wrote: On Thu, Jul 17, 2008 at 5:26 AM, Xiao Meng [EMAIL PROTECTED] wrote: The patch store hash code only in the index tuple. It based on Neil Conway's patch with an old version of PostgreSQL. It passes the regression test but I

Re: [HACKERS] stat() vs cygwin

2008-06-24 Thread Kenneth Marshall
One motivation for keeping it working on Cygwin, is that in some environments it is not allowed to install native Windows apps but they allow the use of the Cygwin environment. Of course if it takes too many resources to support, then dropping support would be an option. I would check this for

Re: [HACKERS] stat() vs cygwin

2008-06-24 Thread Kenneth Marshall
The case I am referring to has a developer clause that allows Cygwin applications to be used for development only. I agree that the policy is odd. Ken On Tue, Jun 24, 2008 at 02:35:50PM +0200, Magnus Hagander wrote: Kenneth Marshall wrote: One motivation for keeping it working on Cygwin

Re: [HACKERS] Cleaning up cross-type arithmetic operators

2008-06-17 Thread Kenneth Marshall
On Tue, Jun 17, 2008 at 01:29:56PM -0400, Tom Lane wrote: ... What I'm inclined to do is remove the two % operators, which don't seem likely to be performance-critical, and fill in the missing int2-vs-int8 cases for the four basic arithmetic operators. But I could be talked into just nuking

Re: [HACKERS] [GSoC08]some detail plan of improving hash index

2008-05-16 Thread Kenneth Marshall
Hi Xiao Meng, I am glad that you are making some progress. I have added a couple of comments below. Your phased approach is a good way to get it in a position for testing. I had a very basic test for creation time, query time for a simple lookup, and index size that I would like to re-run when

Re: [HACKERS] Commit fest queue

2008-04-11 Thread Kenneth Marshall
We use RT here for our trouble ticket system and the dashboard can easily be configured to display tickets based on any search criteria and you can have multiple views on the same screen. The search functionality can be viewed as the tool for configuring your views into the system, for whatever

Re: [HACKERS] machine-dependent hash_any vs the regression tests

2008-04-06 Thread Kenneth Marshall
On Sat, Apr 05, 2008 at 05:57:35PM -0400, Tom Lane wrote: So the proposed changes in hash_any make its hash values different between big-endian and little-endian machines (at least for string keys; for keys that are really arrays of int, I think the changes will unify the behavior). This

Re: [HACKERS] [GSoC] Need for advice on improving hash index performance

2008-03-26 Thread Kenneth Marshall
On Wed, Mar 26, 2008 at 10:55:44PM +0800, Xiao Meng wrote: Hello, Hackers: I've post a question about GSoC before. [GSoC] (Is it OK to choose items without % mark in theToDoList) (is it an acceptable idea to build index on Flash Disk)

Re: [HACKERS] [GSoC] (Is it OK to choose items without % mark in theToDoList) (is it an acceptable idea to build index on Flash Disk)

2008-03-25 Thread Kenneth Marshall
On Tue, Mar 25, 2008 at 01:46:51PM +0100, Zeugswetter Andreas OSB SD wrote: So, I finally decide to focus on the project idea of improving hash index now. It's more valuable , and also challenging. Any suggestion about the project idea of improving hash index? Imho one thing to look

Re: [HACKERS] Maximum statistics target

2008-03-20 Thread Kenneth Marshall
On Thu, Mar 20, 2008 at 11:17:10AM -0500, Decibel! wrote: On Mar 10, 2008, at 1:26 PM, Peter Eisentraut wrote: At some point I think it makes a lot more sense to just have VACUUM gather stats as it goes, rather than have ANALYZE generate a bunch of random IO. BTW, when it comes to the case

Re: [HACKERS] Unique Constraints using Non-Unique Indexes

2008-03-20 Thread Kenneth Marshall
On Thu, Mar 20, 2008 at 02:35:38PM +, Simon Riggs wrote: The current Unique constraint relies directly upon a Unique index to test for uniqueness. This has two disadvantages: * only B-Trees currently support Uniqueness * We need to create an index on *all* of the columns of the

Re: [HACKERS] [PATCHES] Fix for large file support (nonsegment mode support)

2008-03-19 Thread Kenneth Marshall
On Wed, Mar 19, 2008 at 10:51:12AM +0100, Martijn van Oosterhout wrote: On Wed, Mar 19, 2008 at 09:38:12AM +0100, Peter Eisentraut wrote: Another factor I just thought of is that tar, commonly used as part of a backup procedure, can on some systems only handle files up to 8 GB in size.

Re: [HACKERS] UUID data format 4x-4x-4x-4x-4x-4x-4x-4x

2008-02-28 Thread Kenneth Marshall
On Thu, Feb 28, 2008 at 08:06:46PM +0100, Zeugswetter Andreas ADI SD wrote: Well, I guess the question is: if we don't offer some builtin way to render non-standard formats built into company products, will those companies fix their format or just not use PostgreSQL? Well,

Re: [HACKERS] Reverse key index

2008-02-04 Thread Kenneth Marshall
Pretty neat. It may be a possible alternative to the use of the hash index in some applications. Cheers, Ken On Sun, Feb 03, 2008 at 07:13:23PM -0800, Gurjeet Singh wrote: Hi All, I have wanted to create a reverse key index for some time now, and it seems that an evening of reading and

Re: [HACKERS] [PATCHES] Proposed patch: synchronized_scanning GUCvariable

2008-01-30 Thread Kenneth Marshall
On Wed, Jan 30, 2008 at 10:56:47AM +0100, Zeugswetter Andreas ADI SD wrote: The plural seems better to me; there's no such thing as a solitary synchronized scan, no? The whole point of the feature is to affect the behavior of multiple scans. +1. The plural is important IMHO.

Re: [HACKERS] [PATCHES] Proposed patch: synchronized_scanning GUCvariable

2008-01-29 Thread Kenneth Marshall
On Tue, Jan 29, 2008 at 10:40:40AM +0100, Zeugswetter Andreas ADI SD wrote: It's a good point that we don't want pg_dump to screw up the cluster order, but that's the only use case I've seen this far for disabling sync scans. Even that wouldn't matter much if our estimate for

Re: [HACKERS] WAL logging of hash indexes

2008-01-16 Thread Kenneth Marshall
On Tue, Jan 15, 2008 at 07:18:17PM -0800, John Smith wrote: http://archives.postgresql.org/pgsql-hackers/2007-10/msg01468.php, ... I very much want to encourage authors of new Resource Managers and it looks like we may be getting at least 3 new RMs that produce WAL records: hash indexes

Re: [HACKERS] Index trouble with 8.3b4

2008-01-15 Thread Kenneth Marshall
On Mon, Jan 14, 2008 at 10:10:54PM -0500, Tom Lane wrote: Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: I went through all of the heap_beginscan calls in the code last night. pgstattuple was broken but AFAICS none of the other callers care about the visitation

Re: [HACKERS] Hash index todo list item

2007-11-14 Thread Kenneth Marshall
On Thu, Sep 13, 2007 at 02:02:14PM -0700, Neil Conway wrote: On Fri, 2007-09-07 at 08:29 -0500, Kenneth Marshall wrote: This is a great starting point. I would appreciate it if you have the time and could make it apply cleanly to HEAD. Just to give you an update on this, I'll try to find

Re: [HACKERS] Hash index todo list item

2007-10-22 Thread Kenneth Marshall
On Thu, Sep 13, 2007 at 02:02:14PM -0700, Neil Conway wrote: On Fri, 2007-09-07 at 08:29 -0500, Kenneth Marshall wrote: This is a great starting point. I would appreciate it if you have the time and could make it apply cleanly to HEAD. Just to give you an update on this, I'll try to find

Re: [HACKERS] Hash index todo list item

2007-10-21 Thread Kenneth Marshall
Tom, Thank you for the update. I am currently working on updating the patch Neil Conway sent in against 8.0-ish that stores only the hash in the index and locates the entries within the page using a binary search. Then I will fold in your recent update. On Sun, Oct 21, 2007 at 01:13:48PM -0700,

Re: [HACKERS] Hash index todo list item

2007-10-17 Thread Kenneth Marshall
Tom, That is great. I am looking forward to your patch. After the issues that you needed to address, I think that it would be reasonable to add a few more user settings for the hash index. Fill-factor is too course a knob. The others that I have been considering are: maxtuples - Not really the

Re: [HACKERS] Hash index todo list item

2007-10-12 Thread Kenneth Marshall
On Wed, Sep 05, 2007 at 03:07:03PM -0500, Kenneth Marshall wrote: On Sun, Sep 02, 2007 at 01:04:04PM -0500, Kenneth Marshall wrote: Dear PostgreSQL Hackers: After following the hackers mailing list for quite a while, I am going to start investigating what will need to be done

Re: [HACKERS] Hash index todo list item

2007-09-25 Thread Kenneth Marshall
On Thu, Sep 20, 2007 at 05:12:45PM -0700, Tom Raney wrote: We are pleased to announce an upcoming patch to the hash index code which improves build time and index size, based on this item in the TODO list: During index creation, pre-sort the tuples to improve build speed

Re: [HACKERS] Hash index todo list item

2007-09-25 Thread Kenneth Marshall
On Tue, Sep 25, 2007 at 03:35:47PM +0100, Gregory Stark wrote: Kenneth Marshall [EMAIL PROTECTED] writes: On Thu, Sep 20, 2007 at 05:12:45PM -0700, Tom Raney wrote: Using our implementation, build times and index sizes are comparable with btree index build times and index sizes

Re: [HACKERS] Hash index todo list item

2007-09-09 Thread Kenneth Marshall
On Sun, Sep 02, 2007 at 10:41:22PM -0400, Tom Lane wrote: Kenneth Marshall [EMAIL PROTECTED] writes: ... This is the rough plan. Does anyone see anything critical that is missing at this point? Sounds pretty good. Let me brain-dump one item on you: one thing that hash currently has over

Re: [HACKERS] Hash index todo list item

2007-09-08 Thread Kenneth Marshall
On Fri, Sep 07, 2007 at 10:36:41AM -0400, Brian Hurt wrote: Kenneth Marshall wrote: I understand that a hash value is a many-to-one mapping. That is the point of the flag in the index. The flag means that there is only one item in the heap corresponding to that hash value. In this case we

Re: [HACKERS] Hash index todo list item

2007-09-08 Thread Kenneth Marshall
On Sat, Sep 08, 2007 at 05:14:09PM -0400, Mark Mielke wrote: Kenneth Marshall wrote: Continuing this train of thought While it would make sense for larger keys to store the hash in the index, if the key is smaller, particularly if it is of fixed size, it would make sense to store the key

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Fri, Sep 07, 2007 at 09:50:07AM -0400, Mark Mielke wrote: Kenneth Marshall wrote: On Thu, Sep 06, 2007 at 11:56:25PM -0700, Neil Conway wrote: You might find this patch useful: http://archives.postgresql.org/pgsql-patches/2005-05/msg00164.php ... Unfortunately, the patch doesn't

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Fri, Sep 07, 2007 at 12:55:37PM +0100, Heikki Linnakangas wrote: Neil Conway wrote: You might find this patch useful: http://archives.postgresql.org/pgsql-patches/2005-05/msg00164.php Oh, I had forgot about that. It implements the just store the hash in the index idea; it

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Thu, Sep 06, 2007 at 11:56:25PM -0700, Neil Conway wrote: On Sun, 2007-02-09 at 13:04 -0500, Kenneth Marshall wrote: 2. Evaluate the performance of different hash index implementations and/or changes to the current implementation. My current plan is to keep the implementation

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Thu, Sep 06, 2007 at 11:56:25PM -0700, Neil Conway wrote: On Sun, 2007-02-09 at 13:04 -0500, Kenneth Marshall wrote: 2. Evaluate the performance of different hash index implementations and/or changes to the current implementation. My current plan is to keep the implementation

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Fri, Sep 07, 2007 at 10:36:41AM -0400, Brian Hurt wrote: Kenneth Marshall wrote: I understand that a hash value is a many-to-one mapping. That is the point of the flag in the index. The flag means that there is only one item in the heap corresponding to that hash value. In this case we

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Fri, Sep 07, 2007 at 11:08:13AM -0400, Brian Hurt wrote: Kenneth Marshall wrote: How likely is it that you will get a hash collision, two strings that are different that will hash to the same value? To avoid this requires a very large hash key (128 bits, minimum)- otherwise you

Re: [HACKERS] Hash index todo list item

2007-09-07 Thread Kenneth Marshall
On Fri, Sep 07, 2007 at 10:30:30AM -0400, Mark Mielke wrote: Kenneth Marshall wrote: I understand that a hash value is a many-to-one mapping. That is the point of the flag in the index. The flag means that there is only one item in the heap corresponding to that hash value. In this case we

Re: [HACKERS] Hash index todo list item

2007-09-06 Thread Kenneth Marshall
On Thu, Sep 06, 2007 at 11:53:45AM -0400, Mark Mielke wrote: Hannu Krosing wrote: One approahc is not to mix hashes, but to partition the hash, so that each column gets its N bits in the hash. How does that help? You still need all the keys to find out which bucket to look in.

Re: [HACKERS] Hash index todo list item

2007-09-05 Thread Kenneth Marshall
On Sun, Sep 02, 2007 at 01:04:04PM -0500, Kenneth Marshall wrote: Dear PostgreSQL Hackers: After following the hackers mailing list for quite a while, I am going to start investigating what will need to be done to improve hash index performance. Below are the pieces of this project that I

Re: [HACKERS] Hash index todo list item

2007-09-03 Thread Kenneth Marshall
On Sun, Sep 02, 2007 at 10:41:22PM -0400, Tom Lane wrote: Kenneth Marshall [EMAIL PROTECTED] writes: ... This is the rough plan. Does anyone see anything critical that is missing at this point? Sounds pretty good. Let me brain-dump one item on you: one thing that hash currently has over

Re: [HACKERS] Hash index todo list item

2007-09-03 Thread Kenneth Marshall
On Mon, Sep 03, 2007 at 10:33:54AM +0100, Simon Riggs wrote: This is the rough plan. Does anyone see anything critical that is missing at this point? Please send me any suggestions for test data and various performance test ideas, since I will be working on that first. Sounds good.

Re: [HACKERS] Hash index todo list item

2007-09-03 Thread Kenneth Marshall
On Mon, Sep 03, 2007 at 05:20:34PM -0700, Ben Tilly wrote: That raises a very random thought. One of the nicer features of Oracle is the ability to have function-based indexes. So you could index, say, trim(lower(person.name)). There are a *lot* of practical situations where that comes in

[HACKERS] Hash index todo list item

2007-09-02 Thread Kenneth Marshall
Dear PostgreSQL Hackers: After following the hackers mailing list for quite a while, I am going to start investigating what will need to be done to improve hash index performance. Below are the pieces of this project that I am currently considering: 1. Characterize the current hash index

Re: [HACKERS] tsearch2 in PostgreSQL 8.3?

2007-08-14 Thread Kenneth Marshall
On Tue, Aug 14, 2007 at 03:15:44PM -0400, Alvaro Herrera wrote: Bruce Momjian wrote: A lot of work has been done to try to get /contrib/tsearch2 into the core backend for 8.3, but we have hit a roadblock in how to handle multiple text search configurations. (FYI, the documentation is at

Re: [HACKERS] Query plan and execution time of a query

2007-07-21 Thread Kenneth Marshall
Mickael, Not knowing your query, it sounds like your method is working correctly. It is quite normal to have the initial query take longer than subsequent queries. This is a cache effect and is what databases, in general, strive for performance-wise. I suspect that the second time you run the

Re: [HACKERS] todo: Hash index creation

2007-07-07 Thread Kenneth Marshall
On Thu, Jul 05, 2007 at 12:26:45PM +0100, Heikki Linnakangas wrote: Kenneth Marshall wrote: I definitely agree with Tom's assessment. If we cannot need to make the hash index as performant as it is in theory, none of the other refinements are worth it. You would need to use BTree if you were

Re: [HACKERS] todo: Hash index creation

2007-06-27 Thread Kenneth Marshall
On Wed, Jun 27, 2007 at 08:36:54PM -0400, Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: [EMAIL PROTECTED] wrote: Is anyone currently working on this TODO item? During index creation, pre-sort the tuples to improve build speed If you want to work on hash indexes, though,

Re: [PATCHES] [HACKERS] Full page writes improvement, code update

2007-04-25 Thread Kenneth Marshall
On Wed, Apr 25, 2007 at 10:00:16AM +0200, Zeugswetter Andreas ADI SD wrote: 1) To deal with partial/inconsisitent write to the data file at crash recovery, we need full page writes at the first modification to pages after each checkpoint. It consumes much of WAL space. We need

Re: [HACKERS] Reduction in WAL for UPDATEs

2007-03-28 Thread Kenneth Marshall
On Wed, Mar 28, 2007 at 08:07:14AM +0100, Simon Riggs wrote: On Tue, 2007-03-27 at 20:48 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: It seems possible to reduce overall WAL volume by roughly 25% on common workloads by optimising the way UPDATE statements generate WAL.

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-26 Thread Kenneth Marshall
On Wed, Jan 24, 2007 at 07:30:05PM -0500, Tom Lane wrote: Kenneth Marshall [EMAIL PROTECTED] writes: Not that I am aware of. Even extending the relation by one additional block can make a big difference in performance Do you have any evidence to back up that assertion? It seems a bit

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-24 Thread Kenneth Marshall
On Tue, Jan 23, 2007 at 09:01:41PM -0600, Jim Nasby wrote: On Jan 22, 2007, at 6:53 PM, Kenneth Marshall wrote: The default should be approximately the OS standard read-ahead amount. Is there anything resembling a standard across the OSes we support? Better yet, is there a standard call

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-24 Thread Kenneth Marshall
On Mon, Jan 22, 2007 at 05:11:03PM -0600, Jim C. Nasby wrote: On Mon, Jan 22, 2007 at 12:17:39PM -0800, Ron Mayer wrote: Gregory Stark wrote: Actually no. A while back I did experiments to see how fast reading a file sequentially was compared to reading the same file sequentially but

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Kenneth Marshall
On Mon, Jan 22, 2007 at 06:42:09PM +, Simon Riggs wrote: On Mon, 2007-01-22 at 13:27 -0500, Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Kenneth Marshall
On Mon, Jan 22, 2007 at 07:24:20PM +, Heikki Linnakangas wrote: Kenneth Marshall wrote: On Mon, Jan 22, 2007 at 06:42:09PM +, Simon Riggs wrote: Hold that thought! Read Heikki's Piggyback VACUUM idea on new thread... There may be other functions that could leverage a similar sort

Re: [PATCHES] [HACKERS] [Fwd: Index Advisor]

2007-01-08 Thread Kenneth Marshall
One problem with only putting this information in the system logs is that when we provide database services to a member of our community we do not actually give them an account of the DB server or log server. This means that this very useful information would need to be passed through an

Re: [HACKERS] effective_cache_size vs units

2006-12-20 Thread Kenneth Marshall
On Tue, Dec 19, 2006 at 10:12:34PM +, Gregory Stark wrote: Tom Lane [EMAIL PROTECTED] writes: Magnus Hagander [EMAIL PROTECTED] writes: Oh, you mean MB vs Mb. Man, it had to be that simple :) ISTM we had discussed whether guc.c should accept units strings in a case-insensitive

Re: column ordering, was Re: [HACKERS] [PATCHES] Enums patch v2

2006-12-20 Thread Kenneth Marshall
On Wed, Dec 20, 2006 at 01:26:59PM +0100, Peter Eisentraut wrote: Am Mittwoch, 20. Dezember 2006 04:44 schrieb Tom Lane: If you can show me a reasonably bulletproof or machine-checkable way to keep the two kinds of column numbers distinct, I'd be all for it. The only way I can see is to

Re: [HACKERS] New CRC algorithm: Slicing by 8

2006-10-26 Thread Kenneth Marshall
On Mon, Oct 23, 2006 at 05:23:27PM -0400, Tom Lane wrote: Mark Kirkwood [EMAIL PROTECTED] writes: Right - I think the regression is caused by libc and kernel being built with gcc 3.4.6 and the test program being built with gcc 4.1.2. Why do you think that? The performance of the CRC

Re: [HACKERS] horo(r)logy test fail on solaris (again and solved)

2006-09-27 Thread Kenneth Marshall
On Wed, Sep 27, 2006 at 04:09:18PM +0200, Zdenek Kotala wrote: Tom Lane napsal(a): Zdenek Kotala [EMAIL PROTECTED] writes: The problem was generated, because -fast option was set only for the compiler and not for the linker. Linker takes wrong version of libraries. If -fast is set for

Re: BugTracker (Was: Re: [HACKERS] 8.2 features status)

2006-08-17 Thread Kenneth Marshall
On Wed, Aug 16, 2006 at 06:52:21AM +0200, Peter Eisentraut wrote: Tom Lane wrote: that the bug tracker would have to have a reasonable output email capability, but I'd not necessarily insist on being able to input to it by mail. Red Hat's present bugzilla system could be described that

Re: BugTracker (Was: Re: [HACKERS] 8.2 features status)

2006-08-17 Thread Kenneth Marshall
On Wed, Aug 16, 2006 at 01:22:43PM +0900, Michael Glaesemann wrote: On Aug 16, 2006, at 12:29 , Tom Lane wrote: So my current take on this would be that the bug tracker would have to have a reasonable output email capability, but I'd not necessarily insist on being able to input to it by

Re: BugTracker (Was: Re: [HACKERS] 8.2 features status)

2006-08-17 Thread Kenneth Marshall
, Kenneth Marshall wrote: RT is easy to setup/configure/use and works well with PostgreSQL as the backend. CPAN uses it for their bug tracker. Was there a list of features and requirements? I don't know if we ever came up with one, but I know that the big deal killer for a bug tracker

Re: BugTracker (Was: Re: [HACKERS] 8.2 features status)

2006-08-15 Thread Kenneth Marshall
RT is easy to setup/configure/use and works well with PostgreSQL as the backend. CPAN uses it for their bug tracker. Was there a list of features and requirements? Ken On Tue, Aug 15, 2006 at 10:59:52AM -0300, Marc G. Fournier wrote: On Fri, 11 Aug 2006, Alvaro Herrera wrote: I am suggesting

Re: [HACKERS] 8.2 features status

2006-08-06 Thread Kenneth Marshall
On Fri, Aug 04, 2006 at 12:40:36PM -0400, Tom Lane wrote: Guillaume Smet [EMAIL PROTECTED] writes: And what about compression of on-disk sorting? That's purely a performance issue, which some people seem to want to define as not a new feature ... which is not *my* view of what's important

Re: [HACKERS] Hash indexes (was: On-disk bitmap index patch)

2006-08-03 Thread Kenneth Marshall
On Tue, Aug 01, 2006 at 02:26:18PM -0700, [EMAIL PROTECTED] wrote: Kenneth Marshall wrote: On Fri, Jul 28, 2006 at 12:14:49PM -0500, Jim C. Nasby wrote: On Thu, Jul 27, 2006 at 01:46:01PM -0400, Alvaro Herrera wrote: Jim Nasby wrote: On Jul 25, 2006, at 3:31 PM, Tom Lane wrote

Re: [HACKERS] Hash indexes (was: On-disk bitmap index patch)

2006-07-30 Thread Kenneth Marshall
On Fri, Jul 28, 2006 at 12:14:49PM -0500, Jim C. Nasby wrote: On Thu, Jul 27, 2006 at 01:46:01PM -0400, Alvaro Herrera wrote: Jim Nasby wrote: On Jul 25, 2006, at 3:31 PM, Tom Lane wrote: Hannu Krosing [EMAIL PROTECTED] writes: What would be the use-case for hash indexes ? And what

Re: [HACKERS] A couple thoughts about btree fillfactor

2006-07-11 Thread Kenneth Marshall
On Mon, Jul 10, 2006 at 12:36:34PM -0400, Tom Lane wrote: Now that the index options infrastructure is in, I am having a couple of second thoughts about the specific behavior that's been implemented, particularly for btree fillfactor. 1. ... I'm thinking we could change the nbtsort.c code

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-22 Thread Kenneth Marshall
On Sun, Jun 18, 2006 at 05:26:16PM -0400, Tom Lane wrote: I wrote: PFC [EMAIL PROTECTED] writes: So, the proposal : On executing a command, Backend stores the command string, then overwrites the counter with (counter + 1) and with the timestamp of command start. Periodically, like

Re: [HACKERS] Rethinking stats communication mechanisms

2006-06-22 Thread Kenneth Marshall
On Sun, Jun 18, 2006 at 11:07:41PM -0400, Tom Lane wrote: Bort, Paul [EMAIL PROTECTED] writes: Anyone know a variant of this that really works? Here's a theory: If the counter is bumped to an odd number before modification, and an even number after it's done, then the reader will know

Re: [HACKERS] Duplicate rows sneaking in despite PRIMARY KEY / UNIQUE constraint

2006-06-07 Thread Kenneth Marshall
Travis, We have used postgresql 7.4, 8.0, and 8.1 with DSPAM and have never had a single problem like you are describing. In the past on this mailing list, these sorts of issues have been caused by hardware problems on the DB server in some cases. Good luck with tracking it down. Ken On Tue,

Re: [HACKERS] More thoughts about planner's cost estimates

2006-06-02 Thread Kenneth Marshall
Josh, Greg, and Tom, I do not know how sensitive the plans will be to the correlation, but one thought might be to map the histogram X histogram correlation to a square grid of values. Then you can map them to an integer which would give you 8 x 8 with binary values, a 5 x 5 with 4 values per

Re: [HACKERS] SpeedComparison

2006-02-12 Thread Kenneth Marshall
On Sat, Feb 11, 2006 at 09:21:43PM +0100, Jochem van Dieten wrote: On 2/11/06, Andrej Ricnik-Bay wrote: Has anyone here seen this one before? Do the values appear realistic? http://www.sqlite.org/cvstrac/wiki?p=SpeedComparison The values appear to originate from an intrsinsically

Re: [HACKERS] Improving N-Distinct estimation by ANALYZE

2006-01-08 Thread Kenneth Marshall
On Fri, Jan 06, 2006 at 06:36:52PM -0500, Greg Stark wrote: Josh Berkus josh@agliodbs.com writes: These numbers don't make much sense to me. It seems like 5% is about as slow as reading the whole file which is even worse than I expected. I thought I was being a bit pessimistic to think

Re: [HACKERS] Warm-cache prefetching

2005-12-10 Thread Kenneth Marshall
On Fri, Dec 09, 2005 at 11:32:48AM -0500, Bruce Momjian wrote: Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: I can see that being useful for a single-user application that doesn't have locking or I/O bottlenecks, and doesn't have a multi-stage design like a database. Do

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Kenneth Marshall
On Fri, Dec 09, 2005 at 10:37:25AM -0500, Bruce Momjian wrote: Kenneth Marshall wrote: The main benefit of pre-fetching optimization is to allow just- in-time data delivery to the processor. There are numerous papers illustrating the dramatic increase in data throughput by using

Re: [HACKERS] Warm-cache prefetching

2005-12-09 Thread Kenneth Marshall
The main benefit of pre-fetching optimization is to allow just- in-time data delivery to the processor. There are numerous papers illustrating the dramatic increase in data throughput by using datastructures designed to take advantage of prefetching. Factors of 3-7 can be realized and this can

Re: [HACKERS] Shared locking in slru.c

2005-12-01 Thread Kenneth Marshall
On Wed, Nov 30, 2005 at 03:23:55PM -0500, Tom Lane wrote: Kenneth Marshall [EMAIL PROTECTED] writes: ... In pseudo-code, the operations to read the control information are: WriteControl: 1. Set latch. 2. Update control information 3. Increment latch version number. 4. Unset latch

Re: [HACKERS] Shared locking in slru.c

2005-12-01 Thread Kenneth Marshall
On Wed, Nov 30, 2005 at 01:53:13PM -0500, Tom Lane wrote: I've been looking at various ways to resolve this, but one thing that seems promising is to hack slru.c to take the control lock in shared mode, not exclusive mode, for read-only accesses to pages that are already in memory. The vast

Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.1 RC1

2005-11-06 Thread Kenneth Marshall
Marc, Okay, I found an OpenSSL-0.9.7 and readline library. The IRIX 6.5 IP35 also passed with the OpenSSL and readline included. This is with the IRIX cc and not gcc. Ken On Mon, Oct 24, 2005 at 11:51:26AM -0300, Marc G. Fournier wrote: We have released a Release Candidate 1 of the

Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.1 Beta 4

2005-11-06 Thread Kenneth Marshall
Marc, I just finished a build with the 8.1beta4 for IRIX 6.5 but without the nuances. We do not really use SGI other than in special circumstances but the regression test passed all tests: configure --without-readline using IRIX cc. Ken On Mon, Oct 24, 2005 at 11:51:26AM -0300, Marc G.

Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.1 Beta 4

2005-10-30 Thread Kenneth Marshall
Marc, I just finished a build with the 8.1beta4 for Solaris 8 (SPARC) with: OpenSSL 0.9.8 Heimdal 0.7 tcl/tk-8.4.8 perl-5.8.4 using gcc-3.4.3 and the following compile command: gcc -O3 -DOPENSSL_DISABLE_OLD_DES_SUPPORT -fno-sched-interblock All tests passed successfully. Ken On Mon, Oct

Re: [pgsql-advocacy] [HACKERS] MySQL to PostgreSQL for SugarCRM

2005-08-09 Thread Kenneth Marshall
Compiere also runs on PostgreSQL and has gotten some good press. It previously only support Oracle. Ken On Tue, Aug 02, 2005 at 08:53:52PM +0300, Hannu Krosing wrote: On L, 2005-07-30 at 22:26 -0400, Denis Lussier wrote: Thanks, I'll check it out. I didn't see much evidence on the SugarCRM

Re: [HACKERS] Checkpoint cost, looks like it is WAL/CRC

2005-07-07 Thread Kenneth Marshall
On Thu, Jul 07, 2005 at 11:36:40AM -0400, Tom Lane wrote: Greg Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: What we *could* do is calculate a page-level CRC and store it in the page header just before writing out. Torn pages would then manifest as a wrong CRC on

Re: [HACKERS] 2PC transaction id

2005-07-02 Thread Kenneth Marshall
It certainly helps if you need to debug a process. Ken On Fri, Jul 01, 2005 at 09:06:03PM +0300, Heikki Linnakangas wrote: On Fri, 1 Jul 2005, Oliver Jowett wrote: PS: noticed in passing: psql's help doesn't seem to know about the 2PC command syntax yet. True. Should we add support

Re: [HACKERS] commit_delay, siblings

2005-06-29 Thread Kenneth Marshall
On Wed, Jun 29, 2005 at 08:14:36AM +0100, Simon Riggs wrote: Group commit is a well-documented technique for improving performance, but the gains only show themselves on very busy systems. It is possible in earlier testing any apparent value was actually hidden by the BufMgrLock issues we

<    1   2   3   >