Re: [HACKERS] Bitmap index thoughts

2006-12-28 Thread Jie Zhang
On 12/27/06 3:10 AM, Gavin Sherry [EMAIL PROTECTED] wrote: On Wed, 27 Dec 2006, Heikki Linnakangas wrote: Gavin Sherry wrote: On Tue, 26 Dec 2006, Heikki Linnakangas wrote: for typical bitmap index use cases and most of the needed pages should stay in memory, but could we simplify this?

Re: [HACKERS] Bitmap index thoughts

2006-12-28 Thread Jie Zhang
On 12/27/06 3:16 AM, Gavin Sherry [EMAIL PROTECTED] wrote: On Wed, 27 Dec 2006, Heikki Linnakangas wrote: Jie Zhang wrote: The bitmap data segment sounds good in terms of space. The problem is that one bitmap is likely to occupy more pages than before, which may hurt the query performance

Re: [HACKERS] Bitmap index thoughts

2006-12-27 Thread Jie Zhang
And instead of having separate LOV pages that store a number of LOV items, how about storing each LOV item on a page of it's own, and using the rest of the page to store the last chunk of the bitmap. That would eliminate one page access, but more importantly, maybe we could then get rid of

Re: [HACKERS] Bitmap index status

2006-10-18 Thread Jie Zhang
On 10/18/06 2:41 AM, Heikki Linnakangas [EMAIL PROTECTED] wrote: Hi, I don't want to harass you :), but what's the status with the bitmap index code? Is there something I can do to help? Not at all. We will send you the new patch soon. Your comments are most appreciated. Thanks, Jie

Re: [HACKERS] Bitmap index status

2006-09-25 Thread Jie Zhang
Hi Mark, Thanks for doing the test. I checked out the link you provided below. I am a little confused about the goal of these tests. Do you plan to test the overall performance of postgreSQL on handling TPC-H queries? Thanks, Jie On 9/22/06 3:45 PM, Mark Wong [EMAIL PROTECTED] wrote: Jie

Re: [HACKERS] Bitmap index status

2006-09-23 Thread Jie Zhang
Gavin Heikki, The handling of stream and hash bitmaps looks pretty complicated to me. All the bitmap-related nodes have logic to handle both types slightly differently. It all seems to come down to that if a subnode (or amgetbitmap in a bitmap index scan node) returns a StreamBitmap, the

Re: [HACKERS] Bitmap index status

2006-09-20 Thread Jie Zhang
On 9/19/06 5:15 AM, Gavin Sherry [EMAIL PROTECTED] wrote: On Tue, 19 Sep 2006, Heikki Linnakangas wrote: Jie Zhang wrote: Hi Heikki and all, Please find the latest bitmap index patch in the attachment. This patch is generated against the postgresql cvs head. Thanks. The handling

Re: [HACKERS] Bitmap index status

2006-09-17 Thread Jie Zhang
Hi Heikki and all, I just sent the latest bitmap index patch to the list. I am not sure if there is any size limit for this mailing list. If you have received my previous email, please let me know. Thanks, Jie On 9/12/06 2:43 AM, Heikki Linnakangas [EMAIL PROTECTED] wrote: Hi, What's the

Re: [HACKERS] Bitmap index status

2006-09-12 Thread Jie Zhang
Hi Heikki, Gavin and I are trying to merge our changes together this week. We will post a new patch by the end of this week. This patch will include some style fixes, bug fixes, and the stream bitmap implementation. I will look into the problems you have mentioned in this email. Yes, vacuum

Re: [HACKERS] [PATCHES] WIP: bitmap indexes

2006-08-17 Thread Jie Zhang
It occurs to me that what tbm_begin_iterate really is is a constructor for a stream bitmap object that reads out the contents of a tbm bitmap (we need a decent name for the non-stream data structure ... maybe hash bitmap?). If we think of it like that then we can unify the ideas some more.

Re: [HACKERS] [PATCHES] WIP: bitmap indexes

2006-08-17 Thread Jie Zhang
On 8/17/06 5:54 AM, Tom Lane [EMAIL PROTECTED] wrote: Jie Zhang [EMAIL PROTECTED] writes: This sounds great. One thing I am concern about is that this will add the dependency of node types into the access methods. If we still keep nodeBitmapIndexscan and let it do the bitmap construction

Re: [HACKERS] [PATCHES] WIP: bitmap indexes

2006-08-17 Thread Jie Zhang
On 8/17/06 12:29 PM, Tom Lane [EMAIL PROTECTED] wrote: Jie Zhang [EMAIL PROTECTED] writes: This sounds good. Another problem is about ScalarArrayOpExpr support in current nodeBitmapIndexscan. This will not work for stream bitmaps. Sure it will; it's just an OR. Yes, it is just

Re: [HACKERS] [PATCHES] WIP: bitmap indexes

2006-08-16 Thread Jie Zhang
On 8/15/06 6:18 AM, Tom Lane [EMAIL PROTECTED] wrote: Gavin Sherry [EMAIL PROTECTED] writes: On Mon, 14 Aug 2006, Tom Lane wrote: Correct me if I'm wrong, but isn't the patch's present hacking on the executor intended to make it happen like this? Not really. It reads ahead on the bitmap

Re: [HACKERS] On-disk bitmap index patch

2006-07-27 Thread Jie Zhang
On 7/26/06 10:14 PM, Tom Lane [EMAIL PROTECTED] wrote: Mark Kirkwood [EMAIL PROTECTED] writes: An obvious deduction is that the TPCH dataset is much more amenable to run compression than my synthetic Zipfian data was. The interesting question is how well real datasets are run compressable,

Re: [HACKERS] On-disk bitmap index patch

2006-07-27 Thread Jie Zhang
On 7/26/06 11:50 PM, Tom Lane [EMAIL PROTECTED] wrote: Jie Zhang [EMAIL PROTECTED] writes: On 7/26/06 10:14 PM, Tom Lane [EMAIL PROTECTED] wrote: ... A nonuniform distribution would probably mean that some of the bitmaps compress better-than-expected and others worse. I have no idea how

Re: [HACKERS] On-disk bitmap index patch

2006-07-26 Thread Jie Zhang
On 7/26/06 8:55 PM, Luke Lonergan [EMAIL PROTECTED] wrote: Tom, On 7/26/06 7:26 AM, Tom Lane [EMAIL PROTECTED] wrote: I wonder whether they oughtn't use 16-bit instead of 8-bit HRL_WORDs We tried variations from 8-bit to 32-bit and came to the conclusion that 8-bit was a better match

Re: [HACKERS] On-disk bitmap index patch

2006-07-24 Thread Jie Zhang
Thanks Tom and Gavin for your comments! Yes, this patch is generated against 8.2 in a short time. As long as the code is working, I post the patch to get some comments and help. * The xlog routines need help; they seem to not be updated for recent changes in the API for xlog recovery code.

Re: [HACKERS] On-disk bitmap index patch

2006-07-24 Thread Jie Zhang
On 7/24/06 6:59 AM, Hannu Krosing [EMAIL PROTECTED] wrote: Ühel kenal päeval, P, 2006-07-23 kell 20:25, kirjutas Tom Lane: Gavin Sherry [EMAIL PROTECTED] writes: On Sun, 23 Jul 2006, Tom Lane wrote: However, the main problem I've got with this is that a new index AM is a pretty large

Re: [HACKERS] On-disk bitmap index patch

2006-07-24 Thread Jie Zhang
On 7/24/06 6:04 PM, Bruce Momjian [EMAIL PROTECTED] wrote: Jie Zhang wrote: IIRC they quoted the cardinality of 1 as something that is still faster than btree for several usecases. And also for AND-s of several indexes, where indexes are BIG, your btree indexes may be almost as big

[HACKERS] On-disk bitmap index patch

2006-07-18 Thread Jie Zhang
Hi, I have posted a patch to the CVS head for on-disk bitmap index to pgsql-patches. If this can get in 8.2, that would be great. Any comments and suggestions are welcome. I still need to add several items: (1) README file in src/backend/access/bitmap. (2) Bitmap index documentation. (3) Hiding

[HACKERS] An In-memory Bitmap Index Bug

2005-07-23 Thread Jie Zhang
in this email. Sincerely, Jie Zhang Greenplum pgsql-bitmapscan-bugfix.patch Description: pgsql-bitmapscan-bugfix.patch ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq