Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-13 Thread Robert Haas
On Thu, Jan 9, 2014 at 10:57 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I wrote: Robert Haas wrote: Hmm, fair point. But I'm still not convinced that we really need to add extra accounting for this. What's wrong with just reporting the number of exact and lossy pages? No. I

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-09 Thread Etsuro Fujita
I wrote: Robert Haas wrote: Hmm, fair point. But I'm still not convinced that we really need to add extra accounting for this. What's wrong with just reporting the number of exact and lossy pages? No. I intended to show the desired memory space for a TIDBitmap rather than the peak

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-07 Thread Robert Haas
On Mon, Jan 6, 2014 at 9:40 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Robert Haas wrote: I spent some time looking at this tonight. I don't think the value that is displayed for the bitmap memory tracking will be accurate in complex cases. The bitmap heap scan may sit on top of

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-07 Thread Etsuro Fujita
Robert Haas wrote: On Mon, Jan 6, 2014 at 9:40 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Thank you for taking time to look at this patch. The peak memory usage for the discarded bitmap *can* be reflected in the number displayed for the bitmap heap scan by the following code in

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-06 Thread Etsuro Fujita
Robert Haas wrote: I spent some time looking at this tonight. I don't think the value that is displayed for the bitmap memory tracking will be accurate in complex cases. The bitmap heap scan may sit on top of one or more bitmap-and or bitmap-or nodes. When a bitmap-and operation happens,

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-02 Thread Andres Freund
On 2014-01-01 21:15:46 -0500, Robert Haas wrote: [ sensible reasoning ] However, I'm not sure it's really worth it. I think what people really care about is knowing whether the bitmap lossified or not, and generally how much got lossified. The counts of exact and lossy pages are sufficient

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 4:27 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-01 21:15:46 -0500, Robert Haas wrote: [ sensible reasoning ] However, I'm not sure it's really worth it. I think what people really care about is knowing whether the bitmap lossified or not, and generally

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-01 Thread Robert Haas
On Fri, Dec 27, 2013 at 1:47 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I wrote: Robert Haas wrote: I'd be wary of showing a desired value unless it's highly likely to be accurate. The desired value is accurately estimated based on (a) the total number of exact/lossy pages

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-26 Thread Etsuro Fujita
I wrote: Robert Haas wrote: I'd be wary of showing a desired value unless it's highly likely to be accurate. The desired value is accurately estimated based on (a) the total number of exact/lossy pages stored in the TIDBitmap and (b) the following equation in tbm_create(), except

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-13 Thread Etsuro Fujita
I wrote: Robert Haas wrote: I'd be wary of showing a desired value unless it's highly likely to be accurate. The desired value is accurately estimated based on (a) the total number of exact/lossy pages stored in the TIDBitmap and (b) the following equation in tbm_create(), except for the

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-08 Thread Etsuro Fujita
Robert Haas wrote: On Fri, Dec 6, 2013 at 5:02 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Though at first I agreed on this, while working on this I start to think information about (2) is enough for tuning work_mem. Here are examples using a version under development, where

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-07 Thread Robert Haas
On Fri, Dec 6, 2013 at 5:02 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Though at first I agreed on this, while working on this I start to think information about (2) is enough for tuning work_mem. Here are examples using a version under development, where Bitmap Memory Usage means

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-12-06 Thread Etsuro Fujita
I wrote: Amit Khandekar wrote: Yes, I agree that rather than looking at the bitmap heap scan to track the number of pages, we should look somewhere in the underlying index scan. Yes, we should get a constant number of index pages regardless of the actual parent table rows. I agree with

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-26 Thread Etsuro Fujita
Amit Khandekar wrote: On 25 November 2013 13:37, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: So, my question is, we should show the number of exact/lossy pages in a TIDBitmap, not the number of these pages that has been fetched in the bitmap heap scan? Yes, I agree that rather than

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-25 Thread Etsuro Fujita
From: Amit Khandekar [mailto:amit.khande...@enterprisedb.com] On 1 November 2013 16:32, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: From: Fujii Masao [mailto:masao.fu...@gmail.com] I'm not sure if it's good idea to show the number of the fetches because it seems difficult to tune

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-25 Thread Amit Khandekar
On 25 November 2013 13:37, Etsuro Fujita fujita.ets...@lab.ntt.co.jpwrote: Reconsidering that, I wish to know your opinion. The patch shows the number of exact/lossy pages that has been fetched in a bitmap heap scan. But the number varies with the fraction of tuples to be retrieved like the

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-24 Thread Amit Khandekar
On 1 November 2013 16:32, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: From: Fujii Masao [mailto:masao.fu...@gmail.com] This is what I'm looking for! This feature is really useful for tuning work_mem when using full text search with pg_trgm. I'm not sure if it's good idea to show

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-11-01 Thread Etsuro Fujita
From: Fujii Masao [mailto:masao.fu...@gmail.com] This is what I'm looking for! This feature is really useful for tuning work_mem when using full text search with pg_trgm. I'm not sure if it's good idea to show the number of the fetches because it seems difficult to tune work_mem from that

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2013-10-31 Thread Fujii Masao
On Thu, Oct 31, 2013 at 7:54 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Hi, I think that lossy-heap-block information for a bitmap heap scan, not just Rows Removed by Index Recheck information, would also be a clue used to tune work_mem for better performance especially when the