Re: [HACKERS] Patch queue triage
Simon Riggs wrote: > On Wed, 2007-09-12 at 17:47 -0400, Bruce Momjian wrote: > > For those who have forgotten the progress we have made toward 8.3, here > > are the open patches we had for 8.3 as of May 1, 2006: > > Could you please issue a list of open items for 8.3? > > I want to check whether you are waiting on me for anything and which > things have been deferred to next release. I am working on putting them all in the patch queue now. -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Patch queue triage
Pavan Deolasee wrote: > On 9/13/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > > > > For those who have forgotten the progress we have made toward 8.3, here > > are the open patches we had for 8.3 as of May 1, 2006: > > > > > You mean May 1, 2007 ;-) Yea, sorry. -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Patch queue triage
On Wed, 2007-09-12 at 17:47 -0400, Bruce Momjian wrote: > For those who have forgotten the progress we have made toward 8.3, here > are the open patches we had for 8.3 as of May 1, 2006: Could you please issue a list of open items for 8.3? I want to check whether you are waiting on me for anything and which things have been deferred to next release. Thanks, -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Patch queue triage
On 9/13/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > For those who have forgotten the progress we have made toward 8.3, here > are the open patches we had for 8.3 as of May 1, 2006: > > You mean May 1, 2007 ;-) Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com
Re: [HACKERS] Patch queue triage
For those who have forgotten the progress we have made toward 8.3, here are the open patches we had for 8.3 as of May 1, 2006: --- Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > FYI, Tom, Heikki, I need one of you to post the list of patches and > > where we think we are on each one, even if the list is imperfect. > > This message is an attempt to sort out which patch queue entries have > no hope of getting into 8.3 (and so we shouldn't spend more time on them > right now), which ones can get in but are waiting on their authors for > more work, and which ones are ready for immediate review. > > You'll notice that numerically quite a lot of the patches fall into the > "dead" category. This isn't actually all that surprising because if > they were apply-able they'd have gotten in. (It's not like we've > completely neglected applying patches for the last six months...) > However, many of the remaining live patches are huge ones, like HOT and > delayed commit, and are going to consume considerable review effort > (again, if they didn't, they might have been in already). > > The bottom line is that we are desperately in need of more review > manpower. I'm pleased to report that Heikki Linnakangas has promised to > devote most of the next few weeks to helping review, and has already > picked out some patches he feels qualified to work on, as you'll see > below. I have also been coordinating reviewing assignments off-list with > Neil and Magnus to avoid duplication of effort. But I'd like to encourage > everyone who's done any backend hacking to look at the live patches not > shown as assigned to someone specific. The more eyeballs the better; > anything you catch is something someone else might miss. Also, several > of the open patches are in need of more performance testing, so if you > can help out in that fashion please do so. > > With that, the patches: > > > * Re: [pgsql-patches] [PATCHES] [HACKERS] [Fwd: Index Advisor] >/Gurjeet Singh/ > > I don't think this can be applied in anything like its current state. > The internal interface to the planner is not very good, and ditto for the > user API. What I would suggest trying to do is get a set of plugin hooks > defined for the planner that would allow the advisor to be implemented > entirely as an external module, and then let it be worked on as a > pgfoundry project. I have some ideas about appropriate design of the > plugin hooks (as mentioned in my review message of a month ago), but I > have to say that getting that done for 8.3 is lower-priority than some of > the other patches. > > * [pgsql-patches] Ctid chain following enhancement >/Pavan Deolasee/ > > I'm not very excited about this --- it seems to me to complicate the code > in some places that are not in fact performance-critical. While it > doesn't seem likely to break things, I'm not in favor of reducing code > readability unless a measurable performance improvement can be shown. > Can we have some tests showing this is worthwhile? > > * [PATCHES] Error correction for n_dead_tuples > /ITAGAKI Takahiro/ > > Waiting for OldestXmin patch to be accepted or rejected. However, > regardless of the fate of that patch, I'm concerned that this one creates > an open-loop behavior in which the n_dead_tuples estimate will diverge > arbitrarily far from reality over time. I criticized the original > proposal on that basis, and I'm not convinced this version fixes it, > because of the fact that stats counter updates occur much later than the > actions they count. (My recent patch to rate-limit tabstat messages made > that problem worse, but it existed anyway.) What might make sense is for > vacuum to count the number of dead-but-not-removable tuples it skips over, > and apply that as the value of n_dead_tuples on receipt of the vacuum > message (instead of setting to zero as now). This is likely to be wrong > with respect to the actions of transactions running concurrently with the > vacuum, but I think so is the proposed patch; and at least in this form > the error certainly cannot accumulate across vacuum cycles. > > * [PATCHES] Table function support /Pavel Stehule/ > > Neil has promised to review this. AFAIK there are no showstoppers but > there are some disagreements on the details of the functionality. > > * [HACKERS] Grouped Index Tuples /Heikki Linnakangas/ > * [HACKERS] Grouped Index Tuples / Clustered Indexes >/Heikki Linnakangas/ > > Needs review. I'm not sure how many people besides Heikki have really > looked at this (I know I haven't). > > * Re: [PATCHES] POSIX shared memory support > /Chris Marcellino/ > > I'm not convinced that we want this at all. The original proposal was > to provide an alternative for platforms without SysV shmem support, > but the working versions of the patch fail to remove the SysV dependency, > and the porta
Re: [HACKERS] Patch queue triage
Heikki Linnakangas wrote: There are few things that we can separate easily, like CREATE INDEX related changes, VACUUM and VACUUM FULL related changed, space reuse related changes etc. Let me give it a shot. Did we ever get a broken up patch for this? Yes: http://archives.postgresql.org/pgsql-patches/2007-05/msg00065.php Thanks :). Sincerely, Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Patch queue triage
Joshua D. Drake wrote: Pavan Deolasee wrote: I suppose inserting HOT tuples without index maintenance is useful even if no changes to the space allocation is made is useful. It won't get the space usage but it would save on index thrashing. But that still implies all the code to handle scans, updates, index builds, etc. Those chunks could be separated out but you can't commit without them. There are few things that we can separate easily, like CREATE INDEX related changes, VACUUM and VACUUM FULL related changed, space reuse related changes etc. Let me give it a shot. Did we ever get a broken up patch for this? Yes: http://archives.postgresql.org/pgsql-patches/2007-05/msg00065.php -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Patch queue triage
On 5/17/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: Pavan Deolasee wrote: > > > There are few things that we can separate easily, like CREATE INDEX > related changes, VACUUM and VACUUM FULL related changed, space > reuse related changes etc. Let me give it a shot. Did we ever get a broken up patch for this? I broke the patch into 5 smaller, logical pieces and posted them on May 7th. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com
Re: [HACKERS] Patch queue triage
Joshua D. Drake wrote: > Tom Lane wrote: > > > At this point it seems nothing will be done about this issue for 8.3. > > > > * [PATCHES] plpgpsm /Pavel Stehule/ > > > > I think this has to be held for 8.4: it was submitted too late for the 8.3 > > feature deadline, and in fact I don't recall that there was any community > > discussion/consensus on accepting it into core at all. Another big > > problem is that it largely copies-and-pastes the plpgsql code, which I > > think is an unacceptable maintenance burden in the long run. We need to > > consider whether we can't refactor to avoid code duplication. > > Per my updated patch email to the lists yesterday, plus Toms' above > comments *and* Alvaro's comments to me when I asked Alexey to review > it... may I strongly suggest that we bounce this for further development > in 8.4. Agreed. Done. -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Patch queue triage
Pavan Deolasee wrote: I suppose inserting HOT tuples without index maintenance is useful even if no changes to the space allocation is made is useful. It won't get the space usage but it would save on index thrashing. But that still implies all the code to handle scans, updates, index builds, etc. Those chunks could be separated out but you can't commit without them. There are few things that we can separate easily, like CREATE INDEX related changes, VACUUM and VACUUM FULL related changed, space reuse related changes etc. Let me give it a shot. Did we ever get a broken up patch for this? Joshua D. Drake Thanks, Pavan -- EnterpriseDB http://www.enterprisedb.com -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Patch queue triage
Tom Lane wrote: At this point it seems nothing will be done about this issue for 8.3. * [PATCHES] plpgpsm /Pavel Stehule/ I think this has to be held for 8.4: it was submitted too late for the 8.3 feature deadline, and in fact I don't recall that there was any community discussion/consensus on accepting it into core at all. Another big problem is that it largely copies-and-pastes the plpgsql code, which I think is an unacceptable maintenance burden in the long run. We need to consider whether we can't refactor to avoid code duplication. Per my updated patch email to the lists yesterday, plus Toms' above comments *and* Alvaro's comments to me when I asked Alexey to review it... may I strongly suggest that we bounce this for further development in 8.4. Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Patch queue triage
Sorry for late responce due to very long vacation season in Japan. Tom Lane wrote: > > * Re: [PATCHES] [HACKERS] Full page writes improvement, code update > >/Koichi Suzuki/ > > > > I feel that we have to insist that this be modified to not create any WAL > > bloat in the pre-compression form. That will be more efficient and will > > avoid the whole argument about whether a switch is needed. There was also > > doubt about whether the external programs (pg_compresslog etc) were ready > > for prime time. At this point we could accept a patch that makes whatever > > small tweaks are needed to ensure that an incremental-format WAL record > > can be extracted from a full-page-write record, at least for the most > > common WAL record types for which compression is actually important. I > > think the actual compression/decompression programs could undergo further > > development on pgfoundry with an eye to merging them into core before 8.4 > > release. As suggested by Tom, I agree that WAL should not include "both" full page write and incremental (logical) log. I began to examine WAL record format to see if incremental log can be made from full page writes. It will be okay even before 8.4, if simplified patch to the core is accepted. I will post simplified patch to the core as follows: 1. Mark the flag to indicate that the WAL record is compressable from full page writes to incremental log. This flag will be set if a) It is not written during the hot backup, and b) Archive command is active, and c) WAL contains full page writes, and d) full_page_writes=on. No logical log will be written to WAL in this case, and 2. During recovery, xl_tot_len check will be skipped for compressed WAL records. With this patch, compress/decompress can be developped outside the core. I'd like to post them to PG foundary first for the review for 8.4. I'd be very grateful if this patch can be considered again. Best Regards; Koichi Suzuki ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Patch queue triage
On 5/2/07, Gregory Stark <[EMAIL PROTECTED]> wrote: Can we? I mean, sure you can break the patch up into chunks which might make it easier to read, but are any of the chunks useful alone? Well I agree, it would be a tough job. I can try and break the patch into several self-complete incremental patches which compile and work, but I am worried about missing something somewhere and/or inserting any bugs in the process. I suppose inserting HOT tuples without index maintenance is useful even if no changes to the space allocation is made is useful. It won't get the space usage but it would save on index thrashing. But that still implies all the code to handle scans, updates, index builds, etc. Those chunks could be separated out but you can't commit without them. There are few things that we can separate easily, like CREATE INDEX related changes, VACUUM and VACUUM FULL related changed, space reuse related changes etc. Let me give it a shot. Thanks, Pavan -- EnterpriseDB http://www.enterprisedb.com
Re: [HACKERS] Patch queue triage
On Tue, 2007-05-01 at 22:44 -0400, Tom Lane wrote: Nice summary Tom. > * Re: [PATCHES] [Fwd: Deferred Transactions, Transaction Guarantee > and COMMITwithout waiting] /Simon Riggs/ > > Simon is on the hook to submit an updated patch. I hope this one > makes it in, as it looks like a really nice performance improvement > for those who can use it; but the original patch seems overcomplicated. It will, but it will be next week now. Changes agreed though. > * [PATCHES] Minor recovery changes /Simon Riggs/ > > The submission mentions open issues --- when will those be resolved? > Should we apply the patch-so-far anyway? Patch is OK to go in as-is. There are open issues, but they need more thought and discussion and I regret won't happen in a reasonable timescale due to other work. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Patch queue triage
http://www.sigaev.ru/misc/tsearch_core-0.46.gz Patch is synced with current CVS HEAD and synced with bugfixes in contrib/tsearch2 -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Patch queue triage
Tom Lane wrote: * [pgsql-patches] Ctid chain following enhancement /Pavan Deolasee/ I'm not very excited about this --- it seems to me to complicate the code in some places that are not in fact performance-critical. While it doesn't seem likely to break things, I'm not in favor of reducing code readability unless a measurable performance improvement can be shown. Can we have some tests showing this is worthwhile? IIRC this patch was originally part of an old HOT patch, and it was submitted as a separate patch because it has some benefit on its own but more importantly getting it applied first would make the HOT patch slightly smaller. I'm not sure if the latest HOT patch requires or includes this change anymore. If not we should drop this. If it does, then let's deal with this before attacking the hard core of HOT. * [HACKERS] Grouped Index Tuples /Heikki Linnakangas/ * [HACKERS] Grouped Index Tuples / Clustered Indexes /Heikki Linnakangas/ Needs review. I'm not sure how many people besides Heikki have really looked at this (I know I haven't). The patch is ugly as it is. We need API changes to make it less ugly, I had hoped to discuss and reach consensus on them well before feature freeze, that's what the "indexam API proposal" and "Stream bitmaps" threads in the patch queue are all about. But those discussions and patches stalled, so the clustered index patch is still in the same ugly state. I'm afraid we're getting "past due" on clustered indexes. The patch isn't ready for committing as it is, and we still don't have agreement on the API changes or even on the design in general. :( * [HACKERS] Stream bitmaps /Heikki Linnakangas/ I think this is on hold unless a finished bitmap-index patch shows up; however there was some discussion of using this in support of clustered indexes, so maybe it's live anyway? Heikki? This particular thread is closely related to bitmap indexes. But see next item: * Re: [HACKERS] [PATCHES] Bitmapscan changes /Heikki Linnakangas/ I had objected to this on the grounds that it seemed to be covering only a narrow territory between HOT and bitmap indexes, but given the probability that one or even both of those won't make it, we need to give this one a second look. So: live, needs review. Are you talking about the patch I submitted at the beginning of that thread? Because the mail in the patch queue is actually about whether or not we want clustered indexes. I think the original "bitmapscan changes" patch I submitted is live and needs review, even if clustered indexes and bitmap indexes are rejected. It should give some performance benefit when you do bitmap ANDs with partially lossy bitmaps, and from setting bits directly in the bitmap in the indexam in one call, instead of calling amgetmulti many times. Though I never measured that. * [HACKERS] Indexam interface proposal /Heikki Linnakangas/ AFAICS this discussion died out with no actual patch submitted. This is part of clustered indexes.. This was my proposal of what the indexam API changes would be like. This patch is either live or dead together with clustered indexes. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Patch queue triage
Tom Lane wrote: Bruce Momjian <[EMAIL PROTECTED]> writes: FYI, Tom, Heikki, I need one of you to post the list of patches and where we think we are on each one, even if the list is imperfect. This message is an attempt to sort out which patch queue entries have no hope of getting into 8.3 (and so we shouldn't spend more time on them right now), which ones can get in but are waiting on their authors for more work, and which ones are ready for immediate review. Excellent list. I have a little time available now, so I'll work on some, starting with trying to complete arrays of complex types. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Patch queue triage
Another complexity is testing cases where the table fits in shared memory/RAM, and those that don't, and testing cases where heap fits in RAM only because we pushed things to TOAST, and cases where we have to do lots of TOAST access that doesn't fit in RAM. I wonder if it is even worth testing it because pushing more to TOAST probably means the more frequently accessed data is in RAM. Anyway, how is going to run these tests? --- Gregory Stark wrote: > > "Bruce Momjian" <[EMAIL PROTECTED]> writes: > > > Then, figure out where the gains on the non-TEXT field seem to diminish > > in usefulness. Basically, with a lower TOAST value, we are going to > > spend more time accessing the TEXT field, but the speedup for the > > non-TEXT field should be large enough win that we don't care. As the > > TEXT column becomes shorter, it has less affect on the non-TEXT access. > > I guess the key is to break down what it is we want to measure into several > parts. These can each be measured precisely for various sized TOASTed data. > > Costs: > > 1) cost of retrieving data from TOAST pointer versus retrieving data from >inline tuple. We just want the absolute time difference between the two >operations, not the percentage difference. > > 2) cost of creating TOAST pointer (ie, inserting a new tuple with a TOAST >pointer or updating a previously inlined tuple to have a TOASTed column). > > 3) cost of deleting a TOAST pointer (ie, deleting a tuple or updating a tuple >to no longer have a TOASTed column) > > 3) cost of deleting a tuple with an existing TOAST pointer (or updating a >tuple to be all inlined) versus deleting an plain tuple or updating a plain >tuple. > > Savings: > > 1) time savings accessing a tuple without retrieving the TOAST pointer versus >having to access the tuple with the data inlined. > > 2) time savings updating a tuple without modifying toasted data versus >updating same tuple with the data inlined in both versions. > > The plan you described would be testing costs 1 and savings 1 but I think we > need to continue to the others as well. > > Then the trick is to somehow make some argument about the frequency of the > various operations and the acceptable tradeoff. I think you're right that the > time spent accessing the data would be the most important metric. > > -- > Gregory Stark > EnterpriseDB http://www.enterprisedb.com > > > ---(end of broadcast)--- > TIP 6: explain analyze is your friend -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Patch queue triage
"Bruce Momjian" <[EMAIL PROTECTED]> writes: > Then, figure out where the gains on the non-TEXT field seem to diminish > in usefulness. Basically, with a lower TOAST value, we are going to > spend more time accessing the TEXT field, but the speedup for the > non-TEXT field should be large enough win that we don't care. As the > TEXT column becomes shorter, it has less affect on the non-TEXT access. I guess the key is to break down what it is we want to measure into several parts. These can each be measured precisely for various sized TOASTed data. Costs: 1) cost of retrieving data from TOAST pointer versus retrieving data from inline tuple. We just want the absolute time difference between the two operations, not the percentage difference. 2) cost of creating TOAST pointer (ie, inserting a new tuple with a TOAST pointer or updating a previously inlined tuple to have a TOASTed column). 3) cost of deleting a TOAST pointer (ie, deleting a tuple or updating a tuple to no longer have a TOASTed column) 3) cost of deleting a tuple with an existing TOAST pointer (or updating a tuple to be all inlined) versus deleting an plain tuple or updating a plain tuple. Savings: 1) time savings accessing a tuple without retrieving the TOAST pointer versus having to access the tuple with the data inlined. 2) time savings updating a tuple without modifying toasted data versus updating same tuple with the data inlined in both versions. The plan you described would be testing costs 1 and savings 1 but I think we need to continue to the others as well. Then the trick is to somehow make some argument about the frequency of the various operations and the acceptable tradeoff. I think you're right that the time spent accessing the data would be the most important metric. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Patch queue triage
Gregory Stark wrote:
> "Tom Lane" <[EMAIL PROTECTED]> writes:
>
> > Bruce Momjian <[EMAIL PROTECTED]> writes:
> >> FYI, Tom, Heikki, I need one of you to post the list of patches and
> >> where we think we are on each one, even if the list is imperfect.
> >
> > This message is an attempt to sort out which patch queue entries have
> > no hope of getting into 8.3 (and so we shouldn't spend more time on them
> > right now), which ones can get in but are waiting on their authors for
> > more work, and which ones are ready for immediate review.
>
> Thanks for this. This is exactly what we've been missing recently I think.
100% agree.
> > * Re: [HACKERS] Modifying TOAST thresholds /Tom Lane/
> >
> > At this point it seems nothing will be done about this issue for 8.3.
>
> I'm not sure anyone has an idea how to test it. TPCC isn't really useful
> because it has a fixed size (500 byte) string buffer. Perhaps if we modified
> it to have a random string length uniformly distributed between 0-2k ? But
> even then it never does any scans based on that buffer. But the problem with
> going with something more natural is that it'll be harder to tell exactly what
> it's testing.
My idea on this was to create two backends, one with the default TOAST
value, and a second with a value of 50 bytes. Create a table with one
TEXT field, and several other columns, each column < 50 bytes.
Then, fill the table with random data (script attached that might help),
and the try 2000, 1500, 1000, etc, bytes in the TEXT column for each row
(use random data so the compression code doesn't shrink it). Then run a
test with both backends acessing the TEXT column and non-TEXT column and
measure the difference between the two backends, i.e. the backend with a
TOAST value of 50 should show faster access on the non-TEXT field, but
slower access on the TEXT field.
Then, figure out where the gains on the non-TEXT field seem to diminish
in usefulness. Basically, with a lower TOAST value, we are going to
spend more time accessing the TEXT field, but the speedup for the
non-TEXT field should be large enough win that we don't care. As the
TEXT column becomes shorter, it has less affect on the non-TEXT access.
--
Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
:
ROWS=10
COLSIZE=2500
echo "
DROP TABLE test;
CREATE TABLE test(i SERIAL, t text);
INSERT INTO test (t)
SELECT array_to_string(ARRAY(
SELECT chr(32 + (95 * random())::integer)
FROM generate_series(1,$COLSIZE)),'')
FROMgenerate_series(1, $ROWS);
SELECT pg_relation_size('test') AS "HEAP",
pg_total_relation_size('test') - pg_relation_size('test') AS
"TOAST";
SET log_min_duration_statement = 0;
SET client_min_messages = 'log';
SELECT t FROM test WHERE i = 234329823;" |
sql test
---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at
http://www.postgresql.org/about/donate
Re: [HACKERS] Patch queue triage
On Tue, May 01, 2007 at 10:44:38PM -0400, Tom Lane wrote: > * [PATCHES] Preliminary GSSAPI Patches /Henry B. Hotz/ > > Magnus is reviewing this one. Check. > * [PATCHES] Clear up strxfrm() in UTF-8 with locale on Windows >/ITAGAKI Takahiro/ > > Someone else needs to look at this; I can't test it. Magnus? Yup, it's on my list. //Magnus ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Patch queue triage
"Tom Lane" <[EMAIL PROTECTED]> writes: > Bruce Momjian <[EMAIL PROTECTED]> writes: >> FYI, Tom, Heikki, I need one of you to post the list of patches and >> where we think we are on each one, even if the list is imperfect. > > This message is an attempt to sort out which patch queue entries have > no hope of getting into 8.3 (and so we shouldn't spend more time on them > right now), which ones can get in but are waiting on their authors for > more work, and which ones are ready for immediate review. Thanks for this. This is exactly what we've been missing recently I think. > * [PATCHES] non-recursive WITH clause support > /Gregory Stark/ > > I think the consensus is that we should wait for a more complete WITH > implementation to be submitted, since this one creates compatibility > issues without any real return in the form of added functionality. I submitted it because it filled in a missing ANSI feature but nobody's piped up saying they missed that feature so, sure. > * [PATCHES] Concurrent psql v4 [WIP] /stark/ > > This is waiting on the author to change the API per list discussions; we > can't apply something that is about to have some commands removed ... I did finish the api changes -- though I'm not too happy with the names. I was expecting the list to play the name game so I just put in placeholder names originally. I'm adding documentation and example regression tests now. Also I'm trying to fix the cursor-mode FETCH_COUNT support which it breaks. I'm thinking of once the first batch of rows arrives just going into a synchronous function to fetch the rest of the resultsets. > * Re: [HACKERS] Modifying TOAST thresholds /Tom Lane/ > > At this point it seems nothing will be done about this issue for 8.3. I'm not sure anyone has an idea how to test it. TPCC isn't really useful because it has a fixed size (500 byte) string buffer. Perhaps if we modified it to have a random string length uniformly distributed between 0-2k ? But even then it never does any scans based on that buffer. But the problem with going with something more natural is that it'll be harder to tell exactly what it's testing. > * [HACKERS] tsearch_core patch for inclusion > /Teodor Sigaev/ > > Have we resolved whether the API and the dump/restore strategy are > acceptable? The code needs review too, but not till we've settled > on the basic question whether we like the feature set. Personally I actually do like the idea of promoting tsearch to first-class citizen by giving it keywords and a place in the grammar. I think it's a more fully integrated interface than the function based one. The only reason I might think otherwise was if it was just a crutch for missing features it had exposed that would be better implemented more generically. But I don't think that's the case. > * Re: [PATCHES] [Fwd: Deferred Transactions, Transaction Guarantee > and COMMITwithout waiting] /Simon Riggs/ > > Simon is on the hook to submit an updated patch. I hope this one > makes it in, as it looks like a really nice performance improvement > for those who can use it; but the original patch seems overcomplicated. I know Simon's really busy. I might be able to help with it if he wants. > * Re: [PATCHES] LIMIT/SORT optimization /Gregory Stark/ > * [PATCHES] updated SORT/LIMIT patch /Gregory Stark/ > > I will look at this. I recall being concerned about whether there > wasn't a cleaner way to introduce the required inter-node communication. The next big thing to keep in mind in this area is a unique sort which would need to know if there's a Unique node above it with the same key. If the resulting inter-node communication arrangement has your blessing and can handle that as well then I'll probably do that for 8.4. Incidentally I would prefer it if you want to make changes that you explain the changes to me and let me make them. It gives me a better chance to understand what the changes really are and the motivation than trying to read a patch later and understand why you made the changes you did. I understand sometimes it's easier to just write the code than to explain the idea to someone else and then review the resulting code though and there's already enough work your plate so if that's the case then so be it. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Patch queue triage
"Pavan Deolasee" <[EMAIL PROTECTED]> writes: > On 5/2/07, Tom Lane <[EMAIL PROTECTED]> wrote: >> >> This needs a *lot* of review. Can we break it down into more manageable >> chunks? > > Sure, we can do that. I actually did that when I posted the > incremental versions of the HOT-patch, each version implementing > the next big chunk of the code. I can reverse engineer that again. Can we? I mean, sure you can break the patch up into chunks which might make it easier to read, but are any of the chunks useful alone? I suppose inserting HOT tuples without index maintenance is useful even if no changes to the space allocation is made is useful. It won't get the space usage but it would save on index thrashing. But that still implies all the code to handle scans, updates, index builds, etc. Those chunks could be separated out but you can't commit without them. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Patch queue triage
* [PATCHES] Updateable cursors patch /FAST PostgreSQL/ This is incomplete, and I fear at this point has to be held over to 8.4. It is true that my original patch post said that I need to modify the patch to work with tidscan. Since then I have realized that this modification is not needed as it would have the same result as the 'branching out from sequential scan' solution currently implemented. I was hoping that I could discuss this with whoever picks up the patch for review before doing modifications if any is needed. So in my humble opinion, it would be great if this can be considered for 8.3 as there are not many modifications needed. P.S. Only Simon commented on my original patch. Simon, do you have time to revisit the patch so that we could discuss this? Rgds, Arul Shaji ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Patch queue triage
On Tue, 1 May 2007, Tom Lane wrote: * [HACKERS] tsearch_core patch for inclusion /Teodor Sigaev/ Have we resolved whether the API and the dump/restore strategy are acceptable? The code needs review too, but not till we've settled on the basic question whether we like the feature set. There were several discussion and we tried to satisfy a claims. We added the ability of creation of FTS with one command (just create GIN index on text attribute), which many people like a lot, we changed SQL syntax to be more SQL-ish and we like it. We discussed the new FTS on two conferences in Moscow and got positive opinions. Also, we wrote FTSBOOK ( http://mira.sai.msu.su/~megera/pgsql/ftsdoc/ ) in english and FTS introduction in russian ( http://www.sai.msu.su/~megera/postgres/talks/fts_pgsql_intro.html ). For the period from the patch was announced there were > 3100 unique IP, which downloaded FTSBOOK. Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Patch queue triage
On 5/2/07, Tom Lane <[EMAIL PROTECTED]> wrote: * [PATCHES] HOT Patch - Ready for review /Pavan Deolasee/ This needs a *lot* of review. Can we break it down into more manageable chunks? I'm not sure that anyone's got a full grasp of the implications of this patch, and that's a scary thought. Sure, we can do that. I actually did that when I posted the incremental versions of the HOT-patch, each version implementing the next big chunk of the code. I can reverse engineer that again. When I do that, should I just break the patch into logical pieces without worrying about whether each piece alone builds/works correcttly ? Or should I try to make each piece complete ? I know the second would be a preferred way, but it would be more work. But if that can considerably ease review process, I would do that by all means. In any case, there will be dependecies amongst the patches. I am on leave today, so would start on this tomorrow. Thanks, Pavan -- EnterpriseDB http://www.enterprisedb.com
Re: [HACKERS] Patch queue triage
On 5/2/07, Tom Lane <[EMAIL PROTECTED]> wrote: * [pgsql-patches] Ctid chain following enhancement /Pavan Deolasee/ I'm not very excited about this --- it seems to me to complicate the code in some places that are not in fact performance-critical. While it doesn't seem likely to break things, I'm not in favor of reducing code readability unless a measurable performance improvement can be shown. Can we have some tests showing this is worthwhile? I am OK with dropping this for the time being. It came out of code reading. I would later run some tests to see if it really gives any performance benefit. Thanks, Pavan -- EnterpriseDB http://www.enterprisedb.com
Re: [HACKERS] Patch queue triage
Greg Smith <[EMAIL PROTECTED]> writes: > On Tue, 1 May 2007, Tom Lane wrote: >> * Re: [PATCHES] Synchronized Scan WIP patch >> /Simon Riggs/ >> Heikki is reviewing this one. Also I believe Greg Smith is doing more >> performance testing. > Actually it was the "Automatic adjustment of bgwriter_lru_maxpages" patch > from Itagaki Takahiro I've been working on--which, by the way, Bruce sent > out a note about the other day saying was already booted into 8.4. It's still on the patch queue list, and Heikki and I were assuming it was still live. > I've > integrated a part of the LRU patch usefully into the new pg_stat_bgwriter, > renamed some variables around accordingly, cleanup I felt had to preceed > performance testing. Heikki might want to wait for me to finish that > before putting time into it. The buffer allocation monitoring part would > be useful to apply even if the GUC and behavior changes are deemed outside > of the 8.3 scope. > Also, I think I was the most vocal participant in the "COPY-able sql log > outputs" feature discussion and therefore was planning to do what review > what I can of the final patch API immediately afterwards. I try not to > complain about things unless I'm willing to help fix them. Great, you're on the hook for both of those ;-) regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Patch queue triage
On Tue, 1 May 2007, Tom Lane wrote: * Re: [PATCHES] Synchronized Scan WIP patch /Simon Riggs/ Heikki is reviewing this one. Also I believe Greg Smith is doing more performance testing. Actually it was the "Automatic adjustment of bgwriter_lru_maxpages" patch from Itagaki Takahiro I've been working on--which, by the way, Bruce sent out a note about the other day saying was already booted into 8.4. I've integrated a part of the LRU patch usefully into the new pg_stat_bgwriter, renamed some variables around accordingly, cleanup I felt had to preceed performance testing. Heikki might want to wait for me to finish that before putting time into it. The buffer allocation monitoring part would be useful to apply even if the GUC and behavior changes are deemed outside of the 8.3 scope. Also, I think I was the most vocal participant in the "COPY-able sql log outputs" feature discussion and therefore was planning to do what review what I can of the final patch API immediately afterwards. I try not to complain about things unless I'm willing to help fix them. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
