Re: [HACKERS] Is there a good reason we don't have INTERVAL 'infinity'?
On 2 November 2011 16:35, Brar Piening b...@gmx.de wrote: See http://de.wikipedia.org/wiki/Unendlichkeit#Analysis (Sorry for linking the german wikipedia - the english text is ways less verbose on this.) Google Translate has come a very long way. I can read that whole section easily with my brain automatically fixing up the few grammatical errors that Google doesn't get quite right. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Multiple queries in transit
On Mon, Oct 31, 2011 at 7:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 31.10.2011 17:44, Mark Hills wrote: Could libpq be reasonably modified to allow this? I believe it's doable in theory, no-one has just gotten around to it. Patches are welcome. Can't you do that today with a multi-command string submitted to PQsendQuery, followed by multiple calls to PQgetResult? It's more annoying to to error handling on that, plus it still keeps the blocking behaviour, just with larger blocks. I'm hesitant to think about supporting the case more thoroughly than that, or with any different semantics than that, because I think that the error-case behavior will be entirely unintelligible/unmaintainable unless you abandon all queries-in-flight in toto when an error happens. Furthermore, in most apps it'd be a serious PITA to keep track of which reply is for which query, so I doubt that such a feature is of general usefulness. Thats why query queue and error handling must happen in protocol library, not app. And it seems doable, unless the server eats queries or errors in some situation, breaking simple sequential query-response mapping. Do you know of such behaviour? (And several queries in Simple Queriy are known exception, we can ignore them here.) Also I would ask for opposite feature: multiple rows in flight. That means that when server is sending big resultset, the app can process it row-by-row (or by 10 rows) without stopping the stream and re-requesting. -- marko PS. I think full-duplex is better than pipeline here, latter seems to hint something unidirectional, except yeah, it is used in HTTP 1.1 for similar feature. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types - typo + NULL string constructor
On 17.10.2011 01:09, Jeff Davis wrote: On Sat, 2011-10-15 at 01:46 +0300, Heikki Linnakangas wrote: * Do we really need non_empty(anyrange) ? You can just do NOT empty(x) To make it a searchable (via GiST) condition, I need an operator. I could either remove that operator (as it's not amazingly useful), or I could just not document the function but leave the operator there. Looking at the most recent patch, I don't actually see any GiST support for the empty and non-empty operators (!? and ?). I don't see how those could be accelerated with GiST, anyway; I think if you want to use an index for those operators, you might as well create a partial or functional index on empty(x). So I'm actually inclined to remove not only the nonempty function, but also the ? and !? operators. They don't seem very useful, and ? and !? don't feel very intuitive to me, anyway. I'll just leave the empty(x) function. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade if 'postgres' database is dropped
Robert Haas wrote: On Wed, Nov 2, 2011 at 8:31 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: If nobody objects, I'll go do that. ?Hopefully that should be enough to put this problem to bed more or less permanently. All right, I've worked up a (rather boring and tedious) patch to do this, which is attached. I wonder if we should bother using a flag for this. ?No one has asked for one, and the new code to conditionally connect to databases should function fine for most use cases. True, but OTOH we have such a flag for pg_dumpall, and I've already done the work. Well, every user-visible API option has a cost, and I am not sure there is enough usefulness to overcome the cost of this. I am not sure why you think this is worth the time it takes to argue about it, but if you want to whack the patch around or just forget the whole thing, go ahead. The difference between what you're proposing and what I'm proposing is about 25 lines of code, so it hardly needs an acre of justification. To me, making the tools consistent with each other and not dependent on the user's choice of database names is worth the tiny amount of code it takes to make that happen. Well, it would be good to get other opinions on this. The amount of code isn't really the issue for me, but rather keeping the user API as clean as possible. I don't want someone to say, Oh, here's a new user option. Wonder why I should use it? Hmm, no one can tell me. If an option's use-case is not clear, we have to explain in the docs why to use it, and right now no one can tell me why we should use it. Also, if we are going to add this flag, we should have pg_dumpall use it too and just deprecate the old options. I thought about that, but couldn't think of a compelling reason to break backward compatibility. Well, I figure we better have something compelling to do any change, including a new command-line option. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Reduce checkpoints and WAL traffic on low activity database serv
On Thu, Nov 3, 2011 at 12:21 AM, Greg Smith g...@2ndquadrant.com wrote: With some trivial checkpoints containing a small amount of data skipped now, aren't there some cases where less WAL data will be written than before? In that case, the user visible behavior here would be different. I'd be most concerned about file-based log shipping case. Typical settings are checkpoint_timeout = 300 and archive_timeout = 30. So file-based replication users won't notice any difference. The only people who will see a difference are people with archive_timeout = 0 and who either store or stream WAL. For those people, a keepalive message will be available to ensure we can check the link is up, even if no WAL data flows, which I am working on next. The change also increases durability, since the secondary checkpoint is usually in a separate file. In cases where there are little or no writes to the WAL, checkpoints will be skipped even if checkpoint_timeout has passed. At least one new WAL segment must have been created before an automatic checkpoint occurs. The time between checkpoints and when new WAL segments are created are not related in any other way. If file-based WAL shipping is being used and you want to bound how often files are sent to standby server, to reduce potential data loss you should adjust archive_timeout parameter rather than the checkpoint ones. Committed, thanks. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] heap vacuum cleanup locks
On Sun, Jun 5, 2011 at 4:03 AM, Robert Haas robertmh...@gmail.com wrote: We've occasionally seen problems with VACUUM getting stuck for failure to acquire a cleanup lock due to, for example, a cursor holding a pin on the buffer page. In the worst case, this can cause an undetected deadlock, if the backend holding the buffer pin blocks trying to acquire a heavyweight lock that is in turn blocked by VACUUM. Those deadlocks can be detected in exactly the same way as is used for Hot Standby. Cleanup waiter registers interest in pin, anyone with a lock request that must wait checks to see if they hold a pin that would cause deadlock. I'll look at doing a patch for that. Shouldn't take long. A while back, someone (Greg Stark? me?) floated the idea of not waiting for the cleanup lock. If we can't get it immediately, or within some short period of time, then we just skip the page and continue on. Separately, that sounds like a great idea and it's simple to implement - patch attached. Enhancements to that are that I don't see any particular reason why the heap pages need to be vacuumed in exactly sequential order. If they are on disk, reading sequentially is useful, in which case nobody has a pin and so we will continue. But if the blocks are already in shared_buffers, then the sequential order doesn't matter at all. So we could skip pages and then return to them later on. Also, ISTM that LockBufferForCleanup() waits for just a single buffer, but it could equally well wait for multiple buffers at the same time. By this, we would then be able to simply register our interest in multiple buffers and get woken as soon as one of them were free. That way we could read the blocks sequentially, but lock and clean them out of sequence if necessary. Do this in chunks, so it plays nicely with buffer strategy. (Patch doesn't do that yet). (Not sure if the patch handles vacuum map correctly if we skip the page, but its a reasonable prototype for discussion). -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services vacuum_skip_busy_pages.v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Reduce checkpoints and WAL traffic on low activity database serv
On Wed, Nov 2, 2011 at 8:21 PM, Greg Smith g...@2ndquadrant.com wrote: In cases where there are little or no writes to the WAL, checkpoints will be skipped even if checkpoint_timeout has passed. At least one new WAL segment must have been created before an automatic checkpoint occurs. The time between checkpoints and when new WAL segments are created are not related in any other way. If file-based WAL shipping is being used and you want to bound how often files are sent to standby server, to reduce potential data loss you should adjust archive_timeout parameter rather than the checkpoint ones. I think this is good, although where there are little or no writes to the WAL seems a bit awkward to me - how about where little or no WAL has been written? I would probably delete to reduce potential data loss from the last sentence, since I think that sentence has a few too many clauses to be easily parseable. Should we also put a similar sentence into the documentation for checkpoint_timeout? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: GiST for range types (was Re: [HACKERS] Range Types - typo + NULL string constructor)
On 03.11.2011 10:42, Jeff Davis wrote: On Wed, 2011-11-02 at 22:59 +0200, Heikki Linnakangas wrote: This seems to be coming from the selectivity estimation function. The selectivity function for@ is scalargtsel, which is usually used for scalar and=. That doesn't seem right. But what do we store in the statistics for range types in the first place, and what would be the right thing to do for selectivity estimation? I'll have to think more about that, and it depends on the operator. It seems like an easier problem for contains a point than contains another range or overlaps with another range. Right now I don't have a very good answer, and even for the contains a point case I'll have to think about the representation in pg_statistic. I've committed this now, after some more cleanup. I removed the selectivity estimation functions from operators where they were bogus, so writing those is a clear TODO. But that can well be done as a separate patch. Thanks! -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] heap vacuum cleanup locks
On Thu, Nov 3, 2011 at 7:15 AM, Simon Riggs si...@2ndquadrant.com wrote: A while back, someone (Greg Stark? me?) floated the idea of not waiting for the cleanup lock. If we can't get it immediately, or within some short period of time, then we just skip the page and continue on. Separately, that sounds like a great idea and it's simple to implement - patch attached. Oh, that's kind of clever. I was thinking that you'd have to disable this entirely for anti-wraparound vacuum, but the way you've done it avoids that. You'll still have to wait if there's a competing pin on a buffer that contains tuples actually in need of freezing, but that should be relatively rare. Enhancements to that are that I don't see any particular reason why Also, ISTM that LockBufferForCleanup() waits for just a single buffer, but it could equally well wait for multiple buffers at the same time. By this, we would then be able to simply register our interest in multiple buffers and get woken as soon as one of them were free. That way we could read the blocks sequentially, but lock and clean them out of sequence if necessary. Do this in chunks, so it plays nicely with buffer strategy. (Patch doesn't do that yet). I doubt this would help much. The real issue is with open cursors, and those can easily be left open for long enough that those optimizations won't help. I think the patch as it stands is probably gets just about all of the benefit that can be had from this approach while still being reasonably simple. (Not sure if the patch handles vacuum map correctly if we skip the page, but its a reasonable prototype for discussion). Yeah. I think that should be OK, but: - It looks to me like you haven't done anything about the second heap pass. That should probably get a similar fix. - I think that this is going to screw up the reltuples calculation unless we fudge it somehow. The number of scanned pages has already been incremented by the time your new code is reached. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] heap vacuum cleanup locks
On Thu, Nov 3, 2011 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: I think that should be OK, but: - It looks to me like you haven't done anything about the second heap pass. That should probably get a similar fix. I was assuming this worked with Pavan's patch to remove second pass. Not in any rush to commit this, so will wait till that is thru. - I think that this is going to screw up the reltuples calculation unless we fudge it somehow. The number of scanned pages has already been incremented by the time your new code is reached. Yeh, I'll have a look at that in more detail. Thanks for the review. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing =(text, text) in 9.2
Robert Haas robertmh...@gmail.com writes: Hmm, I was kind of expecting that to be wrong at least in some minor way. +/* contrib/hstore/hstore-1.0-1.1.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use ALTER EXTENSION hstore to load this file. \quit You could mention ALTER EXTENSION hstore UPDATE TO 1.1; in this comment, I think. +++ b/contrib/hstore/hstore--1.1.sql @@ -0,0 +1,524 @@ +/* contrib/hstore/hstore--1.0.sql */ That needs a comment update too. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] heap vacuum cleanup locks
On Thu, Nov 3, 2011 at 9:52 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Nov 3, 2011 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: I think that should be OK, but: - It looks to me like you haven't done anything about the second heap pass. That should probably get a similar fix. I was assuming this worked with Pavan's patch to remove second pass. It's not entirely certain that will make it into 9.2, so I would rather get this done first. If you want I can pick up what you've done and send you back a version that addresses this. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Your review of pg_receivexlog/pg_basebackup
On Tue, Nov 1, 2011 at 05:53, Fujii Masao masao.fu...@gmail.com wrote:
On Tue, Nov 1, 2011 at 3:08 AM, Magnus Hagander mag...@hagander.net wrote:
On Fri, Oct 28, 2011 at 08:46, Fujii Masao masao.fu...@gmail.com wrote:
On Thu, Oct 27, 2011 at 11:14 PM, Magnus Hagander mag...@hagander.net
wrote:
Here's a version that does this. Turns out this requires a lot less
code than what was previously in there, which is always nice.
We still need to solve the other part which is how to deal with the
partial files on restore. But this is definitely a cleaner way from a
pure pg_receivexlog perspective.
Comments/reviews?
Looks good.
Minor comment:
the source code comment of FindStreamingStart() seems to need to be updated.
Here's an updated patch that both includes this update to the comment,
and also the functionality to pre-pad files to 16Mb. This also seems
to have simplified the code, which is a nice bonus.
Here are the comments:
In open_walfile(), zerobuf needs to be free'd after use of it.
Ooops, yes.
+ f = open(fn, O_WRONLY | O_CREAT | PG_BINARY, 0666);
We should use S_IRUSR | S_IWUSR instead of 0666 as a file access modes?
Agreed, changed.
+ if (write(f, zerobuf, XLOG_BLCKSZ) != XLOG_BLCKSZ)
+ {
+ fprintf(stderr, _(%s: could not pad WAL segment %s:
%s\n),
+ progname, fn, strerror(errno));
+ close(f);
+ return -1;
+ }
When write() fails, we should delete the partial WAL file, like
XLogFileInit() does?
Yes, that's probably a good idae. Added a simple unlink() call
directly after the close().
If not, subsequent pg_receivexlog always fails unless a user deletes
it manually.
Because open_walfile() always fails when it finds an existing partial WAL
file.
When open_walfile() fails, pg_receivexlog exits without closing the
connection.
I don't think this is good error handling. But this issue itself is
not what we're
trying to address now. So I think you can commit separately from current
patch.
Wow, when looking into that, there was a nice bug in open_walfile -
when the file failed to open, it would write that error message but
not return - then proceed to write a second error message from fstat.
Oops.
Anyway - yes, the return value of ReceiveXLogStream isn't checked at
all. That's certainly not very nice. I'll go fix that too.
I'll apply the patch with the fixes you've mentioned above. Please
check master again in a few minutes. Thanks!
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Term positions in GIN fulltext index
Hello,
I'm using a GIN index for a text column on a big table. I use it to rank
the rows, but I also need to get the term positions for each document of a
subset of documents for one or more terms. I suppose these positions are stored
in the index as the to_tsvector shows them : 'lexeme':{positions}
I've searched and asked on general postgresql mailing list, and I assume
there is no simple way to get these term positions.
For example, for 2 rows of a 'docs' table with a text column 'text' (indexed
with GIN) :
'I get lexemes and I get term positions.'
'Did you get the positions ?'
I'd need a function like this :
select term_positions(text, 'get') from docs;
id_doc | positions
+---
1 | {2,6}
2 | {3}
I'd like to add this function in my database, for experimental purpose.
I got a look at the source code but didn't find some code example using the GIN
index ;
I can not figure out where the GIN index is read as a tsvector
or where the '@@' operator gets the matching tsvectors for the terms of the
tsquery.
Any help about where to start reading would be very welcome :)
Regards,
Yoann Moreau
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] heap vacuum cleanup locks
On Thu, Nov 3, 2011 at 2:22 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 3, 2011 at 9:52 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Nov 3, 2011 at 1:26 PM, Robert Haas robertmh...@gmail.com wrote: I think that should be OK, but: - It looks to me like you haven't done anything about the second heap pass. That should probably get a similar fix. I was assuming this worked with Pavan's patch to remove second pass. It's not entirely certain that will make it into 9.2, so I would rather get this done first. If you want I can pick up what you've done and send you back a version that addresses this. OK, that seems efficient. Thanks. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Term positions in GIN fulltext index
Yoann Moreau yoann.mor...@univ-avignon.fr wrote:
I'd need a function like this :
select term_positions(text, 'get') from docs;
id_doc | positions
+---
1 | {2,6}
2 | {3}
I'd like to add this function in my database, for experimental
purpose. I got a look at the source code but didn't find some code
example using the GIN index ;
I can not figure out where the GIN index is read as a tsvector
or where the '@@' operator gets the matching tsvectors for the
terms of the tsquery.
Any help about where to start reading would be very welcome :)
I'm not really clear on what you want to read about. Do you need
help creating your own function on the fly, or with how to access
the information to write the function?
If the former, these links might help:
http://www.postgresql.org/docs/9.1/interactive/extend.html
http://www.postgresql.org/docs/9.1/interactive/sql-createfunction.html
If the latter, have you looked at this file?:
src/backend/utils/adt/tsrank.c
Or was it something else that I'm missing?
-Kevin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Refactor xlog.c #2 - xlog functions
Patch strips out all user visible functions into a new xlogfuncs.c file. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services refactor_xlog_funcs.v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: GiST for range types (was Re: [HACKERS] Range Types - typo + NULL string constructor)
On Nov 3, 2011, at 4:59 AM, Heikki Linnakangas wrote: I've committed this now, after some more cleanup. I removed the selectivity estimation functions from operators where they were bogus, so writing those is a clear TODO. But that can well be done as a separate patch. Thanks! Woo! Congrats Jeff. Awesome news. Very excited about this feature. Thanks for getting this in, Heikki. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade if 'postgres' database is dropped
Bruce Momjian wrote:
I fixed this a different way. I originally thought I could skip over
the 'postgres' database in the new cluster if it didn't exist in the old
cluster, but we have do things like check it is empty, so that was going
to be awkward.
It turns out there was only one place that expected a 1-1 mapping of old
and new databases (file transfer), so I just modified that code to allow
skipping a database in the new cluster that didn't exist in the old
cluster.
Attached patch applied. This allows an upgrade if the 'postgres'
database is missing from the old cluster.
OK, I thought some more and didn't like the way the code could loop off
the end of the new cluster without matching all the old cluster
database.
The attached, applied patches improves this.
--
Bruce Momjian br...@momjian.ushttp://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
diff --git a/contrib/pg_upgrade/relfilenode.c b/contrib/pg_upgrade/relfilenode.c
new file mode 100644
index 382588f..d67d01f
*** a/contrib/pg_upgrade/relfilenode.c
--- b/contrib/pg_upgrade/relfilenode.c
*** transfer_all_new_dbs(DbInfoArr *old_db_a
*** 41,51
/* Scan the old cluster databases and transfer their files */
for (old_dbnum = new_dbnum = 0;
! old_dbnum old_db_arr-ndbs new_dbnum new_db_arr-ndbs;
old_dbnum++, new_dbnum++)
{
! DbInfo *old_db = old_db_arr-dbs[old_dbnum];
! DbInfo *new_db = new_db_arr-dbs[new_dbnum];
FileNameMap *mappings;
int n_maps;
pageCnvCtx *pageConverter = NULL;
--- 41,50
/* Scan the old cluster databases and transfer their files */
for (old_dbnum = new_dbnum = 0;
! old_dbnum old_db_arr-ndbs;
old_dbnum++, new_dbnum++)
{
! DbInfo *old_db = old_db_arr-dbs[old_dbnum], *new_db;
FileNameMap *mappings;
int n_maps;
pageCnvCtx *pageConverter = NULL;
*** transfer_all_new_dbs(DbInfoArr *old_db_a
*** 55,67
* but not in the old, e.g. postgres. (The user might
* have removed the 'postgres' database from the old cluster.)
*/
! while (strcmp(old_db-db_name, new_db-db_name) != 0
! new_dbnum new_db_arr-ndbs)
! new_db = new_db_arr-dbs[++new_dbnum];
! if (strcmp(old_db-db_name, new_db-db_name) != 0)
! pg_log(PG_FATAL, old and new databases have different names: old \%s\, new \%s\\n,
! old_db-db_name, new_db-db_name);
n_maps = 0;
mappings = gen_db_file_maps(old_db, new_db, n_maps, old_pgdata,
--- 54,69
* but not in the old, e.g. postgres. (The user might
* have removed the 'postgres' database from the old cluster.)
*/
! for (; new_dbnum new_db_arr-ndbs; new_dbnum++)
! {
! new_db = new_db_arr-dbs[new_dbnum];
! if (strcmp(old_db-db_name, new_db-db_name) == 0)
! break;
! }
! if (new_dbnum = new_db_arr-ndbs)
! pg_log(PG_FATAL, old database \%s\ not found in the new cluster\n,
! old_db-db_name);
n_maps = 0;
mappings = gen_db_file_maps(old_db, new_db, n_maps, old_pgdata,
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Storing hot members of PGPROC out of the band
Hi All,
While working on some of the performance issues on HP-UX, I noticed a
significant data cache misses for accessing PGPROC members. On a close
inspection, it was quite evident that for large number (even few 10s)
of clients, the loop inside GetSnapshotData will cause data cache miss
for almost every PGPROC because the PGPROC structure is quite heavy
and no more than one structure may fit in a single cache line. So I
experimented by separating the most frequently and closely accessed
members of the PGPROC into an out of band array. I call it
PGPROC_MINIMAL structure which contains xid, xmin, vacuumFlags amongst
others. Basically, all the commonly accessed members by
GetSnapshotData find a place in this minimal structure.
When PGPROC array is allocated, we also allocate another array of
PGPROC_MINIMAL structures of the same size. While accessing the
ProcArray, a simple pointer mathematic can get us the corresponding
PGPROC_MINIMAL structure. The only exception being the dummy PGPROC
for prepared transaction. A first cut version of the patch is
attached. It looks big, but most of the changes are cosmetic because
of added indirection. The patch also contains another change to keep
the ProcArray sorted by (PGPROC *) to preserve locality of references
while traversing the array.
I did some tests of a 32 core IA HP-UX box and the results are quite
good. With a scale factor of 100 and -N option of pgbench (updates on
only accounts table), the numbers look something like this:
Clients HEADPGPROC-Patched Gain
1 1098.488663 1059.830369 -3.52%
4 3569.481435 3663.898254 2.65%
32 11627.05922816419.86405641.22%
48 11044.50124415825.13258243.29%
64 10432.20652515408.50304 47.70%
80 10210.57835 15170.61443548.58%
The numbers are quite reproducible with couple of percentage points
variance. So even for single client, I sometimes see no degradation.
Here are some more numbers with the normal pgbench tests (without -N
option).
Clients HEADPGPROC-Patched Gain
1 743 7713.77%
4 1821 2315 27.13%
32 8011 9166 14.42%
48 7282 8959 23.03%
64 6742 8937 32.56%
80 6316 8664 37.18%
Its quite possible that the effect of the patch is more evident on the
particular hardware that I am testing. But the approach nevertheless
seems reasonable. It will very useful if someone else having access to
a large box can test the effect of the patch.
BTW, since I played with many versions of the patch, the exact numbers
with this version might be a little different than what I posted
above. I will conduct more tests, especially with more number of
clients and see if there is any difference in the improvement.
Thanks,
Pavan
--
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 477982d..b907f72 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -114,6 +114,7 @@ int max_prepared_xacts = 0;
typedef struct GlobalTransactionData
{
PGPROC proc; /* dummy proc */
+ PGPROC_MINIMAL proc_minimal; /* dummy proc_minimal */
BackendId dummyBackendId; /* similar to backend id for backends */
TimestampTz prepared_at; /* time of preparation */
XLogRecPtr prepare_lsn; /* XLOG offset of prepare record */
@@ -223,6 +224,9 @@ TwoPhaseShmemInit(void)
* technique.
*/
gxacts[i].dummyBackendId = MaxBackends + 1 + i;
+
+ /* Initialize minimal proc structure from the global structure */
+ gxacts[i].proc.proc_minimal = gxacts[i].proc_minimal;
}
}
else
@@ -310,14 +314,15 @@ MarkAsPreparing(TransactionId xid, const char *gid,
gxact-proc.waitStatus = STATUS_OK;
/* We set up the gxact's VXID as InvalidBackendId/XID */
gxact-proc.lxid = (LocalTransactionId) xid;
- gxact-proc.xid = xid;
- gxact-proc.xmin = InvalidTransactionId;
+ gxact-proc.proc_minimal = gxact-proc_minimal;
+ gxact-proc.proc_minimal-xid = xid;
+ gxact-proc.proc_minimal-xmin = InvalidTransactionId;
gxact-proc.pid = 0;
gxact-proc.backendId = InvalidBackendId;
gxact-proc.databaseId = databaseid;
gxact-proc.roleId = owner;
- gxact-proc.inCommit = false;
- gxact-proc.vacuumFlags = 0;
+ gxact-proc.proc_minimal-inCommit = false;
+ gxact-proc.proc_minimal-vacuumFlags = 0;
gxact-proc.lwWaiting = false;
gxact-proc.lwExclusive = false;
gxact-proc.lwWaitLink = NULL;
@@ -326,8 +331,8 @@ MarkAsPreparing(TransactionId xid, const char *gid,
for (i = 0; i NUM_LOCK_PARTITIONS; i++)
SHMQueueInit((gxact-proc.myProcLocks[i]));
/* subxid data must be filled later by GXactLoadSubxactData */
- gxact-proc.subxids.overflowed = false;
- gxact-proc.subxids.nxids = 0;
+ gxact-proc.proc_minimal-overflowed = false;
+ gxact-proc.proc_minimal-nxids = 0;
gxact-prepared_at = prepared_at;
/* initialize LSN to 0 (start of WAL) */
@@ -361,14 +366,14 @@
Re: [HACKERS] Term positions in GIN fulltext index
On Nov3, 2011, at 16:52 , Yoann Moreau wrote:
I'm using a GIN index for a text column on a big table. I use it to rank
the rows, but I also need to get the term positions for each document of a
subset of documents for one or more terms. I suppose these positions are
stored
in the index as the to_tsvector shows them : 'lexeme':{positions}
There's a difference between values of type tsvector, and what GIN indices
on columns or expressions of type tsvector store.
Values of type tsvector, of course, store weights and positions for each lexem.
But GIN indices store only the bare lexems without weights and positions. In
general, GIN indices work by extracting elements from values to be indexed,
and store these elements in a btree, together with pointers to the rows
containing the indexed values.
Thus, if you created a function index on the results of to_tsvector, i.e.
if you do
CREATE INDEX gin_idx ON docs USING gin (to_tsvector(text))
then the weights and positions aren't stored anywhere - they'll only exists in
the transient, in-memory tsvector value that to_tsvector returns, but not in
the on-disk GIN index gin_idx.
For the positions and weights to be store, you need to store the result of
to_tsvector in a column of type tsvector, say text_tsvector, and create the
index as
CREATE INDEX gin_idx ON docs USING gin (text_tsvector)
The GIN index gin_idx still won't store weights and positions, but the column
text_tsvector will.
For example, for 2 rows of a 'docs' table with a text column 'text' (indexed
with GIN) :
'I get lexemes and I get term positions.'
'Did you get the positions ?'
I'd need a function like this :
select term_positions(text, 'get') from docs;
id_doc | positions
+---
1 | {2,6}
2 | {3}
As I pointed out above, you'll first need to make sure to store the result of
to_tsvector in a columns. Then, what you need seems to be a functions that
takes a tsvector value and returns the contained lexems as individual rows.
Postgres doesn't seem to contain such a function currently (don't believe that,
though - go and recheck the documentation. I don't know all thousands of
built-in
functions by heart). But it's easy to add one. You could either use PL/pgSQL
to parse the tsvector's textual representation, or write a C function. If you
go the PL/pgSQL route, regexp_split_to_table() might come in handy.
I'd like to add this function in my database, for experimental purpose.
I got a look at the source code but didn't find some code example using the
GIN index ;
I can not figure out where the GIN index is read as a tsvector
or where the '@@' operator gets the matching tsvectors for the terms of the
tsquery.
The basic flow of information is:
to_tsvector takes a string, parses and, applies various dictionaries according
to the textsearch configuration, and finally returns a value of type tsvector.
See the files names tsvector* for the implementation of that process, and for
the implementation of the various support functions which work on values of type
tsvector.
The GIN index machinery then calls the tsvector's extractValue() function to
extract
the elements mentioned above from the tsvector value. That function is called
gin_extract_tsvector() and lives in tsginidx.c. The extracted elements are
then added to the GIN index's internal btree.
During query execution, if postgres sees that the operator tsvector @@ tsquery
is used, and that the left argument is a GIN-indexed column, it will use the
extractQuery() and consistent() functions to quickly find matching rows by
scanning the internal btree index. In the case of tsvector and tsquery, the
implementation of these functions are gin_extract_tsquery() and
gin_tsquery_consistent(), found also in tsginidx.c.
I suggest you read http://www.postgresql.org/docs/9.1/interactive/gin.html,
it explains all of this in (much) more detail.
best regards,
Florian Pflug
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Further plans to refactor xlog.c
Next steps in refactoring are bigger steps, but not huge ones. I propose this * everything to do with XLOG rmgr into a file called xlogrmgr.c Thats xlog_redo() and most everything to do with checkpoints * everything to do with reading WAL files into a file called xlogread.c That will allow us to put pg_xlogdump into core * possibly some more stuff into xlogboot.c The above actions will reduce xlog.c to about 7000 lines, about 4000 lines smaller than when I started. That sounds like it could go further, but it moves out most of the areas of recent growth by focusing on the purpose of that code. An obvious split would seem to be move all recovery-side code into its own file. That seems quite likely to take a lot of time, break something, as well as requiring us to share XLogCtl, all of which personally I would rather avoid. Fujii's work is likely to remove another few hundred lines as well. That seems enough to me OK? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: GiST for range types (was Re: [HACKERS] Range Types - typo + NULL string constructor)
On Nov3, 2011, at 18:54 , David E. Wheeler wrote: On Nov 3, 2011, at 4:59 AM, Heikki Linnakangas wrote: I've committed this now, after some more cleanup. I removed the selectivity estimation functions from operators where they were bogus, so writing those is a clear TODO. But that can well be done as a separate patch. Thanks! Woo! Congrats Jeff. Awesome news. Very excited about this feature. Thanks for getting this in, Heikki. +1. Great work, guys! best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Term positions in GIN fulltext index
Yoann Moreau yoann.mor...@univ-avignon.fr writes:
I'm using a GIN index for a text column on a big table. I use it to rank
the rows, but I also need to get the term positions for each document of a
subset of documents for one or more terms. I suppose these positions are
stored
in the index as the to_tsvector shows them : 'lexeme':{positions}
I'm pretty sure that a GIN index on tsvector does *not* store positions
--- it only knows about the strings. Don't know one way or the other
about GIST.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Term positions in GIN fulltext index
On Thu, Nov 3, 2011 at 4:52 PM, Yoann Moreau
yoann.mor...@univ-avignon.fr wrote:
I'd need a function like this :
select term_positions(text, 'get') from docs;
id_doc | positions
+---
1 | {2,6}
2 | {3}
check this out:
http://www.postgresql.org/docs/current/static/textsearch-debugging.html
ts_debug does what You want, and more. Look at it's source - it`s a
plain sql function, You can make something based on it.
Greetings
Marcin Mańk
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Term positions in GIN fulltext index
On Thu, Nov 3, 2011 at 11:01 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Yoann Moreau yoann.mor...@univ-avignon.fr writes:
I'm using a GIN index for a text column on a big table. I use it to rank
the rows, but I also need to get the term positions for each document of
a
subset of documents for one or more terms. I suppose these positions are
stored
in the index as the to_tsvector shows them : 'lexeme':{positions}
I'm pretty sure that a GIN index on tsvector does *not* store positions
--- it only knows about the strings. Don't know one way or the other
about GIST.
GiST index doesn't store positions too. See gtsvector_compress. It converts
tsvector to array of crc32 of words. If that value is anyway too large then
function converts it to signature.
--
With best regards,
Alexander Korotkov.
Re: [HACKERS] warning in pg_upgrade
Kevin Grittner kevin.gritt...@wicourts.gov wrote: Untested patch attached for purposes of discussion. I got in a little testing on it -- not only does this patch eliminate the compile-time warning, but if you try to run pg_upgrade when another session has removed your current working directory, you get a reasonable message instead of the program attempting to proceed with undefined (potential garbage) for a working directory. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: GiST for range types (was Re: [HACKERS] Range Types - typo + NULL string constructor)
On Thu, Nov 3, 2011 at 3:59 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I've committed this now, after some more cleanup. I removed the selectivity estimation functions from operators where they were bogus, so writing those is a clear TODO. But that can well be done as a separate patch. Cool! Patch with GiST on range types improvements from me will be soon. -- With best regards, Alexander Korotkov.
Re: [HACKERS] ts_count
On 06/04/2011 04:51 PM, Oleg Bartunov wrote: Well, there are several functions available around tsearch2. so I suggest somebody to collect all of them and create one extension - ts_addon. For example, these are what I remember: 1. tsvector2array 2. noccurences(tsvector, tsquery) - like your ts_count 3. nmatches(tsvector, tsquery) - # of matched lexems in query Of course, we need to think about better names for functions, since ts_count is a bit ambiguous. Oleg, are you doing this? I'd rather this stuff didn't get dropped on the floor. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Strange behavior on to_tsquery()
Hello PG hackers,
I created a *custom dictionary* (based on dict_int) and a search
configuration and a strange behavior happens on *PostgreSQL 8.4.9*.
When I invoke the following instruction several times,*
to_tsquery()*returns distinct results:
catalog= SELECT to_tsquery('custom', 'pi');
to_tsquery
'pi':*
(1 registro)
catalog= SELECT to_tsquery('custom', 'pi');
to_tsquery
'pi'
(1 registro)
Therefore, when I use *@@ operator* over a *tsvector* column in my table
the result set is not always the same.
1) Do you have any clue on what could be happening and how to solve this
issue, please?
2) Sometimes the value returned by *to_tsquery()* has a :* suffix. What
does that mean?
Thanks in advance.
Best Regards,
--
Rodrigo Hjort
www.hjort.co
Re: [HACKERS] Strange behavior on to_tsquery()
Rodrigo Hjort rodrigo.hj...@gmail.com writes: I created a *custom dictionary* (based on dict_int) and a search configuration and a strange behavior happens on *PostgreSQL 8.4.9*. ... Therefore, when I use *@@ operator* over a *tsvector* column in my table the result set is not always the same. This almost certainly means a bug in your dictionary code. 2) Sometimes the value returned by *to_tsquery()* has a :* suffix. What does that mean? Prefix search request. Possibly you're forgetting to zero out the prefix flag? (Just offhand, it rather looks like dict_int and dict_xsyn are both assuming that palloc will give back zeroed space, which is bogus...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] isolationtester patch
Just noticed that I broke the buildfarm with that isolationtester commit I did earlier today. I'm out for dinner now but I'll try to fix it when I'm back .. sorry. -- Álvaro Herrera alvhe...@alvh.no-ip.org -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Storing hot members of PGPROC out of the band
On Thu, Nov 3, 2011 at 6:12 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: When PGPROC array is allocated, we also allocate another array of PGPROC_MINIMAL structures of the same size. While accessing the ProcArray, a simple pointer mathematic can get us the corresponding PGPROC_MINIMAL structure. The only exception being the dummy PGPROC for prepared transaction. A first cut version of the patch is attached. It looks big, but most of the changes are cosmetic because of added indirection. The patch also contains another change to keep the ProcArray sorted by (PGPROC *) to preserve locality of references while traversing the array. This is very good. If you look at your PGPROC_MINIMAL, its mostly transaction related stuff, so I would rename it PGXACT or similar. Not sure why you talk about pointer math, seems easy enough just to have two arrays protected by one lock, and have each proc use the same offset in both arrays. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
On 10/25/11 5:03 AM, Magnus Hagander wrote: If we want something to go in early, that could be as simple as a version of pg_basebackup that runs against the slave but only if full_page_writes=on on the master. If it's not, it throws an error. Then we can improve upon that by adding handling of fpw=off, first by infrastructure, then by tool. Just to be clear, the idea is to require full_page_writes to do backup from the standby in 9.2, but to remove the requirement later? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Review comments for Patch: exclude-table-data option for pg_dump
Hackers, Andrew produced a version for this patch which builds against 9.0. I've tested that version on a production installation of PostgreSQL, including: * dumping and reloading a production database with over 200 objects, 500GB of data and complex dependancies, 4 times so far * excluding different sets of data * getting table names wrong (didn't mean to test that, but I did) * using the --help and man page In all cases the new pg_dump --exclude-table-data option behaved as expected, without errors, and consistent with other pg_dump options. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Strange behavior on to_tsquery()
I wrote: (Just offhand, it rather looks like dict_int and dict_xsyn are both assuming that palloc will give back zeroed space, which is bogus...) Yeah, this is definitely broken. Patches committed; thanks for the report. http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=e3e3087d8717c26cd1c4581ba29274ac214eb816 regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unite recovery.conf and postgresql.conf
On 09/24/2011 04:49 PM, Joshua Berkus wrote: Well, we *did* actually come up with a reasonable way, but it died under an avalanche of bikeshedding and we-must-do-everything-the-way-we-always-have-done. I refer, of course, to the configuration directory patch, which was a fine solution, and would indeed take care of the recovery.conf issues as well had we implemented it. We can *still* implement it, for 9.2. That actually died from a lack of round-tuits, the consensus at the end of the bike-sheeding was pretty clear. Last night I finally got motivated to fix the bit rot and feature set on that patch, to match what seemed to be the easiest path toward community approval. One known bug left to resolve and I think it's ready to submit for the next CF. I think includeifexists is also a good improvement, too, on a related arc to the main topic here. If I can finish off the directory one (or get someone else to fix my bug) I should be able to follow up with that one. The patches won't be that different. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] heap_page_prune comments
On Nov 2, 2011, at 11:27 AM, Robert Haas wrote: The following comment - or at least the last sentence thereof - appears to be out of date. /* * XXX Should we update the FSM information of this page ? * * There are two schools of thought here. We may not want to update FSM * information so that the page is not used for unrelated UPDATEs/INSERTs * and any free space in this page will remain available for further * UPDATEs in *this* page, thus improving chances for doing HOT updates. * * But for a large table and where a page does not receive further UPDATEs * for a long time, we might waste this space by not updating the FSM * information. The relation may get extended and fragmented further. * * One possibility is to leave fillfactor worth of space in this page * and update FSM with the remaining space. * * In any case, the current FSM implementation doesn't accept * one-page-at-a-time updates, so this is all academic for now. */ The simple fix here is just to delete that last sentence, but does anyone think we ought to do change the behavior, now that we have the option to do so? The fillfactor route seems to make the most sense here... it certainly seems to be the least surprising behavior. -- Jim C. Nasby, Database Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] IDLE in transaction introspection
On Thu, Nov 3, 2011 at 3:18 AM, Scott Mead sco...@openscg.com wrote: ISTM that we're all for: creating a new column: state renaming current_query = query State will display RUNNING, IDLE, IDLE in transaction, etc... query will display the last query that was executed. The greater/less-than-sign is still required in the State display? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DeArchiver process
On Thu, Nov 3, 2011 at 2:52 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Nov 2, 2011 at 5:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Dimitri Fontaine dimi...@2ndquadrant.fr writes: The only part of your proposal that I don't like is the process name, that deArchiver thing. wal restore process or something like that would be better. We already have wal writer process and wal sender process and wal receiver process. +1, restore seems pretty vague in this context. Yeh, walrestore seems more natural than just restore. +1 with this name and whole idea. If we introduce walrestore process, pg_standby seems no longer useful. We should get rid of it? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
On Fri, Nov 4, 2011 at 8:06 AM, Josh Berkus j...@agliodbs.com wrote: On 10/25/11 5:03 AM, Magnus Hagander wrote: If we want something to go in early, that could be as simple as a version of pg_basebackup that runs against the slave but only if full_page_writes=on on the master. If it's not, it throws an error. Then we can improve upon that by adding handling of fpw=off, first by infrastructure, then by tool. Just to be clear, the idea is to require full_page_writes to do backup from the standby in 9.2, but to remove the requirement later? Yes unless I'm missing something. Not sure if we can remove that in 9.2, though. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Further plans to refactor xlog.c
On Fri, Nov 4, 2011 at 3:14 AM, Simon Riggs si...@2ndquadrant.com wrote: Next steps in refactoring are bigger steps, but not huge ones. I propose this * everything to do with XLOG rmgr into a file called xlogrmgr.c Thats xlog_redo() and most everything to do with checkpoints * everything to do with reading WAL files into a file called xlogread.c That will allow us to put pg_xlogdump into core * possibly some more stuff into xlogboot.c The above actions will reduce xlog.c to about 7000 lines, about 4000 lines smaller than when I started. That sounds like it could go further, but it moves out most of the areas of recent growth by focusing on the purpose of that code. An obvious split would seem to be move all recovery-side code into its own file. That seems quite likely to take a lot of time, break something, as well as requiring us to share XLogCtl, all of which personally I would rather avoid. Fujii's work is likely to remove another few hundred lines as well. That seems enough to me OK? Additionally what about moving all built-in functions defined in xlog.c to xlogfuncs.c? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
