Re: [PATCHES] Dead Space Map version 3 (simplified)
Gregory Stark wrote: Hiroki Kataoka [EMAIL PROTECTED] writes: I think there is no problem. Bloating will make pages including the unnecessary area which will not be accessed. Soon, those pages will be registered into DSM. Except the whole point of the DSM is to let us vacuum those pages *before* that happens... You are right. However, expecting perfection will often lose performance. Delaying processing to some extent leads to performance. Even if hot page is not vacuumed, it does not mean generating dead tuples boundlessly. About one hot page, the quantity of dead tuple which continues existing unnecessarily is at most 1 page or its extent. Also that page is soon registered into DSM by checkpoint like fail-safe. Isn't some compromise need as first version of DSM vacuum? -- Hiroki Kataoka [EMAIL PROTECTED] ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Full page writes improvement, code update
3) To maintain crash recovery chance and reduce the amount of archive log, removal of unnecessary full page writes from archive logs is a good choice. Definitely, yes. pg_compresslog could even move the full pages written during backup out of WAL and put them in a different file that needs to be applied before replay of the corresponding WAL after a physical restore. This would further help reduce log shipping volume. To do this, we need both logical log and full page writes in WAL. This is only true in the sense, that it allows a less complex implementation of pg_compresslog. Basically a WAL record consists of info about what happened and currently eighter per tuple new data or a full page image. The info of what happened together with the full page image is sufficient to reconstruct the per tuple new data. There might be a few WAL record types (e.g. in btree split ?) where this is not so, but we could eighter fix those or not compress those. This is why I don't like Josh's suggested name of wal_compressable eighter. WAL is compressable eighter way, only pg_compresslog would need to be more complex if you don't turn off the full page optimization. I think a good name would tell that you are turning off an optimization. (thus my wal_fullpage_optimization on/off) Andreas ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] [HACKERS] Full page writes improvement, code update
Koichi, Andreas, 1) To deal with partial/inconsisitent write to the data file at crash recovery, we need full page writes at the first modification to pages after each checkpoint. It consumes much of WAL space. We need to find a way around this someday. Other DBs don't do this; it may be becuase they're less durable, or because they fixed the problem. I don't think there should be only one setting. It depend on how database is operated. Leaving wal_add_optiomization_info = off default does not bring any change in WAL and archive log handling. I understand some people may not be happy with additional 3% or so increase in WAL size, especially people who dosn't need archive log at all. So I prefer to leave the default off. Except that, is there any reason to turn this off if we are archiving? Maybe it should just be slaved to archive_command ... if we're not using PITR, it's off, if we are, it's on. 1) is there any throughput benefit for platforms with fast CPU but contrained I/O (e.g. 2-drive webservers)? Any penalty for servers with plentiful I/O? I've only run benchmarks with archive process running, because wal_add_optimization_info=on does not make sense if we don't archive WAL. In this situation, total I/O decreases because writes to archive log decreases. Because of 3% or so increase in WAL size, there will be increase in WAL write, but decrease in archive writes makes it up. Yeah, I was just looking for a way to make this a performance feature. I see now that it can't be. ;-) 3) How is this better than command-line compression for log-shipping? e.g. why do we need it in the database? I don't fully understand what command-line compression means. Simon suggested that this patch can be used with log-shipping and I agree. If we compare compression with gzip or other general purpose compression, compression ratio, CPU usage and I/O by pg_compresslog are all quite better than those in gzip. OK, that answered my question. This is why I don't like Josh's suggested name of wal_compressable eighter. WAL is compressable eighter way, only pg_compresslog would need to be more complex if you don't turn off the full page optimization. I think a good name would tell that you are turning off an optimization. (thus my wal_fullpage_optimization on/off) Well, as a PG hacker I find the name wal_fullpage_optimization quite baffling and I think our general user base will find it even more so. Now that I have Koichi's explanation of the problem, I vote for simply slaving this to the PITR settings and not having a separate option at all. -- Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] [HACKERS] Full page writes improvement, code update
Josh Berkus [EMAIL PROTECTED] writes: Well, as a PG hacker I find the name wal_fullpage_optimization quite baffling and I think our general user base will find it even more so. Now that I have Koichi's explanation of the problem, I vote for simply slaving this to the PITR settings and not having a separate option at all. The way to not have a separate option is to not need one, by having the feature not cost anything extra in the first place. Andreas and I have made the point repeatedly about how to do that. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[PATCHES] SPI_psprintf and SPI_pstrdup
The Apache runtime library, which is using a similar concept for
allocating heap-based memory out of a pool, has some since utility
functions, named apr_psprintf and apr_pstrdup.
These functions allocate a memory-block out of a pool and print a
formatted string into that block, or duplicate a string respectively.
Since such useful functions are missing in PostgreSQL's Server
Programming Interface, I created two similar functions:
char *SPI_psprintf(const char *fmt, ...);
allocates a block of memory out of the memory pool and prints a
formatted string into it
SPI_pstrdup(const char *src);
allocates a block of memory out of the memory pool and copies an
existing string into it.
I includes a patch containing these functions to be applied onto version
8.2.3 and 8.2.4. I would appreciate seeing it in one of the next
versions.
Jacob Rief
diff -ur postgresql-8.2.3/src/backend/executor/spi.c postgresql-8.2.3.strdup/src/backend/executor/spi.c
--- postgresql-8.2.3/src/backend/executor/spi.c 2006-12-26 17:56:22.0 +0100
+++ postgresql-8.2.3.strdup/src/backend/executor/spi.c 2007-04-16 17:26:46.0 +0200
@@ -818,7 +818,47 @@
MemoryContextDelete(tuptable-tuptabcxt);
}
+char*
+SPI_psprintf(const char *fmt, ...)
+{
+ int n, size = 0;
+ char *p;
+ va_list ap;
+
+ if (fmt==NULL)
+ return NULL;
+ for (n = 0; fmt[n]!='\0'; n++) {
+ size += (fmt[n]=='%') ? 10 : 1;
+ }
+ if ((p = SPI_palloc(size))==NULL)
+ return NULL;
+ while (1) {
+ va_start(ap, fmt);
+ n = vsnprintf(p, size, fmt, ap);
+ va_end(ap);
+ if (n -1 n size)
+ return p;
+ if (n -1)
+ size = n+1; /* glibc 2.1 */
+ else
+ size *= 2; /* glibc 2.0 */
+ if ((p = SPI_repalloc(p, size))==NULL)
+ return NULL;
+ }
+ return NULL;
+}
+char*
+SPI_pstrdup(const char *src)
+{
+ int size;
+ char *p;
+ size = strlen(src)+1;
+ if ((p = SPI_palloc(size))==NULL)
+ return NULL;
+ strcpy(p, src);
+ return p;
+}
/*
* SPI_cursor_open()
diff -ur postgresql-8.2.3/src/include/executor/spi.h postgresql-8.2.3.strdup/src/include/executor/spi.h
--- postgresql-8.2.3/src/include/executor/spi.h 2006-10-04 02:30:08.0 +0200
+++ postgresql-8.2.3.strdup/src/include/executor/spi.h 2007-04-16 17:03:04.0 +0200
@@ -119,6 +119,8 @@
extern char *SPI_getnspname(Relation rel);
extern void *SPI_palloc(Size size);
extern void *SPI_repalloc(void *pointer, Size size);
+extern char *SPI_psprintf(const char* fmt, ...);
+extern char *SPI_pstrdup(const char* src);
extern void SPI_pfree(void *pointer);
extern void SPI_freetuple(HeapTuple pointer);
extern void SPI_freetuptable(SPITupleTable *tuptable);
---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at
http://www.postgresql.org/about/donate
Re: [PATCHES] SPI_psprintf and SPI_pstrdup
Jacob Rief wrote: char *SPI_psprintf(const char *fmt, ...); allocates a block of memory out of the memory pool and prints a formatted string into it Such matters are best solved using the StringInfo API. SPI_pstrdup(const char *src); allocates a block of memory out of the memory pool and copies an existing string into it. How is that different from pstrdup()? -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] SPI_psprintf and SPI_pstrdup
Jacob Rief [EMAIL PROTECTED] writes: The Apache runtime library, which is using a similar concept for allocating heap-based memory out of a pool, has some since utility functions, named apr_psprintf and apr_pstrdup. SPI_palloc might possibly be worth the trouble, but the other thing is duplicative of the StringInfo routines, and I do not think it's a good idea to provide duplicate ways to accomplish the same thing. It's just more code to maintain and more things for newcomers to learn about when trying to read someone else's code. Also, proposed API additions without any documentation are not acceptable. The docs addition for SPI_palloc would be substantially larger than the code itself; is it worth the trouble? regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
