Re: [HACKERS] could not open relation 1663/16384/16584: No such file or directory in a specific combination of transactions with temp tables

2008-02-29 Thread Heikki Linnakangas
John Smith wrote: Architecture: Intel Core 2 Duo OS: linux-2.6.20-gentoo-r8 Filesystem: ext3 Postgres v8.2.3 compiled with gcc 4.1.1-r3 RAM - 2GB Shared buffers - 24MB [All other Postgres configuration parameters are default values] Problem description: COPY into temp table fails using a

[HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Pavan Deolasee
I remember Heikki mentioned improving redo recovery in one of the emails in the past, so I know people are already thinking about this. I have some ideas and just wanted to get comments here. ISTM that its important to keep the redo recovery time as small as possible in order to reduce the

Re: [HACKERS] CREATE TABLE, load and freezing

2008-02-29 Thread Simon Riggs
On Thu, 2008-02-28 at 14:38 +0530, Pavan Deolasee wrote: I had this idea sometime back. Not sure if this has been discussed before Check the archives for my post to hackers in Jan 2007 and subsequent discussion. It's possible, just a little fiddly. -- Simon Riggs 2ndQuadrant

Re: [HACKERS] could not open relation 1663/16384/16584: No such file or directory in a specific combination of transactions with temp tables

2008-02-29 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Attached is a simple patch to fix that by disallowing CREATE+DROP+PREPARE TRANSACTION more reliably. That patch was missing changes to header files. New patch attached. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com Index:

Re: [HACKERS] could not open relation 1663/16384/16584: No such file or directory in a specific combination of transactions with temp tables

2008-02-29 Thread Gurjeet Singh
Plausible theory, and nice explanation Try the following link (I had to wait for 50 sec for the link to appear, but I guess the trade-off of getting knowledge in return is worth it :) ) http://www5.upload2.net/download/77fa86e16a02e52fd5439c76e148d231/47c7fdce/rfsLfnuVlYjEcCJ/basetables.tgz

[HACKERS] Why we panic in pglz_decompress

2008-02-29 Thread Zdenek Kotala
I'm now looking into toast code and I found following code in pglz_decompress: 00704 if (destsize != source-rawsize) 00705 elog(destsize source-rawsize ? FATAL : ERROR, 00706 compressed data is corrupt); I'm surprise why we there panic? By my opinion is not too good

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Florian Weimer
* Pavan Deolasee: The current redo-recovery is a single threaded, synchronous process. The XLOG is read sequentially, each log record is examined and replayed if required. This requires reading disk blocks in the shared buffers and applying changes to the buffer. The reading happens

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: I know that Simon has some ideas about parallel restored, though I don't know how he wants to solve the dependency issues involved. Perhaps by not parallelizing withon one table or index... I think we should be *extremely* cautious about introducing

Re: [HACKERS] Why we panic in pglz_decompress

2008-02-29 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Zdenek Kotala wrote: I'm now looking into toast code and I found following code in pglz_decompress: 00704 if (destsize != source-rawsize) 00705 elog(destsize source-rawsize ? FATAL : ERROR, 00706 compressed data is

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Florian G. Pflug
Pavan Deolasee wrote: What I am thinking is if we can read ahead these blocks in the shared buffers and then apply redo changes to them, it can potentially improve things a lot. If there are multiple read requests, kernel (or controller ?) can probably schedule the reads more efficiently.

Re: [HACKERS] Why we panic in pglz_decompress

2008-02-29 Thread Zdenek Kotala
Alvaro Herrera napsal(a): Zdenek Kotala wrote: I'm now looking into toast code and I found following code in pglz_decompress: 00704 if (destsize != source-rawsize) 00705 elog(destsize source-rawsize ? FATAL : ERROR, 00706 compressed data is corrupt); I'm surprise

Re: [HACKERS] Why we panic in pglz_decompress

2008-02-29 Thread Alvaro Herrera
Zdenek Kotala wrote: I'm now looking into toast code and I found following code in pglz_decompress: 00704 if (destsize != source-rawsize) 00705 elog(destsize source-rawsize ? FATAL : ERROR, 00706 compressed data is corrupt); I'm surprise why we there panic?

Re: [HACKERS] [BUGS] bug or not bug, xmlvalidate(xml, text) can read and show one line from file

2008-02-29 Thread Tom Lane
Sergey Burladyan [EMAIL PROTECTED] writes: [ xmlvalidate is a security hole ] Given that this function is not documented nor tested in the regression tests, I propose diking it out entirely. regards, tom lane ---(end of

Re: [HACKERS] Why we panic in pglz_decompress

2008-02-29 Thread Zdenek Kotala
Tom Lane napsal(a): Alvaro Herrera [EMAIL PROTECTED] writes: Zdenek Kotala wrote: I'm now looking into toast code and I found following code in pglz_decompress: 00704 if (destsize != source-rawsize) 00705 elog(destsize source-rawsize ? FATAL : ERROR, 00706

[HACKERS] request feature: alter view add column (or something like)

2008-02-29 Thread Olivier Thauvin
Sometimes, I have to add column to table, then adding the column to some view. Currently it impossible to alter a view to add a new column, which is really annoying. The only solution is to drop the view, and all dependented view, then recreating everything. Someone on #postgresqlfr said me

[HACKERS] creating new aggregate function

2008-02-29 Thread Justin
Need help and direction creating new aggregate functions. We need to add more average functions for both scientific and finical purposes RMS for electrical measurement purposes Mode for both electrical and finical Weighted Average finical purposes Generalized mean for electrical

Re: [HACKERS] creating new aggregate function

2008-02-29 Thread Sam Mason
On Fri, Feb 29, 2008 at 12:11:59PM -0500, Justin wrote: Need help and direction creating new aggregate functions. We need to add more average functions for both scientific and finical purposes [ ... ] what would be the best way to create these new functions?? I'd be tempted to look at

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Simon Riggs
On Fri, 2008-02-29 at 15:49 +0100, Florian G. Pflug wrote: I know that Simon has some ideas about parallel restored, though I don't know how he wants to solve the dependency issues involved. Perhaps by not parallelizing withon one table or index... Well, I think that problem is secondary to

Re: [HACKERS] Buildfarm member gypsy_moth seems not to like alignment patch

2008-02-29 Thread Tom Lane
Jorgen Austvik - Sun Norway [EMAIL PROTECTED] writes: Tom Lane wrote: This is unfortunate and surprising, since that patch was intended to prevent compilers from making unsafe alignment assumptions, but it sure looks like this compiler has instead added a new one. Could you poke into it ---

[HACKERS] Re: [BUGS] bug or not bug, xmlvalidate(xml, text) can read and show one line from file

2008-02-29 Thread Alvaro Herrera
Peter Eisentraut escribió: Am Freitag, 29. Februar 2008 schrieb Tom Lane: Sergey Burladyan [EMAIL PROTECTED] writes: [ xmlvalidate is a security hole ] Given that this function is not documented nor tested in the regression tests, I propose diking it out entirely. Yes, it was

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Decibel!
On Feb 29, 2008, at 8:10 AM, Florian Weimer wrote: In the end, I wouldn't be surprised if for most loads, cache warming effects dominated recovery times, at least when the machine is not starved on RAM. Uh... that's exactly what all the synchronous reads are doing... warming the cache. And

Re: [HACKERS] [BUGS] bug or not bug, xmlvalidate(xml, text) can read and show one line from file

2008-02-29 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Peter Eisentraut escribió: Yes, it was accidentally left over from previous work. We should have removed it before the release, but that would have required an initdb. So let's change it for a function that elog(ERROR)s on entry. Yeah, I was just

Re: [HACKERS] creating new aggregate function

2008-02-29 Thread Webb Sprague
This probably belongs on General, but On Fri, Feb 29, 2008 at 9:11 AM, Justin [EMAIL PROTECTED] wrote: Need help and direction creating new aggregate functions. We need to add more average functions for both scientific and finical purposes RMS for electrical measurement purposes Mode

Re: [HACKERS] [BUGS] bug or not bug, xmlvalidate(xml, text) can read and show one line from file

2008-02-29 Thread Peter Eisentraut
Am Freitag, 29. Februar 2008 schrieb Tom Lane: Sergey Burladyan [EMAIL PROTECTED] writes: [ xmlvalidate is a security hole ] Given that this function is not documented nor tested in the regression tests, I propose diking it out entirely. Yes, it was accidentally left over from previous

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Heikki Linnakangas
Decibel! wrote: On Feb 29, 2008, at 8:10 AM, Florian Weimer wrote: In the end, I wouldn't be surprised if for most loads, cache warming effects dominated recovery times, at least when the machine is not starved on RAM. Uh... that's exactly what all the synchronous reads are doing... warming

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Tom Lane
Decibel! [EMAIL PROTECTED] writes: Perhaps a good short-term measure would be to have recovery allocate a 16M buffer and read in entire xlog files at once. If that isn't entirely useless, you need a better kernel. The system should *certainly* be bright enough to do read-ahead for our reads

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Aidan Van Dyk
* Tom Lane [EMAIL PROTECTED] [080229 15:49]: If that isn't entirely useless, you need a better kernel. The system should *certainly* be bright enough to do read-ahead for our reads of the source xlog file. The fetches that are likely to be problematic are the ones for pages in the data

Re: [HACKERS] could not open relation 1663/16384/16584: No such file or directory in a specific combination of transactions with temp tables

2008-02-29 Thread Heikki Linnakangas
Gurjeet Singh wrote: Plausible theory, and nice explanation Try the following link (I had to wait for 50 sec for the link to appear, but I guess the trade-off of getting knowledge in return is worth it :) )

Re: [HACKERS] pg_dump additional options for performance

2008-02-29 Thread Decibel!
On Feb 26, 2008, at 4:36 PM, Tom Lane wrote: I think a sane way to think about what Simon would like to accomplish is not turn psql into a parallel job scheduler My $0.02: I often find myself wishing I could perform parallel operations in psql. There was a proposal for that that came up

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Gregory Stark
Florian G. Pflug wrote: The same holds true for index scans, though. Maybe we can find a solution that benefits both cases - something along the line of a bgreader process I posted a patch to do readahead for bitmap index scans using posix_fadvise. Experiments showed it works great on raid

Re: [HACKERS] Read-ahead and parallelism in redo recovery

2008-02-29 Thread Florian G. Pflug
Greg Stark wrote: Florian G. Pflug wrote: The same holds true for index scans, though. Maybe we can find a solution that benefits both cases - something along the line of a bgreader process I posted a patch to do readahead for bitmap index scans using posix_fadvise. Experiments showed it