Re: [HACKERS] Reducing the overhead of NUMERIC data
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Could someone please quantify how much bang we might get for what seems > like quite a lot of bucks? > I appreciate the need for speed, but the saving here strikes me as > marginal at best, unless my instincts are all wrong (quite possible) Two bytes per numeric value is not a lot, agreed. If we were willing to invent the "varlena2" datum format then we could save four bytes per numeric, plus reduce numeric's alignment requirement from int to short which would probably save another byte per value on average. I'm not sure that that's worth doing if numeric and inet are the only beneficiaries, but it might be. >From a speed perspective the only thing favoring UNKNOWNNUMERIC is the possibility for saving some conversion operations when the grammar's initial guess about datatype is wrong. That's probably pretty marginal though. I was thinking of it more as a vehicle for helping us clean up the type-assignment behavior some more. The example I have in my notes is that "float4col = 1.8" is certain to fail since 1.8 will be taken as float8, not float4, and then you have precision mismatch problems. You can make it work by quoting: "float4col = '1.8'" but that's surely pretty ugly. If the constant were initially UNKNOWNNUMERIC then it would end up as float4 without that hack. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Assert failure found in 8.1RC1
Robert Creager <[EMAIL PROTECTED]> writes: > Ran with both for an hour with no problem, where I could produce the ASSERT > failure within minutes for the non patched version. Great. I'll go ahead and commit the smaller fix into HEAD and the back branches, and hold the larger fix for 8.2. It's curious that two different people stumbled across this just recently, when the bug has been there since 7.2. I suppose that the addition of pg_subtrans increased the probability of seeing the bug by a considerable amount, but I'm still surprised it wasn't identified before. At the very least, we should have heard about it earlier in the 8.0 release cycle ... regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Tuna is running Dave-Cramers-Computer:~ davec$ uname -a Darwin Dave-Cramers-Computer.local 8.2.0 Darwin Kernel Version 8.2.0: Fri Jun 24 17:46:54 PDT 2005; root:xnu-792.2.4.obj~3/RELEASE_PPC Power Macintosh powerpc gcc --version powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. build 4061) On 2-Nov-05, at 1:57 PM, Jim C. Nasby wrote: On Wed, Nov 02, 2005 at 06:05:19PM +0100, Idar Tollefsen wrote: Tom Lane wrote: I notice however that you seem to have a different version of gcc: gcc --version: powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 (Apple Computer, Inc. build 5026) Copyright (C) 2005 Free Software Foundation, Inc. Mine says powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. build 4061) Copyright (C) 2004 Free Software Foundation, Inc. which is evidently older. Where'd you get yours from? I believe 5026 came as part of the Xcode 2.1 package when I upgraded it from Xcode 2.0. Xcode et. al. can be downloaded from ADC (http://developer.apple.com/, choose "Log in"). The 4061 build you have, that came with Xcode 2.0, is a pre- release of GCC 4.0. The 5026 build is synchronized with the official 4.0 release. Good catch tough. Could you try the Xcode 2.1 upgrade and see if you get the same errors I'm seeing? Or could anyone tell us what GCC version "tuna" is running? If tuna is upgraded to the latest version of the buildfarm script it will report config output even on sucessful builds, which will contain gcc info. IE: http://lnk.nu/pgbuildfarm.org/5ii.pl (search for 'configure:2078'). -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Reducing the overhead of NUMERIC data
[patches removed] Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: It seems straightforward enough to put in an additional test, similar to the ones already there so that if its too big for a decimal we make it a float straight away - only a float can be that big in that case. After that I can't really see what the problem is? Wrong answer. You'll be introducing weird corner cases into the type resolution behavior. An approach that would actually have some credibility would be to not resolve constants to NUMERIC right away, but to invent an UNKNOWNNUMERIC pseudotype with coercion behavior comparable to the existing UNKNOWN type for string literals. This has been proposed before but hasn't really been needed so far. Of course, this converts the project from a minor localized hack on NUMERIC into a major piece of fiddling with the type resolution rules, with the potential for unforeseen side-effects on the behavior of other data types. It might be worth doing anyway --- I don't recall at the moment what problems UNKNOWNNUMERIC was intended to solve, but if they're still open issues then it's something we ought to get around to sometime. Could someone please quantify how much bang we might get for what seems like quite a lot of bucks? I appreciate the need for speed, but the saving here strikes me as marginal at best, unless my instincts are all wrong (quite possible) cheers andrew ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Wed, Nov 02, 2005 at 06:12:37PM -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > It seems straightforward enough to put in an additional test, similar to > > the ones already there so that if its too big for a decimal we make it a > > float straight away - only a float can be that big in that case. After > > that I can't really see what the problem is? > > Wrong answer. You'll be introducing weird corner cases into the type > resolution behavior. > > An approach that would actually have some credibility would be to not > resolve constants to NUMERIC right away, but to invent an UNKNOWNNUMERIC > pseudotype with coercion behavior comparable to the existing UNKNOWN > type for string literals. This has been proposed before but hasn't > really been needed so far. Of course, this converts the project from a > minor localized hack on NUMERIC into a major piece of fiddling with the > type resolution rules, with the potential for unforeseen side-effects on > the behavior of other data types. It might be worth doing anyway --- I > don't recall at the moment what problems UNKNOWNNUMERIC was intended to > solve, but if they're still open issues then it's something we ought to > get around to sometime. Thought I'd look to see if I could find anything about UNKNOWNNUMERIC, but no such luck (ISTM we really need a better way to find discussion on old ideas...) But while looking I did find this TODO, which might be relevant to the current discussion: # Change NUMERIC to enforce the maximum precision, and increase it Unfortunately I can't find any reference to that in the archives... -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Assert failure found in 8.1RC1
On Wed, 02 Nov 2005 15:19:44 -0500
Tom Lane <[EMAIL PROTECTED]> wrote:
> Robert Creager <[EMAIL PROTECTED]> writes:
> > TRAP: FailedAssertion("!(shared->page_number[slotno] == pageno &&
> > shared->page_status[slotno] == SLRU_PAGE_READ_IN_PROGRESS)", File: "slru.c",
> > Line: 309)
>
> http://archives.postgresql.org/pgsql-hackers/2005-10/msg01385.php
>
> If you can reproduce the failure with any reliability, please try
> one or both of the proposed patches:
>
> http://archives.postgresql.org/pgsql-patches/2005-10/msg00240.php
> http://archives.postgresql.org/pgsql-patches/2005-10/msg00248.php
>
Ran with both for an hour with no problem, where I could produce the ASSERT
failure within minutes for the non patched version.
Thanks,
Rob
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Reducing the overhead of NUMERIC data
Simon Riggs <[EMAIL PROTECTED]> writes: > It seems straightforward enough to put in an additional test, similar to > the ones already there so that if its too big for a decimal we make it a > float straight away - only a float can be that big in that case. After > that I can't really see what the problem is? Wrong answer. You'll be introducing weird corner cases into the type resolution behavior. An approach that would actually have some credibility would be to not resolve constants to NUMERIC right away, but to invent an UNKNOWNNUMERIC pseudotype with coercion behavior comparable to the existing UNKNOWN type for string literals. This has been proposed before but hasn't really been needed so far. Of course, this converts the project from a minor localized hack on NUMERIC into a major piece of fiddling with the type resolution rules, with the potential for unforeseen side-effects on the behavior of other data types. It might be worth doing anyway --- I don't recall at the moment what problems UNKNOWNNUMERIC was intended to solve, but if they're still open issues then it's something we ought to get around to sometime. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Wed, 2005-11-02 at 15:09 -0500, Tom Lane wrote: > [ thinks for a moment... ] Actually, neither proposal is going to get > off the ground, because the parser's handling of numeric constants is > predicated on the assumption that type NUMERIC can handle any valid > value of FLOAT8, and so we can get away with converting to NUMERIC on > sight and then coercing to float later if parse analysis finds out the > constant should be float. If the dynamic range of NUMERIC is less than > 10^308 then this fails. So we have to find another bit somewhere, or > the idea is dead in the water. We convert a Value node to a Const in backend/parser/parse_node.c:make_const It seems straightforward enough to put in an additional test, similar to the ones already there so that if its too big for a decimal we make it a float straight away - only a float can be that big in that case. After that I can't really see what the problem is? There is only a single call where numeric_float8() is called anywhere in the code, which is during selectivity calculations. In that case we actually call numeric_float8_no_overflow(). If its a FLOAT8OID, then we can simply avoid the conversion, since it already would be one. Can you explain further? Thanks, Best Regards, Simon Riggs ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Idar Tollefsen <[EMAIL PROTECTED]> writes: > *blush* Found it. > Any one the following flags will produce the described problems (alone > or in combination): > -faltivec > -mabi=altivec > -maltivec > -mcpu= Fascinating. I tried this on my own machine, and found out that -faltivec causes Darwin's gcc to add about a dozen predefined macros (compare -dM output with and without it). The gem that's killing us is #define bool bool but it tromps on user identifier space in some other ways too, including #define'ing "pixel" and "vector". I cannot imagine any sane use for a macro defined as above, so I'd suggest filing a bug report with Apple. We could kluge our way around this with something like #ifdef __APPLE_CC__ #undef bool #endif in c.h, but I'm a little worried what impact this might have on the system header files. Why in the world have they got it doing that? Meanwhile, this is a good tip to have in our archives. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags
On Wed, Nov 02, 2005 at 02:04:09PM -0500, Tom Lane wrote: > "Jim C. Nasby" <[EMAIL PROTECTED]> writes: > > BTW, that's a reversal from what I was originally arguing for, which was > > due to the performance penalty associated with --enable-cassert. My > > client is now running with Tom's suggestion of commenting out > > CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING and performance is > > good. It appears to be as good as it was with asserts disabled. > > Interesting. I've always wondered whether the "debug_assertions" GUC > variable is worth the electrons it's printed on. If you are running > with asserts active, that variable actually slows things down, by > requiring an additional bool test for every Assert. I suppose the > motivation was to allow the same compiled executable to be used for both > assert-enabled and assert-disabled runs, but how many people really need > that capability? Not sure how that relates to CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING :P, but I agree that it doesn't make sense to have a GUC, at least not if asserts default to being compiled out. Hrm... does debug_assertions end up changing assert_enabled? BTW, is MEMORY_CONTEXT_CHECKING that expensive? It seems like it shouldn't be, but I'm only guessing at what exactly it does... -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Tom Lane wrote: Also it would be worth showing in more detail exactly what happens when you try to build the (unpatched) sources. *blush* Found it. Any one the following flags will produce the described problems (alone or in combination): -faltivec -mabi=altivec -maltivec -mcpu= I have these, and others, set because of other development work I do. And I always forget them when I just want to build something else. My bad. As a curious side note, MySQL (4.1.13a and pre-releases of 5.0) had trouble with the exact same flags. Other pieces of software I have compiled (CommonC++2, TAO, wxWidgets, bison, curl, OmniORB, Apache 2, etc., etc.) had no trouble with these, even if they were there unintentionally. - IT ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] ODBC link to other databases
Edward, > To help with my work, I've created a basic plugin that allows Postgres > to retreive data from tables on other databases via ODBC. So far I've > only tested it against MS SQL Server 2000. It's rather crude at the > moment, as the only operation supported is a SELECT * query, and it > currently requires the username/password to be hard coded into the > source code. http://pgfoundry.org/projects/dbi-link/ http://pgfoundry.org/projects/dblink-tds/ -- --Josh Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Idar Tollefsen <[EMAIL PROTECTED]> writes: > I'll fire up grep in the morning and search for headers defining bool, > true and false. Also it would be worth showing in more detail exactly what happens when you try to build the (unpatched) sources. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Tom Lane wrote: Well, I updated to Xcode 2.1 and ... it still works. So that's not the explanation. I suppose it was a long shot. But thanks for the effort. I'm currently wondering if you have any nonstandard versions of system headers floating about, perhaps in /usr/local/include. Well... Yes and no. I do have some in /usr/pkg/include (pkgsrc from NetBSD), but that's not a location the compiler should pick up unless instructed to. I see tuna is using Fink, but I don't know what it's picking up from there. Readline? I'll fire up grep in the morning and search for headers defining bool, true and false. - IT ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Wed, 2005-11-02 at 15:09 -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > I wasn't trying to claim the bit assignment made sense. My point was > > that the work to mangle the two fields together to make it make sense > > looked like it would take more CPU (since the standard representation of > > signed integers is different for +ve and -ve values). It is the more CPU > > I'm worried about, not the wasted bits on the weight. > > I think that's purely hypothetical. The existing representation, as > well as the one you propose, both require shift-and-mask operations > to pull the fields out of the packed format. The format I'm suggesting > would require some different shift-and-mask operations. As a first > approximation it'd be a wash, and any actual differences would be > CPU-specific enough that we shouldn't put a whole lot of weight on the > point. (C compilers tend to be fairly bright about optimizing field > extraction operations.) OK > Moreover, you've forgotten the basic gating factor here, which is > whether such a proposal will get accepted at all. Reducing the > available range from 1000 digits to 255 might pass without too much > objection, but dropping it down another factor of 4 to 63 starts to > bring it uncomfortably close to mattering to people. > > [ thinks for a moment... ] Actually, neither proposal is going to get > off the ground, because the parser's handling of numeric constants is > predicated on the assumption that type NUMERIC can handle any valid > value of FLOAT8, and so we can get away with converting to NUMERIC on > sight and then coercing to float later if parse analysis finds out the > constant should be float. If the dynamic range of NUMERIC is less than > 10^308 then this fails. So we have to find another bit somewhere, or > the idea is dead in the water. Well, that certainly is obscure, I'll give you that. 308 huh? The middle ground between 64 and 308 is somewhere around 255, yes? :-) I'll get on it. Including Catch-308. Best Regards, Simon Riggs ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
Bruce Momjian wrote: Andrew Dunstan wrote: There is no fseeko in the Windows libraries, nor any provision in the mingw headers that I can see for a 64 bit off_t. So we would need to roll our own to some extent - I think we need more than just a bit of configure cleverness. However, there is a Windows library routine to do a 64bit seek and return the file position, so we could fairly easily implement fseeko and ftello based on that. See http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp See src/port/fseeko.c for a version built on fsetpos(). Yeah, I'm not made warm and fuzzy by the comments about fpos_t in mingw's stdio.h, though: /* * An opaque data type used for storing file positions... The contents of * this type are unknown, but we (the compiler) need to know the size * because the programmer using fgetpos and fsetpos will be setting aside * storage for fpos_t structres. Actually I tested using a byte array and * it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL). * Perhaps an unsigned long? TODO? It's definitely a 64-bit number in * MSVCRT however, and for now `long long' will do. */ But the example program on MSDN contains "pos = 14", which leads one to assume that it really is some simple int underneath. cheers andrew cheers andrew ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] ODBC link to other databases
Hello all, To help with my work, I've created a basic plugin that allows Postgres to retreive data from tables on other databases via ODBC. So far I've only tested it against MS SQL Server 2000. It's rather crude at the moment, as the only operation supported is a SELECT * query, and it currently requires the username/password to be hard coded into the source code. I was wondering if other people are interested in this enough for me to bother with the trouble of polishing this off. I also have no idea what would be a sane security model for this. I'd appreciate people letting me know if there is interest in this... Ed ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
If tuna is upgraded to the latest version of the buildfarm script it will report config output even on sucessful builds, which will contain gcc info. IE: http://lnk.nu/pgbuildfarm.org/5ii.pl (search for 'configure:2078'). She is apparently not. I can't find any stage logs in there. But thanks for the hint, I'll be sure to look for them in the future. - IT ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
Andrew Dunstan wrote: > There is no fseeko in the Windows libraries, nor any provision in the > mingw headers that I can see for a 64 bit off_t. So we would need to > roll our own to some extent - I think we need more than just a bit of > configure cleverness. > > However, there is a Windows library routine to do a 64bit seek and > return the file position, so we could fairly easily implement fseeko and > ftello based on that. See > http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp See src/port/fseeko.c for a version built on fsetpos(). -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: There is no fseeko in the Windows libraries, nor any provision in the mingw headers that I can see for a 64 bit off_t. So we would need to roll our own to some extent - I think we need more than just a bit of configure cleverness. However, there is a Windows library routine to do a 64bit seek and return the file position, so we could fairly easily implement fseeko and ftello based on that. See http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp Is there any risk that the mingw libraries would fail when manipulating a file whose current offset exceeds 32 bits? I'm wondering if we'd have to roll our own stdio in toto, not just fseeko/ftello :-( AFAIK all this is coming straight from the base Windows libraries. I guess we'll have to test it when we get around to it. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Assert failure found in 8.1RC1
On Wed, 02 Nov 2005 15:37:05 -0500 Tom Lane <[EMAIL PROTECTED]> wrote: > Robert Creager <[EMAIL PROTECTED]> writes: > > I can reproduce very quickly. Looks like I should try the patch in 248 > > first to see if it fixes 8.1RC1? > > Excellent. Yes, the second patch is higher priority, but please try > both while you're at it. > I've put in patch 2. I'm kicking the s**t out of it, with no problems so far. I'll let it run for a while longer. One note is that I did hit the CS switch problem, but with a combination of production app and my test app. But, it took much more activity, wasn't as severe (queries were typically staying < 10 seconds) and the db came out of it a few minutes after my test app stopped. I'll put in the first patch and re-run the tests. Cheers, Rob ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Wed, Nov 02, 2005 at 12:53:07PM -0600, Jim C. Nasby wrote: > > This is one of those issues where we need to run tests and take input. > > We cannot decide this sort of thing just by debate alone. So, I'll leave > > this as a less potentially fruitful line of enquiry. > > Is it worth comming up with some script that users can run against a > table to provide us with real data? Like I said, I have a few columns of numeric(12,4). They're costs in cent, to 4 decimal places. The test is (column = trunc(column)). Sample data: col1 | col2 | col3 | col4 -+-++- 21. | 10.1818 | 0. | 21. 22. | 11.2727 | 0. | 22. 22. | 6.0909 | 0. | 22. For each column (across 17 million rows): Col 1: 83% trailing zeros Col 2: 49% Col 3: 94% Col 4: 83% AIUI, I currently get the four decimal places for free, and the idea is to store them explicitly. Fact is, things that cost fractions of cents are not that common, in this database anyway. As for the argument in general, this table is so wide that any gain will vanish into the slack at the end of a block so it won't actually matter... Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a > tool for doing 5% of the work and then sitting around waiting for someone > else to do the other 95% so you can sue them. pgp3tSADdqX1T.pgp Description: PGP signature
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
Andrew Dunstan <[EMAIL PROTECTED]> writes: > There is no fseeko in the Windows libraries, nor any provision in the > mingw headers that I can see for a 64 bit off_t. So we would need to > roll our own to some extent - I think we need more than just a bit of > configure cleverness. > However, there is a Windows library routine to do a 64bit seek and > return the file position, so we could fairly easily implement fseeko and > ftello based on that. See > http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp Is there any risk that the mingw libraries would fail when manipulating a file whose current offset exceeds 32 bits? I'm wondering if we'd have to roll our own stdio in toto, not just fseeko/ftello :-( regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
Magnus Hagander wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: Hmm. Windows reports an offset size of 4 bytes on a dump I just made ... is that relevant? What governs it? [ looks again... ] Hm, that is a 40Gb dump Kevin is complaining of, isn't it. Do our Windows builds have LARGEFILE support? I think from a cursory investigation the short answer is no, but they probably could. If so, that should definitely go on the TODO list. Windows gurus, any thoughts? Windows certainly supports large files. I don't see why we wouldn't pick this up in autoconf, perhaps the mingw libraries don't? Definitly worth investigating, no time for 8.1, so putting it on TODO seems very good :-) There is no fseeko in the Windows libraries, nor any provision in the mingw headers that I can see for a 64 bit off_t. So we would need to roll our own to some extent - I think we need more than just a bit of configure cleverness. However, there is a Windows library routine to do a 64bit seek and return the file position, so we could fairly easily implement fseeko and ftello based on that. See http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Assert failure found in 8.1RC1
Robert Creager <[EMAIL PROTECTED]> writes:
> TRAP: FailedAssertion("!(shared->page_number[slotno] == pageno &&
> shared->page_status[slotno] == SLRU_PAGE_READ_IN_PROGRESS)", File: "slru.c",
> Line: 309)
http://archives.postgresql.org/pgsql-hackers/2005-10/msg01385.php
If you can reproduce the failure with any reliability, please try
one or both of the proposed patches:
http://archives.postgresql.org/pgsql-patches/2005-10/msg00240.php
http://archives.postgresql.org/pgsql-patches/2005-10/msg00248.php
regards, tom lane
---(end of broadcast)---
TIP 6: explain analyze is your friend
Re: [HACKERS] Reducing the overhead of NUMERIC data
Simon Riggs <[EMAIL PROTECTED]> writes: > I wasn't trying to claim the bit assignment made sense. My point was > that the work to mangle the two fields together to make it make sense > looked like it would take more CPU (since the standard representation of > signed integers is different for +ve and -ve values). It is the more CPU > I'm worried about, not the wasted bits on the weight. I think that's purely hypothetical. The existing representation, as well as the one you propose, both require shift-and-mask operations to pull the fields out of the packed format. The format I'm suggesting would require some different shift-and-mask operations. As a first approximation it'd be a wash, and any actual differences would be CPU-specific enough that we shouldn't put a whole lot of weight on the point. (C compilers tend to be fairly bright about optimizing field extraction operations.) Moreover, you've forgotten the basic gating factor here, which is whether such a proposal will get accepted at all. Reducing the available range from 1000 digits to 255 might pass without too much objection, but dropping it down another factor of 4 to 63 starts to bring it uncomfortably close to mattering to people. [ thinks for a moment... ] Actually, neither proposal is going to get off the ground, because the parser's handling of numeric constants is predicated on the assumption that type NUMERIC can handle any valid value of FLOAT8, and so we can get away with converting to NUMERIC on sight and then coercing to float later if parse analysis finds out the constant should be float. If the dynamic range of NUMERIC is less than 10^308 then this fails. So we have to find another bit somewhere, or the idea is dead in the water. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Wed, 2005-11-02 at 13:46 -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Tue, 2005-11-01 at 17:55 -0500, Tom Lane wrote: > >> I don't think it'd be worth having 2 types. Remember that the weight is > >> measured in base-10k digits. Suppose for instance > >>sign1 bit > >>weight 7 bits (-64 .. +63) > >>dscale 8 bits (0..255) > > > I've coded a short patch to do this, which is the result of two > > alternate patches and some thinking, but maybe not enough yet. > > What your patch does is Thanks for checking this so quickly. > > sign2 bits OK, thats just a mistake in my second patch. Thats easily corrected. Please ignore that for now. > weight 8 bits (-128..127) > dscale 6 bits (0..63) > > which is simply pretty lame: weight effectively has a factor of 8 more > dynamic range than dscale in this representation. What's the point of > being able to represent 1 * 1 ^ -128 (ie, 10^-512) if the dscale > only lets you show 63 fractional digits? You've got to allocate the > bits in a saner fashion. Yes, that takes a little more work. I wasn't trying to claim the bit assignment made sense. My point was that the work to mangle the two fields together to make it make sense looked like it would take more CPU (since the standard representation of signed integers is different for +ve and -ve values). It is the more CPU I'm worried about, not the wasted bits on the weight. Spending CPU cycles on *all* numerics just so we can have numbers with > +/-64 decimal places doesn't seem a good trade. Hence I stuck the numeric sign back on the dscale, and so dscale and weight seem out of balance. So, AFAICS, the options are: 0 (current cvstip) Numeric range up to 1000, with additional 2 bytes per column value 1. Numeric range up to 128, but with overhead to extract last bit 2. Numeric range up to 64 I'm suggesting we choose (2) other views are welcome. (I'll code it whichever way we decide.) > Also, since the internal (unpacked) calculation representation has a > much wider dynamic range than this, it'd probably be appropriate to add > some range checks to the code that forms a packed value from unpacked. Well, there already is one that does that, otherwise I would have added one as you suggest. (The unpacked code has int values, whereas the previous packed format used u/int16 values). Best Regards, Simon Riggs ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] Assert failure found in 8.1RC1
Hey all,
While trying to get a reproducible test case for my CS storm problem (see
http://archives.postgresql.org/pgsql-hackers/2005-10/msg00585.php), I upgraded
to 8.1RC1 and encountered the following assert:
TRAP: FailedAssertion("!(shared->page_number[slotno] == pageno &&
shared->page_status[slotno] == SLRU_PAGE_READ_IN_PROGRESS)", File: "slru.c",
Line: 309)
On the good side, I'm yet unable to get a sustained CS storm anymore with this
level of code. Looks like something might changed for the better in the last 2
weeks?
For the assert, I had 5 sets of my app running, each with 8 potential
outstanding queries. I then threw my test at the db with 20 more queries, and
took the above failure.
creagrs=# select version();
version
---
--
PostgreSQL 8.1RC1 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.3.1
(Mandrake Linux 9.2 3.3.1-2mdk)
BINDIR = /usr/local/pgsql810/bin
DOCDIR = /usr/local/pgsql810/doc
INCLUDEDIR = /usr/local/pgsql810/include
PKGINCLUDEDIR = /usr/local/pgsql810/include
INCLUDEDIR-SERVER = /usr/local/pgsql810/include/server
LIBDIR = /usr/local/pgsql810/lib
PKGLIBDIR = /usr/local/pgsql810/lib
LOCALEDIR =
MANDIR = /usr/local/pgsql810/man
SHAREDIR = /usr/local/pgsql810/share
SYSCONFDIR = /usr/local/pgsql810/etc
PGXS = /usr/local/pgsql810/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--enable-syslog' '--prefix=/usr/local/pgsql810' '--enable-debug'
'--enable-cassert'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wendif-labels
-fno-strict-aliasing -g
CFLAGS_SL = -fpic
LDFLAGS = -Wl,-rpath,/usr/local/pgsql810/lib
LDFLAGS_SL =
LIBS = -lpgport -lz -lreadline -lncurses -lcrypt -lresolv -lnsl -ldl -lm -lbsd
VERSION = PostgreSQL 8.1RC1
Thanks,
Rob
--
Robert Creager
Advisory Software Engineer
Data Management Group
Sun Microsystems
[EMAIL PROTECTED]
303.673.2365 Office
888.912.4458 Pager
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Idar Tollefsen <[EMAIL PROTECTED]> writes: > I believe 5026 came as part of the Xcode 2.1 package when I upgraded it from > Xcode 2.0. Xcode et. al. can be downloaded from ADC > (http://developer.apple.com/, choose "Log in"). Well, I updated to Xcode 2.1 and ... it still works. So that's not the explanation. I'm currently wondering if you have any nonstandard versions of system headers floating about, perhaps in /usr/local/include. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > On Wed, Nov 02, 2005 at 06:05:19PM +0100, Idar Tollefsen wrote: >> Good catch tough. Could you try the Xcode 2.1 upgrade and see if you get >> the same errors I'm seeing? Or could anyone tell us what GCC version "tuna" >> is running? > If tuna is upgraded to the latest version of the buildfarm script it > will report config output even on sucessful builds, which will contain > gcc info. That reminds me --- I was going to ask Andrew to pressure all the buildfarm owners to update to current copies of the farm scripts. Quite a lot of the machines are not reporting nearly as much info as I would like to see. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > BTW, that's a reversal from what I was originally arguing for, which was > due to the performance penalty associated with --enable-cassert. My > client is now running with Tom's suggestion of commenting out > CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING and performance is > good. It appears to be as good as it was with asserts disabled. Interesting. I've always wondered whether the "debug_assertions" GUC variable is worth the electrons it's printed on. If you are running with asserts active, that variable actually slows things down, by requiring an additional bool test for every Assert. I suppose the motivation was to allow the same compiled executable to be used for both assert-enabled and assert-disabled runs, but how many people really need that capability? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
On Wed, Nov 02, 2005 at 06:05:19PM +0100, Idar Tollefsen wrote: > Tom Lane wrote: > >I notice however that you seem to have a different version of gcc: > > > >>gcc --version: > >>powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 (Apple Computer, Inc. build > >>5026) > >>Copyright (C) 2005 Free Software Foundation, Inc. > > > >Mine says > > > >powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. > >build 4061) > >Copyright (C) 2004 Free Software Foundation, Inc. > > > >which is evidently older. Where'd you get yours from? > > I believe 5026 came as part of the Xcode 2.1 package when I upgraded it > from Xcode 2.0. Xcode et. al. can be downloaded from ADC > (http://developer.apple.com/, choose "Log in"). > > The 4061 build you have, that came with Xcode 2.0, is a pre-release of GCC > 4.0. The 5026 build is synchronized with the official 4.0 release. > > Good catch tough. Could you try the Xcode 2.1 upgrade and see if you get > the same errors I'm seeing? Or could anyone tell us what GCC version "tuna" > is running? If tuna is upgraded to the latest version of the buildfarm script it will report config output even on sucessful builds, which will contain gcc info. IE: http://lnk.nu/pgbuildfarm.org/5ii.pl (search for 'configure:2078'). -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Wed, Nov 02, 2005 at 08:48:25AM +, Simon Riggs wrote: > On Tue, 2005-11-01 at 18:15 -0500, Tom Lane wrote: > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > Anybody like to work out a piece of SQL to perform data profiling and > > > derive the distribution of values with trailing zeroes? > > > > Don't forget leading zeroes. And all-zero (we omit digits entirely in > > that case). I don't think you can claim that zero isn't a common case. > > The question is: how common? > > For INTEGERs I would accept that many are often zero. For NUMERIC, these > are seldom exactly zero, IMHO. > > This is one of those issues where we need to run tests and take input. > We cannot decide this sort of thing just by debate alone. So, I'll leave > this as a less potentially fruitful line of enquiry. Is it worth comming up with some script that users can run against a table to provide us with real data? -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Reducing the overhead of NUMERIC data
Simon Riggs <[EMAIL PROTECTED]> writes: > On Tue, 2005-11-01 at 17:55 -0500, Tom Lane wrote: >> I don't think it'd be worth having 2 types. Remember that the weight is >> measured in base-10k digits. Suppose for instance >> sign1 bit >> weight 7 bits (-64 .. +63) >> dscale 8 bits (0..255) > I've coded a short patch to do this, which is the result of two > alternate patches and some thinking, but maybe not enough yet. What your patch does is sign2 bits weight 8 bits (-128..127) dscale 6 bits (0..63) which is simply pretty lame: weight effectively has a factor of 8 more dynamic range than dscale in this representation. What's the point of being able to represent 1 * 1 ^ -128 (ie, 10^-512) if the dscale only lets you show 63 fractional digits? You've got to allocate the bits in a saner fashion. Yes, that takes a little more work. Also, since the internal (unpacked) calculation representation has a much wider dynamic range than this, it'd probably be appropriate to add some range checks to the code that forms a packed value from unpacked. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags
On Wed, Nov 02, 2005 at 07:03:57AM -0500, Greg Stark wrote: > > I would bet that ninety percent of the Asserts in the existing code are on > > conditions that could represent, at worst, corruption of backend-local or > > even transaction-local data structures. Taking down the entire database > > cluster for that is not something that sounds like a stability-enhancing > > tradeoff to me. > > It may be minor corruption or it may be that the reason for the minor > corruption comes from some larger bug. It may also be backend-local or > transaction-local corruption at the time the assert catches it but cause major > damage by the time it actually crashes a non-assert-enabled database. Agreed. Personally I'd want to know about anything that corrupts my data, no matter what the locality. I would also argue that if people are seeing 'minor' asserts firing in production that there's a bug that needs to be tracked down. IF it comes out that there's some asserts that can be fired even though there's not anything really bad happening, they could always be relegated to a second class of assert that's not normally turned on. BTW, that's a reversal from what I was originally arguing for, which was due to the performance penalty associated with --enable-cassert. My client is now running with Tom's suggestion of commenting out CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING and performance is good. It appears to be as good as it was with asserts disabled. So I think it would definately be good to break those options out from --enable-cassert. That makes it viable to run with asserts in production, at least from a performance standpoint. BTW, they're also running with patch2 now. Previously, with asserts turned on and without the patch, they were seeing assert failures on average of 2/day. So hopefully tomorrow we'll have an idea if the patch fixed this or not. -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Tue, 2005-11-01 at 17:55 -0500, Tom Lane wrote:
> "Jim C. Nasby" <[EMAIL PROTECTED]> writes:
> > FWIW, most databases I've used limit NUMERIC to 38 digits, presumably to
> > fit length info into 1 or 2 bytes. So there's something to be said for a
> > small numeric type that has less overhead and a large numeric (what we
> > have today).
>
> I don't think it'd be worth having 2 types. Remember that the weight is
> measured in base-10k digits. Suppose for instance
> sign1 bit
> weight 7 bits (-64 .. +63)
> dscale 8 bits (0..255)
> This gives us a dynamic range of 1e-256 to 1e255 as well as the ability
> to represent up to 255 displayable fraction digits. Does anyone know
> any real database applications where that's not enough?
>
> (I'm neglecting NaN here in supposing we need only 1 bit for sign,
> but we could have a special encoding for NaN. Perhaps disallow the
> weight = -64 case and use that to signal NaN.)
I've coded a short patch to do this, which is the result of two
alternate patches and some thinking, but maybe not enough yet.
The patch given here is different on two counts from above:
This sets...
#define NUMERIC_MAX_PRECISION 64
since
#define NUMERIC_MAX_RESULT_SCALE(NUMERIC_MAX_PRECISION * 2)
We don't seem to be able to use all of the bits actually available to us
in the format. Perhaps we need to decouple these now? Previously, we had
room for 14 bits, which gave a maximum of 16384. We were using
NUMERIC_MAX of 1000, so doubling it didn't give problems.
The above on-disk format showed sign & weight together, whereas the
current code has sign and dscale together. Trying to put sign and weight
together is somewhat difficult, since weight is itself a signed value.
I coded it up that way around, which is reasonably straightforward but
harder than the patch enclosed here. But AFAICS - which isn't that far
normally I grant you, doing things that way around would require some
twos-complement work to get things correct when weight is negative. That
worries me.
IMHO we should accept the step down in maximum numeric precision (down
to "only" 64 digits) rather than put extra processing into every
manipulation of a NUMERIC datatype. With luck, I've misunderstood and we
can have both performance and precision.
If not, I commend 64 digits to you as sufficient for every imaginable
purpose - saving 2 bytes off every numeric column. (And still 28 decimal
places more accurate than Oracle).
Best Regards, Simon Riggs
Index: src/include/utils/numeric.h
===
RCS file: /projects/cvsroot/pgsql/src/include/utils/numeric.h,v
retrieving revision 1.20
diff -c -c -r1.20 numeric.h
*** src/include/utils/numeric.h 1 Jan 2005 05:43:09 - 1.20
--- src/include/utils/numeric.h 2 Nov 2005 18:06:03 -
***
*** 15,24
#define _PG_NUMERIC_H_
/*
! * Hardcoded precision limit - arbitrary, but must be small enough that
! * dscale values will fit in 14 bits.
*/
! #define NUMERIC_MAX_PRECISION 1000
/*
* Internal limits on the scales chosen for calculation results
--- 15,24
#define _PG_NUMERIC_H_
/*
! * Hardcoded precision limit - must be small enough that
! * dscale values will fit into the number of bits available in NumericData
*/
! #define NUMERIC_MAX_PRECISION 64
/*
* Internal limits on the scales chosen for calculation results
***
*** 39,49
/*
* Sign values and macros to deal with packing/unpacking n_sign_dscale
*/
! #define NUMERIC_SIGN_MASK 0xC000
! #define NUMERIC_POS 0x
! #define NUMERIC_NEG 0x4000
! #define NUMERIC_NAN 0xC000
! #define NUMERIC_DSCALE_MASK 0x3FFF
#define NUMERIC_SIGN(n) ((n)->n_sign_dscale & NUMERIC_SIGN_MASK)
#define NUMERIC_DSCALE(n) ((n)->n_sign_dscale & NUMERIC_DSCALE_MASK)
#define NUMERIC_IS_NAN(n) (NUMERIC_SIGN(n) != NUMERIC_POS && \
--- 39,49
/*
* Sign values and macros to deal with packing/unpacking n_sign_dscale
*/
! #define NUMERIC_SIGN_MASK 0xC0
! #define NUMERIC_POS 0x00
! #define NUMERIC_NEG 0x40
! #define NUMERIC_NAN 0xC0
! #define NUMERIC_DSCALE_MASK 0x3F
#define NUMERIC_SIGN(n) ((n)->n_sign_dscale & NUMERIC_SIGN_MASK)
#define NUMERIC_DSCALE(n) ((n)->n_sign_dscale & NUMERIC_DSCALE_MASK)
#define NUMERIC_IS_NAN(n) (NUMERIC_SIGN(n) != NUMERIC_POS && \
***
*** 61,74
typedef struct NumericData
{
int32 varlen; /* Variable size (std varlena header) */
! int16 n_weight; /* Weight of 1st digit */
! uint16 n_sign_dscale; /* Sign + display scale */
char n_data[1]; /* Digits (really array of NumericDigit) */
} NumericData;
typedef NumericData *Numeric;
! #define NUMERIC_HDRSZ (sizeof(int32) + sizeof(int16) + sizeof(uint16))
/*
--- 61,74
typedef struct NumericData
{
int32 varlen; /* Variable size (std varlena header) */
! int8 n_weight; /* Weight of 1st dig
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Tom Lane wrote: I notice however that you seem to have a different version of gcc: gcc --version: powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 (Apple Computer, Inc. build 5026) Copyright (C) 2005 Free Software Foundation, Inc. Mine says powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. build 4061) Copyright (C) 2004 Free Software Foundation, Inc. which is evidently older. Where'd you get yours from? I believe 5026 came as part of the Xcode 2.1 package when I upgraded it from Xcode 2.0. Xcode et. al. can be downloaded from ADC (http://developer.apple.com/, choose "Log in"). The 4061 build you have, that came with Xcode 2.0, is a pre-release of GCC 4.0. The 5026 build is synchronized with the official 4.0 release. Good catch tough. Could you try the Xcode 2.1 upgrade and see if you get the same errors I'm seeing? Or could anyone tell us what GCC version "tuna" is running? - IT ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Idar Tollefsen <[EMAIL PROTECTED]> writes: > Trying to build 8.1RC1 on the above configuration fails because it seems to > have > defined bool, but still doesn't seem to know what bool is. Well, having just updated my laptop to 10.4.3, I tried it again, using the same configure switches you mention ... and it still works fine for me. I notice however that you seem to have a different version of gcc: > gcc --version: > powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 (Apple Computer, Inc. build 5026) > Copyright (C) 2005 Free Software Foundation, Inc. Mine says powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. build 4061) Copyright (C) 2004 Free Software Foundation, Inc. which is evidently older. Where'd you get yours from? regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Tom Lane wrote: OS X 10.4.what-exactly? And did you install the 10.4 Xcode tools? PG builds fine for me on 10.4.2, and there's at least one 10.4.something machine in the buildfarm, so it's not broken in general. With all due respect, please read my original posting again. uname output is there, indicating 10.4.3 (although I had the same problem with beta1 on 10.4.2), and the GCC --version output indicates that it's the Apple version of GCC, which is only installed with the Xcode package. Furthermore, it also states that I'm aware of the OS X machines in the build farm. - IT ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Retrieving all possible join trees from PLANNER..
Gayathri TK <[EMAIL PROTECTED]> writes: > How do i efficiently retrieve this information? You don't, because the planner doesn't actually calculate all possible join trees --- it wouldn't run in a reasonable amount of time if it tried. It only considers trees that are built from subtrees that appear optimal for their subset of the relations. If that's enough for your purpose, you could hack the planner to save aside the info somewhere. Note that the representation is not a finished Plan tree, only a Path tree. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] pg_restore [archiver] file offset in dump file
I think you're on to something, Magnus. From a Windows cmd prompt the size matches Linux: 11/01/2005 12:35 PM40,874,747,876 dtr.dump >From an msys command prompt, ti does not look good: -rw-r--r--1 Administ Administ 18446744071634626532 Nov 1 12:35 dtr.dump I believe we got the latest msys and mingw when we set up the PostgreSQL build environment, but that was a couple months ago. -Kevin >>> "Magnus Hagander" <[EMAIL PROTECTED]> >>> Windows certainly supports large files. I don't see why we wouldn't pick this up in autoconf, perhaps the mingw libraries don't? Definitly worth investigating ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Retrieving all possible join trees from PLANNER..
Hello all, I am using Postgres to do my Masters' thesis work. Given a query, i want to know all possible join plan trees for that query. This information can be got from src/backend/optimizer/path/allpaths.c/make_one_rel_by_joins() method. [ie] more specifically, I need joinitems[levels_needed] The information i need is a temporary result generated by planner and is not passed on to later stages. Hence If i add a command, i wont have this infomation by the time ProcessUtility() method is called. How do i efficiently retrieve this information? The input to my method is a file containing queries, and output should be all possible join plan trees for each query in the file... Thanks in advance, Gayathri TK ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] 8.1RC1 fails to build on OS X (10.4)
Idar Tollefsen <[EMAIL PROTECTED]> writes: > Trying to build 8.1RC1 on the above configuration fails OS X 10.4.what-exactly? And did you install the 10.4 Xcode tools? PG builds fine for me on 10.4.2, and there's at least one 10.4.something machine in the buildfarm, so it's not broken in general. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
> > Andrew Dunstan <[EMAIL PROTECTED]> writes: > >> Hmm. Windows reports an offset size of 4 bytes on a dump I > just made > >> ... is that relevant? What governs it? > > > > [ looks again... ] Hm, that is a 40Gb dump Kevin is > complaining of, > > isn't it. Do our Windows builds have LARGEFILE support? > > > > I think from a cursory investigation the short answer is no, > but they probably could. If so, that should definitely go on > the TODO list. Windows gurus, any thoughts? Windows certainly supports large files. I don't see why we wouldn't pick this up in autoconf, perhaps the mingw libraries don't? Definitly worth investigating, no time for 8.1, so putting it on TODO seems very good :-) //Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] PGXS on VPATH?
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Is PGXS on VPATH builds supported? Probably not in the way you're thinking. IIRC pgxs.mk explicitly resets VPATH, and I think it has to do that because any VPATH embedded in Makefile.global would represent what was done at the time of building the PG installation --- which has zip to do with where the PGXS-using software is located. It might work to explicitly set VPATH from the make command line: make USE_PGXS=1 VPATH=whatever as this should override the assignments in the makefiles. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] 8.1RC1 fails to build on OS X (10.4)
Hello, uname -a: Darwin fulcrum.local 8.3.0 Darwin Kernel Version 8.3.0: Mon Oct 3 20:04:04 PDT 2005; root:xnu-792.6.22.obj~2/RELEASE_PPC Power Macintosh powerpc gcc --version: powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 (Apple Computer, Inc. build 5026) Copyright (C) 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. configuration options: --prefix=/Library/PostgreSQL \ --localstatedir=/var/db/pgsql \ --build=powerpc-apple-darwin8 \ --host=powerpc-apple-darwin8 \ --target=powerpc-apple-darwin8 \ --disable-debug \ --with-openssl \ --with-bonjour \ --with-java \ --enable-thread-safety Trying to build 8.1RC1 on the above configuration fails because it seems to have defined bool, but still doesn't seem to know what bool is. This causes it to fail the thread safety test during configuration and then bomb out during build with errors like these: /Users/idart/src/postgresql-8.1RC1/build/src/../../src/interfaces/ecpg/include/ecpglib.h:84: warning: data definition has no type or storage class /Users/idart/src/postgresql-8.1RC1/build/src/../../src/interfaces/ecpg/include/ecpglib.h:85: error: parse error before 'ECPGdescribe' /Users/idart/src/postgresql-8.1RC1/build/src/../../src/interfaces/ecpg/include/ecpglib.h:85: error: parse error before 'bool' /Users/idart/src/postgresql-8.1RC1/build/src/../../src/interfaces/ecpg/include/ecpglib.h:85: warning: type defaults to 'int' in declaration of 'ECPGdescribe' /Users/idart/src/postgresql-8.1RC1/build/src/../../src/interfaces/ecpg/include/ecpglib.h:85: warning: data definition has no type or storage class make[4]: *** [informix.o] Error 1 make[3]: *** [all] Error 2 make[2]: *** [all] Error 2 make[1]: *** [all] Error 2 make: *** [all] Error 2 The attached patches simply makes sure bool, true and false always gets defined on OS X. This fixes the problems, but I'm afraid it might be a bit naive as the patches don't consider OS X version, GCC version, or any other factors that migth impact the build. Not to mention that it might break things if the patched headers were included in a C++ program. I see that the build farm contains "tuna" with 10.4.2 and GCC 4.0, and apparently it builds just fine on that configuration (albeit without thread safety). If there are other solutions to this, I would appreciate it if someone could point me in the right direction. - IT --- src/include/c.h.orig2005-10-15 04:49:41.0 +0200 +++ src/include/c.h 2005-11-02 10:44:07.0 +0100 @@ -178,15 +178,19 @@ #ifndef __cplusplus -#ifndef bool +#if defined(__APPLE__) && defined(__MACH__) +#define MACOSX +#endif + +#if !defined(bool) || defined(MACOSX) typedef char bool; #endif -#ifndef true +#if !defined(true) || defined(MACOSX) #define true ((bool) 1) #endif -#ifndef false +#if !defined(false) || defined(MACOSX) #define false ((bool) 0) #endif #endif /* not C++ */ --- src/interfaces/ecpg/include/ecpglib.h.orig 2005-11-02 11:01:05.0 +0100 +++ src/interfaces/ecpg/include/ecpglib.h 2005-11-02 12:01:50.0 +0100 @@ -9,19 +9,27 @@ #include "libpq-fe.h" #include "ecpgtype.h" #include - #ifndef __BEOS__ +#ifndef C_H #ifndef __cplusplus -#ifndef bool -#define bool char + +#if defined(__APPLE__) && defined(__MACH__) +#define MACOSX +#endif + +#if !defined(bool) || defined(MACOSX) +typedef char bool; #endif /* ndef bool */ -#ifndef true +#if !defined(true) || defined(MACOSX) #define true ((bool) 1) #endif /* ndef true */ -#ifndef false + +#if !defined(false) || defined(MACOSX) #define false ((bool) 0) #endif /* ndef false */ + +#endif /* C_H */ #endif /* not C++ */ #else /* __BEOS__ */ #include --- src/interfaces/ecpg/pgtypeslib/extern.h.orig2005-11-02 11:01:37.0 +0100 +++ src/interfaces/ecpg/pgtypeslib/extern.h 2005-11-02 12:03:14.0 +0100 @@ -37,10 +37,18 @@ char *pgtypes_alloc(long); char *pgtypes_strdup(char *); -#ifndef bool +#ifndef C_H + +#if defined(__APPLE__) && defined(__MACH__) +#define MACOSX +#endif + +#if !defined(bool) || defined(MACOSX) #define bool char #endif /* ndef bool */ +#endif /* ndef C_H */ + #ifndef FALSE #define FALSE 0 #endif /* FALSE */ --- src/tools/thread/thread_test.c.orig 2005-11-02 10:44:30.0 +0100 +++ src/tools/thread/thread_test.c 2005-11-02 10:40:44.0 +0100 @@ -24,15 +24,19 @@ #include "postgres.h" #else /* From src/include/c.h" */ -#ifndef bool +#if defined(__APPLE__) && defined(__MACH__) +#define MACOSX +#endif + +#if !defined(bool) || defined(MACOSX) typedef char bool; #endif -#ifndef true +#if !defined(true) || defined(MACOSX) #define true ((bool) 1) #endif -#ifndef false +#if !defined(false) || defined(MACOSX) #define false ((bool) 0) #endif #endif
Re: [HACKERS] Reducing the overhead of NUMERIC data
I am not able to quickly find your numeric format, so I'll just throw
this in. MaxDB (I only mention this because the format and algorithms
are now under the GPL, so they can be reviewed by the public) uses a
nifty number format that allows the use memcpy to compare two numbers
when they are in the same precision and scale. Basically, the first
byte contains the sign and number of digits in the number (number of
digits is complemented if the number is negative), then the next N bytes
contain the actual decimal digits, where N is the number of decimal
digits / 2 (so two decimal digits per byte). Trailing 0's are removed
to save space. So,
0 is stored as {128}
1 is stored as {193, 16}
1000 is stored as {196, 16}
1001 is stored as {196, 16, 1} x{C4 10 01}
-1 is stored as {63, 144}
-1001 is stored as {60, 144}
Their storage allows for a max of 63 digits in a number, but it should
be no problem to increase the size to 2 bytes, thus allowing up to
16,385 digits.
The advantages are:
- ability to memcmp two numbers.
- compact storage (can be made more compact if you choose to
save hex digits instead of decimal, but I'm not sure you want to do
that).
The disadvantages are as follows:
- this format does not remember the database definition for the
number (that is, no precision or scale); numeric functions must be told
what they are. It would be nice if the number kept track of that as
well...
- comparing two numbers that are not the same precision and
scale means converting one or both (if both precision and scale are
different you may have to convert both)
- calculations (addition, subtraction, etc) require functions to
extract the digits and do the calculation a digit at a time.
- I do not know of any trig functions, so they would need to be
written
If any one is interested, I would be happy to discuss this further.
Mike Pollard
SUPRA Server SQL Engineering and Support
Cincom Systems, Inc
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
Re: [HACKERS] pg_restore [archiver] file offset in dump file is too
Tom Lane said: > Andrew Dunstan <[EMAIL PROTECTED]> writes: >> Hmm. Windows reports an offset size of 4 bytes on a dump I just made >> ... is that relevant? What governs it? > > [ looks again... ] Hm, that is a 40Gb dump Kevin is complaining of, > isn't it. Do our Windows builds have LARGEFILE support? > I think from a cursory investigation the short answer is no, but they probably could. If so, that should definitely go on the TODO list. Windows gurus, any thoughts? Meanwhile, the answer seems to be to use text dumps if you run into this problem. cheers andrew ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] [ANNOUNCE] PostgreSQL 8.1.0 Release Candidate 1
Am I a bad person for running postresql under windows? I'd like to help you test but I'd a)need a windows binary and b)a list of features to test (perhaps a simple changelog would suffice.) I could be missing the binary, but I don't think so. On 10/30/05, Marc G. Fournier <[EMAIL PROTECTED]> wrote: > > After a couple of months of testing, and alot of bug reports (with fixes), > we are pleased to announce the first Release Candidate of PostgreSQL > 8.1.0. > > As with all pre-releases, but especially now that we are in the final > stretch, testing is paramount to a successful, and bug free, release. As > such, we ask everyone able who is able to do so to, to run RC1 through its > paces and report any bugs to us through [EMAIL PROTECTED] > > At this time, our plans for full release are the week of November 7th, but > this depends on the success of this Release Candidate. > > To download via FTP, please go to: > > http://www.postgresql.org/ftp/source/v8.1beta > > For RPMs, please visit: > > http://developer.postgresql.org/~devrim/rpms/8.1/rc1 > > Windows Binaries to be announced shortly. > > Marc G. Fournier > PostgreSQL Core Group > > ---(end of broadcast)--- > TIP 3: Have you checked our extensive FAQ? > >http://www.postgresql.org/docs/faq > -- "All the soarings of my mind begin in my blood." -Rainer Maria Rilke ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] PGXS on VPATH?
Hi, Is PGXS on VPATH builds supported? I was just testing tsearch2 and it isn't working. Note that I'm first standing on the builddir and it doesn't work; then I'm back at the srcdir and it does work. 07:02 jovan tsearch2 2$ pwd /pgsql/build/00orig/contrib/tsearch2 07:02 jovan tsearch2 0$ LANG=C make USE_PGXS=1 make: *** No rule to make target `tsearch.sql.in', needed by `tsearch2.sql'. Stop. 07:02 jovan tsearch2 2$ cdsrc 07:02 jovan tsearch2 0$ pwd /pgsql/source/00orig/contrib/tsearch2 07:02 jovan tsearch2 0$ LANG=C make USE_PGXS=1 sed -e 's,MODULE_PATHNAME,$libdir/tsearch2,g' tsearch.sql.in >tsearch2.sql cp untsearch.sql.in untsearch2.sql gcc -O2 -Wall -Wmissing-prototypes [...] -- Alvaro Herrera http://www.amazon.com/gp/registry/CTMLCN8V17R4 "Changing the world ... one keyboard at a time!" (www.DVzine.org) ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Reducing the overhead of NUMERIC data
On 11/2/05, Simon Riggs <[EMAIL PROTECTED]> wrote: > On Tue, 2005-11-01 at 18:15 -0500, Tom Lane wrote: > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > Anybody like to work out a piece of SQL to perform data profiling and > > > derive the distribution of values with trailing zeroes? > > > > Don't forget leading zeroes. And all-zero (we omit digits entirely in > > that case). I don't think you can claim that zero isn't a common case. > > The question is: how common? > > For INTEGERs I would accept that many are often zero. For NUMERIC, these > are seldom exactly zero, IMHO. Seconded. My INTEGER data does have a quite a few zeros but most of my NUMERIC columns hold debits and credits. Those are almost never zero. > > This is one of those issues where we need to run tests and take input. > We cannot decide this sort of thing just by debate alone. So, I'll leave > this as a less potentially fruitful line of enquiry. > > Best Regards, Simon Riggs > > > ---(end of broadcast)--- > TIP 5: don't forget to increase your free space map settings > -- Mike Rylander [EMAIL PROTECTED] GPLS -- PINES Development Database Developer http://open-ils.org ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: slru.c race condition (was Re: [HACKERS] TRAP: FailedAssertion("!((itemid)->lp_flags
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > I happen to think that except for the rare assertion that has major > > performance impact all the assertions should be on in production builds. The > > goal of assertions is to catch corruption quickly and that's something > > that's > > just as important in production as it is in debugging. > > You seem not to have read the documentation: Sure I have, I just disagree. > I would bet that ninety percent of the Asserts in the existing code are on > conditions that could represent, at worst, corruption of backend-local or > even transaction-local data structures. Taking down the entire database > cluster for that is not something that sounds like a stability-enhancing > tradeoff to me. It may be minor corruption or it may be that the reason for the minor corruption comes from some larger bug. It may also be backend-local or transaction-local corruption at the time the assert catches it but cause major damage by the time it actually crashes a non-assert-enabled database. -- greg ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Reducing the overhead of NUMERIC data
On Tue, 2005-11-01 at 18:15 -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > Anybody like to work out a piece of SQL to perform data profiling and > > derive the distribution of values with trailing zeroes? > > Don't forget leading zeroes. And all-zero (we omit digits entirely in > that case). I don't think you can claim that zero isn't a common case. The question is: how common? For INTEGERs I would accept that many are often zero. For NUMERIC, these are seldom exactly zero, IMHO. This is one of those issues where we need to run tests and take input. We cannot decide this sort of thing just by debate alone. So, I'll leave this as a less potentially fruitful line of enquiry. Best Regards, Simon Riggs ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
