Re: port/snprintf.c (was Re: [PATCHES] Numeric 508 datatype)
Bruce Momjian writes: > Tom Lane wrote: >> Done ... let me know whether the back branches still pass regression >> for you ;-) > I checked back to 7.3 and everything passed. I did a cvs update, > configure, gmake, and regression run for each branch. [ digs a bit deeper... ] Actually, it appears that that bug didn't exist before 8.1; it was introduced here: 2005-03-16 22:18 momjian * src/port/snprintf.c: Factor duplicate snprintf code into functions. by an ill-considered removal of an unsigned local variable. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: port/snprintf.c (was Re: [PATCHES] Numeric 508 datatype)
Tom Lane wrote: > I wrote: > > Hm. One of the main problems I found was incorrect results for > > LONGLONG_MIN (-2^63). I'm rather tempted to add a test case for > > that to the int8 regression test and see if any platforms fail ;-) > > Done ... let me know whether the back branches still pass regression > for you ;-) I checked back to 7.3 and everything passed. I did a cvs update, configure, gmake, and regression run for each branch. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: port/snprintf.c (was Re: [PATCHES] Numeric 508 datatype)
I wrote: > Hm. One of the main problems I found was incorrect results for > LONGLONG_MIN (-2^63). I'm rather tempted to add a test case for > that to the int8 regression test and see if any platforms fail ;-) Done ... let me know whether the back branches still pass regression for you ;-) regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: port/snprintf.c (was Re: [PATCHES] Numeric 508 datatype)
Tom Lane wrote: > Bruce Momjian writes: > > OK, snprintf.c fixed. I added a 'stream' and outlen parameter to all > > the calls, and cleaned up the switch() statement that was outputing > > twice. When we were outputing just to a string, it didn't matter, but > > now that we are also outputting to a stream, it does. > > I found a whole bunch more problems than this :-(. I've committed a > cleaned-up version that seems to work correctly in a simple standalone > testbed, but it'd be a good idea to exercise it inside PG as well. > Can you try regression tests and the factorial() problem on CVS tip? Thanks. Tested 8.1.1 and CVS tip and all compile, and regression pass. I also tested the factorial test and the result looks perfect, thanks! > The problems are sufficiently bad that it might be a good idea to > backport the fixes into 8.0 and before as well --- but I note that > the ABI is different (pg_snprintf vs snprintf, etc) so this requires > a bit of investigation rather than just committing the file as-is. Not as many 8.0.X platforms used *printf because we didn't test %$ for its use on that release, so my bet is that very few platforms would be using it. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: port/snprintf.c (was Re: [PATCHES] Numeric 508 datatype)
Bruce Momjian writes: > Tom Lane wrote: >> The problems are sufficiently bad that it might be a good idea to >> backport the fixes into 8.0 and before as well --- but I note that >> the ABI is different (pg_snprintf vs snprintf, etc) so this requires >> a bit of investigation rather than just committing the file as-is. > Not as many 8.0.X platforms used *printf because we didn't test %$ for > its use on that release, so my bet is that very few platforms would be > using it. Hm. One of the main problems I found was incorrect results for LONGLONG_MIN (-2^63). I'm rather tempted to add a test case for that to the int8 regression test and see if any platforms fail ;-) regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
port/snprintf.c (was Re: [PATCHES] Numeric 508 datatype)
Bruce Momjian writes: > OK, snprintf.c fixed. I added a 'stream' and outlen parameter to all > the calls, and cleaned up the switch() statement that was outputing > twice. When we were outputing just to a string, it didn't matter, but > now that we are also outputting to a stream, it does. I found a whole bunch more problems than this :-(. I've committed a cleaned-up version that seems to work correctly in a simple standalone testbed, but it'd be a good idea to exercise it inside PG as well. Can you try regression tests and the factorial() problem on CVS tip? The problems are sufficiently bad that it might be a good idea to backport the fixes into 8.0 and before as well --- but I note that the ABI is different (pg_snprintf vs snprintf, etc) so this requires a bit of investigation rather than just committing the file as-is. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Bruce Momjian writes: > > Tom Lane wrote: > >> I think it'd be a lot cleaner with the struct. Mind if I take another > >> pass at it? > > > OK, you want my patch or want me to apply and then you can modify? > > You sent out your patch already --- I've been working from that. OK, thanks. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Tom Lane wrote: >> I think it'd be a lot cleaner with the struct. Mind if I take another >> pass at it? > OK, you want my patch or want me to apply and then you can modify? You sent out your patch already --- I've been working from that. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Bruce Momjian writes: > > (I could have done the struct but that seemed too invasive.) > > I think it'd be a lot cleaner with the struct. Mind if I take another > pass at it? OK, you want my patch or want me to apply and then you can modify? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > (I could have done the struct but that seemed too invasive.) I think it'd be a lot cleaner with the struct. Mind if I take another pass at it? regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote:
> Bruce Momjian writes:
> > Yep, I am digging through snprintf.c now to try find a solution.
>
> The cleanest solution is probably to fix things so that dopr_outch is
> aware of whether it's working for sprintf or fprintf, and can dump the
> buffer directly to the file when it gets full in the fprintf case.
> Its existing API would need to be changed a bit ... maybe pass it a
> struct containing what it needs, instead of having all the layers of
> code know what to pass.
OK, snprintf.c fixed. I added a 'stream' and outlen parameter to all
the calls, and cleaned up the switch() statement that was outputing
twice. When we were outputing just to a string, it didn't matter, but
now that we are also outputting to a stream, it does.
Passed regression and initdb tests, and factorial(4000) works!
(I could have done the struct but that seemed too invasive.)
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
Index: src/port/snprintf.c
===
RCS file: /cvsroot/pgsql/src/port/snprintf.c,v
retrieving revision 1.29
diff -c -c -r1.29 snprintf.c
*** src/port/snprintf.c 15 Oct 2005 02:49:51 - 1.29
--- src/port/snprintf.c 3 Dec 2005 04:23:00 -
***
*** 64,70
/*static char _id[] = "$PostgreSQL: pgsql/src/port/snprintf.c,v 1.29
2005/10/15 02:49:51 momjian Exp $";*/
! static void dopr(char *buffer, const char *format, va_list args, char *end);
/* Prevent recursion */
#undefvsnprintf
--- 64,70
/*static char _id[] = "$PostgreSQL: pgsql/src/port/snprintf.c,v 1.29
2005/10/15 02:49:51 momjian Exp $";*/
! static int dopr(FILE *stream, char *buffer, const char *format, va_list args,
char *end);
/* Prevent recursion */
#undefvsnprintf
***
*** 73,89
#undeffprintf
#undefprintf
! int
! pg_vsnprintf(char *str, size_t count, const char *fmt, va_list args)
{
char *end;
! str[0] = '\0';
! end = str + count - 1;
! dopr(str, fmt, args, end);
! if (count > 0)
end[0] = '\0';
! return strlen(str);
}
int
--- 73,100
#undeffprintf
#undefprintf
!
! static int
! pg_fvsnprintf(FILE *stream, char *str, size_t count, const char *fmt, va_list
args)
{
char *end;
+ int len;
! if (str)
! {
! str[0] = '\0';
! end = str + count - 1;
! }
! len = dopr(stream, str, fmt, args, end);
! if (str && count > 0)
end[0] = '\0';
! return len;
! }
!
! int
! pg_vsnprintf(char *str, size_t count, const char *fmt, va_list args)
! {
! return pg_fvsnprintf(NULL, str, count, fmt, args);
}
int
***
*** 93,99
va_list args;
va_start(args, fmt);
! len = pg_vsnprintf(str, count, fmt, args);
va_end(args);
return len;
}
--- 104,110
va_list args;
va_start(args, fmt);
! len = pg_fvsnprintf(NULL, str, count, fmt, args);
va_end(args);
return len;
}
***
*** 103,115
{
int len;
va_list args;
! charbuffer[4096];
va_start(args, fmt);
! len = pg_vsnprintf(buffer, (size_t) 4096, fmt, args);
va_end(args);
/* limit output to string */
! StrNCpy(str, buffer, (len + 1 < 4096) ? len + 1 : 4096);
return len;
}
--- 114,126
{
int len;
va_list args;
! charbuffer[8192]; /* arbitrary limit */
va_start(args, fmt);
! len = pg_fvsnprintf(NULL, buffer, (size_t) 4096, fmt, args);
va_end(args);
/* limit output to string */
! StrNCpy(str, buffer, (len + 1 < 8192) ? len + 1 : 8192);
return len;
}
***
*** 118,131
{
int len;
va_list args;
- charbuffer[4096];
- char *p;
va_start(args, fmt);
! len = pg_vsnprintf(buffer, (size_t) 4096, fmt, args);
va_end(args);
- for (p = buffer; *p; p++)
- putc(*p, stream);
return len;
}
--- 129,138
{
int len;
va_list args;
va_start(args, fmt);
! len = pg_fvsnprintf(stream, NULL, 0, fmt, args);
va_end(args);
return len;
}
***
*** 134,166
{
int len;
va_list args;
- charbuffer[4096];
- char *p;
va_start(args, fmt);
! l
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Yep, I am digging through snprintf.c now to try find a solution. The cleanest solution is probably to fix things so that dopr_outch is aware of whether it's working for sprintf or fprintf, and can dump the buffer directly to the file when it gets full in the fprintf case. Its existing API would need to be changed a bit ... maybe pass it a struct containing what it needs, instead of having all the layers of code know what to pass. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Alvaro Herrera <[EMAIL PROTECTED]> writes: > So this is a psql bug? Not here. Do you see it? regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Tom Lane wrote: >> Doh. OK, we gotta fix it then. But what are you going to do when you >> can't malloc enough memory? You can't ereport in a client environment, >> and there's no API for printf to report failure. > Yep, I am digging through snprintf.c now to try find a solution. Well, there's always the option of going back to plan B, which is not using printf for strings that might be long. AFAICS it would only be a convenience for psql, since it's not doing any real conversions, just inserting the right number of spaces. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Bruce Momjian writes: > > Tom Lane wrote: > >> That would only affect a standalone backend, however, not normal > >> operation. > > > Ah, psql itself is using the same buggy snprintf.c. > > Doh. OK, we gotta fix it then. But what are you going to do when you > can't malloc enough memory? You can't ereport in a client environment, > and there's no API for printf to report failure. Yep, I am digging through snprintf.c now to try find a solution. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Tom Lane wrote: >> That would only affect a standalone backend, however, not normal >> operation. > Ah, psql itself is using the same buggy snprintf.c. Doh. OK, we gotta fix it then. But what are you going to do when you can't malloc enough memory? You can't ereport in a client environment, and there's no API for printf to report failure. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian wrote: > > If you run the query that fails in a standalone backend, do you get > > something like "(typeid = 1700, len = -1, typmod = -1, byval = f)" > > at the end of the line, or is that part truncated too? > > I found the cause. I traced into printf then realized I was not in libc > but port/snprintf.c, and I see 4096 defined for those buffers. I will > work on a patch to make it dynamic. At the time I think there was > thought that 4096 was as large as it ever needed to be, but obviously > this was wrong. I think Win32 would see the same failure because it used > port/snprintf.c too. My Win32 8.1 platform is fine so Win32 must not use port/snprintf.c. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Bruce Momjian writes: > > Tom Lane wrote: > >> Uh, how is control getting to snprintf? I don't see that used either > >> in numeric.c or in printtup.c. > > > I am seeing it in the standalone backend here: > > > debugtup (slot=0x856e0b0, self=0x84306d0) at printtup.c:548 > > 548 printatt((unsigned) i + 1, typeinfo->attrs[i], > > value); > > That would only affect a standalone backend, however, not normal > operation. Ah, psql itself is using the same buggy snprintf.c. That would explain the long dash line, but short digits. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Tom Lane wrote: >> Uh, how is control getting to snprintf? I don't see that used either >> in numeric.c or in printtup.c. > I am seeing it in the standalone backend here: > debugtup (slot=0x856e0b0, self=0x84306d0) at printtup.c:548 > 548 printatt((unsigned) i + 1, typeinfo->attrs[i], > value); That would only affect a standalone backend, however, not normal operation. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote:
> Bruce Momjian writes:
> > I found the cause. I traced into printf then realized I was not in libc
> > but port/snprintf.c, and I see 4096 defined for those buffers.
>
> Uh, how is control getting to snprintf? I don't see that used either
> in numeric.c or in printtup.c.
I am seeing it in the standalone backend here:
debugtup (slot=0x856e0b0, self=0x84306d0) at printtup.c:548
548 printatt((unsigned) i + 1, typeinfo->attrs[i],
value);
(gdb) s
printatt (attributeId=1, attributeP=0x856efa4, value=0x857201c "1", '0'
...) at printtup.c:480
480 printf("\t%2d: %s%s%s%s\t(typeid = %u, len = %d, typmod
= %d, byval = %c)\n",
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > I found the cause. I traced into printf then realized I was not in libc > but port/snprintf.c, and I see 4096 defined for those buffers. Uh, how is control getting to snprintf? I don't see that used either in numeric.c or in printtup.c. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Michael Fuhr wrote: > On Fri, Dec 02, 2005 at 04:30:54PM -0500, Tom Lane wrote: > > Bruce Momjian writes: > > > Wow, check this out: > > > test=> SELECT CAST (pow(10::numeric, 1) + 1 AS TEXT) > > > It works fine! I have all the digits, and the trailing 1.0: > > > 01. > > > while SELECT pow(10::numeric, 1) fails. > > > > That's just about as wacky as can be, because numeric_text() is > > implemented on top of numeric_out() ... there's no way that numeric_out > > can be delivering the wrong answer if the cast produces the right text. > > So somewhere between numeric_out and the delivery to the client, > > something's getting confused. I think it's time you got out your > > debugger and started tracing through the backend ... > > Bruce, have you run a process trace on the backend to see if write() > (or whatever) is writing the correct number of characters? What > exactly is your output device and how are you connected to the > machine that runs the backend (ssh to a remote box from an xterm, > sitting in front of the box's VT52 serial console, etc.)? > > If you run the query that fails in a standalone backend, do you get > something like "(typeid = 1700, len = -1, typmod = -1, byval = f)" > at the end of the line, or is that part truncated too? I found the cause. I traced into printf then realized I was not in libc but port/snprintf.c, and I see 4096 defined for those buffers. I will work on a patch to make it dynamic. At the time I think there was thought that 4096 was as large as it ever needed to be, but obviously this was wrong. I think Win32 would see the same failure because it used port/snprintf.c too. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
On Fri, Dec 02, 2005 at 04:30:54PM -0500, Tom Lane wrote: > Bruce Momjian writes: > > Wow, check this out: > > test=> SELECT CAST (pow(10::numeric, 1) + 1 AS TEXT) > > It works fine! I have all the digits, and the trailing 1.0: > > 01. > > while SELECT pow(10::numeric, 1) fails. > > That's just about as wacky as can be, because numeric_text() is > implemented on top of numeric_out() ... there's no way that numeric_out > can be delivering the wrong answer if the cast produces the right text. > So somewhere between numeric_out and the delivery to the client, > something's getting confused. I think it's time you got out your > debugger and started tracing through the backend ... Bruce, have you run a process trace on the backend to see if write() (or whatever) is writing the correct number of characters? What exactly is your output device and how are you connected to the machine that runs the backend (ssh to a remote box from an xterm, sitting in front of the box's VT52 serial console, etc.)? If you run the query that fails in a standalone backend, do you get something like "(typeid = 1700, len = -1, typmod = -1, byval = f)" at the end of the line, or is that part truncated too? -- Michael Fuhr ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Wow, check this out: > test=> SELECT CAST (pow(10::numeric, 1) + 1 AS TEXT) > It works fine! I have all the digits, and the trailing 1.0: > 01. > while SELECT pow(10::numeric, 1) fails. That's just about as wacky as can be, because numeric_text() is implemented on top of numeric_out() ... there's no way that numeric_out can be delivering the wrong answer if the cast produces the right text. So somewhere between numeric_out and the delivery to the client, something's getting confused. I think it's time you got out your debugger and started tracing through the backend ... regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote:
> Bruce Momjian writes:
> > Tom Lane wrote:
> >> If that gives the right answer then the NUMERIC code is off the hook,
> >> and what you've got is a strange limitation on output column length.
>
> > test=> select length((pow(10::numeric, 131071))::text);
> > length
> >
> > 131089
> > (1 row)
>
> > Looks good. From psql I just tried:
>
> > SELECT repeat('x', 4000);
>
> > and got 4k x's, and SELECT repeat('x', 8000) returns 8k x's, so that works.
>
> Curiouser and curiouser. How about if you repeat 4k or 8k '1's? If the
1's print just fine too.
> behavior is different for letters and digits then I'd look at the column
> justification logic in psql's printing code.
Again, I checked on a stand-alone backend and saw the same failures, so
it isn't psql.
Wow, check this out:
test=> SELECT CAST (pow(10::numeric, 1) + 1 AS TEXT)
It works fine! I have all the digits, and the trailing 1.0:
01.
while SELECT pow(10::numeric, 1) fails.
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes:
> Tom Lane wrote:
>> If that gives the right answer then the NUMERIC code is off the hook,
>> and what you've got is a strange limitation on output column length.
> test=> select length((pow(10::numeric, 131071))::text);
>length
>
>131089
> (1 row)
> Looks good. From psql I just tried:
> SELECT repeat('x', 4000);
> and got 4k x's, and SELECT repeat('x', 8000) returns 8k x's, so that works.
Curiouser and curiouser. How about if you repeat 4k or 8k '1's? If the
behavior is different for letters and digits then I'd look at the column
justification logic in psql's printing code.
regards, tom lane
---(end of broadcast)---
TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Simon Riggs wrote: > On Fri, 2005-12-02 at 14:09 -0500, Bruce Momjian wrote: > > > I ran your SELECT pow(10::numeric, 131071), and gain, 4096 0's are > > displayed on my screen. SELECT pow(10::numeric, 7000) and SELECT > > pow(10::numeric, 1) generate identical displays on my screen. > > Are you saying there is a bug with or without my patch? Without. > Can we get the usual release levels/cvstip, ports, etc. Sorry, BSD/OS 4.3.1, CVS HEAD, no patches. > Are you connecting across network/ protocol differences... psql and the backend are on the same machine. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Numeric 508 datatype
On Fri, 2005-12-02 at 14:09 -0500, Bruce Momjian wrote: > I ran your SELECT pow(10::numeric, 131071), and gain, 4096 0's are > displayed on my screen. SELECT pow(10::numeric, 7000) and SELECT > pow(10::numeric, 1) generate identical displays on my screen. Are you saying there is a bug with or without my patch? Can we get the usual release levels/cvstip, ports, etc. Are you connecting across network/ protocol differences... [Just out of interest, has anybody ever used a number bigger than 10^20 in an application? My understanding is that the number of atoms in the universe is around 10^80. Accuracy is needed during calculation, but much less so during storage.] Thanks, Best Regards, Simon Riggs ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote:
> Bruce Momjian writes:
> > I just tested from a standalone backend:
> > backend> select pow(10::numeric, 131071) + 1
> > and got 4095 zeros and no trailing '1' (wrong), so it isn't psql, it
> > must be something in the backend.
>
> If the backend is truncating the result length, I don't see why psql
> would decide it needs 12K dashes for the header. There's something
> awfully fishy going on in your machine.
>
> Try something like
>
> regression=# select length((pow(10::numeric, 131071))::text);
> length
>
> 131089
> (1 row)
>
> If that gives the right answer then the NUMERIC code is off the hook,
> and what you've got is a strange limitation on output column length.
test=> select length((pow(10::numeric, 131071))::text);
length
131089
(1 row)
Looks good. From psql I just tried:
SELECT repeat('x', 4000);
and got 4k x's, and SELECT repeat('x', 8000) returns 8k x's, so that works.
--
Bruce Momjian| http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > I just tested from a standalone backend: > backend> select pow(10::numeric, 131071) + 1 > and got 4095 zeros and no trailing '1' (wrong), so it isn't psql, it > must be something in the backend. If the backend is truncating the result length, I don't see why psql would decide it needs 12K dashes for the header. There's something awfully fishy going on in your machine. Try something like regression=# select length((pow(10::numeric, 131071))::text); length 131089 (1 row) If that gives the right answer then the NUMERIC code is off the hook, and what you've got is a strange limitation on output column length. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Alvaro Herrera wrote: > Tom Lane wrote: > > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > > Actually, no. If I cut'n paste the number from psql to > > > cat > foo > > > > > > then only 4096 chars are copied. (Amusingly, I can't add a newline to > > > ^D and close the file. I must delete one char to do that.) > > > > Hmm, cut buffer limitation in X or someplace? I definitely get the > > right number of characters into the file written with \g, and what looks > > like a reasonable number of screensful of plain psql output. If Bruce > > is seeing the right number of dashes and the wrong number of data > > characters in his \g output then *something* is pretty weird there. > > Well, I just tried the \g test and it is correct (12675 digits or so). I just tested from a standalone backend: backend> select pow(10::numeric, 131071) + 1 and got 4095 zeros and no trailing '1' (wrong), so it isn't psql, it must be something in the backend. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Actually, no. If I cut'n paste the number from psql to > > cat > foo > > > > then only 4096 chars are copied. (Amusingly, I can't add a newline to > > ^D and close the file. I must delete one char to do that.) > > Hmm, cut buffer limitation in X or someplace? I definitely get the > right number of characters into the file written with \g, and what looks > like a reasonable number of screensful of plain psql output. If Bruce > is seeing the right number of dashes and the wrong number of data > characters in his \g output then *something* is pretty weird there. Well, I just tried the \g test and it is correct (12675 digits or so). -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Actually, no. If I cut'n paste the number from psql to > cat > foo > > then only 4096 chars are copied. (Amusingly, I can't add a newline to > ^D and close the file. I must delete one char to do that.) Hmm, cut buffer limitation in X or someplace? I definitely get the right number of characters into the file written with \g, and what looks like a reasonable number of screensful of plain psql output. If Bruce is seeing the right number of dashes and the wrong number of data characters in his \g output then *something* is pretty weird there. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Please try Tom's \g test: test=> select factorial(4000) test-> \g /tmp/x test=> \q $ wc -c /tmp/x 20881 /tmp/x Do you see a number greater than 20881, something like 3? --- Alvaro Herrera wrote: > Tom Lane wrote: > > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > > So this is a psql bug? > > > > Not here. Do you see it? > > Actually, no. If I cut'n paste the number from psql to > > cat > foo > > > then only 4096 chars are copied. (Amusingly, I can't add a newline to > ^D and close the file. I must delete one char to do that.) > > However if I open vim and paste there, the whole 12000+delta chars are > copied. > > So it must be a bug in Bruce's procedure I guess ... > > -- > Alvaro Herrerahttp://www.CommandPrompt.com/ > PostgreSQL Replication, Consulting, Custom Development, 24x7 support > > ---(end of broadcast)--- > TIP 3: Have you checked our extensive FAQ? > >http://www.postgresql.org/docs/faq > -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > So this is a psql bug? > > Not here. Do you see it? Actually, no. If I cut'n paste the number from psql to cat > foo then only 4096 chars are copied. (Amusingly, I can't add a newline to ^D and close the file. I must delete one char to do that.) However if I open vim and paste there, the whole 12000+delta chars are copied. So it must be a bug in Bruce's procedure I guess ... -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian wrote:
> I tried your \g test and the file size difference is the length of the
> dashed line in the file, not the number of digits display, which are
> both 4096. One has 12550 dashes, the other 19950 dashes.
So this is a psql bug? I can count the correct number of chars with
SPI:
alvherre=# create or replace function factorial_length(int) returns int
language plphp
alvherre-# as $$ $r = spi_exec("select factorial($args[0])");
alvherre$# $row = spi_fetch_row($r);
alvherre$# return strlen($row['factorial']); $$;
alvherre=# select factorial_length(4000);
factorial_length
--
12674
(1 fila)
alvherre=# select factorial_length(6000);
factorial_length
--
20066
(1 fila)
--
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Both are 4096 characters. You forgot the part that scrolled off the screen. Or else your installation is broken. I get this for factorial(4000) 182880195151406501331474317557391904421737771073043921970645269542089597979731773648503702868704841073364430415692855717546724618615435573339426156179569967167452848315973174988187609374828049804195765129487206105589281297880978006205934295377053267406244538842850917439517567461736237872246943619457592957990011421297336065899807397771469726120504866372593633749040406609796663717025402134880094428034228535594664968131626016345974380357717590339473317007684176477908216689118452932423003341414549780183259821851840655225709739253002458273898291910440678216870887149560350190586739996629879853487774792317919579141650440805487897477030865070712087883762498657607334044941485457836738330171570635819412740084985560408047330519683348240807942096427518753888911529665552239772392488715462481065978832100562055836960477865790477191838805431925151398195429674168844724618502125040222501011643301681858803669018017769146177971310430164039570827473470118677275696606461102365652876513873570! 4190876200697145804692125236821066805337175220605745377557452592208653939853278523841448314026548802309860391087939783218946129582647928430739985554883806198749831633640196211202756086016039171607744078776876219661603702759454887945247605749205543464095883664514960293873244842409803801480566600124415293789831496309554117113888561569494314926134470477513516416560102984058751906208865570183683850791317395702861350821464653600469443279077733978568711404244774089509216727922510660941411716412467443445414001188915966547283773988670739792818897314762082568914041952211779194055311405259158538932388745292324386826830135904886472292289993848482289254307628467614523292519222687689180219788377184005246290896703260524910362136627321135976515358528150143796798116836263053229733971612275184896139539613129329008449214723196703789119820971205922195513915546814704778682373487718946560822811623038853887054357316290622378472322045316639418491798517077275839637525427601452961835674484434498885! 6988406924685508257651316109259665853395618544561542290482957422747251 2621879974544803139182629522111438189060068320844155808827122861800658905944410880665299278785463449748715867577098342261093659060062717050097248139944414539852275687062609725023022919579927729992184495471569088324255356925665713251566354493183039331751882898644394213897160914262139764680835180969460373487297798414800269996513787044819986616716294925643504041614688682394214445910517503348839586991040520752132901684267316856383753151891833962772406615293362723673056115541822788867351393745450810382610282770612156033090601640416242005137331365457002003319577878502216919170112074608722852376799943191590480651623958062982829452035227119036502426583752512199824089725611711059153935434418985109241404135069047109527514730648502064630431371185922523036941621026392783813435540195800531988645430344745298845640017082732623248838473771603478336326662579219137601422632057648758807935233915527562817942378675243919886800056209434731407685691942327092464101136254795499159351103542747723! 43454443636631310499637366165498946549818089271646280504222703822210478406262602748015156737784182131629209529568636861930041786332753076430132308190243597116592516351322551117625891947167343755332093491691057399902096608720766313387151653039178753575542034817451995401301599919333520503257117646010500571611530574866936468267526501431022327176280762024280561743559492789067640895305738489071968122584040039669815562479316156585043604782961704971397764959404751358445856914581957186533573207690355894150776647727994156425641953755517727965486096675384323441855372798887805708540920842219936607611547883597774397984908511480312751237592867932246601885937688972917491803572971856504307350631260352783217496299957022001157223866000467728830189631662734637151868125433561735503412333101756008177714471706565175053852587720690671396654785062638983805263943882180363887689991615386454329321164711463925629922042594508892919282617805319256181483111512590885198697876379136078660799208307! 814332752984685342485954714853544207940898546931676443201159004047 1266030942746376074187982809857292874382829734306879420308783010740784767155894363033186129183748698373599293267795620690003900348542140553208244771165593739117369452474295075445169488356550100854423929030967163401348223503674045821834526710830392520955478859202303175309815706259592416130388770535449570146586161954289529234209342432075461261173246383180702862172009624828852517861214008242944903162464816939278645210656431141653019224783906038062818006434258491952705341708781231767324026544165240012431457995565227331912056216396141277926932316783631899833106810492890045890947294731794016389385348294079360921757589715698098491612254898203787546124091057862387229987442131738499764391062483957360997512778806
Re: [PATCHES] Numeric 508 datatype
Uh, I tried factorial(4000) and for display I got: 1828801951514065013314743175573919044217377710730439219706452695420895979797317736485037028687048410733644304156928557175467246186154355733394261561795699671674528483159731749881876093748280498041957651294872061055892812978809780062059342953770532674062445388428509174395175674617362378722469436194575929579900114212973360658998073977714697261205048663725936337490404066097966637170254021348800944280342285355946649681316260163459743803577175903394733170076841764779082166891184529324230033414145497801832598218518406552257097392530024582738982919104406782168708871495603501905867399966298798534877747923179195791416504408054878974770308650707120878837624986576073340449414854578367383301715706358194127400849855604080473305196833482408079420964275187538889115296655522397723924887154624810659788321005620558369604778657904771918388054319251513981954296741688447246185021250402225010116433016818588036690180177691461779713104301640395708274734701186772756966064611023656528765138735704190876200697145804692125236821066805337175220605745377557452592208653939853278523841448314026548802309860391087939783218946129582647928430739985554883806198749831633640196211202756086016039171607744078776876219661603702759454887945247605749205543464095883664514960293873244842409803801480566600124415293789831496309554117113888561569494314926134470477513516416560102984058751906208865570183683850791317395702861350821464653600469443279077733978568711404244774089509216727922510660941411716412467443445414001188915966547283773988670739792818897314762082568914041952211779194055311405259158538932388745292324386826830135904886472292289993848482289254307628467614523292519222687689180219788377184005246290896703260524910362136627321135976515358528150143796798116836263053229733971612275184896139539613129329008449214723196703789119820971205922195513915546814704778682373487718946560822811623038853887054357316290622378472322045316639418491798517077275839637525427601452961835674484434498885698840692468550825765131610925966585339561854456154229048295742274725126218799745448031391826295221114381890600683208441558088271228618006589059444108806652992787854634497487158675770983422610936590600627170500972481399444145398522756870626097250230229195799277299921844954715690883242553569256657132515663544931830393317518828986443942138971609142621397646808351809694603734872977984148002699965137870448199866167162949256435040416146886823942144459105175033488395869910405207521329016842673168563837531518918339627724066152933627236730561155418227888673513937454508103826102827706121560330906016404162420051373313654570020033195778785022169191701120746087228523767999431915904806516239580629828294520352271190365024265837525121998240897256117110591539354344189851092414041350690471095275147306485020646304313711859225230369416210263927838134355401958005319886454303447452988456400170827326232488384737716034783363266625792191376014226320576487588079352339155275628179423786752439198868000562094347314076856919423270924641011362547954991593511035427477234345444363663131049963736616549894654981808927164628050422270382221047840626260274801515673778418213162920952956863686193004178633275307643013230819024359711659251635132255111762589194716734375533209349169105739990209660872076631338715165303917875357554203481745199540130159991933352050325711764601050057161153057486693646826752650143102232717628076202428056174355949278906764089530573848907196812258404003966981556247931615658504360478296170497139776495940475135844585691458195718653357320769035589415077664772799415642564195375551772796548609667538432344185537279888780570854092084221993660761154788359777439798490851148031275123759286793224660188593768897291749180357297185650430735063126035278321749629995702200115722386600046772883018963166273463715186812543356173550341233310175600817771447170656517505385258772069067139665478506263898380526394388218036388768999161538645432932116471146392562992204259450889291928261780531925618148311151259088519869787637913607866079920830781433275298468534248595471485354420794089854693167644320115900404712 and did factorial(6000) and see: 26839997657267395961163166474627355122050186046884725685824261445661070965868578859479409748346329836182703089128852632609939583160469201526637034786976396625075825887529584878861523984702266296018595427551580174654216923424895644549809732165397549500287960097044436776098097112285935902002318300817766074944639865419157280524790653203290253917356133633993978535760253069357555142219669129998317026518032402810751448010557484004254745274140263837210122325107062378927575415179640798325598920907738968963426553833519168033920097139867697345016081563676374441238664213080213666256929097791375199485350231595456497900149202746728755327879812971546373369370677385700341673192285360910112728679385205719710580583641371671892447976818829243817888754721448962887123232268424960754147650482415903026606790
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > Sorry, I am confused. If our computational range is that high, why does > SELECT factorial(4000) and SELECT factorial(6000) produce the same > number of digits on my screen. Are you counting correctly? regression=# select log(factorial(4000)); log 12673.2621666764869773 (1 row) regression=# select log(factorial(6000)); log 20065.428782473590 (1 row) regression=# select factorial(4000) regression-# \g z4000 regression=# select factorial(6000) regression-# \g z6000 regression=# \q $ wc z4000 z6000 5 5 38039 z4000 5 5 60215 z6000 The actual representation limit at the moment (with int16 weights in the storable format) is 10^128K, as you can soon prove with pow(): regression=# select pow(10::numeric, 131071); << lots o zeroes >> regression=# select pow(10::numeric, 131072); ERROR: value overflows numeric format I don't recall what factorial that might correspond to, but it's considerably above 6000. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Bruce Momjian writes: > > I am confused by your use of the term "dynamic" range. From what you > > say above that we are just moving from 1000 to 508 for storage, and that > > computational range would still be 4096? > > No, computational range would still be on the order of 10^16G ... in the > computational format, the weight is an int. The restriction to 1000 > digits was never anything but an artificial limit. (Of course, you > might not have the patience to actually do any arithmetic with that many > digits, but the point is there was a whole lot of headroom before, and > now there won't be.) Sorry, I am confused. If our computational range is that high, why does SELECT factorial(4000) and SELECT factorial(6000) produce the same number of digits on my screen. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > I am confused by your use of the term "dynamic" range. From what you > say above that we are just moving from 1000 to 508 for storage, and that > computational range would still be 4096? No, computational range would still be on the order of 10^16G ... in the computational format, the weight is an int. The restriction to 1000 digits was never anything but an artificial limit. (Of course, you might not have the patience to actually do any arithmetic with that many digits, but the point is there was a whole lot of headroom before, and now there won't be.) regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: > Bruce Momjian writes: > > So we are really decreasing the specified precision from 1000 to 508, > > and the computational precision from 4096 to 508. > > The internal computational precision isn't any less, the limit is only > on the result of a function (ie, partial results within one of the > numeric.c routines could still exceed 10^508). Not sure how much that > distinction matters though. Agreed. > > Is there any plan to > > fix the silent overflow problem? Is that in the patch? I don't see it. > > It will get fixed before application ;-) > > I haven't reviewed the patch yet; I think the gating factor at this > point is whether anyone protests losing dynamic range in NUMERIC, > and we ought to go ahead and ask that. After that we can look at the > code more closely. I am confused by your use of the term "dynamic" range. From what you say above that we are just moving from 1000 to 508 for storage, and that computational range would still be 4096? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian writes: > So we are really decreasing the specified precision from 1000 to 508, > and the computational precision from 4096 to 508. The internal computational precision isn't any less, the limit is only on the result of a function (ie, partial results within one of the numeric.c routines could still exceed 10^508). Not sure how much that distinction matters though. > Is there any plan to > fix the silent overflow problem? Is that in the patch? I don't see it. It will get fixed before application ;-) I haven't reviewed the patch yet; I think the gating factor at this point is whether anyone protests losing dynamic range in NUMERIC, and we ought to go ahead and ask that. After that we can look at the code more closely. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian wrote: > OK, seems all objections have been dealt with so it goes into the patch > queue. I will ask on 'general'. > > The only downside I see is that I can't impress people by doing: > > SELECT factorial(4000); > > I don't suppose the _impression_ factor is worth two bytes per value. > Shame. > > I suppose people wanting to do such manipulations will have to store the > numbers as text and use a server-side library like perl to do > calculations. Oops, I was wrong about this. The patch changes the maximum _specified_ precision: /* ! * Hardcoded precision limit - arbitrary, but must be small enough that ! * dscale values will fit in 14 bits. */ ! #define NUMERIC_MAX_PRECISION 1000 /* * Internal limits on the scales chosen for calculation results --- 15,23 #define _PG_NUMERIC_H_ /* ! * Hardcoded precision limit - maximum that can fit in Numeric storage */ ! #define NUMERIC_MAX_PRECISION 508 but in fact, our computational precision is 4096, and we silently overflow for values greater than that: test=> create table test(x numeric); CREATE TABLE test=> insert into test values (factorial(4000)); INSERT 0 1 The length is 4096 digits, and so is factorial(1) --- clearly wrong. I now see in the TODO: * Change NUMERIC to enforce the maximum precision, and increase it So we are really decreasing the specified precision from 1000 to 508, and the computational precision from 4096 to 508. Is there any plan to fix the silent overflow problem? Is that in the patch? I don't see it. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Simon Riggs wrote: > > Now we're into 8.2devel mode, its time to submit the previously > discussed patch that: > > - reduces Numeric storage format by 2 bytes > - limits scale to +/- 508 decimal places > > This is sufficient to allow Numeric to continue to be used as the > default numeric representation for all numbers in the parser. > > Passes: make check on cvstip, as well as some tests not in there. > > Code comments explain the new format and consequences. > > As previously agreed, reviewing this is a 2 stage process: > 1. review/possibly agree OK to commit > 2. check with everybody on GENERAL that the restriction to 508 is > acceptable > > Figure there's no point doing (2) until we agree the proposal/code is > workable. OK, seems all objections have been dealt with so it goes into the patch queue. I will ask on 'general'. The only downside I see is that I can't impress people by doing: SELECT factorial(4000); I don't suppose the _impression_ factor is worth two bytes per value. Shame. I suppose people wanting to do such manipulations will have to store the numbers as text and use a server-side library like perl to do calculations. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
On Thu, 2005-12-01 at 23:34 -0500, Bruce Momjian wrote: > Where are we on this patch? It is ready for the patch queue? It's good to be applied, AFAIK. > Simon Riggs wrote: > > As previously agreed, reviewing this is a 2 stage process: > > 1. review/possibly agree OK to commit > > 2. check with everybody on GENERAL that the restriction to 508 is > > acceptable > > > > Figure there's no point doing (2) until we agree the proposal/code is > > workable. I was hoping you'd give me a "this looks good enough to apply" thumbs-up, then I'll ask for comments via a Weekly News item. If I ask for comments and then it is technically rejected we would be wasting everybody's time. Best Regards, Simon Riggs ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Where are we on this patch? It is ready for the patch queue? --- Simon Riggs wrote: > > Now we're into 8.2devel mode, its time to submit the previously > discussed patch that: > > - reduces Numeric storage format by 2 bytes > - limits scale to +/- 508 decimal places > > This is sufficient to allow Numeric to continue to be used as the > default numeric representation for all numbers in the parser. > > Passes: make check on cvstip, as well as some tests not in there. > > Code comments explain the new format and consequences. > > As previously agreed, reviewing this is a 2 stage process: > 1. review/possibly agree OK to commit > 2. check with everybody on GENERAL that the restriction to 508 is > acceptable > > Figure there's no point doing (2) until we agree the proposal/code is > workable. > > As Atsushi-san point out, there is also come CPU optimization to be done > on Numeric comparison, and also on other areas such as aggregation. I've > not done this yet. > > Best Regards, Simon Riggs [ Attachment, skipping... ] > > ---(end of broadcast)--- > TIP 2: Don't 'kill -9' the postmaster -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Numeric 508 datatype
Bruce Momjian wrote: Let's go ahead and apply the patch. While this change isn't very significant, I bet there will be other changes in 8.2 where we will want to change the database for a significant benefit, like reducing the tuple header by 4 bytes by recompressing the four xid/cid fields back into three. All of which makes me very uncomfortable about doing 64 bit enums. I am still trying to think of a better solution. cheers andrew ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Andreas Pflug wrote: > Alvaro Herrera wrote: > > > > Is anybody working or considering to work on pg_upgrade, or is all this > > hypothetical? Our past history has seen lots of people offering to work > > on pg_upgrade, and none has produced a working version. Is it fair or > > useful to impose restrictions on development just because it's remotely > > possible that somebody is going to be motivated enough to consider > > producing it? > > Depends on the impact the restriction imposes. If > stability/scalability/functionality or so is affected, this sounds not > tolerable. If it's about not saving two bytes that have been spoiled for > ages before, or keeping a backward compatibility type, it appears > feasible to me. > Changing on-disk structures at the start of the 8.2 dev cycle is a > guarantee that nobody will implement pg_upgrade for 8.2. Let's go ahead and apply the patch. While this change isn't very significant, I bet there will be other changes in 8.2 where we will want to change the database for a significant benefit, like reducing the tuple header by 4 bytes by recompressing the four xid/cid fields back into three. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Numeric 508 datatype
Alvaro Herrera wrote: Is anybody working or considering to work on pg_upgrade, or is all this hypothetical? Our past history has seen lots of people offering to work on pg_upgrade, and none has produced a working version. Is it fair or useful to impose restrictions on development just because it's remotely possible that somebody is going to be motivated enough to consider producing it? Depends on the impact the restriction imposes. If stability/scalability/functionality or so is affected, this sounds not tolerable. If it's about not saving two bytes that have been spoiled for ages before, or keeping a backward compatibility type, it appears feasible to me. Changing on-disk structures at the start of the 8.2 dev cycle is a guarantee that nobody will implement pg_upgrade for 8.2. Regards, Andreas ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Numeric 508 datatype
Andreas Pflug wrote: > Tom Lane wrote: > >Andrew Dunstan <[EMAIL PROTECTED]> writes: > > > >>However, I don't think we can promise never to change the ondisk > >>representation of data, nor the page layout. Sometimes an inplace > >>upgrade just won't work, ISTM. > > > >We have talked about batching on-disk changes so that they'd only occur > >once every few release cycles. But until we have a pg_upgrade, there is > >no reason to adopt such a policy. > > IMHO such a policy is a _prerequisite_ for somebody to come up > implementing pg_upgrade. Why spend time on pg_upgrade if there's no > policy to support it? Is anybody working or considering to work on pg_upgrade, or is all this hypothetical? Our past history has seen lots of people offering to work on pg_upgrade, and none has produced a working version. Is it fair or useful to impose restrictions on development just because it's remotely possible that somebody is going to be motivated enough to consider producing it? -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: However, I don't think we can promise never to change the ondisk representation of data, nor the page layout. Sometimes an inplace upgrade just won't work, ISTM. We have talked about batching on-disk changes so that they'd only occur once every few release cycles. But until we have a pg_upgrade, there is no reason to adopt such a policy. IMHO such a policy is a _prerequisite_ for somebody to come up implementing pg_upgrade. Why spend time on pg_upgrade if there's no policy to support it? Regards, Andreas ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Numeric 508 datatype
They're not exactly easy to setup for the casual user (well, I assume replicator isn't but it's not relevant to the majority of our users anyway). Well FYI any half way competent person can set up replicator in 30 minutes or less ;), that said in place upgrades are a feature that will take up a lot of man power EVERY release I would think. I could be wrong but I would rather see the resources put into other things. J Regards, Dave. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Simon Riggs <[EMAIL PROTECTED]> writes: > If the patch is accepted technically, in can be applied at any time, > right up to code freeze for this or the next release. It's a fairly > independent patch. > I'd suggest we check it out now, then put it in a holding pen for awhile > to see if an upgrade tool emerges. I'm disinclined to do that unless there's a pretty firm commitment from someone to work on pg_upgrade in the near future. Patches that are not in the tree tend to suffer from code drift; if we wait six months or a year to apply what you've done then we'll likely be looking at significantly more work to get it in. We'd also be losing the direct and indirect testing that the patch would get were it in the tree over that length of time. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
On Thu, 2005-11-17 at 11:20 -0500, Tom Lane wrote: > Andrew Dunstan <[EMAIL PROTECTED]> writes: > > However, I don't think we can promise never to change the ondisk > > representation of data, nor the page layout. Sometimes an inplace > > upgrade just won't work, ISTM. > > We have talked about batching on-disk changes so that they'd only occur > once every few release cycles. But until we have a pg_upgrade, there is > no reason to adopt such a policy. If the patch is accepted technically, in can be applied at any time, right up to code freeze for this or the next release. It's a fairly independent patch. I'd suggest we check it out now, then put it in a holding pen for awhile to see if an upgrade tool emerges. Best Regards, Simon Riggs ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Numeric 508 datatype
Andrew Dunstan <[EMAIL PROTECTED]> writes: > However, I don't think we can promise never to change the ondisk > representation of data, nor the page layout. Sometimes an inplace > upgrade just won't work, ISTM. We have talked about batching on-disk changes so that they'd only occur once every few release cycles. But until we have a pg_upgrade, there is no reason to adopt such a policy. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Numeric 508 datatype
Joshua D. Drake wrote: With Slony and Replicator I don't really see the need for in place upgrades. Maintaining a replica is hardly a cost-free exercise. However, I don't think we can promise never to change the ondisk representation of data, nor the page layout. Sometimes an inplace upgrade just won't work, ISTM. cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Numeric 508 datatype
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Joshua D. Drake > Sent: 17 November 2005 15:58 > To: Andreas Pflug > Cc: Tom Lane; Simon Riggs; pgsql-patches@postgresql.org > Subject: Re: [PATCHES] Numeric 508 datatype > > > > > > Hm, so if this patch is applied now, and in 5 months or so somebody > > implements pg_upgrade, this numeric storage patch would be > rolled back? > > OTOH, an upgrade mechanism that's compatible for future > 8.3+ versions > > only seems not too attractive. > With Slony and Replicator I don't really see the need for in place > upgrades. They're not exactly easy to setup for the casual user (well, I assume replicator isn't but it's not relevant to the majority of our users anyway). Regards, Dave. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Hm, so if this patch is applied now, and in 5 months or so somebody implements pg_upgrade, this numeric storage patch would be rolled back? OTOH, an upgrade mechanism that's compatible for future 8.3+ versions only seems not too attractive. With Slony and Replicator I don't really see the need for in place upgrades. Joshua D. Drake A solution might be to keep the current numeric implementation under a different name (deprecatednumeric or so), for backward compatibility (this should apply to future storage format changes as well). Regards, Andreas ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Numeric 508 datatype
Tom Lane wrote: Andreas Pflug <[EMAIL PROTECTED]> writes: Simon Riggs wrote: Now we're into 8.2devel mode, its time to submit the previously discussed patch that: - reduces Numeric storage format by 2 bytes This makes the often discussed binary upgrade impossible, so I wonder if two bytes savings are worth the trouble. Unless someone actually steps forward and produces a working pg_upgrade in the 8.2 timeframe, this objection is moot. Hm, so if this patch is applied now, and in 5 months or so somebody implements pg_upgrade, this numeric storage patch would be rolled back? OTOH, an upgrade mechanism that's compatible for future 8.3+ versions only seems not too attractive. A solution might be to keep the current numeric implementation under a different name (deprecatednumeric or so), for backward compatibility (this should apply to future storage format changes as well). Regards, Andreas ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Numeric 508 datatype
Andreas Pflug <[EMAIL PROTECTED]> writes: > Simon Riggs wrote: >> Now we're into 8.2devel mode, its time to submit the previously >> discussed patch that: >> - reduces Numeric storage format by 2 bytes > This makes the often discussed binary upgrade impossible, so I wonder if > two bytes savings are worth the trouble. Unless someone actually steps forward and produces a working pg_upgrade in the 8.2 timeframe, this objection is moot. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Numeric 508 datatype
Simon Riggs wrote: Now we're into 8.2devel mode, its time to submit the previously discussed patch that: - reduces Numeric storage format by 2 bytes This makes the often discussed binary upgrade impossible, so I wonder if two bytes savings are worth the trouble. Regards, Andreas ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
