Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
On August 28, 2002 11:07 pm, Tom Lane wrote:
D'Arcy J.M. Cain [EMAIL PROTECTED] writes:
Hmm. I did give it a harder look and look what jumped out. Both
chkpass_out and chkpass_rout return PG_RETURN_CSTRING but chkpass_out
builds a standard c string while chkpass_rout builds a variable text
structure. That can't be right.
It's not, but chkpass_rout is declared to return text, so it should be
saying PG_RETURN_TEXT_P. It turns out both macros do the same thing,
so this is just a cosmetic issue.
OK, I will make that cosmetic change.
Oh, one more datapoint - the error only happens on vacuum analyze, not
just vacuum. Not sure what that means exactly.
That is odd. You only have the chkpass operators shown in the contrib
module, right? No chkpass = chkpass operator? Without one, vacuum
analyze should pretty much ignore the chkpass column ...
YES! Well, sort of. I didn't have any other operators but while I thought
that both were the same (after all, I contributed it) someone must have fixed
the one in CVS before adding it. The one I was working with had the
operators working with chkpass on both sides. As soon as I fixed that it
worked again.
In 7.2 the cstring and chkpass types fail in the function definitions because
they have not been defined so I had to stay with opaque. In fact, how will
that work in 7.3 anyway? We declare the functions to take or return a
chkpass before we define it.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: Type definition process (was Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735)
On August 29, 2002 09:45 am, Tom Lane wrote:
D'Arcy J.M. Cain [EMAIL PROTECTED] writes:
YES! Well, sort of. I didn't have any other operators but while I
thought that both were the same (after all, I contributed it) someone
must have fixed the one in CVS before adding it. The one I was working
with had the operators working with chkpass on both sides. As soon as I
fixed that it worked again.
Ah-hah, so vacuum was trying to use the chkpass = text operator to
compare two chkpass values. That explains the whole problem --- the
text code of course would take the first four bytes of the chkpass
string as a length word.
Exactly.
In 7.2 the cstring and chkpass types fail in the function definitions
because they have not been defined so I had to stay with opaque. In
fact, how will that work in 7.3 anyway? We declare the functions to take
or return a chkpass before we define it.
Yeah, you'll get warnings about the type not being defined yet, but it
will take them anyway. There's a fundamental circularity involved in
defining these things with any sort of accuracy, so we're going to have
to live with either warnings or kluges :-(.
I suppose that if the warnings really irritate people, we could think
about exposing the shell-type-entry mechanism more explicitly. For
example, if you did something like
-- make a shell pg_type entry
CREATE TYPE chkpass;
-- make the I/O functions
CREATE FUNCTION chkpass_in(cstring) RETURNS chkpass ...;
CREATE FUNCTION chkpass_out(chkpass) RETURNS cstring ...;
-- replace shell entry with real one
CREATE TYPE chkpass(input = chkpass_in, output = ...);
This looks rather ugly to me but it would be pretty easy to make it
work and not give any warnings. Comments?
Well, magic (DWIM) parsing would be nice but this doesn't seem all that ugly
to me. One thing I do see though is that there is a completion issue. Maybe
we should specify that this can only happen within a transaction and add some
code to the transaction handling. Some simple rules (not to suggest that
they are necessarily simple to implement of course) I see are;
1. An incomplete CREATE TYPE raises an error if not inside a transaction
block.
2. CREATE TYPE and CREATE FUNCTION will be backed out on an abort.
3. Closing a transaction aborts if an incomplete type has not been completed.
I think that this closes the loop without leaving functions defined on
incomplete types. With enough clever programming perhaps we can even make
the incomplete declarion automatic when it is used in the CREATE FUNCTION.
We just don't raise an error until the COMMIT.
BEGIN TRANSACTION
-- an incomplete type chkpass is conditionally created here
CREATE FUNCTION chkpass_in(cstring) RETURNS chkpass;
-- the existing conditional type is used here
CREATE FUNCTION chkpass_out(chkpass) RETURNS cstring;
-- define the actual type
CREATE TYPE chkpass(input = chkpass_in, output = chkpass_out);
END TRANSACTION
Does this make sense?
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: Type definition process (was Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735)
D'Arcy J.M. Cain [EMAIL PROTECTED] writes: One thing I do see though is that there is a completion issue. Well, (a) the shell type can't be used for anything till you turn it into a real type, and (b) the completion issue already exists, and has for a long time; you've always been able to create a shell type by using a not-yet-known type name as the return type of a function. It's just not well documented. 1. An incomplete CREATE TYPE raises an error if not inside a transaction block. I have no intention of implementing this. (1) It wouldn't really simplify life anyway, since we'd still need all the same guard code to prevent you from using the shell type within the creating transaction. (2) It would break existing pg_dump scripts, which don't know they'd need to do this. Wrapping the sequence inside a transaction is a good practice, but I don't feel that we have to try to force good practice on people. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: Type definition process (was Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735)
On August 29, 2002 03:37 pm, Tom Lane wrote:
D'Arcy J.M. Cain [EMAIL PROTECTED] writes:
One thing I do see though is that there is a completion issue.
Well, (a) the shell type can't be used for anything till you turn it
into a real type, and (b) the completion issue already exists, and has
for a long time; you've always been able to create a shell type by using
a not-yet-known type name as the return type of a function. It's just
not well documented.
And gives a warning, right?
1. An incomplete CREATE TYPE raises an error if not inside a transaction
block.
I have no intention of implementing this. (1) It wouldn't really
simplify life anyway, since we'd still need all the same guard code to
prevent you from using the shell type within the creating transaction.
(2) It would break existing pg_dump scripts, which don't know they'd
need to do this.
Yes, I see your point.
Wrapping the sequence inside a transaction is a good practice, but
I don't feel that we have to try to force good practice on people.
OK but how about a little reward if they do. Do everything as we do now
except that if they wrap it in a transaction then they don't get the warning
unless they exit the transaction without completing the type? Some people
(e.g. me) like to code as if warnings were as bad as errors.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
On August 28, 2002 09:23 am, Tom Lane wrote:
The behavior looks a lot like a memory clobber, so perhaps the key
variable is some difference in malloc's allocation strategy, causing
two items to be adjacent in NetBSD where they are not on the other
platforms we've tried.
Hmm. I might try adding some buffer in MemoryContextAlloc() and see if that
changes anything. One thing that may be different is that NetBSD is 64 bit
clean. I don't think the other i386 systems are.
I eyeballed the chkpass code and didn't see any sign of buffer overruns,
but maybe it needs a harder look.
It's pretty simple. Not even indexing. In fact, I wondered if I should add
some just like my other type that does do indexing. That seemed to be the
only real difference between the two and the other works.
Hm --- I guess another possible variable is behavior of the local
crypt() function. Is NetBSD's crypt perhaps willing to return strings
longer than 13 chars?
Well, the value that it stores is the correct 13 character DES string.
Did you try CVS tip both with and without --enable-cassert? That turns
on memory context checking which might alter the failure in interesting
ways.
No difference. It seems that PostgreSQL is too good at catching the problem
before the assert macros see it.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
On August 28, 2002 12:48 pm, D'Arcy J.M. Cain wrote:
On August 28, 2002 09:23 am, Tom Lane wrote:
The behavior looks a lot like a memory clobber, so perhaps the key
variable is some difference in malloc's allocation strategy, causing
two items to be adjacent in NetBSD where they are not on the other
platforms we've tried.
Hmm. I might try adding some buffer in MemoryContextAlloc() and see if
Nope. Tried adding size += 64; into MemoryContextAlloc() and it made no
difference.
I had also tried changing palloc.h and mcxt.c to turn MemoryContextAlloc()
into a macro that called a modified version of the real one to try to narrow
down where it was being called from but that wouldn't even run. Is there
another file I have to modify as well if I try that? Here are the changes I
tried. See any reason that it shouldn't have worked if I rebuilt everything?
*** postgresql-server/work.i386/postgresql-7.2.2/src/backend/utils/mmgr/mcxt.c
Wed Aug 28 14:01:31 2002
--- mcxt.c Mon Aug 26 21:51:31 2002
***
*** 409,416
* nodes/memnodes.h into postgres.h which seems a bad idea.
*/
void *
! MemoryContextAlloc(MemoryContext context, Size size)
{
AssertArg(MemoryContextIsValid(context));
if (!AllocSizeIsValid(size))
--- 409,417
* nodes/memnodes.h into postgres.h which seems a bad idea.
*/
void *
! _MemoryContextAlloc(MemoryContext context, Size size, const char *f, int l)
{
+ elog(NOTICE, _MemoryContextAlloc called from %s line %d, f, l);
AssertArg(MemoryContextIsValid(context));
if (!AllocSizeIsValid(size))
*** postgresql-server/work.i386/postgresql-7.2.2/src/include/utils/palloc.h
Mon
Nov 5 12:46:36 2001
--- palloc.hMon Aug 26 21:51:46 2002
***
*** 45,51
/*
* Fundamental memory-allocation operations (more are in utils/memutils.h)
*/
! extern void *MemoryContextAlloc(MemoryContext context, Size size);
#define palloc(sz)MemoryContextAlloc(CurrentMemoryContext, (sz))
--- 45,52
/*
* Fundamental memory-allocation operations (more are in utils/memutils.h)
*/
! extern void *_MemoryContextAlloc(MemoryContext context, Size size, const
char
*s, int l);
! #define MemoryContextAlloc(context, size) _MemoryContextAlloc(context,
size, __FILE__, __LINE__)
#define palloc(sz)MemoryContextAlloc(CurrentMemoryContext, (sz))
that changes anything. One thing that may be different is that NetBSD is
64 bit clean. I don't think the other i386 systems are.
I eyeballed the chkpass code and didn't see any sign of buffer overruns,
but maybe it needs a harder look.
It's pretty simple. Not even indexing. In fact, I wondered if I should
add some just like my other type that does do indexing. That seemed to be
the only real difference between the two and the other works.
Hm --- I guess another possible variable is behavior of the local
crypt() function. Is NetBSD's crypt perhaps willing to return strings
longer than 13 chars?
Well, the value that it stores is the correct 13 character DES string.
Did you try CVS tip both with and without --enable-cassert? That turns
on memory context checking which might alter the failure in interesting
ways.
No difference. It seems that PostgreSQL is too good at catching the
problem before the assert macros see it.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
On August 28, 2002 09:23 am, Tom Lane wrote:
The behavior looks a lot like a memory clobber, so perhaps the key
variable is some difference in malloc's allocation strategy, causing
two items to be adjacent in NetBSD where they are not on the other
platforms we've tried.
Here's some other wackiness. The following is various encrypted passwords
split for convenience of analyzing them along with the hex version of the big
number in the error. The OK ones were ones that didn't trigger the error.
The first line following that is the last byte in the number in hex and
binary. The next is the first character of the second chunk of the password.
Note how the integer basically is the 4 bytes of the second chunk except for
the first byte which differs in a somewhat regular way. Looks like some sort
of bit mask operation somewhere.
wvx8 42kQ 34jyY (OK)
0x34 00110100
Q/Jz mdRb HSwE. 0x62526471
0x71 01110001
0x6d 01101101
ccIx mriB VsviU 0x42697271
0x71 01110001
0x6d 01101101
wsnr TAub uIelw 0x62754158
0x58 01011000
0x54 01010100
tGep W3d5 EX5pU 0x3564335b
0x5b 01011011
0x57 01010111
gJTk uYzh fb3LM 0x687a5979
0x79 0001
0x75 01110101
EfFt qWDL RgVjY 0x4c445775
0x75 01110101
0x71 01110001
My2J GCTv 8A3GI 0x7654434b
0x4b 01001011
0x47 01000111
uWPk 7xcQ ZpTi. 0x5163783b
0x3b 00111011
0x37 00110111
AlD5 naNP oDKdc (OK)
0x6e 01101110
I eyeballed the chkpass code and didn't see any sign of buffer overruns,
but maybe it needs a harder look.
Hmm. I did give it a harder look and look what jumped out. Both chkpass_out
and chkpass_rout return PG_RETURN_CSTRING but chkpass_out builds a standard c
string while chkpass_rout builds a variable text structure. That can't be
right. It's odd that this always worked before. It seems to me that
chkpass_rout should be changed to build a c string like chkpass_out given the
name of the return macro. I tried that and it made no difference. I'm not
entirely surprised since I never used the chkpass_rout function in any of the
tests.
Is it possible that my thinking is wrong and I should be creating a text type
for both?
Still doesn't explain why no one else sees this though.
Oh, one more datapoint - the error only happens on vacuum analyze, not just
vacuum. Not sure what that means exactly.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
On August 27, 2002 02:01 am, Tom Lane wrote:
D'Arcy J.M. Cain [EMAIL PROTECTED] writes:
I have been getting the subject message ever since upgrading to 7.2.1. I
tried 7.2.2 with the same thing. It seems to be related to my chkpass
type (see contrib) as it only happens on tables with that type.
FWIW, I couldn't see any problem in CVS tip. Could you provide an exact
sequence-to-reproduce?
Surely. Create a database (chkpass_test) and, after loading the chkpass
type, follow this bouncing ball.
1. CREATE TABLE x (c chkpass);
2. INSERT INTO x VALUES ('a'); [Repeat 20 times]
3. VACUUM ANALYZE x;
4. UPDATE x SET c = ':a';
5. VACUUM ANALYZE x;
6. UPDATE x SET c = 'a';
7. VACUUM ANALYZE x;
8. GOTO 4
Note that 3 and 7 should fail most (95%?) of the time. When it does, convert
the invalid size to hex and compare the bytes of the integer with the
encrypted value in the table.
Note that the fact that 5 does not fail has nothing to do with the path
through chkpass.c. If you take a failing password and insert it raw with
UPDATE x SET c = ':2w3dhratDt7xo'; then you get the same failure (even the
same number - 1952543340 in my case) as you did when chkpass encrypted with
that salt.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
D'Arcy J.M. Cain [EMAIL PROTECTED] writes: On August 27, 2002 02:01 am, Tom Lane wrote: FWIW, I couldn't see any problem in CVS tip. Could you provide an exact sequence-to-reproduce? Surely. Create a database (chkpass_test) and, after loading the chkpass type, follow this bouncing ball. ... Note that 3 and 7 should fail most (95%?) of the time. Zero failures in a dozen iterations here. Anyone else see it? regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
On August 27, 2002 09:13 am, Tom Lane wrote:
D'Arcy J.M. Cain [EMAIL PROTECTED] writes:
On August 27, 2002 02:01 am, Tom Lane wrote:
FWIW, I couldn't see any problem in CVS tip. Could you provide an exact
sequence-to-reproduce?
Surely. Create a database (chkpass_test) and, after loading the chkpass
type, follow this bouncing ball.
...
Note that 3 and 7 should fail most (95%?) of the time.
Zero failures in a dozen iterations here. Anyone else see it?
NetBSD issue? It does it on all the NetBSD systems I tried it on. I will be
putting 7.2.2 on AIX shortly. I can try again. Can you think of any OS
issue here?
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
D'Arcy J.M. Cain [EMAIL PROTECTED] writes: On August 27, 2002 09:13 am, Tom Lane wrote: Zero failures in a dozen iterations here. Anyone else see it? NetBSD issue? It does it on all the NetBSD systems I tried it on. Hm. The first thing I thought was portability problem --- I had been testing on HPUX. But I just tried it on a Linux Intel box and see no failure there either. I will be putting 7.2.2 on AIX shortly. Note I'm testing CVS tip, not 7.2. Could we have fixed whatever the bug is? Seems unlikely, if it just appeared between 7.2 and 7.2.1 for you. Another possible reason for differences is configuration. I used a pretty plain-vanilla configure: ./configure --with-CXX --with-tcl --enable-cassert --enable-debug How about you? regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
D'Arcy J.M. Cain [EMAIL PROTECTED] writes: How about that --enable-multibyte? --enable-multibyte is default (indeed only) option on CVS tip, so that's not it. Ditto locale. Could you try CVS tip on one of the boxes where you see the failure? That'd help to narrow down the issue. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
[HACKERS] MemoryContextAlloc: invalid request size 1934906735
I have been getting the subject message ever since upgrading to 7.2.1. I
tried 7.2.2 with the same thing. It seems to be related to my chkpass type
(see contrib) as it only happens on tables with that type. I tried it on a
new database with a very simple table and still see it. After compiling
chkpass.c and running the SQL to create the type create a table with one
field with chkpass type. Add a number of rows, I did 24, then vacuum it.
You get something similar to the above. Sometimes you get Memory exhausted
in AllocSetAlloc(929654141) instead and once in a while there is no error.
Given table x with field c as chkpass run UPDATE x SET c = ':a'; on it.
This never fails. Now try UPDATE x SET c = 'a'; and let chkpass crypt the
value. This usually fails with one of the above messages. The number is
constant until you run the UPDATE again.
Somehow the value of the password string is polluting the size storage. I
know this because every time this happens, the first 2nd, 3rd and 4th bytes
(after adjusting for endianness) of the integer are the 6th, 7th and 8th
characters of the encrypted password.
I have another type which is built like this except that it is an indexable
type and that doesn't seem to have any problem. It is constructed the same
way otherwise. The palloc calls appear to be correct. Can anyone see why
this would suddenly be a problem?
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] MemoryContextAlloc: invalid request size 1934906735
D'Arcy J.M. Cain [EMAIL PROTECTED] writes: I have been getting the subject message ever since upgrading to 7.2.1. I tried 7.2.2 with the same thing. It seems to be related to my chkpass type (see contrib) as it only happens on tables with that type. FWIW, I couldn't see any problem in CVS tip. Could you provide an exact sequence-to-reproduce? regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
