Re: [HACKERS] v7.3 Branched ...
Perhaps one could just create a PostgreSQL Powertools section on techdocs, naming the packages and where to get them. This would eliminate the need to maintain a package that just duplicates other packages... Let ye-old package managers make a shell package which simply points to the others as dependencies. I'd be willing to do this for FreeBSD (think Sean? would help as well) if someone comes up with the list. There is a postgresql-devel port in FreeBSD now that I am maintaining that is where DBAs and anxious developers can cut their teeth on the new features/bugs/interactions in PostgreSQL. As soon as we get out of beta here, I'm going to likely get in the habbit of updating the port once a month or so with snapshots from the tree. FWIW, at some point I'm going to SPAM the CVS tree with a POSTGRESQL_PORT tunable that will let users decide which PostgreSQL instance they want (stable version vs -devel). I've been really busy recently and haven't gotten around to double checking things since I made the changes a month ago during the freeze. Maybe this weekend I'll get around to touching down on all of the various files no promises, I'm getting ready to move. -sc -- Sean Chittenden ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Multiple Key Clustering In Db2 8.1 - Interesting FYI
Dear hackers, I have recently been playing with DB2 8.1 Beta. It has introduced a feature to enable index clustering on more than one key. This reminded me of a previous thread on HACKERS about index access anding/bitmaps in Firebird. So anyway, here is a little snip from the 8.1 manual as a FYI. -- snip As the name implies, MDC tables cluster data on more than one dimension. Each dimension is determined by a column or set of columns that you specify in the ORGANIZE BY DIMENSIONS clause of the CREATE TABLE statement. When you create an MDC table, the following two kinds of indexes are created automatically: * A dimension-block index, which contains pointers to each occupied block for a single dimension. * A composite block index, which contains all dimension key columns. The composite block index is used to maintain clustering during insert and update activity. The optimizer considers dimension-block index scan plans when it determines the most efficient access plan for a particular query. When queries have predicates on dimension values, the optimizer can use the dimension block index to identify, and fetch from, only extents that contain these values. In addition, because extents are physically contiguous pages on disk, this results in more efficient performance and minimizes I/O. -- snipped regards Mark ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Current CVS is broken
%gmake bison -y -d preproc.y preproc.y:5560: fatal error: maximum table size (32767) exceeded gmake[4]: *** [preproc.h] Error 1 gmake[4]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces/ecpg/preproc' gmake[3]: *** [all] Error 2 gmake[3]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces/ecpg' gmake[2]: *** [all] Error 2 gmake[2]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces' gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory `/spool/home/teodor/pgsql/src' gmake: *** [all] Error 2 % bison -V bison (GNU Bison) 1.35 Copyright 1984, 1986, 1989, 1992, 2000, 2001, 2002 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. % uname -a FreeBSD xor 4.6-STABLE FreeBSD 4.6-STABLE #2: Tue Jun 18 20:48:48 MSD 2002 teodor@xor:/usr/src/sys/compile/XOR i386 -- Teodor Sigaev [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Current CVS is broken
Teodor Sigaev [EMAIL PROTECTED] wrote: %gmake bison -y -d preproc.y preproc.y:5560: fatal error: maximum table size (32767) exceeded gmake[4]: *** [preproc.h] Error 1 gmake[4]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces/ecpg/preproc' gmake[3]: *** [all] Error 2 gmake[3]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces/ecpg' gmake[2]: *** [all] Error 2 gmake[2]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces' gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory `/spool/home/teodor/pgsql/src' gmake: *** [all] Error 2 % bison -V bison (GNU Bison) 1.35 Although I am not a hacker, I think you just need to upgrade bison to version 1.75 if you want to build from CVS. The ecpg interface is broken with version 1.35 of bison. Best Regards, Michael Paesold ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] New SET/autocommit problem
Sean Chittenden [EMAIL PROTECTED] writes: WARNING: COMMIT: no transaction in progress I've got tons of these warnings in my logs... is there a programmatic way of determining if the current session is in a transaction? Not at present: you have to track it for yourself. One of the suggestions on the list for the next frontend/backend protocol revision (probably in 7.4) is to add a way for the backend to signal its transaction state: no transaction, in transaction, or in failed transaction seems like the set of possible states. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_dump and large files - is this a problem?
Philip Warner [EMAIL PROTECTED] writes: None, but it will be compatible with itself (the most we can hope for), and will work even if shifting is not supported for off_t (how likely is that?). I agree shift is definitely the way to go if it works on arbitrary data - ie. it does not rely on off_t being an integer. Can I shift a struct? You can't. If there are any platforms where in fact off_t isn't an arithmetic type, then shifting code would break there. I am not sure there are any; can anyone provide a counterexample? It would be simple enough to add a configure test to see whether off_t is arithmetic (just try to compile off_t x; x = 8;). How about #ifdef OFF_T_IS_ARITHMETIC_TYPE // cross-platform compatible use shifting method #else // not cross-platform compatible read or write bytes of struct in storage order #endif regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Current CVS is broken
Oleg Bartunov [EMAIL PROTECTED] writes: install bison 1.75 I've fixed the INSTALL and installation.sgml docs to say you need bison 1.50 or later (it used to say 1.28 or later). Is there anyplace else we should point this out? regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [GENERAL] [HACKERS] Hot Backup
On Wed, Oct 09, 2002 at 09:42:56AM -0400, Sandeep Chadha wrote: I'd say yes replication can solve lot of issues, but is there a way to do replication in postgres(active-active or active-passive) Yes. Check out the rserv code in contrib/, the (?) dbmirror code in contrib/, or contact PostgreSQL, Inc for a commercial version of the rserv code. A -- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada [EMAIL PROTECTED] M2P 2A8 +1 416 646 3304 x110 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] autocommit vs TRUNCATE et al
What I just committed uses your idea of auto-committing TRUNCATE et al, but now that I review the thread I think that everyone else thought that that was a dangerous idea. How do you feel about simply throwing an error in autocommit-off mode, instead? (At least it's a localized change now) Well, if I can throw in another opinion, I think what you did is perfect. It will make Oracle users happy too. Only very shrewd applications would commit previous changes with a truncate statement, and those will learn to issue a commit before truncate. I don't like the solutions involving set autocommit Andreas ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: Disabling triggers (was Re: [HACKERS] pgsql 7.2.3 crash)
On Mon, Oct 14, 2002 at 12:04:14AM -0400, Tom Lane wrote: implication is that its effects would be global to all backends. But the uses that I've seen for suspending trigger invocations would be happier with a local, temporary setting that only affects the current backend. Any thoughts about that? None except that it would indeed be a big help. A -- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada [EMAIL PROTECTED] M2P 2A8 +1 416 646 3304 x110 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] One 7.3 item left
Schema handling - ready? interfaces? client apps? What is the state of the Perl interface? Will it work when 7.3 is released Will it work, but no schema support Will it pop up later on CPAN -- Kaare Rasmussen--Linux, spil,--Tlf:3816 2582 Kaki Datatshirts, merchandize Fax:3816 2501 Howitzvej 75 Åben 12.00-18.00Email: [EMAIL PROTECTED] 2000 FrederiksbergLørdag 12.00-16.00 Web: www.suse.dk ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] CVS split problems
Marc G. Fournier writes: Okay, this is the only message(s) I have on this ... since merging earthdistance back into the tree will most likely cause more headaches, breakage and outcries, and since I see no reason why anyone would want to 'checkout' a module that has already been checked out (instead of doing an update like the rest of us), there is no way I'm going to put earthdistance back in ... ... unless there is, in fact, a completely different problem? It causes a useless and confusing divergence between the module names used to check out things and the names that appear in various messages, files, and the online views. Certainly it'd be a bad idea to do this now, but please do it after 7.3 is released. Just because removing a silliness causes a brief inconvenience is no reason to hang on to a silliness. Once v7.3 is released, I'd like to see a continuation of moving the non-core stuff over to GBorg, as well, so this will likely disappear at that time ... The issues I point out would continue to exist. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] One 7.3 item left
Bruce Momjian writes: OK, we are down to one open item, related to pg_dumping on 64-bit off_t. We had discussion today on this so it should be completed shortly. I hate to spoil the fun, but we have at least the Linux + Perl 5.8.1 + _GNU_SOURCE and the AIX + Large File + Flex problems to fix. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] One 7.3 item left
Peter Eisentraut [EMAIL PROTECTED] writes: Bruce Momjian writes: OK, we are down to one open item, related to pg_dumping on 64-bit off_t. We had discussion today on this so it should be completed shortly. I hate to spoil the fun, but we have at least the Linux + Perl 5.8.1 + _GNU_SOURCE and the AIX + Large File + Flex problems to fix. We should not, however, wait longer before pushing out a beta3 release. Portability problems on individual platforms may hold up RC1, but we're overdue to put out a final beta... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Silly little tool for making parallel_schedule variants
I got tired of wondering what timing dependencies might still be lurking
in the parallel regression tests, and wrote a little Perl script to find
out by making variant versions of the parallel_schedule file.
Essentially, the script forces each test in a parallel group to run
before all the other ones in its group, and also after all the other
ones. This isn't a completely bulletproof check: you could imagine
that test A might be unhappy about some intermediate state created by
test B, while not being unhappy with either the starting or ending
states. But it's a lot better than guess-and-hope.
The script successfully exposed the problem reported earlier today by
Robert Hentosh (create_index test depends on create_misc having run),
and didn't find any other problems, which I guess is a good sign.
(I've now committed a fix for that mistake, BTW.)
I'm not sure if the script has any long-term usefulness (anyone feel it
deserves to get into CVS in src/tools/?). But I'll attach it anyway
just so it gets into the pghackers archives.
To use it you'd do something like
mkdir scheds
./sched_variants parallel_schedule scheds/sch
for f in scheds/sch*
do
echo $f
/bin/sh ./pg_regress --schedule=$f
ff=`basename $f`
mv regression.out scheds/regression.out.$ff
mv regression.diffs scheds/regression.diffs.$ff
done
regards, tom lane
#! /usr/bin/perl
# Generate variants of parallel_schedule file to verify that there are
# no order dependencies between tests executed in parallel.
# Usage: sched_variants input_file output_prefix
# Output files are named with 'output_prefix' suffixed .1, .2, etc.
die Usage: sched_variants input_file output_prefix\n if ($#ARGV != 1);
$infile = $ARGV[0];
$outprefix = $ARGV[1];
$outcount = 0; # number of output files created
# We scan the input file repeatedly. On each pass we generate two
# output files, one where the k'th entry of each parallel test set
# has been extracted and forced to run first, and one where it's been
# forced to run last. The number of passes needed is the same as the
# largest number of tests in a parallel test set.
$k = 1; # test index we are currently hacking
$more = 1; # true if we need another pass
while ($more) {
$more = 0; # until proven differently
open(INFILE, $infile) || die $infile: $!\n;
$outcount++;
$outbefore = $outprefix . . . $outcount;
open(OUTBEFORE, $outbefore) || die $outbefore: $!\n;
$outcount++;
$outafter = $outprefix . . . $outcount;
open(OUTAFTER, $outafter) || die $outafter: $!\n;
while (INFILE) {
if (! /^test:/) {
# comment line
print OUTBEFORE $_;
print OUTAFTER $_;
next;
}
@tests = split;
shift(@tests); # remove test:
if ($#tests $k-1 || $#tests == 0) {
# too few tests in this set, just repeat as-is
print OUTBEFORE $_;
print OUTAFTER $_;
next;
}
if ($#tests = $k) {
$more = 1; # need more passes to process this set
}
@thistest = splice(@tests, $k-1, 1);
print OUTBEFORE test: @thistest\n;
print OUTBEFORE test: @tests\n;
print OUTAFTER test: @tests\n;
print OUTAFTER test: @thistest\n;
}
close OUTBEFORE;
close OUTAFTER;
close INFILE;
$k++;
}
print $outcount test files generated.\n;
exit 0;
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
[HACKERS] Memory leaks
I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Is this done periodically? If so, what tools are others using? I'm currently using dmalloc for my curiosity. [1] Not sure yet as I'm really wanting to find culumative leaks rather than one shot allocations which are simply never freed prior to process termination. Regards, Greg Copeland ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Tue, Oct 22, 2002 at 04:16:16PM -0500, Greg Copeland wrote: I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Is this done periodically? If so, what tools are others using? I'm currently using dmalloc for my curiosity. valgrind is a great tool I used -- didn't get the time to try it out on Postgres yet, though. Besides leaks, it also catches uninitialized variable access and stuff like that. Petru ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Memory leaks
Greg Copeland [EMAIL PROTECTED] writes: I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Yes, this has been dealt with before. Have you read src/backend/utils/mmgr/README? AFAIK the major problems these days are not in the backend as a whole, but in the lesser-used PL language modules (cf recent discussions). plpgsql has some issues too, I suspect, but not as bad as pltcl etc. Possibly the best answer is to integrate the memory-context notion into those modules; if they did most of their work in a temp context that could be freed once per PL statement or so, the problems would pretty much go away. It's fairly difficult to get anywhere with standard leak-tracking tools, since they don't know anything about palloc. What's worse, it is *not* a bug for a routine to palloc space it never pfrees, if it knows that it's palloc'ing in a short-lived memory context. The fact that a context may be released with much still-allocated memory in it is not a bug but a feature; but try teaching that to any standard leak checker... regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Thinking about IN/EXISTS optimization
I've been thinking about how to convert x IN (subselect) and EXISTS constructs into join-like processing, and I've run into a small problem in getting the planner to do it nicely. The issue is that I need to take the subselect and push it into the jointree --- essentially, make it look like a subselect-in-FROM --- so that the join planner can deal with it. Basically, I need to rearrange SELECT ... FROM ... WHERE ... AND x IN (SELECT y FROM ...) into SELECT ... FROM ..., (SELECT y FROM ...) ss WHERE ... AND x =* ss.y where =* represents some specially-marked RestrictInfo node. (NOT IN is the same except that the RestrictInfo node will be marked differently.) The difficulty is that there's no good place to do this in subquery_planner(). We should push the subselect into FROM before we run the pull_up_subqueries() and preprocess_jointree() operations; if we don't pull up the subselect into the main query then we won't have accomplished very much. But the WHERE clause isn't simplified into a form that makes it easy to spot top-level IN() expressions until after that. We can't simply switch the order of the subselect and WHERE-clause processing, because pulling up subqueries typically adds conditions to the WHERE clause. I haven't been able to think of a solution to this that doesn't involve wasting a lot of cycles by repeating some of these processing steps, or missing some optimization possibilities. (For example, if we pull up a subquery that came from a view, it might contain an IN where-clause, which ideally we'd want to be able to optimize. It almost seems like we need to be able to loop around the whole operation; but most of the time this will just waste cycles.) Anyone see a nice way to do this? regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Memory leaks
Petru Paler [EMAIL PROTECTED] writes: valgrind is a great tool I used -- didn't get the time to try it out on Postgres yet, though. Besides leaks, it also catches uninitialized variable access and stuff like that. I've used Valgrind with PostgreSQL a little bit, and it's been fairly useful (I used it to fix some memory leaks in psql and pg_dump and a couple of uninitialized memory accesses in the backend). If you want to use it on the backend, you'll need to stop postgres from clobbering ARGV (as this causes valgrind problems, for some reason) -- add '-DPS_USE_NONE -UPS_USE_CLOBBER_ARGV' to CFLAGS. I mentioned it to the author of valgrind, but IIRC he didn't mention any plans to change this behavior. Cheers, Neil -- Neil Conway [EMAIL PROTECTED] || PGP Key ID: DB3C29FC ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
On Tue, 2002-10-22 at 17:09, Tom Lane wrote: Greg Copeland [EMAIL PROTECTED] writes: I've started playing a little with Postgres to determine if there were memory leaks running around. After some very brief checking, I'm starting[1] to think that the answer is yes. Has anyone already gone through a significant effort to locate and eradicate memory leaks? Yes, this has been dealt with before. What tools, aside from noggin v1.0, did they use? Do we know? Have you read src/backend/utils/mmgr/README? Yes...but it's been some time since I last opened it. It was because I knew there were some caveats that I wasn't attempting to claim for sure that there were leaks. I then moved on to psql, again, just for fun. Here, I'm thinking that I started to find some other leaks...but again, I've not spent any real time on it. So again, I'm not really sure it they are meaningful at this point. AFAIK the major problems these days are not in the backend as a whole, but in the lesser-used PL language modules (cf recent discussions). Ya, that's what made me wonder about the topic on a larger scale. plpgsql has some issues too, I suspect, but not as bad as pltcl etc. Possibly the best answer is to integrate the memory-context notion into those modules; if they did most of their work in a temp context that could be freed once per PL statement or so, the problems would pretty much go away. Interesting. Having not looked at memory management schemes used in the pl implementations, can you enlighten me by what you mean by integrate the memory-context notion? Does that mean they are not using palloc/pfree stuff? It's fairly difficult to get anywhere with standard leak-tracking tools, since they don't know anything about palloc. What's worse, it is *not* a bug for a routine to palloc space it never pfrees, if it knows that it's palloc'ing in a short-lived memory context. The fact that a context may be released with much still-allocated memory in it is not a bug but a feature; but try teaching that to any standard leak checker... regards, tom lane Well, the thing that really got my attention is that dmalloc is reporting frees on null pointers. While that may be safe on specific platforms, IIRC, it's actually undefined. Again, this is as reported by dmalloc so I've yet to actually track it down to determine if it's possibly something else going on (magic voodoo of some heap manager, etc). Greg ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Memory leaks
On 22 Oct 2002, Greg Copeland wrote: On Tue, 2002-10-22 at 17:09, Tom Lane wrote: plpgsql has some issues too, I suspect, but not as bad as pltcl etc. Possibly the best answer is to integrate the memory-context notion into those modules; if they did most of their work in a temp context that could be freed once per PL statement or so, the problems would pretty much go away. Interesting. Having not looked at memory management schemes used in the pl implementations, can you enlighten me by what you mean by integrate the memory-context notion? Does that mean they are not using palloc/pfree stuff? I saw use of a couple of malloc (or Python specific malloc) calls the other day but I also seem to recall that, after consideration, I decided the memory needed to survive for the duration of the backend. Should I have created a new child of the top context and changed these malloc calls? I was going to ask about thoughts on redirecting malloc etc to palloc etc and thereby intercepting memory allocation within the languages and automatically bringing them into the memory context realm. However, that would just be making life way too awkward, bearing in mind the above paragraph. Can't we get Sir Mongle (or whatever the name was) to test these things under the auspices of them being DoS attacks? -- Nigel J. Andrews ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] One 7.3 item left
On Tue, Oct 22, 2002 at 19:01:20 +0200, Kaare Rasmussen [EMAIL PROTECTED] wrote: Schema handling - ready? interfaces? client apps? What is the state of the Perl interface? Will it work when 7.3 is released Will it work, but no schema support Will it pop up later on CPAN I am using Pg with 7.3b1 and it works OK for what I am doing. I am not explicitly naming schemas when referencing objects though. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Memory leaks
Nigel J. Andrews [EMAIL PROTECTED] writes: I saw use of a couple of malloc (or Python specific malloc) calls the other day but I also seem to recall that, after consideration, I decided the memory needed to survive for the duration of the backend. Should I have created a new child of the top context and changed these malloc calls? If there is truly no reason to release the memory before the backend dies, then I see no reason to avoid malloc(). Basically what the memory context stuff gives you is the ability to bunch allocations together and release a whole tree of stuff for little work. An example: currently, the plpgsql parser builds its parsed syntax tree with a bunch of malloc's. It has no way to free a syntax tree, so that tree lives till the backend exits, whether it's ever used again or not. It would be better to build the tree via palloc's in a context created specially for the specific function: then we could free the whole context if we discovered that the function had gone away or been redefined. (Compare what relcache does for rule syntaxtrees for rules associated with relcache entries.) But right at the moment, there's no mechanism to discover that situation, and so there's not now any direct value in using palloc instead of malloc for plpgsql syntaxtrees. Yet that's surely the direction to go in for the long term. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Memory leaks
Greg Copeland [EMAIL PROTECTED] writes: On Tue, 2002-10-22 at 17:09, Tom Lane wrote: Yes, this has been dealt with before. What tools, aside from noggin v1.0, did they use? Do we know? s/they/me/ ... none. But I don't know of any that I think would be useful. I then moved on to psql, again, just for fun. Here, I'm thinking that I started to find some other leaks...but again, I've not spent any real time on it. So again, I'm not really sure it they are meaningful at this point. psql might well have some internal leaks; the backend memory-context design doesn't apply to it. Possibly the best answer is to integrate the memory-context notion into those modules; if they did most of their work in a temp context that could be freed once per PL statement or so, the problems would pretty much go away. Interesting. Having not looked at memory management schemes used in the pl implementations, can you enlighten me by what you mean by integrate the memory-context notion? Does that mean they are not using palloc/pfree stuff? Not everywhere. plpgsql is full of malloc's and I think the other PL modules are too --- and that's not to mention the allocation policies of the perl, tcl, etc, language interpreters. We could use a thorough review of that whole area. Well, the thing that really got my attention is that dmalloc is reporting frees on null pointers. AFAIK that would dump core on many platforms (it sure does here...), so I don't think I believe it without seeing chapter and verse. But if you can point out where it's really happening, then we must fix it. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] pg_dump and large files - is this a problem?
Bruce Momjian [EMAIL PROTECTED] writes: I wonder if any other platforms have this limitation. I think we need to add some type of test for no-fseeko()/ftello() and sizeof(off_t) sizeof(long). This fseeko/ftello/off_t is just too fluid, and the failure modes too serious. I am wondering why pg_dump has to depend on either fseek or ftell. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] pg_dump and large files - is this a problem?
At 12:32 AM 23/10/2002 -0400, Tom Lane wrote: I am wondering why pg_dump has to depend on either fseek or ftell. It doesn't - it just works better and has more features if they are available, much like zlib etc. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /() __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_dump and large files - is this a problem?
At 12:29 AM 23/10/2002 -0400, Bruce Momjian wrote: This fseeko/ftello/off_t is just too fluid, and the failure modes too serious. I agree. Can you think of a better solution than the one I suggested??? Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /() __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] pg_dump and large files - is this a problem?
OK, you are saying if we don't have fseeko(), there is no reason to use off_t, and we may as well use long. What limitations does that impose, and are the limitations clear to the user. What has me confused is that I only see two places that use a non-zero fseeko, and in those cases, there is a non-fseeko code path that does the same thing, or the call isn't actually required. Both cases are in pg_dump/pg_dump_custom.c. It appears seeking in the file is an optimization that prevents all the blocks from being read. That is fine, but we shouldn't introduce failure cases to do that. If BSD/OS is the only problem OS, I can deal with that, but I have no idea if other OS's have the same limitation, and because of the way our code exists now, we are not even checking to see if there is a problem. I did some poking around, and on BSD/OS, fgetpos/fsetpos use fpos_t, which is actually off_t, and interestingly, lseek() uses off_t too. Seems only fseek/ftell is limited to long. I can easily implemnt fseeko/ftello using fgetpos/fsetpos, but that is only one OS. One idea would be to patch up BSD/OS in backend/port/bsdi and add a configure tests that actually fails if fseeko doesn't exist _and_ sizeof(off_t) sizeof(long). That would at least catch OS's before they make 2gig backups that can't be restored. --- Philip Warner wrote: At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote: Uh, not exactly. I have off_t as a quad, and I don't have fseeko, so the above conditional doesn't work. I want to use off_t, but can't use fseek(). Then when you create dumps, they will be invalid since I assume that ftello is also broken in the same way. You need to fix _getFilePos as well. And any other place that uses an off_t needs to be looked at very carefully. The code was written assuming that if 'hasSeek' was set, then we could trust it. Given that you say you do have support for some kind of 64 bt offset, I would be a lot happier with these changes if you did something akin to my original sauggestion: #if defined(HAVE_FSEEKO) #define FILE_OFFSET off_t #define FSEEK fseeko #elseif defined(HAVE_SOME_OTHER_FSEEK) #define FILE_OFFSET some_other_offset #define FSEEK some_other_fseek #else #define FILE_OFFSET long #define FSEEK fseek #end if ...assuming you have a non-broken 64 bit fseek/tell pair, then this will work in all cases, and make the code a lot less ugly (assuming of course the non-broken version can be shifted). Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED]) -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_dump and large files - is this a problem?
At 01:02 AM 23/10/2002 -0400, Bruce Momjian wrote: OK, you are saying if we don't have fseeko(), there is no reason to use off_t, and we may as well use long. What limitations does that impose, and are the limitations clear to the user. What I'm saying is that if we have not got fseeko then we should use any 'seek-class' function that returns a 64 bit value. We have already made the assumption that off_t is an integer; the same logic that came to that conclusion, applies just as validly to the other seek functions. Secondly, if there is no 64 bit 'seek-class' function, then we should probably use a size_t, but a long would probably be fine too. I am not particularly attached to this part; long, int etc etc. Whatever is most likely to return an integer and work with whatever function we choose. As to implications: assuming they are all integers (which as you know I don't like), we should have no problems. If a system does not have any function to access 64 bit file offsets, then I'd say they are pretty unlikely to have files 2GB. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Memory leaks
On Tue, Oct 22, 2002 at 11:28:23PM -0400, Tom Lane wrote: I then moved on to psql, again, just for fun. Here, I'm thinking that I started to find some other leaks...but again, I've not spent any real time on it. So again, I'm not really sure it they are meaningful at this point. psql might well have some internal leaks; the backend memory-context design doesn't apply to it. But why? In the Mape project is used mmgr based on PostgreSQL's mmgr and it's used for BE and FE. There is not problem with it (BTW backend is multithread:-). IMHO use memory-context design for FE is good idea if FE a lot works with memory. I already long time think about shared lib with PostgreSQL mmgr... Karel -- Karel Zak [EMAIL PROTECTED] http://home.zf.jcu.cz/~zakkr/ C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] New SET/autocommit problem
WARNING: COMMIT: no transaction in progress COMMIT The WARNING happens with SHOW and RESET too. I wonder if we should suppress the WARNING of a COMMIT with no statements when autocommit is off. I don't think so; that will make it quite hard to check whether a transaction is open. I've got tons of these warnings in my logs... is there a programmatic way of determining if the current session is in a transaction? Unless I misunderstood the fix and the commit message, I'm pretty sure that most of my problem has been fixed in CVS with SET's auto-committing if it's not in a transaction, but now that there are some intricate rules with regards to starting transactions, I'd love to provide a DBI interface into a call that returns whether or not we're in a transaction to prevent millions of these: NOTICE: ROLLBACK: no transaction in progress -sc -- Sean Chittenden ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_dump and large files - is this a problem?
At 09:52 PM 21/10/2002 -0400, Bruce Momjian wrote: 4) pg_restore -Fc /tmp/x pg_restore /tmp/x is enough; it will determine the file type, and by avoiding the pipe, you allow it to do seeks which are not much use here, but are usefull when you only restore one table in a very large backup. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] integer array, push and pop
regression=# select '{124,567,66}'::int[] + 345;
?column?
--
{124,567,66,345}
(1 row)
regression=# select '{124,567,66}'::int[] + '{345,1}'::int[];
?column?
{124,567,66,345,1}
(1 row)
select '{124,567,66}'::int[] - 567;
?column?
--
{124,66}
(1 row)
regression=# select '{124,567,66}'::int[] - '{567,66}';
?column?
--
{124}
(1 row)
Ryan Mahoney wrote:
Hi Oleg (and pgsql hackers!),
Recently I encountered a problem attempting to use the integer array
function for pushing an integer onto an integer array field.
Can you write an example of a sql statement for pushing a single value onto
an integer array and for popping a specific value off of an integer array?
I see the function in the documentation, but the actual statement syntax to
use is not clear to me.
Thanks for any help you can provide!
Ryan Mahoney
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
--
Teodor Sigaev
[EMAIL PROTECTED]
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Current CVS is broken
install bison 1.75 On Tue, 22 Oct 2002, Teodor Sigaev wrote: %gmake bison -y -d preproc.y preproc.y:5560: fatal error: maximum table size (32767) exceeded gmake[4]: *** [preproc.h] Error 1 gmake[4]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces/ecpg/preproc' gmake[3]: *** [all] Error 2 gmake[3]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces/ecpg' gmake[2]: *** [all] Error 2 gmake[2]: Leaving directory `/spool/home/teodor/pgsql/src/interfaces' gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory `/spool/home/teodor/pgsql/src' gmake: *** [all] Error 2 % bison -V bison (GNU Bison) 1.35 Copyright 1984, 1986, 1989, 1992, 2000, 2001, 2002 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. % uname -a FreeBSD xor 4.6-STABLE FreeBSD 4.6-STABLE #2: Tue Jun 18 20:48:48 MSD 2002 teodor@xor:/usr/src/sys/compile/XOR i386 Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] pg_dump and large files - is this a problem?
At 10:16 AM 21/10/2002 -0400, Tom Lane wrote: What are the odds that dumping the bytes in it, in either order, will produce something that's compatible with any other platform? None, but it will be compatible with itself (the most we can hope for), and will work even if shifting is not supported for off_t (how likely is that?). I agree shift is definitely the way to go if it works on arbitrary data - ie. it does not rely on off_t being an integer. Can I shift a struct? Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /() __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] pg_dump and large files - is this a problem?
At 12:00 PM 22/10/2002 -0400, Bruce Momjian wrote: It does have the advantage of being more portable on systems that do have integral off_t I suspect it is no more portable than determining storage order by using 'int i = 256', then writing in storage order, and has the disadvantage that it may break as discussed. AFAICT, using storage order will not break under any circumstances within one OS/architecture (unlike using shift), and will not break any more often than using shift in cases where off_t is integral. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_dump and large files - is this a problem?
Bruce Momjian [EMAIL PROTECTED] writes: However, since we don't know if we support any non-integral off_t platforms, and because a configure test would require us to have two code paths for with/without integral off_t, I suggest we apply my version of Philip's patch and let's see if everyone can compile it cleanly. Actually, it looks to me like configure will spit up if off_t is not an integral type: /* Check that off_t can represent 2**63 - 1 correctly. We can't simply define LARGE_OFF_T to be 9223372036854775807, since some C++ compilers masquerading as C compilers incorrectly reject 9223372036854775807. */ #define LARGE_OFF_T (((off_t) 1 62) - 1 + ((off_t) 1 62)) int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721 LARGE_OFF_T % 2147483647 == 1) ? 1 : -1]; So I think we're wasting our time to debate whether we need to support non-integral off_t ... let's just apply Bruce's version and wait to see if anyone has a problem before doing more work. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] pg_dump and large files - is this a problem?
Philip Warner wrote: At 12:00 PM 22/10/2002 -0400, Bruce Momjian wrote: It does have the advantage of being more portable on systems that do have integral off_t I suspect it is no more portable than determining storage order by using 'int i = 256', then writing in storage order, and has the disadvantage that it may break as discussed. AFAICT, using storage order will not break under any circumstances within one OS/architecture (unlike using shift), and will not break any more often than using shift in cases where off_t is integral. Your version will break more often because we are assuming we can determine the endian-ness of the OS, _and_ for quad off_t types, assuming we know that is stored the same too. While we have ending for int's, I have no idea if quads are always stored the same. By accessing it as an integral type, we make certain it is output the same way every time for every OS. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_dump and large files - is this a problem?
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: However, since we don't know if we support any non-integral off_t platforms, and because a configure test would require us to have two code paths for with/without integral off_t, I suggest we apply my version of Philip's patch and let's see if everyone can compile it cleanly. Actually, it looks to me like configure will spit up if off_t is not an integral type: /* Check that off_t can represent 2**63 - 1 correctly. We can't simply define LARGE_OFF_T to be 9223372036854775807, since some C++ compilers masquerading as C compilers incorrectly reject 9223372036854775807. */ #define LARGE_OFF_T (((off_t) 1 62) - 1 + ((off_t) 1 62)) int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721 LARGE_OFF_T % 2147483647 == 1) ? 1 : -1]; So I think we're wasting our time to debate whether we need to support non-integral off_t ... let's just apply Bruce's version and wait to see if anyone has a problem before doing more work. I am concerned about one more thing. On BSD/OS, we have off_t of quad (8 byte), but we don't have fseeko, so this call looks questionable: if (fseeko(AH-FH, tctx-dataPos, SEEK_SET) != 0) In this case, dataPos is off_t (8 bytes), while fseek only accepts long in that parameter (4 bytes). When this code is hit, a file 4 gigs will seek to the wrong offset, I am afraid. Also, I don't understand why the compiler doesn't produce a warning. I wonder if I should add a conditional test so this code is hit only if HAVE_FSEEKO is defined. There is alternative code for all the non-zero fseeks. Comments? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] autocommit vs TRUNCATE et al
Tom Lane wrote: Joe Conway [EMAIL PROTECTED] writes: I just noticed that this afternoon's changes cause dblink's regression test to fail due to: CREATE OR REPLACE FUNCTION conditional_drop() [...] IF FOUND THEN DROP DATABASE regression_slave; END IF; [...] ' LANGUAGE 'plpgsql'; SELECT conditional_drop(); I'm wondering what is the best alternative? Well, the *best* alternative would be to make CREATE/DROP DATABASE transaction-safe ;-). I was speculating to myself earlier today about how we might do that. It seems like it's not that far out of reach: we could make smgr's list of files-to-remove-at-xact-commit-or-abort include whole database subdirectories. But I'm not sure how that would interact with upcoming features like tablespaces, so I don't want to go off and implement it right now. FYI, the MSWin port in 7.4 will have C versions of 'cp' and 'rm -r', so those can be used to hook into the smgr layer for all platforms. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] One 7.3 item left
Kaare Rasmussen wrote: Schema handling - ready? interfaces? client apps? What is the state of the Perl interface? Will it work when 7.3 is released Will it work, but no schema support Will it pop up later on CPAN We have a separate gborg project for the old perl5 in interface and dbd-pg. The DBD group is making improvements right now. Not sure how it works with 7.3 but I am sure they will get to testing it soon. David Wheeler is working on it, and he is involved in 7.3. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] New SET/autocommit problem
WARNING: COMMIT: no transaction in progress I've got tons of these warnings in my logs... is there a programmatic way of determining if the current session is in a transaction? Not at present: you have to track it for yourself. One of the suggestions on the list for the next frontend/backend protocol revision (probably in 7.4) is to add a way for the backend to signal its transaction state: no transaction, in transaction, or in failed transaction seems like the set of possible states. That would be fabulous because with autocommit set to off, the complexity for tracking that in application is getting pretty gnarly. -sc -- Sean Chittenden ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_dump and large files - is this a problem?
Bruce Momjian [EMAIL PROTECTED] writes: Your version will break more often because we are assuming we can determine the endian-ness of the OS, _and_ for quad off_t types, assuming we know that is stored the same too. While we have ending for int's, I have no idea if quads are always stored the same. There is precedent for problems of that ilk, too, cf PDP_ENDIAN: years ago someone made double-word-integer software routines and did not think twice about which word should appear first in storage, with the consequence that the storage order was neither little-endian nor big-endian. (We have exactly the same issue with our CRC routines for compilers without int64: the two-int32 struct is defined in a way that's compatible with little-endian storage, and on a big-endian machine it'll produce a funny storage order.) Unless someone can point to a supported (or potentially interesting) platform on which off_t is indeed not integral, I think the shift-based code is our safest bet. (The precedent of the off_t checking code in configure makes me really doubt that there are any platforms with non-integral off_t.) regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] pg_dump and large files - is this a problem?
Patch applied with shift / changes by me. Thanks. --- Philip Warner wrote: I have put the latest patch at: http://downloads.rhyme.com.au/postgresql/pg_dump/ along with two dump files of the regression DB, one with 4 byte and the other with 8 byte offsets. I can read/restore each from the other, so it looks pretty good. Once the endianness is tested, we should be OK. Known problems: - will not cope with 4GB files and size_t not 64 bit. - when printing data position, it is assumed that off_t is UINT64 (we could remove this entirely - it's just for display) - if seek is not supported, then an intXX is assigned to off_t when file offsets are needed. This *should* not cause a problem since without seek, the offsets will not be written to the file. Changes from Prior Version: - No longer stores or outputs data length - Assumes result of ftello is correct if it disagrees with internally kept tally. - 'pg_restore -l' now shows sizes of int and offset. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_dump and large files - is this a problem?
Bruce Momjian wrote:
So I think we're wasting our time to debate whether we need to support
non-integral off_t ... let's just apply Bruce's version and wait to
see if anyone has a problem before doing more work.
I am concerned about one more thing. On BSD/OS, we have off_t of quad
(8 byte), but we don't have fseeko, so this call looks questionable:
if (fseeko(AH-FH, tctx-dataPos, SEEK_SET) != 0)
In this case, dataPos is off_t (8 bytes), while fseek only accepts long
in that parameter (4 bytes). When this code is hit, a file 4 gigs
will seek to the wrong offset, I am afraid. Also, I don't understand
why the compiler doesn't produce a warning.
I wonder if I should add a conditional test so this code is hit only if
HAVE_FSEEKO is defined. There is alternative code for all the non-zero
fseeks.
Here is a patch that I think fixes the problem I outlined above. If
there is no fseeko(), it will not call fseek with a non-zero offset
unless sizeof(off_t) = sizeof(long).
--
Bruce Momjian| http://candle.pha.pa.us
[EMAIL PROTECTED] | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup.| Newtown Square, Pennsylvania 19073
Index: src/bin/pg_dump/pg_backup_custom.c
===
RCS file: /cvsroot/pgsql-server/src/bin/pg_dump/pg_backup_custom.c,v
retrieving revision 1.22
diff -c -c -r1.22 pg_backup_custom.c
*** src/bin/pg_dump/pg_backup_custom.c 22 Oct 2002 19:15:23 - 1.22
--- src/bin/pg_dump/pg_backup_custom.c 22 Oct 2002 21:36:30 -
***
*** 431,437
if (tctx-dataState == K_OFFSET_NO_DATA)
return;
! if (!ctx-hasSeek || tctx-dataState == K_OFFSET_POS_NOT_SET)
{
/* Skip over unnecessary blocks until we get the one we want. */
--- 431,441
if (tctx-dataState == K_OFFSET_NO_DATA)
return;
! if (!ctx-hasSeek || tctx-dataState == K_OFFSET_POS_NOT_SET
! #if !defined(HAVE_FSEEKO)
! || sizeof(off_t) sizeof(long)
! #endif
! )
{
/* Skip over unnecessary blocks until we get the one we want. */
***
*** 809,815
* be ok to just use the existing self-consistent block
* formatting.
*/
! if (ctx-hasSeek)
{
fseeko(AH-FH, tpos, SEEK_SET);
WriteToc(AH);
--- 813,823
* be ok to just use the existing self-consistent block
* formatting.
*/
! if (ctx-hasSeek
! #if !defined(HAVE_FSEEKO)
!sizeof(off_t) = sizeof(long)
! #endif
! )
{
fseeko(AH-FH, tpos, SEEK_SET);
WriteToc(AH);
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_dump and large files - is this a problem?
Bruce Momjian writes: I am concerned about one more thing. On BSD/OS, we have off_t of quad (8 byte), but we don't have fseeko, so this call looks questionable: if (fseeko(AH-FH, tctx-dataPos, SEEK_SET) != 0) Maybe you want to ask your OS provider how the heck this is supposed to work. I mean, it's great to have wide types, but what's the point if the API can't handle them? -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Memory leaks
On Wed, 2002-10-23 at 03:09, Tom Lane wrote: It's fairly difficult to get anywhere with standard leak-tracking tools, since they don't know anything about palloc. What's worse, it is *not* a bug for a routine to palloc space it never pfrees, if it knows that it's palloc'ing in a short-lived memory context. The fact that a context may be released with much still-allocated memory in it is not a bug but a feature; but try teaching that to any standard leak checker... Seems that Valgrind should have no problems with it, as it tracks actual usage of _memory_ (down to single bits :)) , not malloc/free. See: http://developer.kde.org/~sewardj/ --- Hannu ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_dump and large files - is this a problem?
Peter Eisentraut wrote: Bruce Momjian writes: I am concerned about one more thing. On BSD/OS, we have off_t of quad (8 byte), but we don't have fseeko, so this call looks questionable: if (fseeko(AH-FH, tctx-dataPos, SEEK_SET) != 0) Maybe you want to ask your OS provider how the heck this is supposed to work. I mean, it's great to have wide types, but what's the point if the API can't handle them? Excellent question. They do have fsetpos/fgetpos, and I think they think you are supposed to use those. However, they don't do seek from current position, and they don't take an off_t, so I am confused myself. I did ask on the mailing list and everyone kind of agreed it was a missing feature. However, because of the way we call fseeko not knowing if it is a quad or a long, I think we have to add the checks to prevent such wild seeks from happening. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] One 7.3 item left
On Tue, 22 Oct 2002, Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Bruce Momjian writes: OK, we are down to one open item, related to pg_dumping on 64-bit off_t. We had discussion today on this so it should be completed shortly. I hate to spoil the fun, but we have at least the Linux + Perl 5.8.1 + _GNU_SOURCE and the AIX + Large File + Flex problems to fix. We should not, however, wait longer before pushing out a beta3 release. Portability problems on individual platforms may hold up RC1, but we're overdue to put out a final beta... Was just about to ask that ... Friday sound reasonable for beta3 then? Bruce, can you have all your files updated by then? ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] One 7.3 item left
Marc G. Fournier wrote: On Tue, 22 Oct 2002, Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Bruce Momjian writes: OK, we are down to one open item, related to pg_dumping on 64-bit off_t. We had discussion today on this so it should be completed shortly. I hate to spoil the fun, but we have at least the Linux + Perl 5.8.1 + _GNU_SOURCE and the AIX + Large File + Flex problems to fix. We should not, however, wait longer before pushing out a beta3 release. Portability problems on individual platforms may hold up RC1, but we're overdue to put out a final beta... Was just about to ask that ... Friday sound reasonable for beta3 then? Bruce, can you have all your files updated by then? I can, sure. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_dump and large files - is this a problem?
At 05:37 PM 22/10/2002 -0400, Bruce Momjian wrote: ! if (ctx-hasSeek ! #if !defined(HAVE_FSEEKO) !sizeof(off_t) = sizeof(long) ! #endif ! ) Just to clarify my understanding: - HAVE_FSEEKO is tested defined in configure - If it is not defined, then all calls to fseeko will magically be translated to fseek calls, and use the 'long' parameter type. Is that right? If so, why don't we: #if defined(HAVE_FSEEKO) #define FILE_OFFSET off_t #define FSEEK fseeko #else #define FILE_OFFSET long #define FSEEK fseek #end if then replace all refs to off_t with FILE_OFFSET, and fseeko with FSEEK. Existing checks etc will then refuse to load file offsets with significant bytes after the 4th byte, we will still use fseek/o in broken OS implementations of off_t. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /() __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Thinking about IN/EXISTS optimization
This sounds like one of those classic optimizer problems we have had to deal with in the past. I suggest you go through the optimizer pass and set a boolean in Query whenever you do something that may require another loop through, then at the end, you check the boolean and loop if required. I think the rules system has to do something similar. I don't see any way around that, but because you are setting the boolean you only loop when you need to. --- Tom Lane wrote: I've been thinking about how to convert x IN (subselect) and EXISTS constructs into join-like processing, and I've run into a small problem in getting the planner to do it nicely. The issue is that I need to take the subselect and push it into the jointree --- essentially, make it look like a subselect-in-FROM --- so that the join planner can deal with it. Basically, I need to rearrange SELECT ... FROM ... WHERE ... AND x IN (SELECT y FROM ...) into SELECT ... FROM ..., (SELECT y FROM ...) ss WHERE ... AND x =* ss.y where =* represents some specially-marked RestrictInfo node. (NOT IN is the same except that the RestrictInfo node will be marked differently.) The difficulty is that there's no good place to do this in subquery_planner(). We should push the subselect into FROM before we run the pull_up_subqueries() and preprocess_jointree() operations; if we don't pull up the subselect into the main query then we won't have accomplished very much. But the WHERE clause isn't simplified into a form that makes it easy to spot top-level IN() expressions until after that. We can't simply switch the order of the subselect and WHERE-clause processing, because pulling up subqueries typically adds conditions to the WHERE clause. I haven't been able to think of a solution to this that doesn't involve wasting a lot of cycles by repeating some of these processing steps, or missing some optimization possibilities. (For example, if we pull up a subquery that came from a view, it might contain an IN where-clause, which ideally we'd want to be able to optimize. It almost seems like we need to be able to loop around the whole operation; but most of the time this will just waste cycles.) Anyone see a nice way to do this? regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED]) -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_dump and large files - is this a problem?
Philip Warner wrote: At 05:37 PM 22/10/2002 -0400, Bruce Momjian wrote: ! if (ctx-hasSeek ! #if !defined(HAVE_FSEEKO) !sizeof(off_t) = sizeof(long) ! #endif ! ) Just to clarify my understanding: - HAVE_FSEEKO is tested defined in configure - If it is not defined, then all calls to fseeko will magically be translated to fseek calls, and use the 'long' parameter type. Is that right? If so, why don't we: #if defined(HAVE_FSEEKO) #define FILE_OFFSET off_t #define FSEEK fseeko #else #define FILE_OFFSET long #define FSEEK fseek #end if then replace all refs to off_t with FILE_OFFSET, and fseeko with FSEEK. Existing checks etc will then refuse to load file offsets with significant bytes after the 4th byte, we will still use fseek/o in broken OS implementations of off_t. Uh, not exactly. I have off_t as a quad, and I don't have fseeko, so the above conditional doesn't work. I want to use off_t, but can't use fseek(). As it turns out, the code already has options to handle no fseek, so it seems to work anyway. I think what you miss may be the table of contents in the archive, if I am reading the code correctly. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pg_dump and large files - is this a problem?
At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote: Uh, not exactly. I have off_t as a quad, and I don't have fseeko, so the above conditional doesn't work. I want to use off_t, but can't use fseek(). Then when you create dumps, they will be invalid since I assume that ftello is also broken in the same way. You need to fix _getFilePos as well. And any other place that uses an off_t needs to be looked at very carefully. The code was written assuming that if 'hasSeek' was set, then we could trust it. Given that you say you do have support for some kind of 64 bt offset, I would be a lot happier with these changes if you did something akin to my original sauggestion: #if defined(HAVE_FSEEKO) #define FILE_OFFSET off_t #define FSEEK fseeko #elseif defined(HAVE_SOME_OTHER_FSEEK) #define FILE_OFFSET some_other_offset #define FSEEK some_other_fseek #else #define FILE_OFFSET long #define FSEEK fseek #end if ...assuming you have a non-broken 64 bit fseek/tell pair, then this will work in all cases, and make the code a lot less ugly (assuming of course the non-broken version can be shifted). Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] pg_dump and large files - is this a problem?
Sounds messy. Let me see if I can code up an fseeko/ftello for BSD/OS and add that to /port. No reason to hold up beta for that, though. I wonder if any other platforms have this limitation. I think we need to add some type of test for no-fseeko()/ftello() and sizeof(off_t) sizeof(long). This fseeko/ftello/off_t is just too fluid, and the failure modes too serious. --- Philip Warner wrote: At 10:46 PM 22/10/2002 -0400, Bruce Momjian wrote: Uh, not exactly. I have off_t as a quad, and I don't have fseeko, so the above conditional doesn't work. I want to use off_t, but can't use fseek(). Then when you create dumps, they will be invalid since I assume that ftello is also broken in the same way. You need to fix _getFilePos as well. And any other place that uses an off_t needs to be looked at very carefully. The code was written assuming that if 'hasSeek' was set, then we could trust it. Given that you say you do have support for some kind of 64 bt offset, I would be a lot happier with these changes if you did something akin to my original sauggestion: #if defined(HAVE_FSEEKO) #define FILE_OFFSET off_t #define FSEEK fseeko #elseif defined(HAVE_SOME_OTHER_FSEEK) #define FILE_OFFSET some_other_offset #define FSEEK some_other_fseek #else #define FILE_OFFSET long #define FSEEK fseek #end if ...assuming you have a non-broken 64 bit fseek/tell pair, then this will work in all cases, and make the code a lot less ugly (assuming of course the non-broken version can be shifted). Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED]) -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Brazilian Portuguese version of the PostgreSQL Advocacy and Marketing site is ready
Hi everyone, Thanks to Diogo Biazus [EMAIL PROTECTED], the Brazilian Portuguese translation of the PostgreSQL Advocacy and Marketing site is now completed and ready for public use: http://advocacy.postgresql.org/?lang=br :-) Wow, that's 6 languages already, and more are coming along. Am very, very proud of our community members. :-) Regards and best wishes, Justin Clift -- My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there. - Indira Gandhi ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Security question : Database access control
Is there any way to prevent superuser to acces the database ? I mean something like "GRANT / REVOKE CONNECT" MECHANISM I have no idea how to prevent root from access data in one of this ways : root @ linux:~#su - postgres postgres @ linux:/usr/local/pgsql/bin$pg_dump or edit pg_hba.conf # Allow any user on the local system to connect to any # database under any username, but only via an IP connection: host all 127.0.0.1 255.255.255.255 trust # The same, over Unix-socket connections: local all trustor my nightmare a cygwin on Win 98 everybody can can access everything :-
Re: [HACKERS] [ADMIN] Security question : Database access control
On Tue, Oct 22, 2002 at 17:05:38 +0200, Igor Georgiev [EMAIL PROTECTED] wrote: Is there any way to prevent superuser to acces the database ? I mean something like GRANT / REVOKE CONNECT MECHANISM I have no idea how to prevent root from access data in one of this ways : root @ linux:~#su - postgres postgres @ linux:/usr/local/pgsql/bin$pg_dump or edit pg_hba.conf # Allow any user on the local system to connect to any # database under any username, but only via an IP connection: host all 127.0.0.1 255.255.255.255trust # The same, over Unix-socket connections: localall trust or my nightmare a cygwin on Win 98 everybody can can access everything :- They can just read the raw database files as well. You have to be able to trust whoever has root access to the system, as well as anyone who has physical access to the system. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Security question : Database access control
edit *pg_hba.conf * # Allow any user on the local system to connect to any # database under any username, but only via an IP connection: host all 127.0.0.1 255.255.255.255 trust # The same, over Unix-socket connections: local all trust what about reading pg_hba.conf comments? local all md5 Ok, but my question actually isn't about pg_hba.conf comments, i read enough but what will stop root from adding this lines or doing su - postgres ??
Re: [HACKERS] Security question : Database access control
On Tue, 22 Oct 2002, Igor Georgiev wrote: edit *pg_hba.conf * # Allow any user on the local system to connect to any # database under any username, but only via an IP connection: host all 127.0.0.1 255.255.255.255trust # The same, over Unix-socket connections: localall trust what about reading pg_hba.conf comments? localall md5 Ok, but my question actually isn't about pg_hba.conf comments, i read enough but what will stop root from adding this lines or doing su - postgres ?? Nothing, root is GOD in unix. He can do whatever he wants. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] [ADMIN] Security question : Database access control
On Tue, 22 Oct 2002, Igor Georgiev wrote: edit *pg_hba.conf * # Allow any user on the local system to connect to any # database under any username, but only via an IP connection: host all 127.0.0.1 255.255.255.255trust # The same, over Unix-socket connections: localall trust what about reading pg_hba.conf comments? localall md5 Ok, but my question actually isn't about pg_hba.conf comments, i read enough but what will stop root from adding this lines or doing su - postgres ?? Not much really. But given that they have access to the raw data files, preventing them access to the server doesn't gain you that much if they really want to get the data. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] [ADMIN] Security question : Database access control
They can just read the raw database files as well. wow I'm not sure how about this edit pg_hba.conf # Allow any user on the local system to connect to any # database under any username local all trust su - posgres psql test -U dba or pg_dump test You have to be able to trust whoever has root access to the system, as well as anyone who has physical access to the system.
Re: [HACKERS] [ADMIN] Security question : Database access control
Igor Georgiev [EMAIL PROTECTED] writes: Ok, but my question actually isn't about pg_hba.conf comments, i read enough but what will stop root from adding this lines or doing su - postgres ?? As somebody already pointed out, you *must* trust the people with root access to your machine; there is not anything you can do to defend yourself against them. If you can't trust the admins on the machine you're using, better get your own machine. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Security question : Database access control
Next your going to ask what will stop root from stopping your PostgreSQL, compiling a second copy with authentication disabled and using your data directory as it's source :) He he i'm enough paranoic :)) If you want to prevent root from accomplishing these things, you're going to have to look to your kernel for help. The kernel must prevent root from changing users, starting / stopping applications, or touching certain filesystems. PostgreSQL will let you put a password on the data. But that only works if they actually try to use PostgreSQL to get at the data. use PostgreSQL to get at the data -Yeah this will be enough i want just only REVOKE CONNECT PRIVILEGES on database
