Re: [HACKERS] ECPG and bison
going to be dealing with this this weekend ... Bruce already asked me as well, just been busy during the week, so had to defer to the weekend ... if its not in place by tomorrow around, so, 8pm GMT,please feel free to email-nag me :) On Fri, 18 Oct 2002, Michael Meskes wrote: Could anyone please tell me if we found a solution? It seems I missed any discussion or whatsoever. The only things I know is that postgresql.org still has bison 1.35 installed and 7.3 release is nearing. Michael -- Michael Meskes [EMAIL PROTECTED] Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL! ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: Antw: Re: [HACKERS] Postgresql and multithreading
On 18 Oct 2002 at 18:10, Ulrich Neumann wrote: Marc, not only the global variables are a problem. PostgreSQL doesn't clean up all the memory before a process terminates and you must deal with signals between threads. OK, first of all let me say this. I am interested in seeing postgresql mutlithreaded but I don't think it's a good idea to provide a thread per connection. On slick unices like freebsd/linux, threads and processes are almost equally heavy. Besides if connection opening time for a database is an issue, application should cache the pool of connections. Databases are not web servers which should cater to thousands of connections per minute. I am interested in seeing postgresql multithreaded so that it performs better on SMP machine. Right now I am building/designing a *huge* database which is going to use at least 4 way machines. With number of connections very small, around 10-15 and being persistent, I am afraid that postgresql would not use all the available horsepower. I've modified PostgreSQL that it is completely thread based and nearly clean with allocating/deallocating memory and I'm using signals between threads. I am sure it must have been massive task. I'm still VERY interested in giving those changes back to the community, but i think that my changes are still not wanted. Well, I understand your feeling but I am sure it's not the cases. People here have stated that there are other optimisation areas which needs to be addresses first. Threading is not ruled out but it's not on current radar either. I would be more than happy to assist you to make this happen. I have some experience in threads(pthreads on linux) but absolute no experience in postgresql code. Can we make a patch that converts a single module/operation to multithreaded? Let's say sequential table scan. I propose that such a conversion should be done under conditional compilation with default threaded behaviour as off. This would not disturb existing setup and behaviour where threads are not good enough. Besides since we are distributing almost exclusive work to threads, there should be little syncronisation issues. I request postgresql hackers to look up to this proposal favourably. This would be a good and gradual way to get threading in postgresql. Awaiting your feedback.. Bye Shridhar -- The Consultant's Curse: When the customer has beaten upon you long enough, give him what he asks for, instead of what he needs. This is very strong medicine, and is normally only required once. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Design decision curiosity
Dann Corbit [EMAIL PROTECTED] writes: I am curious as to why it is necessary to add PL/pgSQL as a language instead of including it by default. I believe security was one of the reasons. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Design decision curiosity
I am curious as to why it is necessary to add PL/pgSQL as a language instead of including it by default. Is it to save space on small systems or??? Since the code necessary PL/pgSQL is very small, it does not seem like there would be much savings. I imagine that there is occasionally some confusion with manuals describing the use of PL/pgSQL and yet the language does not exist until installation. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] /contrib/retep to gborg
Bruce Momjian [EMAIL PROTECTED] writes: I am concerned that removing them may cause instability, particularly in some of the java build scripts. That's why I am suggesting not doing it during beta. Actually, it looks to me like they belong with the JDBC driver. As long as JDBC is in the main distro, I think we should leave contrib/retep there too. There was some talk of moving JDBC to gborg, but I'm not sure what the plan is, if any. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Freeing plan memory
I notice there's a leak of memory in SPI_prepare().
The full fix is nontrival and I don't want to submit a half solution so I
thought I'd check whether people think it's worth worrying about.
The leak is that memory is grabbed in SPI_prepare() for a plan within whatever
context is current when it does the palloc(). It may be the caller's or it may
be the relevent SPI one. The plan is then copied out of this memory [and
context] into a child of the procedure's context and forgotten about, or just
plain forgotten. Obviously the intention is that this memory is freed when the
context is deleted and is probably not a problem unless someone does something
like:
i = 10;
while (i--)
{
plan = SPI_prepare(SELECT 1, 0, (Oid *)NULL);
SPI_freeplan(plan); /* SPI_freeplan() is not just for SPI_saveplan() */
}
Is this worth worrying about?
Any busy person can stop reading now as the above defines the problem while the
below only shows an easily reproducable example.
FWIW, I found it while testing something like, which is a little less daft
than the above example:
create function atest1 ( ) returns int as '
a = 0
while a 1:
plan = plpy.prepare(SELECT + repr(a))
a = a + 1
' language 'plpython';
Here the plpython code uses SPI_freeplan to release the context holding the
plan memory when each plan object returned by plpy.prepare() is garbage
collected. This seems sensibly to happen when the plan variable is
reassigned. However I was baffled why the process still had an obvious memory
leak so looked a little closer at SPI.
--
Nigel J. Andrews
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Freeing plan memory
On Sat, 19 Oct 2002, Tom Lane wrote: Nigel J. Andrews [EMAIL PROTECTED] writes: The leak is that memory is grabbed in SPI_prepare() for a plan within whatever context is current when it does the palloc(). It may be the caller's or it may be the relevent SPI one. The plan is then copied out of this memory [and context] into a child of the procedure's context and forgotten about, or just plain forgotten. Au contraire: SPI_prepare builds the plan in its execCxt, which is reset before returning (look at _SPI_begin_call and _SPI_end_call). So I see no leak there. Ah, yes, I see that now. I'm not sure where the leak is in your plpython example, but I'd be inclined to look to plpython itself, perhaps even just the string concatenation expression in plan = plpy.prepare(SELECT + repr(a)) Well it's not that string operation. plpgsql used to have terrible intra-function memory leaks, and only by dint of much hard work has it been brought to the point where you can expect a long loop in a plpgsql function not to chew up memory. AFAIK, no one has yet done similar work for the other PL languages. Hmmm...my test case should boil down to a fairly small number of other calls in the SPI_prepare wrapper and a quick looks doesn't show anything interesting. Not sure I've got the time to dedicate to investigating this but I'll look at it as and when I can. I'm sending a patch for plpython.c to -patches which fixes a mistake I made in the previous patch. -- Nigel J. Andrews ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Freeing plan memory
Nigel J. Andrews [EMAIL PROTECTED] writes: On Sat, 19 Oct 2002, Tom Lane wrote: I'm not sure where the leak is in your plpython example, but I'd be inclined to look to plpython itself, perhaps even just the string concatenation expression in plan = plpy.prepare(SELECT + repr(a)) Well it's not that string operation. Actually, I'll bet it's this code in PLy_spi_prepare: plan-plan = SPI_prepare(query, plan-nargs, plan-types); // error check plan-plan = SPI_saveplan(plan-plan); // error check The copy of the plan that's returned by SPI_prepare is being blithely lost --- and since it's in the procCxt, it won't go away until the plpython function is exited. Need a SPI_freeplan() here, I think. Can you check it out and send a patch? regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] autocommit vs TRUNCATE et al
On Fri, 18 Oct 2002, Mike Mascari wrote: Gavin Sherry wrote: On Fri, 18 Oct 2002, Tom Lane wrote: Anyone see a way out of this catch-22? If not, which is the least bad alternative? Ultimately, fix TRUNCATE to be transaction safe. This is non-trivial, I know :-). Regardless, the first option seems the less of the two evils. Even though TRUNCATE was modeled after Oracle's TRUNCATE and Oracle's TRUNCATE commits the running tx, truncates the relation, and starts a new tx, regardless of whether or not TRUNCATE is the first statement of the tx? Why should we be *only* as good as Oracle? :-) Gavin ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] ECPG and bison
On Fri, 2002-10-18 at 22:03, Marc G. Fournier wrote: On Fri, 18 Oct 2002, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Larry Rosenman wrote: Looks like threre is a 1.75 out (based on a FreeBSD PR I just saw fly by). I can confirm that: http://ftp.gnu.org/gnu/bison/ Attached is the Changelog since 1.50. Urgh. Nothing like major releases 2 weeks apart to give one a nice warm comfy feeling about the stability of one's tools. Offhand it looks like these are nearly all bug fixes, and so we'd better standardize on 1.75 not 1.50. But I wonder what will be out next week... that's one of hte reasons I've been holding off (the other being a busy week) ... I *try* to keep stuff like this to what the FreeBSD ports collection considers a stable release, and it hasn't been updated yet :( The PR I referred to above is the PR to update to 1.75. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: [EMAIL PROTECTED] US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] ECPG and bison
On Fri, 18 Oct 2002, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Larry Rosenman wrote: Looks like threre is a 1.75 out (based on a FreeBSD PR I just saw fly by). I can confirm that: http://ftp.gnu.org/gnu/bison/ Attached is the Changelog since 1.50. Urgh. Nothing like major releases 2 weeks apart to give one a nice warm comfy feeling about the stability of one's tools. Offhand it looks like these are nearly all bug fixes, and so we'd better standardize on 1.75 not 1.50. But I wonder what will be out next week... that's one of hte reasons I've been holding off (the other being a busy week) ... I *try* to keep stuff like this to what the FreeBSD ports collection considers a stable release, and it hasn't been updated yet :( ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Postgresql and multithreading
On Fri, Oct 18, 2002 at 10:28:38AM -0400, Tom Lane wrote: Greg Copeland [EMAIL PROTECTED] writes: On Thu, 2002-10-17 at 22:20, Tom Lane wrote: Simple: respond to 'em all with a one-line answer: convince us why we should use it. The burden of proof always seems to fall on the wrong end in these discussions. ... Now, it seems, that people don't want to answer questions at all as it's bothering the developers. Not at all. But rehashing issues that have been talked out repeatedly is starting to bug some of us ;-). Perhaps the correct standard answer is more like this has been discussed before, please read the list archives. Let me explain my posting which started this `thread': - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use threads (and many times it has been discussed on the list). - The TODO list has an item `Experiment with multi-threaded backend' and points to a mailing list discussion about the implementation by Myron Scott. His final comment is that he didn't `gain much performance' and `ended up with some pretty unmanagable code'. He also says that he wouldn't `personally try this again ... but there probably was a better way'. - I was going through the TODO list, and was wondering if I should try on this. But before doing that, naturally, I wanted to figure out if any of the core developers themselves have any plans of doing it. Now, I am trying hard to figure out why this `are you going to do this? otherwise I can try it', type posting was not differentiated from numerous `why don't YOU implement this feature' type postings ;) Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) Life is too important to take seriously. -- Corky Siegel ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] pg_dump and large files - is this a problem?
At 12:07 AM 19/10/2002 +0200, Peter Eisentraut wrote: Any old machine has a 4-byte off_t if you configure with --disable-largefile. Thanks - done. I just dumped to a custom backup file, then dumped it do SQL, and compared each version (V7.2.1, 8 byte 4 byte offsets), and they all looked OK. Also, the 4 byte version reads the 8 byte offset version correctly - although I have not checked reading 4GB files with 4 byte offset, but it's not a priority for obvious reasons. So once Giles gets back to me (Monday), I'll commit the changes. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /() __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] pg_dump and large files - is this a problem?
I have put the latest patch at: http://downloads.rhyme.com.au/postgresql/pg_dump/ along with two dump files of the regression DB, one with 4 byte and the other with 8 byte offsets. I can read/restore each from the other, so it looks pretty good. Once the endianness is tested, we should be OK. Known problems: - will not cope with 4GB files and size_t not 64 bit. - when printing data position, it is assumed that off_t is UINT64 (we could remove this entirely - it's just for display) - if seek is not supported, then an intXX is assigned to off_t when file offsets are needed. This *should* not cause a problem since without seek, the offsets will not be written to the file. Changes from Prior Version: - No longer stores or outputs data length - Assumes result of ftello is correct if it disagrees with internally kept tally. - 'pg_restore -l' now shows sizes of int and offset. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /() __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] /contrib/retep to gborg
Bruce Momjian [EMAIL PROTECTED] writes: I think we should move /contrib/retep to gborg. It is 400k and isn't anywhere near our core functionality. I will add to TODO: Move /contrib/retep to gborg Can I move this during beta? I don't think so. Why not? Feature addition during beta is frowned on, but I never heard anyone say you couldn't remove features during beta. (In fact that's a pretty standard response to serious bugs, no?) Whether retep *should* be moved is not something I feel qualified to comment on, but I don't put any stock in an argument that it can't be done simply because we're in beta. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Current CVS has strange parser for float type
Peter Eisentraut wrote: Teodor Sigaev writes: wow=# select 5.3::float; ERROR: Bad float8 input format '5.3' Does it accept '5,4'::float? Yes, it accepted '5,4'::float format. -- Teodor Sigaev [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Open items
Bruce Momjian [EMAIL PROTECTED] writes: Michael, please use bison 1.75 to update ecpg Why is this necessary? AFAIK we don't keep bison-derived files in CVS... Cheers, Neil -- Neil Conway [EMAIL PROTECTED] || PGP Key ID: DB3C29FC ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Postgresql and multithreading
Anuradha Ratnaweera [EMAIL PROTECTED] writes: Let me explain my posting which started this `thread': - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use threads (and many times it has been discussed on the list). - The TODO list has an item `Experiment with multi-threaded backend' and points to a mailing list discussion about the implementation by Myron Scott. His final comment is that he didn't `gain much performance' and `ended up with some pretty unmanagable code'. He also says that he wouldn't `personally try this again ... but there probably was a better way'. - I was going through the TODO list, and was wondering if I should try on this. But before doing that, naturally, I wanted to figure out if any of the core developers themselves have any plans of doing it. Now, I am trying hard to figure out why this `are you going to do this? otherwise I can try it', type posting was not differentiated from numerous `why don't YOU implement this feature' type postings ;) Well, if you'd actually said the above, we'd probably have replied to the effect of we still think it's an unpromising project, but try it if you like. By my reading, your earliest postings in this thread showed no sign of any familiarity at all with the history: http://archives.postgresql.org/pgsql-hackers/2002-10/msg00704.php http://archives.postgresql.org/pgsql-hackers/2002-10/msg00707.php http://archives.postgresql.org/pgsql-hackers/2002-10/msg00711.php and so you got the sort of response that's usually given to clueless newbies... regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Open items
Neil Conway wrote: Bruce Momjian [EMAIL PROTECTED] writes: Michael, please use bison 1.75 to update ecpg Why is this necessary? AFAIK we don't keep bison-derived files in CVS... Right, they aren't in CVS, but they are shipped in the tarball so we don't require everyone to have bison. Developers and people building from CVS do need bison, and will need 1.75 for ecpg. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] /contrib/retep to gborg
-Original Message- From: Tom Lane [mailto:tgl;sss.pgh.pa.us] Sent: 19 October 2002 17:17 To: Bruce Momjian Cc: PostgreSQL-development Subject: Re: [HACKERS] /contrib/retep to gborg Bruce Momjian [EMAIL PROTECTED] writes: I am concerned that removing them may cause instability, particularly in some of the java build scripts. That's why I am suggesting not doing it during beta. Actually, it looks to me like they belong with the JDBC driver. As long as JDBC is in the main distro, I think we should leave contrib/retep there too. There was some talk of moving JDBC to gborg, but I'm not sure what the plan is, if any. Marc asked me to sort it along with some other things, but when I spoke to Barry he didn't want to move. I passed it back to Marc. Regards, Dave. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
