Re: [HACKERS] Remove xmin and cmin from frozen tuples
On Fri, 2 Sep 2005 20:41:48 -0400 (EDT), Bruce Momjian pgman@candle.pha.pa.us wrote: Once I had a patch based on 7.4 that stored cmin and cmax in backend-local memory. Interesting idea, but how would you record the cmin/xmin values without requiring unlimited memory? That's exactly the reason for not sending it to -patches. Without spilling to disk this is just not ready for real life. The problem is that -- unlike other data structures that build up during a transaction, e.g. trigger queues -- cmin/cmax lookup requires random access, so we'd need some form of tree or hash. Unfornunately I never got beyond brainstorming :-( BTW, is there anything else that'd need spilling to disk during long transactions? Servus Manfred ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] Adding a new node to the executor
Hey guys, I am taking my chances of posting a devel doubt on this list (couldn't find any other relevant list). I think I have spent too much time on this and need a fresh pair of eyes to look in it.. So this is my situation: 1. I have added a new field to RangeTblEntry (bool tablesample). 2. I have created a new executor node (this is to do system sampling [Ref: Query sampling thread]). In short, this new node goes over a table block by block, skipping some blocks as required. 3. Have made the appropriate changes to the parser+planner so that the information about a tablesample is passed from the parser to the executor. But the RangeTblEntry received by my executor node does not have the tablesample set, inspite of the fact that I do set it in parse_relation.c:addRangeTableEntry(). I have made the appropriate changes to copyfuncs.c, but still in vain.. Any help would be highly appreciated. I can send a diff if someone is interested. Thanks Varun ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Remove xmin and cmin from frozen tuples
On Sat, Sep 03, 2005 at 10:59:31AM +0200, Manfred Koizar wrote: On Fri, 2 Sep 2005 20:41:48 -0400 (EDT), Bruce Momjian pgman@candle.pha.pa.us wrote: Once I had a patch based on 7.4 that stored cmin and cmax in backend-local memory. Interesting idea, but how would you record the cmin/xmin values without requiring unlimited memory? That's exactly the reason for not sending it to -patches. Without spilling to disk this is just not ready for real life. The problem is that -- unlike other data structures that build up during a transaction, e.g. trigger queues -- cmin/cmax lookup requires random access, so we'd need some form of tree or hash. Unfornunately I never got beyond brainstorming :-( BTW, is there anything else that'd need spilling to disk during long transactions? Yes, the queue of pending deferred triggers. -- Alvaro Herrera -- Valdivia, Chile Architect, www.EnterpriseDB.com El hombre nunca sabe de lo que es capaz hasta que lo intenta (C. Dickens) ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Adding a new node to the executor
On Sat, Sep 03, 2005 at 02:43:20AM -0700, Varun Kacholia wrote: But the RangeTblEntry received by my executor node does not have the tablesample set, inspite of the fact that I do set it in parse_relation.c:addRangeTableEntry(). I have made the appropriate changes to copyfuncs.c, but still in vain.. Did you change all the other files in src/backend/nodes? You need to change not only copyfuncs.c, but equalfuncs, outfuncs and readfuncs as well. -- Alvaro Herrera -- Valdivia, Chile Architect, www.EnterpriseDB.com El miedo atento y previsor es la madre de la seguridad (E. Burke) ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Adding a new node to the executor
Alvaro Herrera wrote: On Sat, Sep 03, 2005 at 02:43:20AM -0700, Varun Kacholia wrote: But the RangeTblEntry received by my executor node does not have the tablesample set, inspite of the fact that I do set it in parse_relation.c:addRangeTableEntry(). I have made the appropriate changes to copyfuncs.c, but still in vain.. Did you change all the other files in src/backend/nodes? You need to change not only copyfuncs.c, but equalfuncs, outfuncs and readfuncs as well. Yep. My suggestion is to pick an existing field in RangeTblEntry and find all occurances of that, and determine if your new field has to be added in that place. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Remove xmin and cmin from frozen tuples
Manfred Koizar wrote: On Fri, 2 Sep 2005 20:41:48 -0400 (EDT), Bruce Momjian pgman@candle.pha.pa.us wrote: Once I had a patch based on 7.4 that stored cmin and cmax in backend-local memory. Interesting idea, but how would you record the cmin/xmin values without requiring unlimited memory? That's exactly the reason for not sending it to -patches. Without spilling to disk this is just not ready for real life. The problem is that -- unlike other data structures that build up during a transaction, e.g. trigger queues -- cmin/cmax lookup requires random access, so we'd need some form of tree or hash. Unfornunately I never got beyond brainstorming :-( What we do for shared row locks is much more compact because we create a phantom xid for combination of xids and store that in a hash. I think the problem with having cmin/cmax in local memory is that without a way to stamp a row with some fake cid (as is suggested in the TODO item now) there is really no way to _group_ rows together to store their values as a single item in local memory, meaning we would have to store ctids or something for each row, and that seems unscalable. I have added your memory-only idea to the TODO item because it highlights that cmin/cmax is never looked at outside the backend that is modifying the row. If there was a tuple header field we can use just while the row is being created or expired we could use that, but I don't see one. Could we recalculated one of the fields after it is created/expired? I don't know. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] core dump with last CVS
Hi there,
following query cause postmaster to fail.
tp=# select query from rquery('newyorkhotel') as query;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
this happens *only* if rquery marks as immutable.
in postgresql console I see
TRAP: BadArgument(!(((context) != ((void *)0) (Node*)((context)))-type) ==
T_AllocSetContext, File: mcxt.c, Line: 502)
(gdb) bt
#0 0x40101ac1 in kill () from /lib/libc.so.6
#1 0x401016eb in raise () from /lib/libc.so.6
#2 0x40103127 in abort () from /lib/libc.so.6
#3 0x08251f02 in ExceptionalCondition (
conditionName=0x6 Address 0x6 out of bounds,
errorType=0x8278b22 BadArgument, fileName=0x4020ad24 8,\023,
lineNumber=0) at assert.c:51
#4 0x082691db in MemoryContextAlloc (context=0x8278b22, size=24) at mcxt.c:502
#5 0x40018191 in QTNCopy () from /u/megera/app/pgsql/cvs/run/lib/tprewriter.so
#6 0x40018e90 in tp_rewrite_accum ()
from /u/megera/app/pgsql/cvs/run/lib/tprewriter.so
#7 0x08151962 in advance_transition_function (aggstate=0x83e5b3c,
peraggstate=0x83ebab8, pergroupstate=0x83ec7d8, newVal=138568552,
isNull=36 '$') at nodeAgg.c:385
#8 0x08151aef in advance_aggregates (aggstate=0x83e5b3c, pergroup=0x83ec7d8)
at nodeAgg.c:447
#9 0x081521c2 in agg_retrieve_direct (aggstate=0x83e5b3c) at nodeAgg.c:785
#10 0x0814844b in ExecProcNode (node=0x83e5b3c) at execProcnode.c:377
#11 0x081469bc in ExecutePlan (estate=0x83e58a4, planstate=0x83e5b3c,
operation=CMD_SELECT, numberTuples=1, direction=NoMovementScanDirection,
dest=0x8321144) at execMain.c:1107
#12 0x08145b3b in ExecutorRun (queryDesc=0x83da174,
direction=ForwardScanDirection, count=0) at execMain.c:231
#13 0x0815e8bc in _SPI_pquery (queryDesc=0x83da174, tcount=0) at spi.c:1536
#14 0x0815e631 in _SPI_execute_plan (plan=0x83dc1c4, Values=0x83d1134,
Nulls=0x83d1148 ~, snapshot=0x0, crosscheck_snapshot=0x0,
read_only=1 '\001', tcount=1) at spi.c:1464
#15 0x0815c969 in SPI_execute_plan (plan=0x83dc1c4, Values=0x83d1134,
Nulls=0x0, read_only=1 '\001', tcount=1) at spi.c:335
#16 0x4c852195 in exec_run_select (estate=0xbfffe400, expr=0x83d38dc,
maxtuples=0, portalP=0x0) at pl_exec.c:3716
#17 0x4c84e5ae in exec_stmt_select (estate=0xbfffe400, stmt=0x83d3928)
at pl_exec.c:1654
#18 0x4c84d98f in exec_stmt (estate=0xbfffe400, stmt=0x83d3928)
at pl_exec.c:1045
#19 0x4c84d86e in exec_stmts (estate=0x, stmts=0x4020ad24)
at pl_exec.c:981
#20 0x4c84d73b in exec_stmt_block (estate=0xbfffe400, block=0x83d39ec)
at pl_exec.c:937
#21 0x4c84c68a in plpgsql_exec_function (func=0x83d2904, fcinfo=0xbfffe4e0)
at pl_exec.c:286
#22 0x4c848a15 in plpgsql_call_handler (fcinfo=0xbfffe4e0) at pl_handler.c:123
#23 0x08149c07 in ExecMakeFunctionResult (fcache=0x83c8720,
econtext=0x83c8b78, isNull=0xbfffe749 \001S, isDone=0x0)
at execQual.c:1031
#24 0x0814c375 in ExecEvalExprSwitchContext (expression=0x6,
econtext=0x4020ad24, isNull=0x0, isDone=0x0) at execQual.c:2808
#25 0x0819c275 in evaluate_expr (expr=0x0, result_type=16977) at clauses.c:2654
#26 0x0819b948 in simplify_function (funcid=5439893, result_type=16977,
args=0x83bf004, allow_inline=1 '\001', context=0xbfffe7f0)
at clauses.c:2162
#27 0x0819b4b4 in eval_const_expressions_mutator (node=0x530195,
context=0xbfffe7f0) at clauses.c:1308
#28 0x0819aa2a in eval_const_expressions (node=0x0) at clauses.c:1214
#29 0x081916f7 in preprocess_expression (root=0x83bf004, expr=0x4020ad24,
kind=1283697108) at planner.c:443
#30 0x0819165f in subquery_planner (parse=0x83be7a0, tuple_fraction=0,
subquery_pathkeys=0x0) at planner.c:290
#31 0x0819124c in planner (parse=0x83be7a0, isCursor=0 '\0', cursorOptions=0,
boundParams=0x0) at planner.c:135
#32 0x081d574e in pg_plan_query (querytree=0x83be7a0, boundParams=0x0)
at postgres.c:725
#33 0x081d580d in pg_plan_queries (querytrees=0x4020ad24, boundParams=0x0,
needSnapshot=0 '\0') at postgres.c:793
#34 0x081d5a3f in exec_simple_query (
query_string=0x83be1e4 select query from rquery('newyorkhotel') as
query;)
at postgres.c:958
#35 0x081d8778 in PostgresMain (argc=4, argv=0x8364388,
username=0x8364360 megera) at postgres.c:3160
#36 0x081acc42 in BackendRun (port=0x837d350) at postmaster.c:2864
#37 0x081ac630 in BackendStartup (port=0x837d350) at postmaster.c:2505
#38 0x081aa8d0 in ServerLoop () at postmaster.c:1232
#39 0x081a9c19 in PostmasterMain (argc=1, argv=0x8363578) at postmaster.c:941
#40 0x0816715e in main (argc=1, argv=0x8363578) at main.c:268
#41 0x400edd06 in __libc_start_main () from /lib/libc.so.6
create or replace function rquery(text) returns tsquery as
$$
DECLARE
inquery ALIAS FOR $1;
outquery tsquery;
BEGIN
select into outquery tp_rewrite(ARRAY[query, place.name_tsquery])
from
Re: [HACKERS] core dump with last CVS
Oleg Bartunov oleg@sai.msu.su writes: following query cause postmaster to fail. You haven't shown us your own code, but I'd guess that you are doing something you shouldn't with memory contexts. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
On Fri, Sep 02, 2005 at 11:42:21AM -0400, Tom Lane wrote: The objection is fundamentally that a platform-specific implementation cannot be our long-term goal, and so expending effort on creating one seems like a diversion. If there were a plan put forward showing how this is just a useful way-station, and we could see how we'd later get rid of the glibc dependency without throwing away the work already done, then it would be a different story. Well, my patch showed that useful locale work can be acheived with precisely two functions: newlocale and strxfrm_l. I'm going to talk about two things: one, the code from Apple. Two, how we present locale support to users. --- Now, it would be really nice to take Apple's implementation in Darwin and use that. What I don't understand is the licence of the code in Darwin. My interpretation is that stuff in: http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/ is Apple stuff under APSL, useless to us. And that stuff in: http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/FreeBSD/ are just patches to FreeBSD and this under the normal BSD license (no big header claiming the licence change). The good news is that the majority of what we need is in patch form. The bad news is that the hub of the good stuff (newlocale, duplocale, freelocale) is under a big fat APSL licence. Does anyone know if this code can be used at all by BSD projects or did they blanket relicence everything? --- Now, I want to bring up some points relating to including a locale library in PostgreSQL. Given that none of the BSDs seem really interested in fixing the issue we'll have to do it ourselves (I don't see anyone else doing it). We can save ourselves effort by basing it on FreeBSDs locale code, because then we can use their datafiles, which we *definitly* don't want to maintain ourselves. Now: 1. FreeBSDs locale list is short, some 48 compared with glibc's 217. Hopefully Apple can expand on that in a way we can use. But given the difference we should probably give people a way of falling back to the system libraries in case there's a locale we don't support. On the other hand, lots of locales are similar so maybe people can find ones close enough to work. No, glibc and FreeBSD use different file formats, so you can't copy them. Do we want this locale data just for collation, or do we want to be able to use it for formatting monetary amounts too? This is even more info to store. Lots of languages use ISO/IEC 14651 for order. 2. Locale data needs to be combined with a charset and compiled to work with the library. PostgreSQL supports at least 15 charsets but we don't want to ship compiled versions of all of these (Debian learnt that the hard way). So, how do we generate the files people need. a. Auto-compile on demand. First time a locale is referenced spawn the compiler to create the locale, then continue. (Ugh) b. Add a CREATE LOCALE english AS 'en_US' WITH CHARSET 'utf8'. Then require the COLLATE clause to refer to this identifier. This has some appeal, seperating the system names from the PostgreSQL names. It also gives some info regarding charsets. c. Should users be allowed to define new locales? d. Should admins be required to create the external files using a program, say pg_createlocale. Remember, if you use a latin1 locale to sort utf8 you'll get the wrong result, so we want to avoid that. 3. Compiled locale files are large. One UTF-8 locale datafile can exceed a megabyte. Do we want the option of disabling it for small systems? 4. Do we want the option of running system locale in parallel with the internal ones? 5. I think we're going to have to deal with the very real possibility that our locale database will not be as good as some of the system provided ones. The question is how. This is quite unlike timezones which are quite standardized and rarely change. That database is quite well maintained. Would people object to a configure option that selected: --with-locales=internal (use pg database) --with-locales=system (use system database for win32, glibc or MacOS X) --with-locales=none (what we support now, which is neither) I don't think it will be much of an issue to support this, all the functions take the same parameters and have almost the same names. 6. Locales for SQL_ASCII. Seems to me you have two options, either reject COLLATE altogether unless they specify a charset, or don't care and let the user shoot themselves in the foot if they wish... BTW, this MacOS locale supports seems to be new for 10.4.2 according to the CVS log info, can anyone confirm this? Anyway, I hope this post didn't bore too much. Locale support has been one of those things that has bugged me for a long time and it would be nice if there could be some real movement. Have a nice weekend, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
Re: [HACKERS] Proof of concept COLLATE support with patch
Tom Lane [EMAIL PROTECTED] writes: Greg Stark [EMAIL PROTECTED] writes: I still doesn't get where the hostility towards this functionality comes from. We're not really willing to say here is a piece of syntax REQUIRED BY THE SQL SPEC which we only support on some platforms. readline, O_DIRECT, and the like are a completely inappropriate analogy, because those are inherently platform-dependent (and not in the spec). But that's not the case at all. The syntax can be supported everywhere it would just be somewhat faster on some platforms than others. It's already reasonably fast on any platform that caches locale information which includes glibc and presumably other free software libcs. It would be slightly faster if there are _l functions. And much slower if the libc setlocale implementation is braindead. But there's nothing wrong with saying it's slow because your libc is slow. Compile with this freely available library which has a better implementation. The programming syntax would still be exactly 100% the same. The objection is fundamentally that a platform-specific implementation cannot be our long-term goal, and so expending effort on creating one seems like a diversion. If there were a plan put forward showing how this is just a useful way-station, and we could see how we'd later get rid of the glibc dependency without throwing away the work already done, then it would be a different story. It's not like the actual calls to setlocale are going to be much code. One day presumably some variant of these _l functions will become entirely standard. In which case you're talking about potentially throwing away 50 lines of code. The bulk of the code is going to be parsing and implementing the actual syntax and behaviour of the SQL spec. And in any case I wouldn't expect it to ever get thrown away. There will be people compiling on RH9 or similar vintage systems for a long time. -- greg ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
Martijn van Oosterhout kleptog@svana.org writes: 2. Locale data needs to be combined with a charset and compiled to work with the library. PostgreSQL supports at least 15 charsets but we don't want to ship compiled versions of all of these (Debian learnt that the hard way). So, how do we generate the files people need. That's just one of many lessons learned the hard way by distributions. Nor will it be the last innovation in this area. I really find this instinct of wanting to reimplement large swaths of the OS inside Postgres (and annoying detail-ridden swaths that are hard to get right and continually evolving too) to be a bad idea. I can't believe it's harder to maintain an #ifdef HAVE_STRCOL_L #else #endif than it is to try to maintain an entire independent locale library. Nor is it simpler for sysadmins to have to maintain an entirely separate set of locales independently from the system locales. If you really are unhappy enough with OS setlocale implementations to want to try to do this then it would be more helpful to do it outside of Postgres. Package up the Apple setlocale library as a separate package that anyone can install on Solaris, BSD, Linux or whatever. Then Postgres can just say it works fine with your OS library but your OS library might be very slow. Here's a third-party library that you can install that is fast and may relieve any problems you have with collation performance. But I think that's getting ahead of things. Until Postgres even supports collations using the OS libraries you won't even know if that's even necessary. -- greg ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] OT: SCALE 4x -- Call For Papers
Hello, The call for papers for SCALE 4x, the 2006 Southern California Linux Expo, is now open. This event will be our fourth annual show. It will be held on Feb 11-12, 2006 at the Los Angeles Airport Westin. We are expecting 1,300+ in attendance this year. We are non-profit, community run Linux, open-source and free software conference. If you are working on something you believe the community would be interested in, please consider submitting a presentation to our call for papers. I am including details bellow. Past presentations are available online (including slides audio in most cases): 2005 - http://www.socallinuxexpo.org/past/2005/hours.php 2003 - http://www.socallinuxexpo.org/past/2003/presentations.php 2002 - http://www.socallinuxexpo.org/past/2002/presentations.php If you have any questions please feel free to contact the Call For Papers team at cfp =at= socallinuxexp.org CFP Link: http://www.socallinuxexpo.org/pr/pr_20050620.php CFP PDF: http://www.socallinuxexpo.org/pr/cfp4x.pdf Best regards, Ilan Rabinovitch Conference Chair Southern California Linux Expo http://www.socallinuxexpo.org 2006 Southern CAlifornia Linux Expo The USC, Simi/Conejo, and UCLA Linux User Groups are proud to announce the 4th annual Southern California Linux Expo scheduled for February 11-12, 2006 at the Westin Hotel near the Los Angeles International Airport. Building on the tremendous success of last three years' SCALE, we will continue to promote Linux and the Open Source Software community. We invite you to share your work on Linux and Open Source projects with the rest of the community as well as exchange ideas with some of the leading experts in these field. Details about SCALE 4X as well as archives for the last three years can be found at http://www.socallinuxexpo.com. Topics of interest include, but are not limited to: * Linux kernel * Linux Networking * Linux for embedded systems * Linux for Desktops * LAMP * Multimedia in Linux * Security in Linux * VoIP * Wireless tools in Linux * Linux Games * GIMP other graphics software * Administration techniques for specific distributions * Custom Configurations * Linux Deployments and experiences: Case studies * Open source Licensing * Government policies with Open Source * Other open source projects The proposals should comprise a 1-page (maximum) description containing the following: 1] Title for the talk. 2] Name, Affiliation, Bio, a passport size picture (optional) and contact email address of the Presenter. 3] What will be covered? A bulleted list of the main points of the presentation will be ideal. Please include enough detail as will be necessary. 4] Any specific requirements needed for the presentation other than an overhead projector and a microphone. Presentations are alloted a time slot of about 45 minutes. All proposals are to be sent to cfp =at= socallinuxexpo.com. Important Dates: 20 Jun, 2005: CFP Opens 20 Nov, 2005: Last date for abstracts/proposals 20 Dec, 2005: Last date for notification of acceptance 11 Feb, 2006: Conference starts ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
