Re: [HACKERS] Initial review of xslt with no limits patch
On 09/08/10 04:07, Tom Lane wrote: Robert Haasrobertmh...@gmail.com writes: On Sun, Aug 8, 2010 at 11:36 AM, Mike Fowlerm...@mlfowler.com wrote: 1) XML2 is largely undocumented, giving rise to the problems encountered. Since the module is deprecated anyways, does it make more sense to get xslt handling moved into core and get it fully documented? Yes, I think that would be better. I'm hesitant to consider pulling this into core when there's so little consensus on how it ought to act. It'd be better to have a solid, widely used contrib module *first*, rather than imagine that pulling it into core is somehow a cure for its problems. Perhaps the first step forward is to pull xslt_process out of xml2 and create a standalone xslt contrib module. Then at least it can lose the stigma of being in a deprecated module and perhaps make it more visible to users. 2) Pavel's regression test exposes a bug in libxslt. The stylesheet declares 5 parameters, but uses 12. Simplifying, take the stylesheet: I'm not sure whether there's anything we can do about this. We should file a bug report with the libxslt authors, obviously. Turns out the bug was filed in 2005 (see https://bugzilla.gnome.org/show_bug.cgi?id=307061). They are currently taking a fairly loose interpretation of the XSLT spec. However that was only one aspect of the concern. The other was that no errors were being reported back in psql when the libxslt is generating errors. Is this desirable? Regards, -- Mike Fowler Registered Linux user: 379787 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: psql: edit function, show function commands patch
2010/8/9 Tom Lane t...@sss.pgh.pa.us: Robert Haas robertmh...@gmail.com writes: On Sun, Aug 8, 2010 at 1:14 PM, Tom Lane t...@sss.pgh.pa.us wrote: What exactly is the point of the \sf command? I rather like \sf, actually; in fact, I think there's a decent argument to be made that it's more useful than the line-numbering stuff for \ef. I don't particularly like the name \sf, but that's more because I think backslash commands are a fundamentally unscalable approach to providing administrative functionality than because I think there's a better option in this particular case. It's rather hard right now to get a function definition out of the database in easily cut-and-pastable format. Um, but \sf *doesn't* give you anything that's usefully copy and pasteable. And if that were the goal, why doesn't it have an option to write to a file? there are not a line numbers. And you can't to use a result of \df+ too. But it's really the line numbers shoved in front that I'm on about here. I can't see *any* use for that behavior except to figure out what part of your function an error message with line number is referring to; and as I said upthread, there are better ways to be attacking that problem. If you've got a thousand-line function (yes, they're out there) do you really want to be scrolling through \sf output to find out what line 714 \sf supports a pager \sf can show lines from entered number so \sf foo 700 -- show from line 700 Best regards Pavel Stehule is? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: psql: edit function, show function commands patch
Hello 2010/8/8 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: updated patch attached What exactly is the point of the \sf command? It seems like quite a lot of added code for a feature that nobody has requested, and whose definition is about as ad-hoc as could be. Personally I'd much sooner use \ef for looking at a function definition. I think if \sf had been submitted as a separate patch, rather than being snuck in with a feature people do want, it wouldn't be accepted. The current patch doesn't even compile warning-free :-( command.c: In function `exec_command': command.c:559: warning: `lineno' might be used uninitialized in this function command.c: In function `editFile': command.c:1729: warning: `editor_lineno_switch' might be used uninitialized in this function This warnings depends on gcc version, probably :(. On new fedora I see nothing. So updated patch attached - these variables are initialised in declaration now. Regards Pavel Stehule regards, tom lane *** ./doc/src/sgml/ref/psql-ref.sgml.orig 2010-08-03 09:00:48.384710383 +0200 --- ./doc/src/sgml/ref/psql-ref.sgml 2010-08-03 10:44:57.312835131 +0200 *** *** 1339,1345 varlistentry ! termliteral\edit/literal (or literal\e/literal) literaloptional replaceable class=parameterfilename/replaceable /optional/literal/term listitem para --- 1339,1345 varlistentry ! termliteral\edit/literal (or literal\e/literal) literaloptional replaceable class=parameterfilename/replaceable /optional optional linenumber /optional/literal/term listitem para *** *** 1369,1380 systems, filenamenotepad.exe/filename on Windows systems. /para /tip /listitem /varlistentry varlistentry ! termliteral\ef optional replaceable class=parameterfunction_description/replaceable /optional/literal/term listitem para --- 1369,1387 systems, filenamenotepad.exe/filename on Windows systems. /para /tip + + para + If replaceable class=parameterlinenumber/replaceable is + specified, then cursor is moved on this line after start of + editor (The psql's variable varnameEDITOR_LINENUMBER_SWITCH/varname + have to be filled). + /para /listitem /varlistentry varlistentry ! termliteral\ef optional replaceable class=parameterfunction_description/replaceable /optional optional linenumber /optional /literal/term listitem para *** *** 1397,1402 --- 1404,1417 If no function is specified, a blank commandCREATE FUNCTION/ template is presented for editing. /para + + para + If replaceable class=parameterlinenumber/replaceable is + specified, then cursor is moved on this line after start of + editor. It count lines from start of function body, not from + start of text (The psql's variable varnameEDITOR_LINENUMBER_SWITCH/varname + have to be filled). + /para /listitem /varlistentry *** *** 2116,2121 --- 2131,2148 varlistentry + termliteral\sf[+] replaceable class=parameterfunction_description/replaceable optional linenumber /optional /literal/term + + listitem + para + This command fetches and shows the definition of the named function, + in the form of a commandCREATE OR REPLACE FUNCTION/ command. + If the form literal\sf+/literal is used, then lines are numbered. + /para + /listitem + /varlistentry + + varlistentry termliteral\t/literal/term listitem para *** *** 2123,2128 --- 2150,2161 footer. This command is equivalent to literal\pset tuples_only/literal and is provided for convenience. /para + + para + If replaceable class=parameterlinenumber/replaceable is + specified, then cursor is moved on this line after start of + editor. + /para /listitem /varlistentry *** *** 2459,2464 --- 2492,2511 /varlistentry varlistentry + termvarnameEDITOR_LINENUMBER_SWITCH/varname/term + listitem + para + Option used for navigation (go to line command) in external + editor. When it isn't defined, then you cannot to specify + line numbers for command\edit/command and command\ef/command + commands. On unix platforms are possible to use a 'option+/option' + or 'option--line /option'. The space after literalline/literal + is required. + /para + /listitem +
Re: [HACKERS] parallel quicksort
Hi, On 08/09/2010 12:04 AM, Mark Wong wrote: I've been playing around with a process based parallel quicksort (http://github.com/markwkm/quicksort) and I tried to shoehorn it into postgres because I wanted to see if I could sort more than integers. I've attached a patch that creates a new GUC to control the degree of parallelism and only modified the quicksort algorithm in quicksort.c. Trying to 'make install' quickly shows me the patch breaks zic and Andrew Gierth further pointed out on irc (a couple months back now) that user defined comparison functions won't work as expected in the forked processes (if I remember that correctly). I'm not sure what the problems are, but the background worker infrastructure I recently posted could possibly solve this problem, as those are more like normal backends. (Assuming you were forking from the backend). Hoping this could be useful, I wanted to put out what I had so far and see how far away this is from something workable. Not to mention that there are probably some improvements that could be make to the parallel quicksort algorithm. Thanks for sharing. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_stat_transaction patch
2010/8/8 Tom Lane t...@sss.pgh.pa.us Itagaki Takahiro itagaki.takah...@gmail.com writes: Accessor functions to get so far collected statistics for the current transaction https://commitfest.postgresql.org/action/patch_view?id=301 The only issue in the patch is too long view and function names: - pg_stat_transaction_user_tables (31 chars) - pg_stat_get_transaction_tuples_hot_updated (42 chars) - pg_stat_get_transaction_function_self_time (42 chars) Since we've already used _xact_ in some system objects, we could replace _transaction_ parts with _xact_. It will save 7 key types per query ;-) Applied, with assorted corrections - * Renamed *_transaction_* to *_xact_* as suggested by Itagaki-san. * Removed functions and view columns for delta live/dead tuple counts. * Marked functions as volatile ... they certainly aren't stable. * Got rid of use of get_tabstat_entry() to fetch table entries. That function forcibly creates tabstat entries if they weren't there before, which was absolutely not what we want here: it'd result in bloating the tabstat arrays with entries for tables the current transaction actually never touched. Worse, since you weren't passing the correct isshared flag for the particular relation, the entries could be created with the wrong isshared setting, leading to misbehavior if they did get used later in the transaction. We have to use a find-don't-create function here. * Fixed bogus handling of inserted/updated/deleted counts --- you need to add on the pending counts for all open levels of subtransaction. * Assorted docs improvement and other minor polishing. BTW, I notice that the patch provides pg_stat_get_xact_blocks_fetched() and pg_stat_get_xact_blocks_hit(), but doesn't build any views on top of them. Was this intentional? Providing a full complement of pg_statio_xact_* views seems like overkill to me, but maybe that was where you were intending to go and forgot. If the functions are there then anyone who needs the functionality can easily build their own views atop them, so this might be an intentional compromise position, but I'm not sure. Or maybe we should decide that intratransaction statio numbers aren't likely to be of interest to anybody, and drop the functions too. When I created the views, I just copied the existing pg_stat_user_* views without knowing if any columns where irrelevant for current transaction data. I guess if someone would need the blocks_fetched/hit, they could build their own view. regards, tom lane -- Best regards, Joel Jacobson Glue Finance E: j...@gluefinance.com T: +46 70 360 38 01 Postal address: Glue Finance AB Box 549 114 11 Stockholm Sweden Visiting address: Glue Finance AB Birger Jarlsgatan 14 114 34 Stockholm Sweden
Re: [HACKERS] GROUPING SETS revisited
Hello I was confused when I though so I found a solution of 1 shift/reduce conflict :( All identificators used for buildin functions have to be a col_name_keywords or reserved keyword. There is conflict with our (probably obsolete) feature SELECT colname(tabname). So for this moment the real solution is removing CUBE and ROLLUP from keywords and dynamically testing a funcname in transformation stage - what is slower and more ugly. ideas? Regards Pavel Stehule 2010/8/7 Pavel Stehule pavel.steh...@gmail.com: 2010/8/7 Joshua Tolley eggyk...@gmail.com: On Thu, Aug 05, 2010 at 04:46:51PM +0200, Pavel Stehule wrote: I am sending a updated version. I've been looking at the changes to gram.y, and noted the comment under func_expr where you added CUBE and ROLLUP definitions. It says that CUBE can't be a reserved keyword because it's already used in the cube contrib module. But then the changes to kwlist.h include this: I am little bit confused now - it's bad comment - and I have to verify it. What I remember, we cannot to use a two parser's rules, because it going to a conflict. So there have to be used a trick with a moving to decision to transform stage, where we have a context info. I have to recheck a minimal level - probably it can't be a RESERVED_KEYWORD. Because then we can't to create a function cube. + PG_KEYWORD(cube, CUBE, RESERVED_KEYWORD) ... + PG_KEYWORD(rollup, ROLLUP, RESERVED_KEYWORD) ...and CUBE and ROLLUP are added in gram.y under the reserved_keyword list. I realize things like CURRENT_TIME, that also have special entries in the func_expr grammar, are also reserved keywords, but this all seems at odds with the comment. What am I missing? Is the comment simply pointing out that the designation of CUBE and ROLLUP as reserved keywords will have to change at some point, but it hasn't been implemented yet (or no one has figured out how to do it)? -- Joshua Tolley / eggyknap End Point Corporation http://www.endpoint.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkxcjSIACgkQRiRfCGf1UMPpCwCcCHBh/1NiLykIcVYgPyfbIegF xq0AoID75rCPiW8yf29OSkaJVza1FQt5 =PcLs -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] more personal copyrights
On Mon, Aug 9, 2010 at 1:18 AM, Jaime Casanova ja...@2ndquadrant.com wrote: A few months ago Bruce was doing a hunting of personal Copyrights notices, but i still found a lot of files copyrighted to: Regents of the University of California and other files copyrighted to individuals (ej: almost everything inside src/backend/regex is copyrighted to Henry Spencer) there's something we can/need to do about it? I can make a list if anyone is interested showing what files are still copyrighted to something different to 'PostgreSQL Global Development Group' Henry Spencer probably didn't intend to become a member of the PostgreSQL Global Development Group; he wrote a regex library, which we included because the license was sufficiently permissive for our needs. Or at least, that's how I understand it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] is syntax columname(tablename) necessary still?
Hello I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. But then we have a gram conflict with mentioned obsolete syntax. I am thinking so after removing add_missing_from this syntax is useless. Without this feature we can clean a gramatic. The cube issue can be solved without removing this feature too. We have to check every funcname on equality to cube or rollup string. Is this method acceptable? Regards Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: psql: edit function, show function commands patch
On Sun, Aug 8, 2010 at 11:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Um, but \sf *doesn't* give you anything that's usefully copy and pasteable. Works for me. \sf ts_debug(regconfig, text) And if that were the goal, why doesn't it have an option to write to a file? Well, you cut-and-paste from a terminal window, not a file, so that's a slightly different problem, although perhaps also a good one to solve. But I'd rather see us solve that problem via some new pg_dump functionality. Hmm... or perhaps \sf should respect \o. I notice that \d does. But it's really the line numbers shoved in front that I'm on about here. I can't see *any* use for that behavior except to figure out what part of your function an error message with line number is referring to; and as I said upthread, there are better ways to be attacking that problem. If you've got a thousand-line function (yes, they're out there) do you really want to be scrolling through \sf output to find out what line 714 is? Well, as Pavel points out, I guess you could use the line number argument to \sf to start at around the place you're interested in, athough I suspect that I would probably choose to use \ef in that case. I suspect \sf is in general most useful with somewhat shorter functions (I'd copy and paste a 100 line function, perhaps, but for a 1000 line function I'd probably try to get the definition into a file and scp it or whatever). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: psql: edit function, show function commands patch
2010/8/9 Robert Haas robertmh...@gmail.com: On Sun, Aug 8, 2010 at 11:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Um, but \sf *doesn't* give you anything that's usefully copy and pasteable. Works for me. \sf ts_debug(regconfig, text) And if that were the goal, why doesn't it have an option to write to a file? Well, you cut-and-paste from a terminal window, not a file, so that's a slightly different problem, although perhaps also a good one to solve. But I'd rather see us solve that problem via some new pg_dump functionality. Hmm... or perhaps \sf should respect \o. I notice that \d does. it's not a bad idea. updated patch attached But it's really the line numbers shoved in front that I'm on about here. I can't see *any* use for that behavior except to figure out what part of your function an error message with line number is referring to; and as I said upthread, there are better ways to be attacking that problem. If you've got a thousand-line function (yes, they're out there) do you really want to be scrolling through \sf output to find out what line 714 is? Well, as Pavel points out, I guess you could use the line number argument to \sf to start at around the place you're interested in, athough I suspect that I would probably choose to use \ef in that case. I suspect \sf is in general most useful with somewhat shorter functions (I'd copy and paste a 100 line function, perhaps, but for a 1000 line function I'd probably try to get the definition into a file and scp it or whatever). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company *** ./doc/src/sgml/ref/psql-ref.sgml.orig 2010-08-03 09:00:48.384710383 +0200 --- ./doc/src/sgml/ref/psql-ref.sgml 2010-08-03 10:44:57.312835131 +0200 *** *** 1339,1345 varlistentry ! termliteral\edit/literal (or literal\e/literal) literaloptional replaceable class=parameterfilename/replaceable /optional/literal/term listitem para --- 1339,1345 varlistentry ! termliteral\edit/literal (or literal\e/literal) literaloptional replaceable class=parameterfilename/replaceable /optional optional linenumber /optional/literal/term listitem para *** *** 1369,1380 systems, filenamenotepad.exe/filename on Windows systems. /para /tip /listitem /varlistentry varlistentry ! termliteral\ef optional replaceable class=parameterfunction_description/replaceable /optional/literal/term listitem para --- 1369,1387 systems, filenamenotepad.exe/filename on Windows systems. /para /tip + + para + If replaceable class=parameterlinenumber/replaceable is + specified, then cursor is moved on this line after start of + editor (The psql's variable varnameEDITOR_LINENUMBER_SWITCH/varname + have to be filled). + /para /listitem /varlistentry varlistentry ! termliteral\ef optional replaceable class=parameterfunction_description/replaceable /optional optional linenumber /optional /literal/term listitem para *** *** 1397,1402 --- 1404,1417 If no function is specified, a blank commandCREATE FUNCTION/ template is presented for editing. /para + + para + If replaceable class=parameterlinenumber/replaceable is + specified, then cursor is moved on this line after start of + editor. It count lines from start of function body, not from + start of text (The psql's variable varnameEDITOR_LINENUMBER_SWITCH/varname + have to be filled). + /para /listitem /varlistentry *** *** 2116,2121 --- 2131,2148 varlistentry + termliteral\sf[+] replaceable class=parameterfunction_description/replaceable optional linenumber /optional /literal/term + + listitem + para + This command fetches and shows the definition of the named function, + in the form of a commandCREATE OR REPLACE FUNCTION/ command. + If the form literal\sf+/literal is used, then lines are numbered. + /para + /listitem + /varlistentry + + varlistentry termliteral\t/literal/term listitem para *** *** 2123,2128 --- 2150,2161 footer. This command is equivalent to literal\pset tuples_only/literal and is provided for convenience. /para + + para + If replaceable class=parameterlinenumber/replaceable is + specified, then cursor is moved on this line after start of + editor. + /para /listitem /varlistentry *** *** 2459,2464 --- 2492,2511
Re: [HACKERS] MERGE Specification
On 07/08/10 10:58, Boxuan Zhai wrote: I have just finished a new patch, with the following feature: Please include the regression tests in the patch too. Also, I note that there's a few merge conflicts when applied over CVS HEAD from today, can you please fix the bitrot? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] MERGE Specification
On 09/08/10 14:47, Heikki Linnakangas wrote: On 07/08/10 10:58, Boxuan Zhai wrote: I have just finished a new patch, with the following feature: Please include the regression tests in the patch too And the docs changes too. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
On Mon, Aug 09, 2010 at 12:18:33PM +0200, Pavel Stehule wrote: Hello I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. The cube contrib module was only ever meant to be replaced by the real feature, which you're working on, so +1 for dropping everything in it that you are not replacing with the one which complies with the SQL standard. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] more personal copyrights
On Mon, Aug 09, 2010 at 02:27:50PM +0900, Tatsuo Ishii wrote: A few months ago Bruce was doing a hunting of personal Copyrights notices, but i still found a lot of files copyrighted to: Regents of the University of California and other files copyrighted to individuals (ej: almost everything inside src/backend/regex is copyrighted to Henry Spencer) there's something we can/need to do about it? I can make a list if anyone is interested showing what files are still copyrighted to something different to 'PostgreSQL Global Development Group' I am not sure if we could arbitrarily change the copyright notices without explicit permissions from the original authors. We can't. What we should probably do is come up with a list of exceptions (Henry Spencer's regex library, e.g.) and a script to find all non-exception copyright notices. This will help us keep the copyrights in the code base to the standard we set. I can even see this becoming a commit- or push- hook. :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
On Mon, Aug 9, 2010 at 2:02 PM, David Fetter da...@fetter.org wrote: I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. The cube contrib module was only ever meant to be replaced by the real feature, which you're working on, so +1 for dropping everything in it that you are not replacing with the one which complies with the SQL standard. That's not right. The cube contrib module is a kind of vector data type. It's not related in any way to the SQL CUBE or ROLLUP syntax. Personally I think cube is uncommonly used and CUBE an important enough SQL feature that we should just bite the bullet and kill/rename the contrib module. Partly that's because I find the name quite strange and non-intuitive anyways. Something like vector or ntuple would be far clearer. Doing nasty hacks to make CUBE a non-reserved word doesn't seem justified by the contrib module. Now conceivably it's a word users might be using in their schema and that might be a good enough reason to hack up the grammar -- but it's not like it's a new keyword in SQL so it shouldn't come as a surprise to users when they get an error. I think more people are surprised when we *don't* support CUBE than will be when we start doing so. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
On Mon, Aug 09, 2010 at 02:23:55PM +0100, Greg Stark wrote: On Mon, Aug 9, 2010 at 2:02 PM, David Fetter da...@fetter.org wrote: I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. The cube contrib module was only ever meant to be replaced by the real feature, which you're working on, so +1 for dropping everything in it that you are not replacing with the one which complies with the SQL standard. That's not right. That's what I get for posting before coffee :P +1 for renaming the stuff in the contrib/cube module, along with a loud warning about this in the release notes, release announcement, etc. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
2010/8/9 Greg Stark gsst...@mit.edu: On Mon, Aug 9, 2010 at 2:02 PM, David Fetter da...@fetter.org wrote: I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. The cube contrib module was only ever meant to be replaced by the real feature, which you're working on, so +1 for dropping everything in it that you are not replacing with the one which complies with the SQL standard. That's not right. The cube contrib module is a kind of vector data type. It's not related in any way to the SQL CUBE or ROLLUP syntax. Personally I think cube is uncommonly used and CUBE an important enough SQL feature that we should just bite the bullet and kill/rename the contrib module. Partly that's because I find the name quite strange and non-intuitive anyways. Something like vector or ntuple would be far clearer. Doing nasty hacks to make CUBE a non-reserved word doesn't seem justified by the contrib module. Now conceivably it's a word users might be using in their schema and that might be a good enough reason to hack up the grammar -- but it's not like it's a new keyword in SQL so it shouldn't come as a surprise to users when they get an error. I think more people are surprised when we *don't* support CUBE than will be when we start doing so. ok - with reserved keyword the life is little bit nicer, but still if we remove obsolete columnname(tablename) syntax, we can remeve a few hack in parser - and implement a GROUPING SETS grammar little bit cleaner. Pavel -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
Greg Stark gsst...@mit.edu writes: Personally I think cube is uncommonly used and CUBE an important enough SQL feature that we should just bite the bullet and kill/rename the contrib module. Yeah. It looks to me like CUBE will have to be a type_function_name keyword (but hopefully not fully reserved), which will mean that we can't have a contrib module defining a type by that name. Ergo, rename. ... Now conceivably it's a word users might be using in their schema and that might be a good enough reason to hack up the grammar -- but it's not like it's a new keyword in SQL so it shouldn't come as a surprise to users when they get an error. As long as we can avoid making it fully reserved, tables/columns named cube will still work, so the damage should be limited. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] more personal copyrights
Jaime Casanova ja...@2ndquadrant.com writes: A few months ago Bruce was doing a hunting of personal Copyrights notices, but i still found a lot of files copyrighted to: Regents of the University of California and other files copyrighted to individuals (ej: almost everything inside src/backend/regex is copyrighted to Henry Spencer) there's something we can/need to do about it? No. The regex code was lifted whole hog from Spencer's package; it would be most uncivilized, as well as illegal, to remove his name from it. The case that we are trying to eliminate is where people have put individual copyrights on code that was written specifically for Postgres. There's no good reason to have such files look like they might have a license different from the rest of Postgres. However, Spencer's code isn't in that category --- it's also in Tcl, and I imagine it was once distributed as a standalone library. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Initial review of xslt with no limits patch
Mike Fowler m...@mlfowler.com writes: Turns out the bug was filed in 2005 (see https://bugzilla.gnome.org/show_bug.cgi?id=307061). They are currently taking a fairly loose interpretation of the XSLT spec. However that was only one aspect of the concern. The other was that no errors were being reported back in psql when the libxslt is generating errors. Is this desirable? Uh, no; if we're failing to detect an error that the library does report, that's our bug (and another indication of the immaturity of this code :-()). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Initial review of xslt with no limits patch
On Mon, Aug 9, 2010 at 10:14 AM, Tom Lane t...@sss.pgh.pa.us wrote: Mike Fowler m...@mlfowler.com writes: Turns out the bug was filed in 2005 (see https://bugzilla.gnome.org/show_bug.cgi?id=307061). They are currently taking a fairly loose interpretation of the XSLT spec. However that was only one aspect of the concern. The other was that no errors were being reported back in psql when the libxslt is generating errors. Is this desirable? Uh, no; if we're failing to detect an error that the library does report, that's our bug (and another indication of the immaturity of this code :-()). Right. So, what about Mike's idea of extracting this into a new contrib module, perhaps contrib/xslt? That might also provide a good excuse to jettison any details of the existing interfaces that we happen to find unfortunate. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: xml_is_well_formed
On lör, 2010-08-07 at 16:47 +0100, Mike Fowler wrote: To be honest I'm happiest with returning a boolean, even if there is some confusion over content only being valid. Though changing the return value to DOCUMENT/CONTENT/NULL makes things a touch more explicit, the same results can be achieved by simply running: SELECT data::xml FROM mixed WHERE xml_is_well_formed(data) AND data::xml IS DOCUMENT; Note that this wouldn't necessarily work because it is not guaranteed that the well-formedness test is executed before the cast to xml. SQL doesn't short-circuit left to right. (A CASE expression could work.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
Pavel Stehule pavel.steh...@gmail.com writes: I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. But then we have a gram conflict with mentioned obsolete syntax. I am thinking so after removing add_missing_from this syntax is useless. Without this feature we can clean a gramatic. That's a documented and useful feature. It's not going away. Even if it did go away, removing it wouldn't do a thing to solve grammar problems, because the grammar isn't involved in that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Initial review of xslt with no limits patch
Robert Haas robertmh...@gmail.com writes: Right. So, what about Mike's idea of extracting this into a new contrib module, perhaps contrib/xslt? That might also provide a good excuse to jettison any details of the existing interfaces that we happen to find unfortunate. Seems like mostly make-work to me --- we could just as easily fix the code where it sits. But if you're excited about it, I won't stand in the way. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: xml_is_well_formed
On Mon, Aug 9, 2010 at 10:20 AM, Peter Eisentraut pete...@gmx.net wrote: On lör, 2010-08-07 at 16:47 +0100, Mike Fowler wrote: To be honest I'm happiest with returning a boolean, even if there is some confusion over content only being valid. Though changing the return value to DOCUMENT/CONTENT/NULL makes things a touch more explicit, the same results can be achieved by simply running: SELECT data::xml FROM mixed WHERE xml_is_well_formed(data) AND data::xml IS DOCUMENT; Note that this wouldn't necessarily work because it is not guaranteed that the well-formedness test is executed before the cast to xml. SQL doesn't short-circuit left to right. (A CASE expression could work.) There's also the fact that it would probably end up parsing the data twice. Given xmloption, I'm inclined to think Tom has it right: provided xml_is_well_formed() that follows xmloption, plus a specific version for each of content and document. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
2010/8/9 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. But then we have a gram conflict with mentioned obsolete syntax. I am thinking so after removing add_missing_from this syntax is useless. Without this feature we can clean a gramatic. That's a documented and useful feature. It's not going away. Even if it did go away, removing it wouldn't do a thing to solve grammar problems, because the grammar isn't involved in that. This isn't a SQL feature and it coming from old times like missing from. Without this we can little bit simplify ParseFuncOrColumn. But I don't know, if this can be a significant win. It is just obsolete. Regards Pavel regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
On Mon, Aug 9, 2010 at 10:45 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2010/8/9 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: I am working on Grouping Sets support. The first issue is cube keyword. Contrib module cube define a few functions cube. So if we want to continue in support this function, then cube have to be a unreserved keyword. But then we have a gram conflict with mentioned obsolete syntax. I am thinking so after removing add_missing_from this syntax is useless. Without this feature we can clean a gramatic. That's a documented and useful feature. It's not going away. Even if it did go away, removing it wouldn't do a thing to solve grammar problems, because the grammar isn't involved in that. This isn't a SQL feature and it coming from old times like missing from. Without this we can little bit simplify ParseFuncOrColumn. But I don't know, if this can be a significant win. It is just obsolete. I think the point is that it's not going to solve the problem you have right now. It might or might not be a good thing to do, but it's not going to help with GROUPING SETS. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Markus Wanner wrote: Hi, On 07/26/2010 07:16 PM, Robert Haas wrote: Of course, there are other parts of the system (a whole bunch of them) that used shared memory also, and perhaps some of those could be modified to use the dynamic allocator as well. But they're getting by without it now, so maybe they don't really need it. The SLRU stuff, I think, works more or less like shared buffers (so you have the same set of issues) and I think most of the other users are allocating small, fixed-size chunks. Yeah, I see your point(s). Note however, that a thread based design doesn't have this problem *at all*. Memory generally is shared (between threads) and you can dynamically allocate more or less (until Linux' OOM killer hits you.. yet another story). The OS reuses memory you don't currently need even for other applications. [ Sorry to be jumping into this thread late.] I am not sure threads would greatly help us. The major problem is that all of our our structures are currently contiguous in memory for quick access. I don't see how threading would help with that. We could use realloc(), but we can do the same in shared memory if we had a chunk infrastructure, though concurrent access to that memory would hurt us in either threads or shared memory. Fundamentally, recreating the libc memory allocation routines is not that hard. (Everyone has to detach from the shared memory segment, but they have to stop using it too, so it doesn't seem that hard.) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
2010/8/9 Tom Lane t...@sss.pgh.pa.us: Greg Stark gsst...@mit.edu writes: Personally I think cube is uncommonly used and CUBE an important enough SQL feature that we should just bite the bullet and kill/rename the contrib module. Yeah. It looks to me like CUBE will have to be a type_function_name keyword (but hopefully not fully reserved), which will mean that we can't have a contrib module defining a type by that name. Ergo, rename. I am afraid, CUBE and ROLLUP have to be a reserved keyword because as type_function_name is in conflict with func_name ( ... Regards Pavel Stehule ... Now conceivably it's a word users might be using in their schema and that might be a good enough reason to hack up the grammar -- but it's not like it's a new keyword in SQL so it shouldn't come as a surprise to users when they get an error. As long as we can avoid making it fully reserved, tables/columns named cube will still work, so the damage should be limited. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] regclass, \d command and temp tables
Hi, I have created regular table t1 and temp table t1. Regclass and \d command do not seem to distinguish them. Is this normal? test=# create table t1(i int); CREATE TABLE test=# select 't1'::regclass::oid; oid - 1470776 (1 row) test=# create temp table t1(i int, j int); CREATE TABLE test=# select 't1'::regclass::oid; oid - 1470776 -- why same oid? (1 row) test=# \d t1 -- \d finds regular table, not temporary table? Table public.t1 Column | Type | Modifiers +-+--- i | integer | test=# insert into t1 values(1,2); -- insert command works as expected of course. INSERT 0 1 test=# select * from t1; i | j ---+--- 1 | 2 (1 row) -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian br...@momjian.us wrote: I am not sure threads would greatly help us. The major problem is that all of our our structures are currently contiguous in memory for quick access. I don't see how threading would help with that. We could use realloc(), but we can do the same in shared memory if we had a chunk infrastructure, though concurrent access to that memory would hurt us in either threads or shared memory. Fundamentally, recreating the libc memory allocation routines is not that hard. (Everyone has to detach from the shared memory segment, but they have to stop using it too, so it doesn't seem that hard.) I actually don't think that's true. The advantage (and disadvantage) of using threads is that everything runs in one address space. So you just allocate more memory and everyone immediately sees it. In a process environment, that's not the case: to expand or shrink the size of the shared memory arena, everyone needs to explicitly change their own mapping. So imagine that thread-or-process A allocates allocates a new chunk of memory and then writes a pointer to the new chunk in a previously allocated section of memory. Thread-or-process B then follows the pointer. In a threaded model, this is guaranteed to be safe. In a process model, it's not: A might have enlarged the shared memory mapping while B has not yet done so. So I think in our model any sort of change to the shared memory segment is going to require extremely careful gymnastics, and be pretty expensive. I don't care to take a position in the religious war over threads vs. processes, but I do think threads simplify the handling of this particular case. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
On Mon, Aug 9, 2010 at 11:06 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2010/8/9 Tom Lane t...@sss.pgh.pa.us: Greg Stark gsst...@mit.edu writes: Personally I think cube is uncommonly used and CUBE an important enough SQL feature that we should just bite the bullet and kill/rename the contrib module. Yeah. It looks to me like CUBE will have to be a type_function_name keyword (but hopefully not fully reserved), which will mean that we can't have a contrib module defining a type by that name. Ergo, rename. I am afraid, CUBE and ROLLUP have to be a reserved keyword because as type_function_name is in conflict with func_name ( ... They name to be type_func_keywords, perhaps, but not fully reserved. And they'd still need that treatment anyway. Even if cube(whatever) can't mean extract a column called cube from table whatever, it can still mean call a function called cube on a column called whatever. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] regclass, \d command and temp tables
Tatsuo Ishii is...@postgresql.org writes: I have created regular table t1 and temp table t1. Regclass and \d command do not seem to distinguish them. Is this normal? It works for me. Are you using clean sources? regression=# create table t1(i int); CREATE TABLE regression=# select 't1'::regclass::oid; oid 127671 (1 row) regression=# create temp table t1(i int, j int); CREATE TABLE regression=# select 't1'::regclass::oid; oid 127674 (1 row) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
2010/8/9 Robert Haas robertmh...@gmail.com: On Mon, Aug 9, 2010 at 11:06 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2010/8/9 Tom Lane t...@sss.pgh.pa.us: Greg Stark gsst...@mit.edu writes: Personally I think cube is uncommonly used and CUBE an important enough SQL feature that we should just bite the bullet and kill/rename the contrib module. Yeah. It looks to me like CUBE will have to be a type_function_name keyword (but hopefully not fully reserved), which will mean that we can't have a contrib module defining a type by that name. Ergo, rename. I am afraid, CUBE and ROLLUP have to be a reserved keyword because as type_function_name is in conflict with func_name ( ... They name to be type_func_keywords, perhaps, but not fully reserved. And they'd still need that treatment anyway. Even if cube(whatever) can't mean extract a column called cube from table whatever, it can still mean call a function called cube on a column called whatever. look to gram.y, please. we can use a GROUP BY CUBE(expr, ..) GROUP BY func_name(expr, ..) so these rules are in conflict, because func_name can have a type_func_keywords symbols. So we have to significantly rewrite a rules about func call or CUBE and ROLLUP have to be a reserved words. There isn't any other possibility. regards Pavel -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] regclass, \d command and temp tables
It works for me. Are you using clean sources? regression=# create table t1(i int); CREATE TABLE regression=# select 't1'::regclass::oid; oid 127671 (1 row) regression=# create temp table t1(i int, j int); CREATE TABLE regression=# select 't1'::regclass::oid; oid 127674 (1 row) Oops. Sorry for noise. I accidentaly accessed through pgpool and it load balanced the SELECT to different PostgreSQL server. Pgpool should have remebered that temporary tables are created in the session. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: psql: edit function, show function commands patch
On Aug 8, 2010, at 8:38 PM, Tom Lane wrote: Um, but \sf *doesn't* give you anything that's usefully copy and pasteable. And if that were the goal, why doesn't it have an option to write to a file? But it's really the line numbers shoved in front that I'm on about here. I can't see *any* use for that behavior except to figure out what part of your function an error message with line number is referring to; and as I said upthread, there are better ways to be attacking that problem. If you've got a thousand-line function (yes, they're out there) do you really want to be scrolling through \sf output to find out what line 714 is? Suggestion: \sf without line numbers \sf+ with line numbers Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: psql: edit function, show function commands patch
2010/8/9 David E. Wheeler da...@kineticode.com: On Aug 8, 2010, at 8:38 PM, Tom Lane wrote: Um, but \sf *doesn't* give you anything that's usefully copy and pasteable. And if that were the goal, why doesn't it have an option to write to a file? But it's really the line numbers shoved in front that I'm on about here. I can't see *any* use for that behavior except to figure out what part of your function an error message with line number is referring to; and as I said upthread, there are better ways to be attacking that problem. If you've got a thousand-line function (yes, they're out there) do you really want to be scrolling through \sf output to find out what line 714 is? Suggestion: \sf without line numbers \sf+ with line numbers it did it :) Pavel Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Hi, On 08/09/2010 05:02 PM, Bruce Momjian wrote: [ Sorry to be jumping into this thread late.] No problem at all. I am not sure threads would greatly help us. Note that I'm absolutely, certainly not advocating the use of threads for Postgres. The major problem is that all of our our structures are currently contiguous in memory for quick access. I don't see how threading would help with that. We could use realloc(), but we can do the same in shared memory if we had a chunk infrastructure, though concurrent access to that memory would hurt us in either threads or shared memory. I don't quite follow what you are trying to say here. Whether or not structures are contiguous in memory might affect performance, but I don't see the relation to programmer's habits and/or knowledge. With our process-based design, the default is private memory (i.e. not shared). If you need shared memory, you must specify a certain amount in advance. That chunk of shared memory then is reserved and can't ever be used by another subsystem. Even if you barely ever need that much shared memory for the subsystem in question. That's opposed to what lots of people are used to with the threaded approach, where shared memory is the default. And where you can easily and dynamically allocate *shared* memory. Whatever chunk of shared memory one subsystem doesn't need is available to another one (modulo fragmentation of the dynamic allocator, perhaps, but..) Fundamentally, recreating the libc memory allocation routines is not that hard. Uh.. well, writing a good, scalable, dynamic allocator certainly poses some very interesting problems. Writing one that doesn't violate any patent or other IP as an additional requirement seems like a pretty tough problem to me. (Everyone has to detach from the shared memory segment, but they have to stop using it too, so it doesn't seem that hard.) So far, I only considered dynamically allocating from a pool of shared memory that's initially fixed in size. So as to be able to make better use of shared memory. Resizing the overall pool the easy way, requiring every backend to detach would cost a lot of performance. So that's certainly not something you want to do often. The purpose of such a dynamic allocator as I see it rather is to be able to re-allocate unused memory of one subsystem to another one *on the fly*. Not just for performance, but also for ease of use for the admin and the developer, IMO. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas robertmh...@gmail.com writes: So imagine that thread-or-process A allocates allocates a new chunk of memory and then writes a pointer to the new chunk in a previously allocated section of memory. Thread-or-process B then follows the pointer. In a threaded model, this is guaranteed to be safe. In a process model, it's not: A might have enlarged the shared memory mapping while B has not yet done so. So I think in our model any sort of change to the shared memory segment is going to require extremely careful gymnastics, and be pretty expensive. ... and on some platforms, it'll be flat out impossible. We looked at this years ago and concluded that changing the size of the shmem segment after postmaster start was impractical from a portability standpoint. I have not seen anything to change that conclusion. I don't care to take a position in the religious war over threads vs. processes, but I do think threads simplify the handling of this particular case. You meant I don't think, right? I agree. The only way threads would simplify this is if we went over to a mysql-style model where there was only one process, period, and all backends were threads inside that. No shared memory as such, at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas wrote: On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian br...@momjian.us wrote: I am not sure threads would greatly help us. ?The major problem is that all of our our structures are currently contiguous in memory for quick access. ?I don't see how threading would help with that. ?We could use realloc(), but we can do the same in shared memory if we had a chunk infrastructure, though concurrent access to that memory would hurt us in either threads or shared memory. Fundamentally, recreating the libc memory allocation routines is not that hard. ?(Everyone has to detach from the shared memory segment, but they have to stop using it too, so it doesn't seem that hard.) I actually don't think that's true. The advantage (and disadvantage) of using threads is that everything runs in one address space. So you just allocate more memory and everyone immediately sees it. In a process environment, that's not the case: to expand or shrink the size of the shared memory arena, everyone needs to explicitly change their own mapping. You can't expand the size of malloc'ed memory --- you have to call realloc(), and then you effectively get a new pointer. Shared memory has a similar limitation. If you allocate shared memory in chunks so you don't need to change the location, you are effectively doing another malloc(), like you would in a threaded process. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Markus Wanner wrote: Hi, On 08/09/2010 05:02 PM, Bruce Momjian wrote: [ Sorry to be jumping into this thread late.] No problem at all. I am not sure threads would greatly help us. Note that I'm absolutely, certainly not advocating the use of threads for Postgres. The major problem is that all of our our structures are currently contiguous in memory for quick access. I don't see how threading would help with that. We could use realloc(), but we can do the same in shared memory if we had a chunk infrastructure, though concurrent access to that memory would hurt us in either threads or shared memory. I don't quite follow what you are trying to say here. Whether or not structures are contiguous in memory might affect performance, but I don't see the relation to programmer's habits and/or knowledge. With our process-based design, the default is private memory (i.e. not shared). If you need shared memory, you must specify a certain amount in advance. That chunk of shared memory then is reserved and can't ever be used by another subsystem. Even if you barely ever need that much shared memory for the subsystem in question. Once multiple threads are using the same local memory, you have the same issues of being unable to resize it because repalloc can change the pointer location. That's opposed to what lots of people are used to with the threaded approach, where shared memory is the default. And where you can easily and dynamically allocate *shared* memory. Whatever chunk of shared memory one subsystem doesn't need is available to another one (modulo fragmentation of the dynamic allocator, perhaps, but..) Well, this could be done with shared memory as well. My point is that you can treat malloc the same as add shared memory, to some extent, with the same limiations. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ERROR: argument to pg_get_expr() must come from system catalogs
Tom Lane wrote: Dave Page dp...@pgadmin.org writes: On Fri, Jul 30, 2010 at 12:17 AM, Tom Lane t...@sss.pgh.pa.us wrote: I've applied a (rather hurried) patch for this for 9.0beta4. Thanks. Bruce seemed to think it affected 8.4.4 as well - would that be the case, or is it something else? He's mistaken. The bug is in all the branches, but there have been no releases with it except 9.0beta3. I will work on back-patching the older branches this morning. I was using 8.4.X CVS with pgAdmin 8.4 and was seeing the error. Is the error in 8.4.X CVS? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Universal B-tree
Dear developers, I'm PhD candidate in Brazil and a newbie on postgresql developement, sorry for any silly questions. I implemented a new algorithm for range search using universal b-tree but I don't have a clue how to integrate it into postgresql. Where I can find the resources about it? I don't need to change B-tree estructure. I just need integrate my encode function that transforms multiple keys into one key by bit-interleaving and to acess elements given several intervals (range search). I've heard about a framework ATOM on the article Native Multidimensional Indexing in Relational Databases written by David Hoksza, Tomas Skopal which says that ATOM abstracts the way that postgres records information... does any one know where I can get it? Best regards, Daniel Oliveira
Re: [HACKERS] dynamically allocating chunks from shared memory
Bruce Momjian wrote: With our process-based design, the default is private memory (i.e. not shared). If you need shared memory, you must specify a certain amount in advance. That chunk of shared memory then is reserved and can't ever be used by another subsystem. Even if you barely ever need that much shared memory for the subsystem in question. Once multiple threads are using the same local memory, you have the same issues of being unable to resize it because repalloc can change the pointer location. Let me be more concrete. Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. How do you do that? If you want it contiguous, you have to use realloc, which might move the pointer. If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ERROR: argument to pg_get_expr() must come from system catalogs
Bruce Momjian br...@momjian.us writes: Tom Lane wrote: Dave Page dp...@pgadmin.org writes: Thanks. Bruce seemed to think it affected 8.4.4 as well - would that be the case, or is it something else? He's mistaken. The bug is in all the branches, but there have been no releases with it except 9.0beta3. I will work on back-patching the older branches this morning. I was using 8.4.X CVS with pgAdmin 8.4 and was seeing the error. Is the error in 8.4.X CVS? It was in CVS, but it never got into a release. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ERROR: argument to pg_get_expr() must come from system catalogs
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Tom Lane wrote: Dave Page dp...@pgadmin.org writes: Thanks. Bruce seemed to think it affected 8.4.4 as well - would that be the case, or is it something else? He's mistaken. The bug is in all the branches, but there have been no releases with it except 9.0beta3. I will work on back-patching the older branches this morning. I was using 8.4.X CVS with pgAdmin 8.4 and was seeing the error. Is the error in 8.4.X CVS? It was in CVS, but it never got into a release. OK, that explains why I see it and no one else does --- never mind. ;-) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
Pavel Stehule wrote: 2010/7/21 Itagaki Takahiro itagaki.takah...@gmail.com: 2010/7/20 Pavel Stehule pavel.steh...@gmail.com: here is a new version - new these functions are not a strict and function to_string is marked as stable. We have array_to_string(anyarray, text) and string_to_array(text, text), and you'll introduce to_string(anyarray, text, text) and to_array(text, text, text). I have to repeat it, the behave of this functions are little bit different. string_to_array and array_to_string are buggy. * it isn't support a NULL * it doesn't differentiate a empty array and NULL * we cannot to change default behave of existing functions * array_to_string is badly marked as IMMUTABLE This email thread linked to from our TODO list explains that arrays combined with NULLs have many inconsistenciess: http://archives.postgresql.org/pgsql-bugs/2008-11/msg9.php -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Universal B-tree
On Mon, Aug 9, 2010 at 5:31 PM, Daniel Oliveira danielmarquesolive...@gmail.com wrote: I don't need to change B-tree estructure. I just need integrate my encode function that transforms multiple keys into one key by bit-interleaving and to acess elements given several intervals (range search). You could build a expression index on the function which will build a regular btree. Then any range clause on a similar expression in the query would be able to use the index. Postgres is capable of detecting multiple range clauses that can use one or more indexes and combine the results. It might not be perfect for your use case, being a general purpose tool. I'm not sure how feasible it would be to implement a special purpose case that matches your needs. Perhaps a special index type and index access method similar to gist tsquery data type. But that's a lot more work and a lot more C coding. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
Apparently, the message I sent (quoted below) didn't make it to -hackers. I know that Pavel received the message, as he replied to it. I'm calling shenanigans on the mailing list server, but in the meantime, here are those diffs again. On 31 July 2010 07:37, Brendan Jurd dire...@gmail.com wrote: Hi Pavel, I've reviewed your latest patch (which I refer to as v3 to keep continuity with previous versions under the to_array naming system). You didn't quite complete the rename of the functions; in-code comments and regression tests still referred to the old names. I cleanup that up for you and also reworded some of the in-code comments for clarity. Otherwise the patch looks good and the functions now work exactly as I would expect. I also went ahead and added some more documentation to explain how (im|ex)plode differ from their foo_to_bar counterparts, and what kind of behaviour you'll get by specifying the arguments as NULL. I have attached v4 of the patch against HEAD, and also an incremental patch showing just my changes against v3. I'll mark this as ready for committer. Cheers, BJ implode_v4.diff.gz Description: GNU Zip compressed data implode_v3-to-4.diff.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] knngist - 0.8
In gist consitent method support only filtering strategies. For such strategies consistent method returns true if subtree can contain matching node and false otherwise. Knngist introduce also order by strategies. For filtering strategies knngist consistent method returns 0 if subtree can contain matching node and -1 otherwise. For order by strategies knngist consistent method returns minimal possible distance in subtree. I think we can use consistent method with order by strategies not only for ordering but also for filtering. If query contain assertion that distance is less than some value, than we can call consistent method with order by strategy and compare result with query value in order to determine whether scan subtree. Such approach can give benefit when we need to filter by similarity. For example, in pg_trgm % is used for similarity filtering, but similarity threshold is global for session. That's why we can't create complex queries which contain similarity filtering with different threshold. With best regards, Alexander Korotkov. On Mon, Aug 2, 2010 at 8:14 PM, Robert Haas robertmh...@gmail.com wrote: 2010/7/29 Alexander Korotkov aekorot...@gmail.com: But, in pg_trgm it makes it possible to combine different similarity levels in one query. For example: select * from test_trgm order by t - 'asdf' 0.5 or t - 'qwer' 0.4; Is there any chance to handle this syntax also? Maybe I'm missing something, but I don't think that ORDER BY clause makes much sense. OR is going to reduce a true or false value - and it's usually not that interesting to order by a column that can only take one of two values. Am I confused? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company
Re: [HACKERS] Universal B-tree
For research purpose, I think that expression index is a good idea. I just want to do a proof of concept. The other issue is that my algorithm break a z-order interval into several intervals that represents the query box. How should I create it without creating any overhead? Best regards, daniel 2010/8/9 Greg Stark gsst...@mit.edu On Mon, Aug 9, 2010 at 5:31 PM, Daniel Oliveira danielmarquesolive...@gmail.com wrote: I don't need to change B-tree estructure. I just need integrate my encode function that transforms multiple keys into one key by bit-interleaving and to acess elements given several intervals (range search). You could build a expression index on the function which will build a regular btree. Then any range clause on a similar expression in the query would be able to use the index. Postgres is capable of detecting multiple range clauses that can use one or more indexes and combine the results. It might not be perfect for your use case, being a general purpose tool. I'm not sure how feasible it would be to implement a special purpose case that matches your needs. Perhaps a special index type and index access method similar to gist tsquery data type. But that's a lot more work and a lot more C coding. -- greg
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, 2010-08-09 at 11:41 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: So imagine that thread-or-process A allocates allocates a new chunk of memory and then writes a pointer to the new chunk in a previously allocated section of memory. Thread-or-process B then follows the pointer. In a threaded model, this is guaranteed to be safe. In a process model, it's not: A might have enlarged the shared memory mapping while B has not yet done so. So I think in our model any sort of change to the shared memory segment is going to require extremely careful gymnastics, and be pretty expensive. ... and on some platforms, it'll be flat out impossible. We looked at this years ago and concluded that changing the size of the shmem segment after postmaster start was impractical from a portability standpoint. I have not seen anything to change that conclusion. As caches get larger, downtime gets longer. Downtime of more than a few minutes per year is enough to blow claims of high availability. At some point, this project will need to face this particular hurdle. We may need to balance utility for the majority against portability for the minority. We should be laying out an architectural roadmap, not just saying no. We can make multi-year plans if we wish to. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Surprising dead_tuple_count from pgstattuple
On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote: So, I guess my real question here is, what happened to the missing 100 items? If it was HOT prune, can anyone summarize what that does? Itagaki already explained that the second DELETE would have removed the 100 dead rows you consider to be missing. Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
On Mon, Aug 9, 2010 at 11:24 AM, Pavel Stehule pavel.steh...@gmail.com wrote: They name to be type_func_keywords, perhaps, but not fully reserved. And they'd still need that treatment anyway. Even if cube(whatever) can't mean extract a column called cube from table whatever, it can still mean call a function called cube on a column called whatever. look to gram.y, please. we can use a GROUP BY CUBE(expr, ..) GROUP BY func_name(expr, ..) so these rules are in conflict, because func_name can have a type_func_keywords symbols. So we have to significantly rewrite a rules about func call or CUBE and ROLLUP have to be a reserved words. There isn't any other possibility. I understand that you have to make CUBE and ROLLUP reserved words. But you would still have to do that even if we changed $SUBJECT. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is syntax columname(tablename) necessary still?
2010/8/9 Robert Haas robertmh...@gmail.com: On Mon, Aug 9, 2010 at 11:24 AM, Pavel Stehule pavel.steh...@gmail.com wrote: They name to be type_func_keywords, perhaps, but not fully reserved. And they'd still need that treatment anyway. Even if cube(whatever) can't mean extract a column called cube from table whatever, it can still mean call a function called cube on a column called whatever. look to gram.y, please. we can use a GROUP BY CUBE(expr, ..) GROUP BY func_name(expr, ..) so these rules are in conflict, because func_name can have a type_func_keywords symbols. So we have to significantly rewrite a rules about func call or CUBE and ROLLUP have to be a reserved words. There isn't any other possibility. I understand that you have to make CUBE and ROLLUP reserved words. But you would still have to do that even if we changed $SUBJECT. I am not sure if I understand well. yes - CUBE and ROLLUP have to be reserved keywords - and I don't calculate with removing a obsolete syntax now. Regards Pavel -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Hi, On 08/09/2010 05:41 PM, Tom Lane wrote: ... and on some platforms, it'll be flat out impossible. We looked at this years ago and concluded that changing the size of the shmem segment after postmaster start was impractical from a portability standpoint. I have not seen anything to change that conclusion. I haven't tried, but I tend to believe that's true. However, I'd like to get back to the original intent of the posted patch. Which is about dynamically allocating memory *within a fixed size pool*. That's something SRLU or shared_buffers do to some extent, but with lots of limitations. And without the ability to move free memory between sub-systems (i.e. between different SLRU buffers). You meant I don't think, right? I agree. The only way threads would simplify this is if we went over to a mysql-style model where there was only one process, period, and all backends were threads inside that. No shared memory as such, at all. That's how the threaded model normally is used, yes. And with that model, allocation of shared memory is very easy. It has none of the pre-allocation requirements we are currently facing. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Hi, On 08/09/2010 06:10 PM, Bruce Momjian wrote: My point is that you can treat malloc the same as add shared memory, to some extent, with the same limiations. Once one of the SLRU buffers is full, it cannot currently allocate from another SLRU buffer's unused memory area. That memory there is plain wasted at that moment. That's my point and the problem the allocator I posted tries to solve. I fail to see how malloc could help here. malloc() only allocates process-local memory. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: So imagine that thread-or-process A allocates allocates a new chunk of memory and then writes a pointer to the new chunk in a previously allocated section of memory. Thread-or-process B then follows the pointer. In a threaded model, this is guaranteed to be safe. In a process model, it's not: A might have enlarged the shared memory mapping while B has not yet done so. So I think in our model any sort of change to the shared memory segment is going to require extremely careful gymnastics, and be pretty expensive. ... and on some platforms, it'll be flat out impossible. We looked at this years ago and concluded that changing the size of the shmem segment after postmaster start was impractical from a portability standpoint. I have not seen anything to change that conclusion. I haven't done extensive research into this, but I did take a look at it briefly. It looked to me like the style of shared memory we're using now (I guess it's System V) has no way to resize a shared memory segment at all, and certainly no way that's portable. However it also looked as though POSIX shm (shm_open, etc.) can be resized using ftruncate(). Whether this is portable to all the platforms we run on, or whether the behavior of ftruncate() in combination with shm_open() is in the standard, I'm not sure. I believe I went back and reread the old threads on this topic and it seems like the sticking point as far as POSIX shm goes it that it lacks a readable equivalent of shm_nattch. I think it was proposed to use a small syv shm and then do the main shared memory arena with shm_open, but at that point you start to wonder you're messing around with at all. But I can't help but be intrigued by it, even so. Suppose, for example, that we kept things that were really fixed-size in shared memory but moved, say, shared_buffers to a POSIX shm. Would that allow you to then make shared_buffers PGC_SIGHUP? The obvious answer is no, because there are a whole bunch of knock-on issues. Changing the size of shared_buffers also means changing the number of LWLocks, changing the number of buffer descriptors, etc. So maybe it can't be done. But I can't stop wondering if there's a way to make it work... I don't care to take a position in the religious war over threads vs. processes, but I do think threads simplify the handling of this particular case. You meant I don't think, right? I agree. The only way threads would simplify this is if we went over to a mysql-style model where there was only one process, period, and all backends were threads inside that. No shared memory as such, at all. I think we're saying the same thing in different ways; I agree with everything in that paragraph that follows the question mark. By this particular case, I meant shared memory allocation; it would amount to just calling malloc() [or palloc()]. But yeah, clearly that only works in a single-process model. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 12:03 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian br...@momjian.us wrote: I am not sure threads would greatly help us. ?The major problem is that all of our our structures are currently contiguous in memory for quick access. ?I don't see how threading would help with that. ?We could use realloc(), but we can do the same in shared memory if we had a chunk infrastructure, though concurrent access to that memory would hurt us in either threads or shared memory. Fundamentally, recreating the libc memory allocation routines is not that hard. ?(Everyone has to detach from the shared memory segment, but they have to stop using it too, so it doesn't seem that hard.) I actually don't think that's true. The advantage (and disadvantage) of using threads is that everything runs in one address space. So you just allocate more memory and everyone immediately sees it. In a process environment, that's not the case: to expand or shrink the size of the shared memory arena, everyone needs to explicitly change their own mapping. You can't expand the size of malloc'ed memory --- you have to call realloc(), and then you effectively get a new pointer. Shared memory has a similar limitation. If you allocate shared memory in chunks so you don't need to change the location, you are effectively doing another malloc(), like you would in a threaded process. The point isn't what happens when you resize individual chunks; it's what happens when you need to expand the arena. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 12:31 PM, Bruce Momjian br...@momjian.us wrote: Bruce Momjian wrote: With our process-based design, the default is private memory (i.e. not shared). If you need shared memory, you must specify a certain amount in advance. That chunk of shared memory then is reserved and can't ever be used by another subsystem. Even if you barely ever need that much shared memory for the subsystem in question. Once multiple threads are using the same local memory, you have the same issues of being unable to resize it because repalloc can change the pointer location. Let me be more concrete. Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. How do you do that? If you want it contiguous, you have to use realloc, which might move the pointer. If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. You probably wouldn't do either of those things. You'd just allocate small chunks here and there for whatever you need them for. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Surprising dead_tuple_count from pgstattuple
On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs si...@2ndquadrant.com wrote: On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote: So, I guess my real question here is, what happened to the missing 100 items? If it was HOT prune, can anyone summarize what that does? Itagaki already explained that the second DELETE would have removed the 100 dead rows you consider to be missing. Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. Where does heap_page_prune() get called from in the DELETE path? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Surprising dead_tuple_count from pgstattuple
On 09/08/10 21:21, Robert Haas wrote: On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggssi...@2ndquadrant.com wrote: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. Where does heap_page_prune() get called from in the DELETE path? heapgetpage() -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Surprising dead_tuple_count from pgstattuple
On Mon, Aug 9, 2010 at 2:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 09/08/10 21:21, Robert Haas wrote: On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggssi...@2ndquadrant.com wrote: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. Where does heap_page_prune() get called from in the DELETE path? heapgetpage() Ah, OK. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Hi, On 08/09/2010 06:31 PM, Bruce Momjian wrote: Let me be more concrete. Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. How do you do that? There's absolutely no need to pre-allocate 20 MB in advance in a threaded environment. You just allocate memory in small chunks. For a threaded-model, that memory is shared by default, so the total amount of shared memory can grow and shrink very easily. (And even makes usused memory available to other processes, not just other threads). If you want it contiguous, you have to use realloc, which might move the pointer. If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. Okay, I think I now understand the requirement of continuity you mentioned earlier already. I agree that with the current approach, we cannot simply use such a dynamic allocator to solve all of our problems. Every subsystem would need to be converted to something that allocates shared memory in smaller chunks for such a dynamic allocator to be of any use. Robert already pointed out that this may be troublesome for shared_buffers, which is by far the largest consumer of shared memory. I didn't look into this, yet. And I'd like to hear more about the feasibility of that approach for other subsystems. Another issue to be discussed would be the limits of sharing free memory between subsystems. Maybe we even reach the conclusion that we absolutely *want* fixed maximum sizes for every single subsystem so as to be able to guarantee a certain amount of multi-xact or SLRU entries at any point in time (otherwise one memory hungry subsystem could possibly eat it all up with another subsystem getting the OOM error when trying to allocate for its very first entry). Thanks for bringing this discussion to live again. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] knngist - 0.8
On Sun, Aug 8, 2010 at 4:28 PM, Alexander Korotkov aekorot...@gmail.com wrote: In gist consitent method support only filtering strategies. For such strategies consistent method returns true if subtree can contain matching node and false otherwise. Knngist introduce also order by strategies. For filtering strategies knngist consistent method returns 0 if subtree can contain matching node and -1 otherwise. For order by strategies knngist consistent method returns minimal possible distance in subtree. I think we can use consistent method with order by strategies not only for ordering but also for filtering. If query contain assertion that distance is less than some value, than we can call consistent method with order by strategy and compare result with query value in order to determine whether scan subtree. I am not an expert on this code, but after thinking this over, I believe you are correct and that this is a good point. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Markus Wanner wrote: Hi, On 08/09/2010 06:10 PM, Bruce Momjian wrote: My point is that you can treat malloc the same as add shared memory, to some extent, with the same limiations. Once one of the SLRU buffers is full, it cannot currently allocate from another SLRU buffer's unused memory area. That memory there is plain wasted at that moment. That's my point and the problem the allocator I posted tries to solve. I fail to see how malloc could help here. malloc() only allocates process-local memory. My point is that we have the same limitations with malloc()/threads, as we have with shared memory. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas wrote: On Mon, Aug 9, 2010 at 12:31 PM, Bruce Momjian br...@momjian.us wrote: Bruce Momjian wrote: With our process-based design, the default is private memory (i.e. not shared). If you need shared memory, you must specify a certain amount in advance. That chunk of shared memory then is reserved and can't ever be used by another subsystem. Even if you barely ever need that much shared memory for the subsystem in question. Once multiple threads are using the same local memory, you have the same issues of being unable to resize it because repalloc can change the pointer location. Let me be more concrete. ?Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. ?How do you do that? ?If you want it contiguous, you have to use realloc, which might move the pointer. ?If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. You probably wouldn't do either of those things. You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Markus Wanner wrote: Hi, On 08/09/2010 06:31 PM, Bruce Momjian wrote: Let me be more concrete. Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. How do you do that? There's absolutely no need to pre-allocate 20 MB in advance in a threaded environment. You just allocate memory in small chunks. For a threaded-model, that memory is shared by default, so the total amount of shared memory can grow and shrink very easily. (And even makes usused memory available to other processes, not just other threads). If you want it contiguous, you have to use realloc, which might move the pointer. If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. Okay, I think I now understand the requirement of continuity you mentioned earlier already. I agree that with the current approach, we cannot simply use such a dynamic allocator to solve all of our problems. Every subsystem would need to be converted to something that allocates shared memory in smaller chunks for such a dynamic allocator to be of any use. Robert already pointed out that this may be troublesome for shared_buffers, which is by far the largest consumer of shared memory. I didn't look into this, yet. And I'd like to hear more about the feasibility of that approach for other subsystems. Another issue to be discussed would be the limits of sharing free memory between subsystems. Maybe we even reach the conclusion that we absolutely *want* fixed maximum sizes for every single subsystem so as to be able to guarantee a certain amount of multi-xact or SLRU entries at any point in time (otherwise one memory hungry subsystem could possibly eat it all up with another subsystem getting the OOM error when trying to allocate for its very first entry). Yep, you would have to use chunks in threads/malloc, and you have to do the same thing with shared memory. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On 08/09/2010 08:33 PM, Bruce Momjian wrote: Robert Haas wrote: You probably wouldn't do either of those things. You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. That's what my patch allows you to do, yes. Currently you are bound to pre-allocate shared memory at startup. Or how would you allocate small chunks from shared memory at the moment? Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 2:28 PM, Markus Wanner mar...@bluegap.ch wrote: Another issue to be discussed would be the limits of sharing free memory between subsystems. Maybe we even reach the conclusion that we absolutely *want* fixed maximum sizes for every single subsystem so as to be able to guarantee a certain amount of multi-xact or SLRU entries at any point in time (otherwise one memory hungry subsystem could possibly eat it all up with another subsystem getting the OOM error when trying to allocate for its very first entry). Yeah, I think that's a real concern. I think we need to distinguish memory needs from memory wants. Ideally, we'd like our entire database to be cached in RAM. But that may or may not be feasible, so we page what we can into shared_buffers and page out as necessary to make room for other things. In contrast, the traditional malloc() approach doesn't give you much flexibility: if it returns NULL, you pretty much have to fail whatever operation you were trying to perform. For some things, that's OK. For other things, it's not. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] host name support in pg_hba.conf
Here is a patch for host name support in pg_hba.conf. I have reviewed various past threads about this, and there appeared to have been a 50/50 split of for and against reverse lookup. I went with the reverse lookup, because 0) I like it. 1) It is more secure. 2) It allows extending it to wildcards in the future. 3) Apache (Allow from) does it that way. To clarify how it works: The client's IP address (known from the kernel) is reverse looked up, which results in a host name. That host name is compared with the line in pg_hba.conf. If it matches, a forward lookup is performed on the host name to check if any of the resulting IP addresses match the client's IP address. If yes, the line is considered to match and the authentication method is selected. Anyway, assuming we will go with this, you will also notice that in the patch I changed the default pg_hba.conf to match against localhost instead of numeric addresses. Initially thought of as a temporary change for testing this patch, I think this might actually have some permanent value because it saves you from having to change the IPv4 and IPv6 lines in tandem most of the times, which is a moderately common mistake. We already rely on localhost being (forward) resolvable for the stats collector. Something to think about: Maybe we need a quoting mechanism in case someone names their hosts samenet. diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index e8995f9..808c468 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -101,9 +101,9 @@ A record can have one of the seven formats synopsis local replaceabledatabase/replaceable replaceableuser/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional -host replaceabledatabase/replaceable replaceableuser/replaceable replaceableCIDR-address/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional -hostsslreplaceabledatabase/replaceable replaceableuser/replaceable replaceableCIDR-address/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional -hostnossl replaceabledatabase/replaceable replaceableuser/replaceable replaceableCIDR-address/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional +host replaceabledatabase/replaceable replaceableuser/replaceable replaceableaddress/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional +hostsslreplaceabledatabase/replaceable replaceableuser/replaceable replaceableaddress/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional +hostnossl replaceabledatabase/replaceable replaceableuser/replaceable replaceableaddress/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional host replaceabledatabase/replaceable replaceableuser/replaceable replaceableIP-address/replaceable replaceableIP-mask/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional hostsslreplaceabledatabase/replaceable replaceableuser/replaceable replaceableIP-address/replaceable replaceableIP-mask/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional hostnossl replaceabledatabase/replaceable replaceableuser/replaceable replaceableIP-address/replaceable replaceableIP-mask/replaceable replaceableauth-method/replaceable optionalreplaceableauth-options/replaceable/optional @@ -218,12 +218,17 @@ hostnossl replaceabledatabase/replaceable replaceableuser/replaceable /varlistentry varlistentry - termreplaceableCIDR-address/replaceable/term + termreplaceableaddress/replaceable/term listitem para Specifies the client machine IP address range that this record - matches. This field contains an IP address in standard dotted decimal - notation and a acronymCIDR/ mask length. (IP addresses can only be + matches. This field can contain either a host name, an IP + address range, one of the special key words mentioned below. + /para + + para + An IP address is specified in standard dotted decimal + notation with a acronymCIDR/ mask length. (IP addresses can only be specified numerically, not as domain or host names.) The mask length indicates the number of high-order bits of the client IP address that must match. Bits to the right of this must @@ -233,14 +238,7 @@ hostnossl replaceabledatabase/replaceable replaceableuser/replaceable /para para - Instead of a replaceableCIDR-address/replaceable, you can write - literalsamehost/literal to match any of the server's own IP - addresses, or literalsamenet/literal to match any address in any -
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian br...@momjian.us wrote: Let me be more concrete. ?Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. ?How do you do that? ?If you want it contiguous, you have to use realloc, which might move the pointer. ?If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. You probably wouldn't do either of those things. You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. Well, I think your point is wrong, then. :-) It's not the same at all. If you have a bunch of threads in one address space, shared memory is really just process-local. You can grow the total amount of allocated space just by calling malloc(). With our architecture, you can't. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [JDBC] Trouble with COPY IN
On Sat, 7 Aug 2010, Kris Jurka wrote:
On Fri, 6 Aug 2010, James William Pye wrote:
I think there's a snag in the patch:
postgres=# COPY data FROM '/Users/jwp/DATA.bcopy' WITH BINARY;
ERROR: row field count is -1, expected 1
CONTEXT: COPY data, line 4
Probably a quick/small fix away, I imagine.
Hmm, not quite sure why that is. That seems to imply that it's not using V3
protocol, but I thought binary copy could only be used with the V3 protocol.
In any case, I think this new patch is more bulletproof.
Oh, duh. It's a server side copy not going through the client at all.
Here's a hopefully final patch.
Kris Jurka*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
***
*** 2058,2069 CopyFrom(CopyState cstate)
int16 fld_count;
ListCell *cur;
! if (!CopyGetInt16(cstate, fld_count) ||
! fld_count == -1)
{
done = true;
break;
}
if (fld_count != attr_count)
ereport(ERROR,
--- 2058,2088
int16 fld_count;
ListCell *cur;
! if (!CopyGetInt16(cstate, fld_count))
{
done = true;
break;
}
+
+ if (fld_count == -1)
+ {
+ /*
+* Received EOF marker. Check to see if this
is really
+* the EOF and complain if we find any more
data.
+* In a V3 protocol copy, this also enforces
that we wait
+* for the protocol end of copy (CopyDone/Fail).
+*/
+ int8 unused;
+ if (CopyGetData(cstate, unused,
sizeof(unused), sizeof(unused)))
+ {
+ ereport(ERROR,
+
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+errmsg(received copy
data after EOF marker)));
+ }
+
+ done = true;
+ break;
+ }
if (fld_count != attr_count)
ereport(ERROR,
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Markus Wanner wrote: On 08/09/2010 08:33 PM, Bruce Momjian wrote: Robert Haas wrote: You probably wouldn't do either of those things. You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. That's what my patch allows you to do, yes. Currently you are bound to pre-allocate shared memory at startup. Or how would you allocate small chunks from shared memory at the moment? We don't --- we allocate it all at startup. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas wrote: On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian br...@momjian.us wrote: Let me be more concrete. ?Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. ?How do you do that? ?If you want it contiguous, you have to use realloc, which might move the pointer. ?If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. You probably wouldn't do either of those things. ?You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. Well, I think your point is wrong, then. :-) It's not the same at all. If you have a bunch of threads in one address space, shared memory is really just process-local. You can grow the total amount of allocated space just by calling malloc(). With our architecture, you can't. You effectively have to add infrastructure to add/remove shared memory segments to match memory requests. It is another step, but it is the same behavior. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] host name support in pg_hba.conf
Peter Eisentraut pete...@gmx.net wrote: The client's IP address (known from the kernel) Some machines have several IP addresses; how is that handled? is reverse looked up, which results in a host name. Some IP addresses have several host names, including in reverse lookup; how is that handled? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] host name support in pg_hba.conf
On 9 August 2010 19:47, Peter Eisentraut pete...@gmx.net wrote: Here is a patch for host name support in pg_hba.conf. I have reviewed various past threads about this, and there appeared to have been a 50/50 split of for and against reverse lookup. I went with the reverse lookup, because 0) I like it. 1) It is more secure. 2) It allows extending it to wildcards in the future. 3) Apache (Allow from) does it that way. To clarify how it works: The client's IP address (known from the kernel) is reverse looked up, which results in a host name. That host name is compared with the line in pg_hba.conf. If it matches, a forward lookup is performed on the host name to check if any of the resulting IP addresses match the client's IP address. If yes, the line is considered to match and the authentication method is selected. Anyway, assuming we will go with this, you will also notice that in the patch I changed the default pg_hba.conf to match against localhost instead of numeric addresses. Initially thought of as a temporary change for testing this patch, I think this might actually have some permanent value because it saves you from having to change the IPv4 and IPv6 lines in tandem most of the times, which is a moderately common mistake. We already rely on localhost being (forward) resolvable for the stats collector. Something to think about: Maybe we need a quoting mechanism in case someone names their hosts samenet. -- A couple things: + matches. This field can contain either a host name, an IP + address range, one of the special key words mentioned below. + /para s/, one/, or one/ + If a host name is specified (anything that is not an IP address + or a special key word is processed as a potential host name), a + reverse DNS lookup is performed on the client's IP address, + then a forward DNS lookup on the resulting name to check if it + matches the original IP address (that is, at least one of the + potentially many IP addresses matches the original one), and + the name found in the reverse lookup is compared with the + specified host name. That's one lng sentence! -- Thom Brown Registered Linux user: #516935 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On 08/09/2010 08:49 PM, Bruce Momjian wrote: Markus Wanner wrote: That's what my patch allows you to do, yes. Currently you are bound to pre-allocate shared memory at startup. Or how would you allocate small chunks from shared memory at the moment? We don't --- we allocate it all at startup. Exactly. And that's the difference to a thread-based approach. The downside of it is that you need to know in advance how much shared memory each of the subsystems is going to need. On the upside is the certainty, that you already have the memory allocated and cannot run out of it. You just have what you have. (Note that you could do that as well with the thread-based approach, if you want. Most other programs I know don't choose that approach, though, but instead try to cope with OOM). Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 2:50 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian br...@momjian.us wrote: Let me be more concrete. ?Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. ?How do you do that? ?If you want it contiguous, you have to use realloc, which might move the pointer. ?If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. You probably wouldn't do either of those things. ?You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. Well, I think your point is wrong, then. :-) It's not the same at all. If you have a bunch of threads in one address space, shared memory is really just process-local. You can grow the total amount of allocated space just by calling malloc(). With our architecture, you can't. You effectively have to add infrastructure to add/remove shared memory segments to match memory requests. It is another step, but it is the same behavior. That would be one way to tackle the problem, but there are difficulties. If we just created new shared memory segments at need, we might end up with a lot of shared memory segments. I suspect that would get complicated and present many management difficulties - which is why I'm so far of the opinion that we should try to architect the system to avoid the need for this functionality. I don't think it's going to be too easy to provide, short of (as Tom says) moving to the MySQL model of many threads working in a single process. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas wrote: On Mon, Aug 9, 2010 at 2:50 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian br...@momjian.us wrote: Let me be more concrete. ?Suppose you are using threads, and you want to increase your shared memory from 20MB to 30MB. ?How do you do that? ?If you want it contiguous, you have to use realloc, which might move the pointer. ?If you allocate another 10MB chunk, you then have shared memory fragments, which is the same as adding another shared memory segment. You probably wouldn't do either of those things. ?You'd just allocate small chunks here and there for whatever you need them for. Well, then we do that with shared memory then --- my point is that it is the same problem with threads or processes. Well, I think your point is wrong, then. ?:-) It's not the same at all. ?If you have a bunch of threads in one address space, shared memory is really just process-local. ?You can grow the total amount of allocated space just by calling malloc(). With our architecture, you can't. You effectively have to add infrastructure to add/remove shared memory segments to match memory requests. ?It is another step, but it is the same behavior. That would be one way to tackle the problem, but there are difficulties. If we just created new shared memory segments at need, we might end up with a lot of shared memory segments. I suspect that would get complicated and present many management difficulties - which is why I'm so far of the opinion that we should try to architect the system to avoid the need for this functionality. I don't think it's going to be too easy to provide, short of (as Tom says) moving to the MySQL model of many threads working in a single process. You could allocate shared memory in chunks and then pass that out to requestors, the same way sbrk() does it. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On 08/09/2010 08:50 PM, Bruce Momjian wrote: You effectively have to add infrastructure to add/remove shared memory segments to match memory requests. It is another step, but it is the same behavior. That's of no use without a dynamic allocator, I think. Or else it is a vague description of a dynamic allocator. I'm approaching the problem from another perspective: trying to implement a dynamic allocator on top of a fixed size memory pool, first. Once we have that, we may start to think about dynamically adding or removing underlying segments. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] host name support in pg_hba.conf
On Mon, Aug 9, 2010 at 2:47 PM, Peter Eisentraut pete...@gmx.net wrote: Here is a patch for host name support in pg_hba.conf. I have reviewed various past threads about this, and there appeared to have been a 50/50 split of for and against reverse lookup. I went with the reverse lookup, because 0) I like it. 1) It is more secure. 2) It allows extending it to wildcards in the future. 3) Apache (Allow from) does it that way. To clarify how it works: The client's IP address (known from the kernel) is reverse looked up, which results in a host name. That host name is compared with the line in pg_hba.conf. If it matches, a forward lookup is performed on the host name to check if any of the resulting IP addresses match the client's IP address. If yes, the line is considered to match and the authentication method is selected. Seems reasonable. Anyway, assuming we will go with this, you will also notice that in the patch I changed the default pg_hba.conf to match against localhost instead of numeric addresses. Initially thought of as a temporary change for testing this patch, I think this might actually have some permanent value because it saves you from having to change the IPv4 and IPv6 lines in tandem most of the times, which is a moderately common mistake. We already rely on localhost being (forward) resolvable for the stats collector. -1 from me, on this part. I think we should be trying to eliminate any dependencies we have on how localhost resolves, and we certainly should not add more. I have spent way too much time fighting with problems caused by localhost being present/absent in /etc/hosts; and more caused by whether it maps to 127.0.0.1 or some IP assigned to one of the boxes' Ethernet cards. What's really special is when you have two parts of the system, one of which will only work with one version of /etc/hosts and the other of which will only work with a different version. I humbly, but fervently, suggest that we try to avoid being polemical about this. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On 08/09/2010 09:00 PM, Bruce Momjian wrote: You could allocate shared memory in chunks and then pass that out to requestors, the same way sbrk() does it. sbrk() is described [1] as a low-level memory allocator, which is typically only used by the high-level malloc memory allocator implemented in the C library. Think of my patch as the high(er)-level variant ;-) It's certainly doable using processes and shared memory. Yes. My patch shows one way of how to go a step into that direction. Regards Markus Wanner [1]: http://www.cs.utah.edu/flux/moss/node39.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 3:00 PM, Bruce Momjian br...@momjian.us wrote: That would be one way to tackle the problem, but there are difficulties. If we just created new shared memory segments at need, we might end up with a lot of shared memory segments. I suspect that would get complicated and present many management difficulties - which is why I'm so far of the opinion that we should try to architect the system to avoid the need for this functionality. I don't think it's going to be too easy to provide, short of (as Tom says) moving to the MySQL model of many threads working in a single process. You could allocate shared memory in chunks and then pass that out to requestors, the same way sbrk() does it. Sure. But I don't think that gets you very far. The management of the chunks is really hard. I go back to my previous example: you can't store a pointer that might point to another chunk, because the chunks won't get mapped into all the address spaces synchronously. Even if you don't care about doing that (and I bet you do), mapping and unmapping chunks is a heavyweight operation that requires every backend to notice that it needs to do something (and, incidentally, if any of them fail, you pretty much have to PANIC). I just can't imagine us building a reliable system this way. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Hi, On 08/09/2010 08:45 PM, Robert Haas wrote: Yeah, I think that's a real concern. I think we need to distinguish memory needs from memory wants. Ideally, we'd like our entire database to be cached in RAM. But that may or may not be feasible, so we page what we can into shared_buffers and page out as necessary to make room for other things. In contrast, the traditional malloc() approach doesn't give you much flexibility: if it returns NULL, you pretty much have to fail whatever operation you were trying to perform. For some things, that's OK. For other things, it's not. Agreed, it's going to be a difficult compromise and it possibly is very hard to find a good one automatically. However, I doubt our current approach with hard limits between subsystems is the best compromise. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Markus Wanner mar...@bluegap.ch writes: However, I'd like to get back to the original intent of the posted patch. Which is about dynamically allocating memory *within a fixed size pool*. That's something SRLU or shared_buffers do to some extent, but with lots of limitations. And without the ability to move free memory between sub-systems (i.e. between different SLRU buffers). As far as SLRU is concerned, the already-agreed-to plan is to get rid of the separate arenas for SLRU and merge those things into the main shared buffers arena. IIRC, the motivation for designing SLRU the way it is was to ensure that SLRU uses couldn't be starved for memory due to high demand for shared buffers. But that was back when people frequently ran PG with only a few meg for shared buffers; I think that worry is obsolete. So I don't see this patch as offering anything at all that we care about so far as the core server is concerned. Maybe there are extensions that need it badly enough to justify such a feature in core, but SLRU is not a good argument for it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas robertmh...@gmail.com writes: On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: ... and on some platforms, it'll be flat out impossible. We looked at this years ago and concluded that changing the size of the shmem segment after postmaster start was impractical from a portability standpoint. I have not seen anything to change that conclusion. I haven't done extensive research into this, but I did take a look at it briefly. It looked to me like the style of shared memory we're using now (I guess it's System V) has no way to resize a shared memory segment at all, and certainly no way that's portable. However it also looked as though POSIX shm (shm_open, etc.) can be resized using ftruncate(). Whether this is portable to all the platforms we run on, or whether the behavior of ftruncate() in combination with shm_open() is in the standard, I'm not sure. It's not portable. That's exactly what we were looking into back when. I believe I went back and reread the old threads on this topic and it seems like the sticking point as far as POSIX shm goes it that it lacks a readable equivalent of shm_nattch. Yeah, that was another little problem. In principle though we only need one SysV-style shmem segment to get the required interlock, and there could be add-on shmem segments using POSIX or other APIs. But that doesn't get you out from under the portability issue or the memory space management issue (it's unlikely you can enlarge a segment without remapping it). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] host name support in pg_hba.conf
Peter Eisentraut pete...@gmx.net writes: Here is a patch for host name support in pg_hba.conf. My recollection is that the previous discussions got stuck on the cost of doing DNS lookups for every connect; and the obvious solution of trying to cache the names was shot down on the basis of not knowing when to flush the cache. Have you decided that people who want this feature will just have to pay that cost? If so, I think the documentation needs to be a bit more explicit about names being more expensive than IP addresses in pg_hba.conf. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Hi, On 08/09/2010 09:14 PM, Tom Lane wrote: As far as SLRU is concerned, the already-agreed-to plan is to get rid of the separate arenas for SLRU and merge those things into the main shared buffers arena. I didn't know about that plan. Sounds good. (I'm personally thinking this is trying to solve the same problem in a more specific fashion). IIRC, the motivation for designing SLRU the way it is was to ensure that SLRU uses couldn't be starved for memory due to high demand for shared buffers. But that was back when people frequently ran PG with only a few meg for shared buffers; I think that worry is obsolete. Good to know. So I don't see this patch as offering anything at all that we care about so far as the core server is concerned. Maybe there are extensions that need it badly enough to justify such a feature in core, but SLRU is not a good argument for it. Fair enough. (Patch is already marked as returned with feedback on the commitfest app, thanks again for additional feedback) Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] host name support in pg_hba.conf
* Kevin Grittner (kevin.gritt...@wicourts.gov) wrote: The client's IP address (known from the kernel) Some machines have several IP addresses; how is that handled? Sounds like he already described it, or I read it wrong. The fact that some machines have several IP addresses hardly matters- whatever IP is used to connect to PG is what gets the reverse DNS lookup. That then returns a host. That host is then looked up, and as long as *one* of the IPs associated with that host matches the IP of the connector, it's good to go. is reverse looked up, which results in a host name. Some IP addresses have several host names, including in reverse lookup; how is that handled? Yea... That's just busted, imnsho. But then, that's probably because it breaks Kerberos too. :) Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] security label support, part.2
2010/7/26 KaiGai Kohei kai...@ak.jp.nec.com: The attached patches are revised ones, as follows. I think this is pretty good, and I'm generally in favor of committing it. Some concerns: 1. Since nobody has violently objected to the comment.c refactoring patch I recently proposed, I'm hopeful that can go in. And if that's the case, then I'd prefer to see that committed first, and then rework this to use that code. That would eliminate some code here, and it would also make it much easier to support labels on other types of objects. 2. Some of this code refers to local security labels. I'm not sure what's local about them - aren't they just security labels? On a related note, I don't like the trivial wrappers you have here, with DeleteSecurityLabel around DeleteLocalSecLabel, SetSecurityLabel around SetLocalSecLabel, etc. Just collapse these into a single set of functions. 3. Is it really appropriate for ExecRelationSecLabel() to have an Exec prefix? I don't think so. 4. Please get rid of the nkeys++ stuff in DeleteLocalSecLabel() and just use fixed offsets as we do everywhere else. 5. Why do we think that the relabel hook needs to be passed the number of expected parents? 6. What are we doing about the assignment of initial security labels? I had initially thought that perhaps new objects would just start out unlabeled, and the user would be responsible for labeling them as needed. But maybe we can do better. Perhaps we should extend the security provider hook API with a function that gets called when a labellable object gets created, and let each loaded security provider return any label it would like attached. Even if we don't do that now, esp_relabel_hook_entry needs to be renamed to something more generic; we will certainly want to add more fields to that structure later. 7. I think we need to write and include in the fine documentation some big picture documentation about enhanced security providers. Of course, we have to decide what we want to say. But the SECURITY LABEL documentation is just kind of hanging out there in space right now; it needs to connect to a broad introduction to the subject. 8. Generally, the English in both the comments and documentation needs work. However, we can address that problem when we're closer to commit. I am going to mark this Returned with Feedback because I don't believe it's realistic to get the comment code committed in the next week, rework this patch, and then get this patch committed also. However, I'm feeling pretty good about this effort and I think we're making good progress toward getting this done. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
Brendan Jurd dire...@gmail.com writes: I have attached v4 of the patch against HEAD, and also an incremental patch showing just my changes against v3. I'll mark this as ready for committer. Looking at this, I want to question the implode/explode naming. I think those names are too cute by half, not particularly mnemonic, not visibly related to the similar existing functions, and not friendly to any future extension in the same area. My first thought is that we should go back to the string_to_array and array_to_string names. The key reason not to use those names was the conflict with the old functions if you didn't specify a third argument, but where is the advantage of not specifying the third argument? It would be a lot simpler for people to understand if we just said the two-argument forms work like this, while the three-argument forms work like that. This is especially reasonable because the difference in behavior is about nulls in the array, which is exactly what the third argument exists to specify. [ Sorry for not complaining about this before, but I was on vacation when the previous naming discussion went on. ] regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: ... and on some platforms, it'll be flat out impossible. We looked at this years ago and concluded that changing the size of the shmem segment after postmaster start was impractical from a portability standpoint. I have not seen anything to change that conclusion. I haven't done extensive research into this, but I did take a look at it briefly. It looked to me like the style of shared memory we're using now (I guess it's System V) has no way to resize a shared memory segment at all, and certainly no way that's portable. However it also looked as though POSIX shm (shm_open, etc.) can be resized using ftruncate(). Whether this is portable to all the platforms we run on, or whether the behavior of ftruncate() in combination with shm_open() is in the standard, I'm not sure. It's not portable. That's exactly what we were looking into back when. Uggh, that sucks. Can you provide any more details? I believe I went back and reread the old threads on this topic and it seems like the sticking point as far as POSIX shm goes it that it lacks a readable equivalent of shm_nattch. Yeah, that was another little problem. In principle though we only need one SysV-style shmem segment to get the required interlock, and there could be add-on shmem segments using POSIX or other APIs. But that doesn't get you out from under the portability issue or the memory space management issue (it's unlikely you can enlarge a segment without remapping it). Unlikely is probably an understatement. Still, enlarging a segment with remapping might be workable for some useful subset of the cases. But, if enlarging it can't be done portably, then we're pretty much dead in the water. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
On Mon, Aug 9, 2010 at 4:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Brendan Jurd dire...@gmail.com writes: I have attached v4 of the patch against HEAD, and also an incremental patch showing just my changes against v3. I'll mark this as ready for committer. Looking at this, I want to question the implode/explode naming. I think those names are too cute by half, not particularly mnemonic, not visibly related to the similar existing functions, and not friendly to any future extension in the same area. My first thought is that we should go back to the string_to_array and array_to_string names. The key reason not to use those names was the conflict with the old functions if you didn't specify a third argument, but where is the advantage of not specifying the third argument? It would be a lot simpler for people to understand if we just said the two-argument forms work like this, while the three-argument forms work like that. This is especially reasonable because the difference in behavior is about nulls in the array, which is exactly what the third argument exists to specify. [ Sorry for not complaining about this before, but I was on vacation when the previous naming discussion went on. ] I can live with that, as long as it's clearly explained in the docs. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas robertmh...@gmail.com writes: On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not portable. That's exactly what we were looking into back when. Uggh, that sucks. Can you provide any more details? You don't really have to go further than consulting the relevant standards, eg SUS says at http://www.opengroup.org/onlinepubs/007908799/xsh/mmap.html If the size of the mapped file changes after the call to mmap() as a result of some other operation on the mapped file, the effect of references to portions of the mapped region that correspond to added or removed portions of the file is unspecified. Particular implementations might cope with such cases in useful ways, or then again they might not. And even if your platform does, you've set an upper limit for the possible segment size in your mmap() call. Further down the page, SUS also takes pains to point out that you probably can't have an unlimited number of mapped regions, so adding more mmap'd segments isn't a way out either. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
On Mon, Aug 9, 2010 at 4:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Brendan Jurd dire...@gmail.com writes: I have attached v4 of the patch against HEAD, and also an incremental patch showing just my changes against v3. I'll mark this as ready for committer. Looking at this, I want to question the implode/explode naming. I think those names are too cute by half, not particularly mnemonic, not visibly related to the similar existing functions, and not friendly to any future extension in the same area. My first thought is that we should go back to the string_to_array and array_to_string names. The key reason not to use those names was the conflict with the old functions if you didn't specify a third argument, but where is the advantage of not specifying the third argument? It would be a lot simpler for people to understand if we just said the two-argument forms work like this, while the three-argument forms work like that. This is especially reasonable because the difference in behavior is about nulls in the array, which is exactly what the third argument exists to specify. Is there any reason why array functions need the type prefix when other type conversion functions don't? Why didn't we name unnest() array_unnest()? merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Hector Beyers wants to stay in touch on LinkedIn
LinkedIn Hector Beyers souhaite se connecter à vous sur LinkedIn : -- Amine, I'd like to add you to my professional network on LinkedIn. - Hector Beyers Accepter l'invitation de Hector Beyers http://www.linkedin.com/e/cm7uxn-gcns21oh-6h/VWCdriNHrcXMfYpwEG6pcCKmodzjIoHmhgnzwrRR5NC/blk/I76828881_6/6lColZJrmZznQNdhjRQnOpBtn9QfmhBt71BoSd1p65Lr6lOfPpvcjwUe38Udzt9bQJzolFTk7BVbPoVej8Od3oQdj4LrCBxbOYWrSlI/EML_comm_afe/ Voir l'invitation de Hector Beyers http://www.linkedin.com/e/cm7uxn-gcns21oh-6h/VWCdriNHrcXMfYpwEG6pcCKmodzjIoHmhgnzwrRR5NC/blk/I76828881_6/dBYNe3wUczwSdQALqnpPbOYWrSlI/svi/ -- Quel est l' intérêt de faire partie du réseau de Hector Beyers ? Vous avez une question ? Le réseau de Hector Beyers a probablement une réponse : La rubrique Réponses de LinkedIn vous permet de vous informer auprès de Hector Beyers et de votre réseau. Obtenez l'information que vous recherchez auprès de professionnels expérimentés. http://www.linkedin.com/e/cm7uxn-gcns21oh-6h/ash/inv19_ayn/ -- (c) 2010, LinkedIn Corporation
Re: [HACKERS] dynamically allocating chunks from shared memory
Robert Haas robertmh...@gmail.com wrote: I don't think it's going to be too easy to provide, short of (as Tom says) moving to the MySQL model of many threads working in a single process. Well, it's a bit misleading to refer to it as the MySQL model. It's used by Microsoft SQL Server, MySQL, Informix, and Sybase. IBM DB2 supports four different process models, and OS threads in a single process is the default for them on an OS with good threading support; otherwise they default to one process per connection. Just because MySQL uses a particular technique doesn't *automatically* mean it's a bad one; it's just not in itself a confidence-builder. ;-) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
Merlin Moncure mmonc...@gmail.com writes: Is there any reason why array functions need the type prefix when other type conversion functions don't? Why didn't we name unnest() array_unnest()? UNNEST() is in the standard, IIRC, so you'd have to ask the SQL committee that. (And no, they're not exactly being consistent either, see array_agg() for example.) But anyway, my point here is that these functions are close enough to the existing string_to_array/array_to_string functions that they should be presented as variants of those, not arbitrarily assigned unrelated new names. Whether we'd have chosen different names if we had it to do over is academic. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: to_string, to_array functions
On Aug 9, 2010, at 1:10 PM, Robert Haas wrote: My first thought is that we should go back to the string_to_array and array_to_string names. The key reason not to use those names was the conflict with the old functions if you didn't specify a third argument, but where is the advantage of not specifying the third argument? It would be a lot simpler for people to understand if we just said the two-argument forms work like this, while the three-argument forms work like that. This is especially reasonable because the difference in behavior is about nulls in the array, which is exactly what the third argument exists to specify. [ Sorry for not complaining about this before, but I was on vacation when the previous naming discussion went on. ] I can live with that, as long as it's clearly explained in the docs. +1 David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
