Re: [HACKERS] Per-column collation, proof of concept
On Mon, Aug 16, 2010 at 10:56 PM, Peter Eisentraut wrote: > On lör, 2010-08-14 at 02:05 -0500, Jaime Casanova wrote: > >> BTW, why the double quotes? > > Because the name contains upper case letters? > why everything seems so obvious once someone else state it? :) >> sorry to state the obvious but this doesn't work on windows, does it? > > Probably not, but hopefully there is some similar API that could be used > on Windows. > good luck with that! ;) seriously, maybe this helps http://msdn.microsoft.com/en-us/library/system.windows.forms.inputlanguage.installedinputlanguages.aspx but probably you will need to write the code yourself... at least i don't think there is something like "locale -a" >> and for some reason it also didn't work on a centos 5 (this error >> ocurred when initdb'ing) >> """ >> loading system objects' descriptions ... ok >> creating collations ...FATAL: invalid byte sequence for encoding >> "UTF8": 0xe56c09 >> CONTEXT: COPY tmp_pg_collation, line 86 >> STATEMENT: COPY tmp_pg_collation FROM >> E'/usr/local/pgsql/9.1/share/locales.txt'; >> """ > > Hmm, what is in that file on that line? > > bokmål ISO-8859-1 (i'm attaching the locales.txt just in case) -- Jaime Casanova www.2ndQuadrant.com Soporte y capacitación de PostgreSQL aa_DJ ISO-8859-1 aa_DJ.iso88591 ISO-8859-1 aa_DJ.utf8 UTF-8 aa_ER UTF-8 aa...@saaho UTF-8 aa_ER.utf8 UTF-8 aa_er.u...@saahoUTF-8 aa_ET UTF-8 aa_ET.utf8 UTF-8 af_ZA ISO-8859-1 af_ZA.iso88591 ISO-8859-1 af_ZA.utf8 UTF-8 am_ET UTF-8 am_ET.utf8 UTF-8 an_ES ISO-8859-15 an_ES.iso885915 ISO-8859-15 an_ES.utf8 UTF-8 ar_AE ISO-8859-6 ar_AE.iso88596 ISO-8859-6 ar_AE.utf8 UTF-8 ar_BH ISO-8859-6 ar_BH.iso88596 ISO-8859-6 ar_BH.utf8 UTF-8 ar_DZ ISO-8859-6 ar_DZ.iso88596 ISO-8859-6 ar_DZ.utf8 UTF-8 ar_EG ISO-8859-6 ar_EG.iso88596 ISO-8859-6 ar_EG.utf8 UTF-8 ar_IN UTF-8 ar_IN.utf8 UTF-8 ar_IQ ISO-8859-6 ar_IQ.iso88596 ISO-8859-6 ar_IQ.utf8 UTF-8 ar_JO ISO-8859-6 ar_JO.iso88596 ISO-8859-6 ar_JO.utf8 UTF-8 ar_KW ISO-8859-6 ar_KW.iso88596 ISO-8859-6 ar_KW.utf8 UTF-8 ar_LB ISO-8859-6 ar_LB.iso88596 ISO-8859-6 ar_LB.utf8 UTF-8 ar_LY ISO-8859-6 ar_LY.iso88596 ISO-8859-6 ar_LY.utf8 UTF-8 ar_MA ISO-8859-6 ar_MA.iso88596 ISO-8859-6 ar_MA.utf8 UTF-8 ar_OM ISO-8859-6 ar_OM.iso88596 ISO-8859-6 ar_OM.utf8 UTF-8 ar_QA ISO-8859-6 ar_QA.iso88596 ISO-8859-6 ar_QA.utf8 UTF-8 ar_SA ISO-8859-6 ar_SA.iso88596 ISO-8859-6 ar_SA.utf8 UTF-8 ar_SD ISO-8859-6 ar_SD.iso88596 ISO-8859-6 ar_SD.utf8 UTF-8 ar_SY ISO-8859-6 ar_SY.iso88596 ISO-8859-6 ar_SY.utf8 UTF-8 ar_TN ISO-8859-6 ar_TN.iso88596 ISO-8859-6 ar_TN.utf8 UTF-8 ar_YE ISO-8859-6 ar_YE.iso88596 ISO-8859-6 ar_YE.utf8 UTF-8 as_IN.utf8 UTF-8 az_AZ.utf8 UTF-8 be_BY CP1251 be_BY.cp1251CP1251 be...@latin UTF-8 be_BY.utf8 UTF-8 be_by.u...@latinUTF-8 bg_BG CP1251 bg_BG.cp1251CP1251 bg_BG.utf8 UTF-8 bn_BD UTF-8 bn_BD.utf8 UTF-8 bn_IN UTF-8 bn_IN.utf8 UTF-8 bokmal ISO-8859-1 bokmål ISO-8859-1 br_FR ISO-8859-1 br...@euro ISO-8859-15 br_FR.iso88591 ISO-8859-1 br_fr.iso885...@euroISO-8859-15 br_FR.utf8 UTF-8 bs_BA ISO-8859-2 bs_BA.iso88592 ISO-8859-2 bs_BA.utf8 UTF-8 byn_ER UTF-8 byn_ER.utf8 UTF-8 C ANSI_X3.4-1968 ca_AD ISO-8859-15 ca_AD.iso885915 ISO-8859-15 ca_AD.utf8 UTF-8 ca_ES ISO-8859-1 ca...@euro ISO-8859-15 ca_ES.iso88591 ISO-8859-1 ca_es.iso885...@euroISO-8859-15 ca_ES.utf8 UTF-8 ca_FR ISO-8859-15 ca_FR.iso885915 ISO-8859-15 ca_FR.utf8 UTF-8 ca_IT ISO-8859-15 ca_IT.iso885915 ISO-8859-15 ca_IT.utf8 UTF-8 catalan ISO-8859-1 croatianISO-8859-2 csb_PL UTF-8 csb_PL.utf8 UTF-8 cs_CZ ISO-8859-2 cs_CZ.iso88592 ISO-8859-2 cs_CZ.utf8 UTF-8 cy_GB ISO-8859-14 cy_GB.iso885914 ISO-8859-14 cy_GB.utf8 UTF-8 czech ISO-8859-2 da_DK ISO-8859-1 da_DK.iso88591 ISO-8859-1 da_DK.iso885915 ISO-8859-15 da_DK.utf8 UTF-8 danish ISO-8859-1 dansk ISO-8859-1 de_AT ISO-8859-1 de...@euro ISO-8859-15 de_AT.iso88591 ISO-8859-1 de_at.iso885...@euroISO-8859-15 de_AT.utf8 UTF-8 de_BE ISO-8859-1 de...@euro ISO-8859-15 de_BE.iso88591 ISO-8859-1 de_be.iso885...@euroISO-8859-15 de_BE.utf8 UTF-8 de_CH ISO-8859-1 de_CH.iso88591 ISO-8859-1 de_CH.utf8 UTF-8 de_DE ISO-8859-1 de...@euro ISO-8859-15 de_DE.iso88591 ISO-8859-1 de_de.iso885...@euroISO-8859-15 de_DE.utf8 UTF-8 de_LU ISO-8859-1 de...@euro ISO-8859-15 de_LU.iso88591 ISO-8859-1 de_lu.iso885...@euroISO-8859-15 de_LU.utf8 UTF-8 deutsch ISO-8859-1 dutch ISO-8859-1 dz_BT UTF-8 dz_BT.utf8 UTF-8 eesti ISO-8859-1 el_CY ISO-8859-7 el_CY.iso88597 ISO-8859-7 el_CY.utf8 UTF-8 el_GR ISO-8859-7 el_GR.iso88597 ISO-8859-7 el_GR.utf8 UTF-8 en_AU ISO-8859-1 en_AU.iso88
[HACKERS] Progress indication prototype
Here is a small prototype for a query progress indicator.
Past discussions seemed to indicate that the best place to report this
would be in pg_stat_activity. So that's what this does. You can try it
out with any of the following (on a sufficiently large table): VACUUM
(lazy) (also autovacuum), COPY out from table, COPY in from file,
table-rewriting ALTER TABLE (e.g., add column with default), or a very
simple query. Run the command, and stare at pg_stat_activity (perhaps
via "watch") in a separate session.
More can be added, and the exact placement of the counters is debatable,
but those might be details to be worked out later. Note, my emphasis
here is on maintenance commands; I don't plan to do a progress
estimation of complex queries at this time.
Some thoughts:
- Are the interfaces OK?
- Is this going to be too slow to be useful?
- Should there be a separate switch to turn it on (currently
track_activities)?
- How to handle commands that process multiple tables? For example,
lazy VACUUM on a single table is pretty easy to track (count the block
numbers), but what about a database-wide lazy VACUUM?
Other comments?
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8852326..8280550 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -341,7 +341,8 @@ CREATE VIEW pg_stat_activity AS
S.xact_start,
S.query_start,
S.waiting,
-S.current_query
+S.current_query,
+S.query_progress
FROM pg_database D, pg_stat_get_activity(NULL) AS S, pg_authid U
WHERE S.datid = D.oid AND
S.usesysid = U.oid;
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 906d547..67e4fe0 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -35,6 +35,7 @@
#include "miscadmin.h"
#include "optimizer/planner.h"
#include "parser/parse_relation.h"
+#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
#include "storage/fd.h"
#include "tcop/tcopprot.h"
@@ -1424,6 +1425,7 @@ CopyTo(CopyState cstate)
bool *nulls;
HeapScanDesc scandesc;
HeapTuple tuple;
+ int64 cnt = 0;
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (bool *) palloc(num_phys_attrs * sizeof(bool));
@@ -1439,6 +1441,8 @@ CopyTo(CopyState cstate)
/* Format and send the data */
CopyOneRowTo(cstate, HeapTupleGetOid(tuple), values, nulls);
+
+ pgstat_report_progress(++cnt, cstate->rel->rd_rel->reltuples);
}
heap_endscan(scandesc);
@@ -1695,6 +1699,8 @@ CopyFrom(CopyState cstate)
CommandId mycid = GetCurrentCommandId(true);
int hi_options = 0; /* start with default heap_insert options */
BulkInsertState bistate;
+ off_t file_size;
+ off_t file_cnt = 0;
Assert(cstate->rel);
@@ -1776,6 +1782,8 @@ CopyFrom(CopyState cstate)
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("\"%s\" is a directory", cstate->filename)));
+
+ file_size = st.st_size;
}
tupDesc = RelationGetDescr(cstate->rel);
@@ -2051,6 +2059,9 @@ CopyFrom(CopyState cstate)
}
Assert(fieldno == nfields);
+
+ file_cnt += cstate->line_buf.len;
+ pgstat_report_progress(file_cnt, file_size);
}
else
{
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 5f6fe41..0d70e86 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -60,6 +60,7 @@
#include "parser/parse_type.h"
#include "parser/parse_utilcmd.h"
#include "parser/parser.h"
+#include "pgstat.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
@@ -3126,6 +3127,7 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap)
MemoryContext oldCxt;
List *dropped_attrs = NIL;
ListCell *lc;
+ int64 cnt = 0;
econtext = GetPerTupleExprContext(estate);
@@ -3253,6 +3255,8 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap)
ResetExprContext(econtext);
+ pgstat_report_progress(++cnt, oldrel->rd_rel->reltuples);
+
CHECK_FOR_INTERRUPTS();
}
@@ -7064,6 +7068,9 @@ copy_relation_data(SMgrRelation src, SMgrRelation dst,
* write; we'll do it ourselves below.
*/
smgrextend(dst, forkNum, blkno, buf, true);
+
+ if (forkNum == MAIN_FORKNUM)
+ pgstat_report_progress(blkno + 1, nblocks);
}
/*
diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 4408db8..2127434 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -350,6 +350,8 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats,
bool all_visible_according_to_vm = false;
bool all_visible;
+ pgstat_report_progress(blkno, nblocks);
+
/*
* Skip pages that don't require vacuuming according to the visibility
* map. But only if we've seen a streak of at least
diff --git a/src/backend/executor/execMain.c
Re: [HACKERS] security label support, part.2
(2010/08/17 13:28), Peter Eisentraut wrote: > On tis, 2010-08-17 at 13:00 +0900, KaiGai Kohei wrote: >> However, isn't it strange if we stand on the perspective that child >> table is a part of parent object? It means an object have multiple >> properties depending on the context. > > Such is the nature of inheritance, isn't it? > Yep, it will return different set of user data depending on the table queried, when we reference either parent or child table. But it seems to me too stretch interpretation to apply this behavior on metadata of the tables also. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
On tis, 2010-08-17 at 13:00 +0900, KaiGai Kohei wrote: > However, isn't it strange if we stand on the perspective that child > table is a part of parent object? It means an object have multiple > properties depending on the context. Such is the nature of inheritance, isn't it? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
(2010/08/17 11:58), Tom Lane wrote: > Stephen Frost writes: >> * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: >>> Indeed, PG does not try to handle child table as an independent object >>> from a parent table. However, if so, it seems to me strange that we can >>> assign individual ownership and access privileges on child tables. > >> I tend to agree. Perhaps we should bring up, in an independent thread, >> the question of if that really makes sense or if we should do something >> to prevent it (or at least issue a warning when we detect it). > > The reason there is still some value in setting permissions state on a > child table is that that controls what happens when you address the > child table directly, rather than implicitly by querying its parent. > However, isn't it strange if we stand on the perspective that child table is a part of parent object? It means an object have multiple properties depending on the context. If we want to allow someone to reference a part of the table (= child table), I think VIEW is more appropriate and flexible tool. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Per-column collation, proof of concept
On lör, 2010-08-14 at 02:05 -0500, Jaime Casanova wrote: > btw, the patch no longer apply cleanly but most are just hunks the > worst it's in src/backend/catalog/namespace.c because > FindConversionByName() is now called get_conversion_oid()... so maybe > this function should be named get_collation_oid(), i guess OK, that will need to be adjusted. > well at least pg_collation should be a shared catalog, no? > and i think we shouldn't be thinking in this without think first how > to integrate this with at least per-database configuration Good point. But one might also want to create "private" collations, so a collation in a schema would also be useful. Tricky. > also, it doesn't recognize C collate although it is in the locales.txt > """ > test3=# create database test4 with template=template0 encoding 'utf-8' > lc_collate='C'; > CREATE DATABASE > test3=# create table t3 (col1 text collate "C" ); > ERROR: collation "C" does not exist > """ I've fixed this in the meantime. Your version of the patch doesn't support the C locale yet. > BTW, why the double quotes? Because the name contains upper case letters? > sorry to state the obvious but this doesn't work on windows, does it? Probably not, but hopefully there is some similar API that could be used on Windows. > and for some reason it also didn't work on a centos 5 (this error > ocurred when initdb'ing) > """ > loading system objects' descriptions ... ok > creating collations ...FATAL: invalid byte sequence for encoding > "UTF8": 0xe56c09 > CONTEXT: COPY tmp_pg_collation, line 86 > STATEMENT: COPY tmp_pg_collation FROM > E'/usr/local/pgsql/9.1/share/locales.txt'; > """ Hmm, what is in that file on that line? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
(2010/08/17 11:37), Robert Haas wrote: >> I might have a reason why the script need to launch in single-user >> mode, but it is not clear right now, sorry. > > Another point here is that I wonder if we really need to label system > objects at all. Are you applying the same label to all of them? If > so, perhaps it might be feasible to set up the code so that it simply > assumes that label for every object in the pg_catalog namespace. > No, SELinux provides APIs to suggest what database object should have what security label on initialization time. (selabel_open(3), selabel_lookup(3) and selabel_close(3)) It depends on configurations by system admin, so we cannot assume a certain label for every object in a certain namespace. > And if you're NOT setting the label the same way on all of them, then > there's a maintenance issue to think about. > Right, I don't want to have multiple way to label them. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
2010/8/17 Robert Haas : > On Sun, Aug 15, 2010 at 7:44 AM, Hitoshi Harada wrote: >> We (Marko, David Fetter and I) discussed on IRC about design of >> writeable CTEs. It does and will contain not only syntax but also >> miscellaneous specifications, general rules and restrictions. I hope >> this will help the patch reviews and stop dangerous design at the >> early stage. If you find something wrong, or have request, please >> notify. >> >> http://wiki.postgresql.org/wiki/WriteableCTEs >> >> We will keep to add details. Any comments are welcome. > > There are really two separate features here, and it might be worth > giving them separate names and separate designs (and separate > patches). Allowing the main query to be an insert, update, or delete > seems easier than allowing the toplevel CTEs to contain those > constructs, although I might be wrong about that. > > Under features, what is DCL? There has been previous talk of allowing > WITH (COPY ...) and I am personally of the opinion that it would be > nice to be able to do WITH (EXPLAIN ...). DDL seems like a poor idea. So, there are three? One for allowing the main query to be an insert, update, or delete, one for the main subject of writeable CTE with insert, update, delete and one for allowing COPY to return rows. I am attracted by such COPY functionality. And the first part seems easier to be committed separately. Is it possible to get it in by only adding syntax and little parser/planner modification, or more executor code? > P.S. Call me a prude, but your choice of shorthand for > insert-update-delete may not be the best. I think so, too :(. But there's no good expression in my mind. Suggestions? Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
2010/8/17 David Fetter : > On Mon, Aug 16, 2010 at 04:34:07PM -0400, Robert Haas wrote: >> On Mon, Aug 16, 2010 at 3:00 PM, David Fetter wrote: >> >> There has been previous talk of allowing WITH (COPY ...) and I am >> >> personally of the opinion that it would be nice to be able to do >> >> WITH (EXPLAIN ...). DDL seems like a poor idea. >> > >> > It may be, but I can see use cases for partitioning... >> >> Like what? > > Managing partitions, or creating partitions in some way the partition > syntax doesn't anticipate, whatever it turns out to be. I don't see the good use cases you might be able to imagine, either. Now we have DO statement, quite various types of DDL can be described at one time. Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
Stephen Frost writes: > * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: >> Indeed, PG does not try to handle child table as an independent object >> from a parent table. However, if so, it seems to me strange that we can >> assign individual ownership and access privileges on child tables. > I tend to agree. Perhaps we should bring up, in an independent thread, > the question of if that really makes sense or if we should do something > to prevent it (or at least issue a warning when we detect it). The reason there is still some value in setting permissions state on a child table is that that controls what happens when you address the child table directly, rather than implicitly by querying its parent. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
Joseph Adams writes: > Others already mentioned that you can't convert 2 billion byte long > JSON strings to BSON. Another issue is that BSON cannot encode all > JSON numbers without precision loss. As somebody already mentioned, the former isn't likely to be an issue for us anytime in the foreseeable future, because we can't push around datum values more than 1GB large anyhow. The latter seems like a pretty nasty problem though. I'm good with just dropping this idea for the moment. The Google hit statistics that were cited earlier show that there's not enough interest in BSON to justify a separate datatype, which is what it would apparently need to be. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] refactoring comment.c
Robert Haas writes: > On Mon, Aug 16, 2010 at 3:48 PM, Tom Lane wrote: >> I think the problem is you're trying to put this into backend/parser >> which is not really the right place for it. > If this isn't parse analysis, then you and I have very different ideas > of what parse analysis is. Maybe so, but the parser is expected to put out a representation that will still be valid when the command is executed some time later. That is exactly why utility statements have the barely-more-than-source parsetree representation they do: because we do not hold locks on the objects from parsing to execution, we could not expect an OID-level representation to remain good. This is a lot different from what we do with DML statements, but there are good reasons for it. I repeat my observation that this code doesn't belong in /parser. The code you're replacing was not in /parser, and that was because it didn't belong there, not because somebody didn't understand the system structure. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
2010/8/16 KaiGai Kohei : >> I don't really see what the advantage of doing this in single-user >> mode is. If the overhead of permissions-checking is enough to matter, >> maybe that's a sign we're doing something wrong. >> > Hmm... I guess the overhead is not a significant matter, because the > number of system obejcts (not only tables) are less than 3,500. > It will be small enough on recent hardware. I would think so. More to the point, what is the cost of checking permissions as a percentage of the cost of applying the new labels? If it isn't pretty small, something's not right. Performance will probably be terrible if anyone actually attempts to use this do to real work. > I might have a reason why the script need to launch in single-user > mode, but it is not clear right now, sorry. Another point here is that I wonder if we really need to label system objects at all. Are you applying the same label to all of them? If so, perhaps it might be feasible to set up the code so that it simply assumes that label for every object in the pg_catalog namespace. And if you're NOT setting the label the same way on all of them, then there's a maintenance issue to think about. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Robert Haas wrote: > > Yeah, it's a bit too slow to do on every sync. ?I run it every week or > > two and keep the output in a text file. ?Usually what I want the history > > for is stuff that happened awhile ago, so the fact that it's not 100% up > > to date is seldom a factor. > > OK, try this. It takes about 14 seconds on my machine on my copy of > Magnus's test repository. Output looks like this: > > Author: Robert Haas > Branch: master [8c5aba824] 2010-07-21 09:23:34 -0400 > Branch: REL9_0_STABLE [00314ceab] 2010-07-21 09:23:34 -0400 > Branch: REL8_4_STABLE [14ddf23a8] 2010-07-21 09:23:34 -0400 > > Compact numeric format, with 2-byte header in common cases. > > Author: Robert Haas > Branch: master [d0706cfd2] 2010-07-21 09:28:08 -0400 > > Standardize get_whatever_oid functions for other object types. > > - Rename TSParserGetPrsid to get_ts_parser_oid. > - Rename TSDictionaryGetDictid to get_ts_dict_oid. > - Rename TSTemplateGetTmplid to get_ts_template_oid. > - Rename TSConfigGetCfgid to get_ts_config_oid. > - Rename FindConversionByName to get_conversion_oid. > - Rename GetConstraintName to get_constraint_oid. > - Add new functions get_opclass_oid, get_opfamily_oid, get_rewrite_oid, > get_rewrite_oid_without_relid, get_trigger_oid, and get_cast_oid. > > The name of each function matches the corresponding catalog. Great. src/tools/pgcvslog -d will delete HEAD commits that were also applied in back-branches. That is home-grown tool so a similar tool will have to be written when I create major release notes, but that is a year away. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
(2010/08/17 10:57), Robert Haas wrote: > 2010/8/16 KaiGai Kohei: >> (2010/08/17 9:52), Robert Haas wrote: >>> 2010/8/16 KaiGai Kohei: (2010/08/16 23:40), Robert Haas wrote: > 2010/8/16 KaiGai Kohei: >> Although nobody paid an attention, it seems to me a problem to be fixed. >> >> The attached patch fixes the problem using a simple idea which adds >> process_shared_preload_libraries() at PostgresMain() when we launched >> it in single-user mode. > > I have no confidence at all that this is a sane thing to do. I think > any enhanced security provider that needs system objects to be > labelled should provide a script to label them after the fact. You > can't count on everyone who wants to use SE-PostgreSQL having made > that decision at initdb time. I think we want to keep single-user > mode as lean and mean as possible, so that people can rely on it when > they need to fix their broken database. > I also agree it is nonsense to make access control decision during initdb phase, but it is not the reason why I want to fix this problem. I plan to provide a script that assigns initial security label after the initdb, but before launching postmaster. This script tries to execute postgres in single-user mode, then labels database objects according to the system setting. But the sepgsql module is not loaded currently. I want to kick this job in single-user mode, not normal processing mode, because we can simplify several stuffs. For example, we don't need to check whether the user has privilege to assign initial labels, because it is obvious people who launch initdb has superpower on whole of the database. In addition, we don't need to consider a possibility that someone create a new database object during initial labeling. >>> >>> I think this is a bad design. Consider someone who has 10 databases >>> for which he does NOT wish to use security labelling. One day he >>> decides to create database #11 and on this one he DOES want security >>> labelling. Ideally, he'd be able to do this without shutting down the >>> database. Of course, that's not going to quite work, since >>> shared_preload_libraries needs to be changed, but that can be done >>> with a very quick server bounce. Requiring him to run the setup >>> scripts in single-user mode is just painful; forcing him to label >>> every database is even worse. >>> >> Hmm. It seems to me a reasonable scenario indeed. >> In this case, the someone who wants to create #11 database with security >> labels may connect on the postmaster via internet, so we need to check >> his privileges to assign security label on individual database objects >> without short-cut like a case when single-user mode. >> >> Perhaps, the best one is to provide two options for users. >> If he wants to label database obejcts just after initdb, he can launch >> a script to label them in single-user mode; without any cumbers. >> If he wants to label only objects within database #11 after several >> months from initdb, he can launch a script to label them via normal >> connection. > > I don't really see what the advantage of doing this in single-user > mode is. If the overhead of permissions-checking is enough to matter, > maybe that's a sign we're doing something wrong. > Hmm... I guess the overhead is not a significant matter, because the number of system obejcts (not only tables) are less than 3,500. It will be small enough on recent hardware. I might have a reason why the script need to launch in single-user mode, but it is not clear right now, sorry. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
Tom Lane wrote: > Magnus Hagander writes: > > According to the decision at the developer meeting, the migration to > > git should happen 17-20 Aug. Here's my proposed timeline. This will > > obviously affect development work some, and since the original > > timeline called for us having already released 9.0 buy then ;) > > > 1. Tuesday evening, around 19:00 central european time, which is 17:00 > > GMT or 12:00 EST, I will freeze the current cvs repository. I will do > > this by disabling committer login on that box, so please note that > > this will also make it impossible for committers to do a "cvs update" > > from the authenticated repository. > > So, per discussion, I'd like to suggest that we have a "quiet time" for > say three hours before the repository is closed off, to give interested > committers a chance to capture final snapshots of the current > repository. > > IOW, please no more CVS commits after 14:00 GMT tomorrow, Tuesday 8/17. OK, would someone please send an email to hackers at that time as a reminder? I might not be available then. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
2010/8/16 KaiGai Kohei : > (2010/08/17 9:52), Robert Haas wrote: >> 2010/8/16 KaiGai Kohei: >>> (2010/08/16 23:40), Robert Haas wrote: 2010/8/16 KaiGai Kohei: > Although nobody paid an attention, it seems to me a problem to be fixed. > > The attached patch fixes the problem using a simple idea which adds > process_shared_preload_libraries() at PostgresMain() when we launched > it in single-user mode. I have no confidence at all that this is a sane thing to do. I think any enhanced security provider that needs system objects to be labelled should provide a script to label them after the fact. You can't count on everyone who wants to use SE-PostgreSQL having made that decision at initdb time. I think we want to keep single-user mode as lean and mean as possible, so that people can rely on it when they need to fix their broken database. >>> I also agree it is nonsense to make access control decision during >>> initdb phase, but it is not the reason why I want to fix this problem. >>> >>> I plan to provide a script that assigns initial security label after >>> the initdb, but before launching postmaster. This script tries to execute >>> postgres in single-user mode, then labels database objects according to >>> the system setting. But the sepgsql module is not loaded currently. >>> >>> I want to kick this job in single-user mode, not normal processing mode, >>> because we can simplify several stuffs. For example, we don't need to >>> check whether the user has privilege to assign initial labels, because >>> it is obvious people who launch initdb has superpower on whole of the >>> database. In addition, we don't need to consider a possibility that >>> someone create a new database object during initial labeling. >> >> I think this is a bad design. Consider someone who has 10 databases >> for which he does NOT wish to use security labelling. One day he >> decides to create database #11 and on this one he DOES want security >> labelling. Ideally, he'd be able to do this without shutting down the >> database. Of course, that's not going to quite work, since >> shared_preload_libraries needs to be changed, but that can be done >> with a very quick server bounce. Requiring him to run the setup >> scripts in single-user mode is just painful; forcing him to label >> every database is even worse. >> > Hmm. It seems to me a reasonable scenario indeed. > In this case, the someone who wants to create #11 database with security > labels may connect on the postmaster via internet, so we need to check > his privileges to assign security label on individual database objects > without short-cut like a case when single-user mode. > > Perhaps, the best one is to provide two options for users. > If he wants to label database obejcts just after initdb, he can launch > a script to label them in single-user mode; without any cumbers. > If he wants to label only objects within database #11 after several > months from initdb, he can launch a script to label them via normal > connection. I don't really see what the advantage of doing this in single-user mode is. If the overhead of permissions-checking is enough to matter, maybe that's a sign we're doing something wrong. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
(2010/08/17 9:52), Robert Haas wrote: > 2010/8/16 KaiGai Kohei: >> (2010/08/16 23:40), Robert Haas wrote: >>> 2010/8/16 KaiGai Kohei: Although nobody paid an attention, it seems to me a problem to be fixed. The attached patch fixes the problem using a simple idea which adds process_shared_preload_libraries() at PostgresMain() when we launched it in single-user mode. >>> >>> I have no confidence at all that this is a sane thing to do. I think >>> any enhanced security provider that needs system objects to be >>> labelled should provide a script to label them after the fact. You >>> can't count on everyone who wants to use SE-PostgreSQL having made >>> that decision at initdb time. I think we want to keep single-user >>> mode as lean and mean as possible, so that people can rely on it when >>> they need to fix their broken database. >>> >> I also agree it is nonsense to make access control decision during >> initdb phase, but it is not the reason why I want to fix this problem. >> >> I plan to provide a script that assigns initial security label after >> the initdb, but before launching postmaster. This script tries to execute >> postgres in single-user mode, then labels database objects according to >> the system setting. But the sepgsql module is not loaded currently. >> >> I want to kick this job in single-user mode, not normal processing mode, >> because we can simplify several stuffs. For example, we don't need to >> check whether the user has privilege to assign initial labels, because >> it is obvious people who launch initdb has superpower on whole of the >> database. In addition, we don't need to consider a possibility that >> someone create a new database object during initial labeling. > > I think this is a bad design. Consider someone who has 10 databases > for which he does NOT wish to use security labelling. One day he > decides to create database #11 and on this one he DOES want security > labelling. Ideally, he'd be able to do this without shutting down the > database. Of course, that's not going to quite work, since > shared_preload_libraries needs to be changed, but that can be done > with a very quick server bounce. Requiring him to run the setup > scripts in single-user mode is just painful; forcing him to label > every database is even worse. > Hmm. It seems to me a reasonable scenario indeed. In this case, the someone who wants to create #11 database with security labels may connect on the postmaster via internet, so we need to check his privileges to assign security label on individual database objects without short-cut like a case when single-user mode. Perhaps, the best one is to provide two options for users. If he wants to label database obejcts just after initdb, he can launch a script to label them in single-user mode; without any cumbers. If he wants to label only objects within database #11 after several months from initdb, he can launch a script to label them via normal connection. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
(2010/08/17 9:51), Stephen Frost wrote: > * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: >> Ahh, yes, the question is "what is an object", not a "MAC vs DAC". >> >> Indeed, PG does not try to handle child table as an independent object >> from a parent table. However, if so, it seems to me strange that we can >> assign individual ownership and access privileges on child tables. > > I tend to agree. Perhaps we should bring up, in an independent thread, > the question of if that really makes sense or if we should do something > to prevent it (or at least issue a warning when we detect it). > >> If we stand on the perspective that child tables are a part of the >> parent table, isn't it necessary to keep same ownership and access >> privileges between parent and children? It seems to me the current >> implementation is in the halfway from the perspective of child >> tables as independent object to the perspective of child tables as >> a part of parent table. > > I tend to agree- PG isn't doing this as cleanly as it should. > >> If PG can keep consistency of ownership and access privileges between >> parent and children, it is quite natural we keep consistency of labels, >> isn't it? > > Yes, but that's something that should be dealt with in PG and not as > part of an external security infrastructure. For example, PG could just > force that the same label is applied to a child table when it's made a > child of a particular parent, perhaps with a check to the external > security system to see if there's a problem changing whatever label is > on the child table before it's changed to be that of the parent, but > once it's a child of the parent (if that operation was permitted and was > successful), it no longer has its own 'identity'. > > Let's not build the complication of dealing with inheiritance/child > relations into the external security system when we're in the middle of > trying to get rid of that distinction in core PG though. > I also agree the idea that PG core handles the recursion of inheritance hierarchy and enforce same label between them. The reason why I handled it within the module is the core does not enforce same labels. OK, I'll rid 'expected_parents' argument from the security hook on relabeling. Right now, it is incomplete, but should be fixed up in the future. In addition, I'll also post a design proposal to keep consistency of ownership and access privileges between parent and children. Please also wait for a while. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] refactoring comment.c
On Mon, Aug 16, 2010 at 3:48 PM, Tom Lane wrote: >> 2. I haven't done anything about moving the definition of >> ObjectAddress elsewhere, as Alvaro suggested, because I'm not sure >> quite where it ought to go. I still think it's a good idea, though >> I'm not dead set on it, either. Suggestions? > > I think the problem is you're trying to put this into backend/parser > which is not really the right place for it. It's an execution-time > animal not a parse-time animal. I would put it into backend/catalog, > perhaps named objectaddress.c, and similarly the header file would be > objectaddress.h. Then it would be reasonable to move struct > ObjectAddress into this header and have dependency.h #include it. > There might be some other stuff in dependency.c that more naturally > belongs here, too. If this isn't parse analysis, then you and I have very different ideas of what parse analysis is. And under this theory, what are routines like LookupAggNameTypeNames() doing in src/backend/parser? I'll make the rest of the changes you suggest... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On Mon, Aug 16, 2010 at 8:38 PM, Josh Berkus wrote: > >> but BSON pidgenholes numeric values to either double, int32, int64, or >> a 12-byte MongoDB Object ID. Thus, for people who expect JSON to be >> able to hold arbitrary-precision numbers (which the JSON data type in >> my patch can), using BSON for transfer or storage will violate that >> expectation. > > Good lord. I'd suggest that maybe we wait for BSON v. 2.0 instead. > > Is BSON even any kind of a standard? You know that if it were a standard, it would be WORSE! :-) This falls into big time "no way!" If there was a standardized binary encoding for XML that was effectively a tree (e.g. - more or less like a set of Lisp objects with CAR/CDR linkages), that would be somewhat interesting, as it could be both comparatively compact, and perhaps offer rapid navigation through the tree. BSON doesn't sound like that! -- http://linuxfinances.info/info/linuxdistributions.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
On Mon, Aug 16, 2010 at 5:08 PM, David Fetter wrote: > On Mon, Aug 16, 2010 at 04:34:07PM -0400, Robert Haas wrote: >> On Mon, Aug 16, 2010 at 3:00 PM, David Fetter wrote: >> >> There has been previous talk of allowing WITH (COPY ...) and I am >> >> personally of the opinion that it would be nice to be able to do >> >> WITH (EXPLAIN ...). DDL seems like a poor idea. >> > >> > It may be, but I can see use cases for partitioning... >> >> Like what? > > Managing partitions, or creating partitions in some way the partition > syntax doesn't anticipate, whatever it turns out to be. So, what is the syntax you're hoping will work? >> >> P.S. Call me a prude, but your choice of shorthand for >> >> insert-update-delete may not be the best. >> > >> > Then I presume you'll be supporting my idea of using the word >> > "span" for temporal data types rather than the current idea whose >> > name appears in academic literature. >> >> Wise guy. > > In all seriousness, the temporal stuff is giving us a fantastic > opportunity, as a project, to break our tradition of lousy naming. > > "Span" is descriptive, mnemonic, and easy to spell. I guess that's a flamewar for another day. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
2010/8/16 KaiGai Kohei : > (2010/08/16 23:40), Robert Haas wrote: >> 2010/8/16 KaiGai Kohei: >>> Although nobody paid an attention, it seems to me a problem to be fixed. >>> >>> The attached patch fixes the problem using a simple idea which adds >>> process_shared_preload_libraries() at PostgresMain() when we launched >>> it in single-user mode. >> >> I have no confidence at all that this is a sane thing to do. I think >> any enhanced security provider that needs system objects to be >> labelled should provide a script to label them after the fact. You >> can't count on everyone who wants to use SE-PostgreSQL having made >> that decision at initdb time. I think we want to keep single-user >> mode as lean and mean as possible, so that people can rely on it when >> they need to fix their broken database. >> > I also agree it is nonsense to make access control decision during > initdb phase, but it is not the reason why I want to fix this problem. > > I plan to provide a script that assigns initial security label after > the initdb, but before launching postmaster. This script tries to execute > postgres in single-user mode, then labels database objects according to > the system setting. But the sepgsql module is not loaded currently. > > I want to kick this job in single-user mode, not normal processing mode, > because we can simplify several stuffs. For example, we don't need to > check whether the user has privilege to assign initial labels, because > it is obvious people who launch initdb has superpower on whole of the > database. In addition, we don't need to consider a possibility that > someone create a new database object during initial labeling. I think this is a bad design. Consider someone who has 10 databases for which he does NOT wish to use security labelling. One day he decides to create database #11 and on this one he DOES want security labelling. Ideally, he'd be able to do this without shutting down the database. Of course, that's not going to quite work, since shared_preload_libraries needs to be changed, but that can be done with a very quick server bounce. Requiring him to run the setup scripts in single-user mode is just painful; forcing him to label every database is even worse. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
* KaiGai Kohei (kai...@ak.jp.nec.com) wrote: > Ahh, yes, the question is "what is an object", not a "MAC vs DAC". > > Indeed, PG does not try to handle child table as an independent object > from a parent table. However, if so, it seems to me strange that we can > assign individual ownership and access privileges on child tables. I tend to agree. Perhaps we should bring up, in an independent thread, the question of if that really makes sense or if we should do something to prevent it (or at least issue a warning when we detect it). > If we stand on the perspective that child tables are a part of the > parent table, isn't it necessary to keep same ownership and access > privileges between parent and children? It seems to me the current > implementation is in the halfway from the perspective of child > tables as independent object to the perspective of child tables as > a part of parent table. I tend to agree- PG isn't doing this as cleanly as it should. > If PG can keep consistency of ownership and access privileges between > parent and children, it is quite natural we keep consistency of labels, > isn't it? Yes, but that's something that should be dealt with in PG and not as part of an external security infrastructure. For example, PG could just force that the same label is applied to a child table when it's made a child of a particular parent, perhaps with a check to the external security system to see if there's a problem changing whatever label is on the child table before it's changed to be that of the parent, but once it's a child of the parent (if that operation was permitted and was successful), it no longer has its own 'identity'. Let's not build the complication of dealing with inheiritance/child relations into the external security system when we're in the middle of trying to get rid of that distinction in core PG though. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 7:01 PM, Tom Lane wrote: > Robert Haas writes: >> On Mon, Aug 16, 2010 at 4:33 PM, Tom Lane wrote: >>> I'd be satisfied with a tool that merges commit reports if they have the >>> same log message and occur at approximately the same time, which is the >>> heuristic that cvs2cl uses. > >> So how do you run cvs2cl? Do you run it once in a while and save the >> output someplace? Or what? > > Yeah, it's a bit too slow to do on every sync. I run it every week or > two and keep the output in a text file. Usually what I want the history > for is stuff that happened awhile ago, so the fact that it's not 100% up > to date is seldom a factor. OK, try this. It takes about 14 seconds on my machine on my copy of Magnus's test repository. Output looks like this: Author: Robert Haas Branch: master [8c5aba824] 2010-07-21 09:23:34 -0400 Branch: REL9_0_STABLE [00314ceab] 2010-07-21 09:23:34 -0400 Branch: REL8_4_STABLE [14ddf23a8] 2010-07-21 09:23:34 -0400 Compact numeric format, with 2-byte header in common cases. Author: Robert Haas Branch: master [d0706cfd2] 2010-07-21 09:28:08 -0400 Standardize get_whatever_oid functions for other object types. - Rename TSParserGetPrsid to get_ts_parser_oid. - Rename TSDictionaryGetDictid to get_ts_dict_oid. - Rename TSTemplateGetTmplid to get_ts_template_oid. - Rename TSConfigGetCfgid to get_ts_config_oid. - Rename FindConversionByName to get_conversion_oid. - Rename GetConstraintName to get_constraint_oid. - Add new functions get_opclass_oid, get_opfamily_oid, get_rewrite_oid, get_rewrite_oid_without_relid, get_trigger_oid, and get_cast_oid. The name of each function matches the corresponding catalog. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company git-topo-order Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
> but BSON pidgenholes numeric values to either double, int32, int64, or > a 12-byte MongoDB Object ID. Thus, for people who expect JSON to be > able to hold arbitrary-precision numbers (which the JSON data type in > my patch can), using BSON for transfer or storage will violate that > expectation. Good lord. I'd suggest that maybe we wait for BSON v. 2.0 instead. Is BSON even any kind of a standard? -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
(2010/08/16 22:14), Stephen Frost wrote: > KaiGai, > > * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: >> The purpose of this restriction is to ensure an access control decision >> using parent's label being also consistent on child tables. > > Robert and I understand the concern that you have. The answer, at least > for now, is that we don't agree with you. PG doesn't consider child > tables to be independent objects when they're being accessed through the > parent. As such, they don't have their own permissions checking. > >> If we control accesses on child tables using child's label, no need to >> restrict an identical label within an entire inheritance hierarchy. >> But it needs to provide the original rte->requiredPerms of child tables. >> Now it is cleared at expand_inherited_rtentry(), so we have no way to >> control accesses on child tables using child's label. :( > > If you want to argue that we should care about the childs permissions, > or do something different with regard to inheritance, then you need to > make that argument for all of PG, not just try to do what you think is > right in the security definer framework. > >> > From viewpoint of MAC, both of the following SQLs should be denied, >> when accesses on parent_tbl is allowed, but child_tbl is denied. > > KaiGai, this is not a MAC vs. DAC difference. This is a question of > "what is an object" and if a child table is really an independent object > from a parent table. In PG, we've decided they're not. We should > probably do more to make that clearer in PG, rather than have different > parts of the system treat them differently. > Ahh, yes, the question is "what is an object", not a "MAC vs DAC". Indeed, PG does not try to handle child table as an independent object from a parent table. However, if so, it seems to me strange that we can assign individual ownership and access privileges on child tables. If we stand on the perspective that child tables are a part of the parent table, isn't it necessary to keep same ownership and access privileges between parent and children? It seems to me the current implementation is in the halfway from the perspective of child tables as independent object to the perspective of child tables as a part of parent table. If PG can keep consistency of ownership and access privileges between parent and children, it is quite natural we keep consistency of labels, isn't it? Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
> Another idea that comes to mind is that you have vacuum_cost_page_dirty > set to an unreasonably large value, so that autovac is blocking whenever > it has to write even one page. Nope. Default. And total cost was raised to 1000. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
(2010/08/17 9:02), Itagaki Takahiro wrote: > 2010/8/17 KaiGai Kohei: >> I want to kick this job in single-user mode, not normal processing mode, > > Does an explicit LOAD work in single-user mode? > I think LOAD just after login works as same as it was preloaded, > unless it allocates shared memory. > Thanks, I never thought this idea. It works and solves the matter, but I think the right way is to fix the problem, rather than a workaround in scripts... Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
2010/8/17 KaiGai Kohei : > I want to kick this job in single-user mode, not normal processing mode, Does an explicit LOAD work in single-user mode? I think LOAD just after login works as same as it was preloaded, unless it allocates shared memory. -- Itagaki Takahiro -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
Magnus Hagander writes: > According to the decision at the developer meeting, the migration to > git should happen 17-20 Aug. Here's my proposed timeline. This will > obviously affect development work some, and since the original > timeline called for us having already released 9.0 buy then ;) > 1. Tuesday evening, around 19:00 central european time, which is 17:00 > GMT or 12:00 EST, I will freeze the current cvs repository. I will do > this by disabling committer login on that box, so please note that > this will also make it impossible for committers to do a "cvs update" > from the authenticated repository. So, per discussion, I'd like to suggest that we have a "quiet time" for say three hours before the repository is closed off, to give interested committers a chance to capture final snapshots of the current repository. IOW, please no more CVS commits after 14:00 GMT tomorrow, Tuesday 8/17. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
(2010/08/16 23:40), Robert Haas wrote: > 2010/8/16 KaiGai Kohei: >> Although nobody paid an attention, it seems to me a problem to be fixed. >> >> The attached patch fixes the problem using a simple idea which adds >> process_shared_preload_libraries() at PostgresMain() when we launched >> it in single-user mode. > > I have no confidence at all that this is a sane thing to do. I think > any enhanced security provider that needs system objects to be > labelled should provide a script to label them after the fact. You > can't count on everyone who wants to use SE-PostgreSQL having made > that decision at initdb time. I think we want to keep single-user > mode as lean and mean as possible, so that people can rely on it when > they need to fix their broken database. > I also agree it is nonsense to make access control decision during initdb phase, but it is not the reason why I want to fix this problem. I plan to provide a script that assigns initial security label after the initdb, but before launching postmaster. This script tries to execute postgres in single-user mode, then labels database objects according to the system setting. But the sepgsql module is not loaded currently. I want to kick this job in single-user mode, not normal processing mode, because we can simplify several stuffs. For example, we don't need to check whether the user has privilege to assign initial labels, because it is obvious people who launch initdb has superpower on whole of the database. In addition, we don't need to consider a possibility that someone create a new database object during initial labeling. So, I'd like to fix the problem. Thanks, -- KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On Mon, Aug 16, 2010 at 7:24 PM, Tom Lane wrote:
>
> Well, if it's not just a binary encoding of JSON, I think we can forget
> about it ... certainly it won't work in the form I was visualizing.
>
> regards, tom lane
I just read the spec, and BSON has a lot of bells and whistles
attached (such as labeling binary data with a subtype like UUID or
MD5). With a couple exceptions (see below), any JSON can be converted
to BSON (but the way BSON does arrays is silly: item indices are
stored as strings), but not all BSONs can be converted to JSON without
losing some type details.
Others already mentioned that you can't convert 2 billion byte long
JSON strings to BSON. Another issue is that BSON cannot encode all
JSON numbers without precision loss. JSON can hold any number
matching
'-'? (0 | [1-9][0-9]*) ('.' [0-9]+)? ([Ee] [+-]? [0-9]+)?
but BSON pidgenholes numeric values to either double, int32, int64, or
a 12-byte MongoDB Object ID. Thus, for people who expect JSON to be
able to hold arbitrary-precision numbers (which the JSON data type in
my patch can), using BSON for transfer or storage will violate that
expectation.
Now that I know more about BSON, my opinion is that it shouldn't be
used as the transfer or storage format of the JSON data type. Maybe
if someone wants to do the work, BSON could be implemented as a
contrib module, and functions could be provided in that module to
convert to/from JSON with documented caveats.
Joey Adams
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
1;2403;0cOn Mon, Aug 16, 2010 at 05:02:47PM -0500, Kevin Grittner wrote: > Charles Pritchard wrote: > > > Storing internally as BSON (if it holds up to its premise) would > > mean more efficient traversal of internal objects in the future, > > if we were to have JSON-related functions/selectors. > How about the fact that not all JSON objects can be represented in > BSON (if the JSON object has a very long string) Any such long string wont be representable in pg anyway. Or am I missing something here? Besides that I have to say that I find it pretty strange to design a supposedly generic file-format with a 32bit signed integer length... > , and not all BSON objects can be represented in JSON (if the BSON object has > an > array). Or do we invent our own flavors of one or both to cover the > mismatch? The BSON representation could be purely internal... Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
"Kevin Grittner" writes: > Charles Pritchard wrote: >> Storing internally as BSON (if it holds up to its premise) would >> mean more efficient traversal of internal objects in the future, >> if we were to have JSON-related functions/selectors. > How about the fact that not all JSON objects can be represented in > BSON (if the JSON object has a very long string), and not all BSON > objects can be represented in JSON (if the BSON object has an > array). Well, if it's not just a binary encoding of JSON, I think we can forget about it ... certainly it won't work in the form I was visualizing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Alex Hunsaker writes: > On Mon, Aug 16, 2010 at 14:33, Tom Lane wrote: >> I'd be satisfied with a tool that merges commit reports if they have the >> same log message and occur at approximately the same time, which is the >> heuristic that cvs2cl uses. > I dont think it would be to hard to code that up (main worry is it > might be dog slow). Well, cvs2cl is pretty dog-slow too. As I was just saying to Robert, it doesn't really matter since I only run it a couple times a month. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Robert Haas writes: > On Mon, Aug 16, 2010 at 4:33 PM, Tom Lane wrote: >> I'd be satisfied with a tool that merges commit reports if they have the >> same log message and occur at approximately the same time, which is the >> heuristic that cvs2cl uses. > So how do you run cvs2cl? Do you run it once in a while and save the > output someplace? Or what? Yeah, it's a bit too slow to do on every sync. I run it every week or two and keep the output in a text file. Usually what I want the history for is stuff that happened awhile ago, so the fact that it's not 100% up to date is seldom a factor. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 14:33, Tom Lane wrote: > Alex Hunsaker writes: >> How exactly patches get applied into back branches? > There was discussion about that before, but I don't know whether we > really have a solution that will work comfortably. I don't either, not being a -commiter I don't really follow that area much :-) > A couple of > comments: > > * My practice has always been to develop a fix in HEAD first and then > work backwards. I'm going to resist any tool that tries to force me > to do it the other way. Yep, I agree and as you pointed out it does not work anyway (in the sense of being able to keep the same commit id/hash) because you end up needing to change things. > I'd be satisfied with a tool that merges commit reports if they have the > same log message and occur at approximately the same time, which is the > heuristic that cvs2cl uses. I dont think it would be to hard to code that up (main worry is it might be dog slow). BTW the point about git cherry-pick -x is that it includes the original commit hash in the commit message. That way we don't have to do any guess work based on commit time and log message. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
Charles Pritchard wrote: > Storing internally as BSON (if it holds up to its premise) would > mean more efficient traversal of internal objects in the future, > if we were to have JSON-related functions/selectors. How about the fact that not all JSON objects can be represented in BSON (if the JSON object has a very long string), and not all BSON objects can be represented in JSON (if the BSON object has an array). Or do we invent our own flavors of one or both to cover the mismatch? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On 8/15/10 8:47 PM, Andrew Dunstan wrote: On 08/15/2010 11:03 PM, Tom Lane wrote: Charles Pritchard writes: I'd originally sent this to Joseph Adams, as he has been working on adding a JSON datatype. I've suggested supporting BSON, as there are many client implementations available, I knew there would be a lot of critters crawling out as soon as we turned over this rock. Which other data-formats-of-the-week shall we immortalize as core PG types? If BSON is simply in effect an efficient encoding of JSON, then it's not clear to me that we would want another type at all. Rather, we might want to consider storing the data in this supposedly more efficient format, and maybe also some conversion routines. I agree that we don't want in core a huge array of general serialization formats. The one thing that JSON has going for it for general use, in my view, is that, unlike hstore, the structure is not flat. That makes it potentially useful for various purposes, especially complex structured function arguments, in places where using hstore can be rather limiting, and xml overly verbose. While I certainly haven't done homework on this -- I agree with Andrew. Storing internally as BSON (if it holds up to its premise) would mean more efficient traversal of internal objects in the future, if we were to have JSON-related functions/selectors. The core type would still be json, and would return as text, a json string, but internally it would be stored as BSON, and a function would be available, json_to_bson(typedjsoncol::json), returning a binary string. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
Excerpts from Alvaro Herrera's message of lun ago 16 16:58:31 -0400 2010:
> I suspect that the problem may lie in the "cost_delay rebalance" code in
> autovacuum.
Hmm, so we have this code:
void
AutoVacuumUpdateDelay(void)
{
if (MyWorkerInfo)
{
VacuumCostDelay = MyWorkerInfo->wi_cost_delay;
VacuumCostLimit = MyWorkerInfo->wi_cost_limit;
}
}
where the MyWorkerInfo bits come from shared memory and can be modified
by other autovac worker processes. We could read an incomplete value
into our variables. But this only makes sense if an "int" variable can
be subject to a partial read/write, which we already assume not to be so
(c.f. GetNewTransactionId).
In any case, if you happen to see this reoccur, could you please attach
GDB to the misbehaving worker and see what VacuumCostDelay and
VacuumCostLimit print out as?
--
Álvaro Herrera
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
On Mon, Aug 16, 2010 at 04:34:07PM -0400, Robert Haas wrote: > On Mon, Aug 16, 2010 at 3:00 PM, David Fetter wrote: > >> There has been previous talk of allowing WITH (COPY ...) and I am > >> personally of the opinion that it would be nice to be able to do > >> WITH (EXPLAIN ...). DDL seems like a poor idea. > > > > It may be, but I can see use cases for partitioning... > > Like what? Managing partitions, or creating partitions in some way the partition syntax doesn't anticipate, whatever it turns out to be. > >> P.S. Call me a prude, but your choice of shorthand for > >> insert-update-delete may not be the best. > > > > Then I presume you'll be supporting my idea of using the word > > "span" for temporal data types rather than the current idea whose > > name appears in academic literature. > > Wise guy. In all seriousness, the temporal stuff is giving us a fantastic opportunity, as a project, to break our tradition of lousy naming. "Span" is descriptive, mnemonic, and easy to spell. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
Excerpts from Joe Conway's message of lun ago 16 16:47:19 -0400 2010: > On 08/16/2010 12:12 PM, Josh Berkus wrote: > > > >> I've also recently heard a report of vacuum hanging on 8.3.x on Solaris > >> Sparc. Any chance you can get a backtrace from a build with debug symbols? > > > > The problem is that we haven't been able to reproduce the bug in > > testing. Like I said, it only seems to happen occasionally ... like > > maybe once in 10 or 20 (or more?) autovacuums. We've never been seen it > > with a manual vacuum at all. > > > > And we can't rebuild the production servers. > > Hmmm, well I don't know how to reproduce it on demand either -- I'll try > to get a backtrace from the wild if possible. I'll keep you posted... FWIW there's also a report of it hanging in FreeBSD, but sadly when the process is inspected under truss, it dies because of its "parent PID" attribute changing underneath and thus triggering the safety feature that makes it die if the parent postmaster disappears. I suspect that the problem may lie in the "cost_delay rebalance" code in autovacuum. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
On 08/16/2010 12:12 PM, Josh Berkus wrote: > >> I've also recently heard a report of vacuum hanging on 8.3.x on Solaris >> Sparc. Any chance you can get a backtrace from a build with debug symbols? > > The problem is that we haven't been able to reproduce the bug in > testing. Like I said, it only seems to happen occasionally ... like > maybe once in 10 or 20 (or more?) autovacuums. We've never been seen it > with a manual vacuum at all. > > And we can't rebuild the production servers. Hmmm, well I don't know how to reproduce it on demand either -- I'll try to get a backtrace from the wild if possible. I'll keep you posted... Joe -- Joe Conway credativ LLC: http://www.credativ.us Linux, PostgreSQL, and general Open Source Training, Service, Consulting, & 24x7 Support signature.asc Description: OpenPGP digital signature
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 4:33 PM, Tom Lane wrote: > I'd be satisfied with a tool that merges commit reports if they have the > same log message and occur at approximately the same time, which is the > heuristic that cvs2cl uses. So how do you run cvs2cl? Do you run it once in a while and save the output someplace? Or what? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
On Mon, Aug 16, 2010 at 3:00 PM, David Fetter wrote: >> There has been previous talk of allowing WITH (COPY ...) and I am >> personally of the opinion that it would be nice to be able to do >> WITH (EXPLAIN ...). DDL seems like a poor idea. > > It may be, but I can see use cases for partitioning... Like what? >> P.S. Call me a prude, but your choice of shorthand for >> insert-update-delete may not be the best. > > Then I presume you'll be supporting my idea of using the word "span" > for temporal data types rather than the current idea whose name > appears in academic literature. Wise guy. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Alex Hunsaker writes: > How exactly patches get applied into back branches? Has that been > spelled out somewhere? There are a lot of ways to do it. For > instance git.git seems to apply the patch to the earliest branch first > and then merge it on up so that everything can share the same > commit/hash. That looks like a royal PITA to me, and I assume the > plan is to just cherry-pick commits back. As long as we use git > cherry-pick -x, I agree with Magnus, it should be fairly easy to write > a short script to do it. II'll even volunteer if the above is > basically the only requirement :-). There was discussion about that before, but I don't know whether we really have a solution that will work comfortably. A couple of comments: * My practice has always been to develop a fix in HEAD first and then work backwards. I'm going to resist any tool that tries to force me to do it the other way. There are a couple of reasons for that: one, I'm generally more familiar with HEAD, and two, I want HEAD to have the cleanest solution. If you do an old branch first, you'll probably come up with a solution that is good for that branch but could be improved in newer ones, eg by using some subroutine or facility that doesn't exist earlier. Forward-patching won't encourage you to find that. * My experience is that a patch that has to go back more than one or two branches is almost never exactly the same on each branch, even without any of the non-trivial changes suggested above. We constantly do things like rearrange the arguments of some function that's used everywhere. So "patch" is definitely not smart enough to back-patch the fixes by itself. Maybe git will be a lot smarter but I'm not expecting miracles. Anything that is based on "same hash" is pretty much guaranteed to not do what I need. I'd be satisfied with a tool that merges commit reports if they have the same log message and occur at approximately the same time, which is the heuristic that cvs2cl uses. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 12:45, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Aug 16, 2010 at 20:27, Tom Lane wrote: >>> Second, does git offer a way to collate matching log entries across >>> multiple branches? > >> But what really is the usecase there? > > Generating back-branch update release notes, mainly. > > 2010-08-13 12:27 tgl > > * src/backend/: catalog/namespace.c, utils/cache/plancache.c > (REL9_0_STABLE), catalog/namespace.c, utils/cache/plancache.c > (REL8_3_STABLE), catalog/namespace.c, utils/cache/plancache.c > (REL8_4_STABLE), catalog/namespace.c, utils/cache/plancache.c: Yeah... it cant really. You can git log --decorate which will add any tags or branches a commit is in, but it breaks for merges and only works if the commit hash is the same (and if its the *current* commit on the branch I think). Skimming the git mailing list, it seems the powers that be think the above is stupid pointless and wrong (out of touch with reality or what?). Basically the argument is if you want to back patch something you probably need to change it in some way and touch up the commit message anyway. So just include any relevant info in the commit message and you can script something to parse and extract that info if you care. This (long) thread sums it up http://thread.gmane.org/gmane.comp.version-control.git/95381/focus=95386. How exactly patches get applied into back branches? Has that been spelled out somewhere? There are a lot of ways to do it. For instance git.git seems to apply the patch to the earliest branch first and then merge it on up so that everything can share the same commit/hash. That looks like a royal PITA to me, and I assume the plan is to just cherry-pick commits back. As long as we use git cherry-pick -x, I agree with Magnus, it should be fairly easy to write a short script to do it. II'll even volunteer if the above is basically the only requirement :-). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] refactoring comment.c
Robert Haas writes: > 5. Since I'm hoping Tom will read this, I ran it through filterdiff. :-) OK, I looked ;-). It mostly looks good, but of course I've got some opinions... > 2. I haven't done anything about moving the definition of > ObjectAddress elsewhere, as Alvaro suggested, because I'm not sure > quite where it ought to go. I still think it's a good idea, though > I'm not dead set on it, either. Suggestions? I think the problem is you're trying to put this into backend/parser which is not really the right place for it. It's an execution-time animal not a parse-time animal. I would put it into backend/catalog, perhaps named objectaddress.c, and similarly the header file would be objectaddress.h. Then it would be reasonable to move struct ObjectAddress into this header and have dependency.h #include it. There might be some other stuff in dependency.c that more naturally belongs here, too. > 3. I fixed the issue Kaigai Kohei spotted, regarding > LargeObjectRelationId vs. LargeObjectMetadataRelationId, by adding a > grotty hack. However, I feel that I'm not so much adding a new grotty > hack as working around an existing grotty hack which was added for > reasons I'm unclear on. Is there a pg_upgrade-related reason not to > revert the original hack instead? It's not pg_upgrade (nor psql, nor pg_dump) we are protecting here. It's third-party applications that might understand the contents of pg_description, pg_depend, etc. I think that hack is quite small and localized enough to live with, rather than causing a flag day for an unknown number of clients by changing the representation. + /* +* Translate the parser representation which identifies this object into +* an ObjectAddress. get_object_address() will throw an error if the +* object does not exist. +*/ + address = get_object_address(stmt->objtype, stmt->objname, stmt->objargs, +&relation, ShareUpdateExclusiveLock); I think this comment should also explicitly mention that we're getting a lock to protect against concurrent DROPs. + /* + * Translate an object name and arguments (as passed by the parser) to an + * ObjectAddress. + * + * The returned object will be locked using the specified lockmode. If a + * sub-object is looked up, the parent object will be locked instead. + * + * We don't currently provide a function to release the locks acquired here; + * typically, the lock must be held until commit to guard against a concurrent + * drop operation. + */ + ObjectAddress + get_object_address(ObjectType objtype, List *objname, List *objargs, + Relation *relp, LOCKMODE lockmode) This comment's a bit shy of a load too, since it totally fails to mention the relp argument, or specify what the caller is required to do with it, or explain how the locking on the relation works. As a matter of style, I'd suggest putting the single externally callable function (ie get_object_address) at the top of the file not the bottom. People shouldn't have to read through the entire file before finding out what API it is supposed to provide to the outside world. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
Josh Berkus writes: > This is something I'd swear we fixed around 8.3.2. However, I'm seeing > it again in production, and was wondering if anyone could remember what > the root cause was and how we fixed it. Hmm, I can't find anything in the 8.3-series CVS logs suggesting that there was a post-8.3.0 fix related to vacuum delays. > The problem is that sometimes (but not the majority of times) autovaccum > with cost_delay is going into a pathological cycle where it polls the > system clock after reading every single disk page of a table. What I find interesting about that trace is the large proportion of writes. That appears to me to indicate that it's *not* a matter of vacuum delays, or at least not just a matter of that. The process seems to be getting involved in having to dump dirty buffers to disk. Perhaps the background writer is malfunctioning? Another idea that comes to mind is that you have vacuum_cost_page_dirty set to an unreasonably large value, so that autovac is blocking whenever it has to write even one page. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
> I've also recently heard a report of vacuum hanging on 8.3.x on Solaris > Sparc. Any chance you can get a backtrace from a build with debug symbols? The problem is that we haven't been able to reproduce the bug in testing. Like I said, it only seems to happen occasionally ... like maybe once in 10 or 20 (or more?) autovacuums. We've never been seen it with a manual vacuum at all. And we can't rebuild the production servers. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
On 08/16/2010 11:24 AM, Josh Berkus wrote: > All, > > This is something I'd swear we fixed around 8.3.2. However, I'm seeing > it again in production, and was wondering if anyone could remember what > the root cause was and how we fixed it. I've also recently heard a report of vacuum hanging on 8.3.x on Solaris Sparc. Any chance you can get a backtrace from a build with debug symbols? Joe -- Joe Conway credativ LLC: http://www.credativ.us Linux, PostgreSQL, and general Open Source Training, Service, Consulting, & 24x7 Support signature.asc Description: OpenPGP digital signature
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
On Mon, Aug 16, 2010 at 02:25:50PM -0400, Robert Haas wrote: > On Sun, Aug 15, 2010 at 7:44 AM, Hitoshi Harada wrote: > > We (Marko, David Fetter and I) discussed on IRC about design of > > writeable CTEs. It does and will contain not only syntax but also > > miscellaneous specifications, general rules and restrictions. I hope > > this will help the patch reviews and stop dangerous design at the > > early stage. If you find something wrong, or have request, please > > notify. > > > > http://wiki.postgresql.org/wiki/WriteableCTEs > > > > We will keep to add details. Any comments are welcome. > > There are really two separate features here, and it might be worth > giving them separate names and separate designs (and separate > patches). Allowing the main query to be an insert, update, or delete > seems easier than allowing the toplevel CTEs to contain those > constructs, although I might be wrong about that. Interesting. I'd kinda seen them as the same thing. > Under features, what is DCL? Data Control Language, i.e. GRANT and REVOKE. > There has been previous talk of allowing WITH (COPY ...) and I am > personally of the opinion that it would be nice to be able to do > WITH (EXPLAIN ...). DDL seems like a poor idea. It may be, but I can see use cases for partitioning... > P.S. Call me a prude, but your choice of shorthand for > insert-update-delete may not be the best. Then I presume you'll be supporting my idea of using the word "span" for temporal data types rather than the current idea whose name appears in academic literature. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
9.0 open issues (was Re: [HACKERS] Git migration timeline)
Robert Haas writes: > That sounds good, except for the part about nobody doing anything > about the 9.0 open issues. Those issues are: > * Backup procedure is wrong? - Nobody's been able to clearly > articulate what the problem is, and according to Fujii Masao it's been > this way since 8.2. Just because it's old doesn't mean it's not a bug ;-). Normally I would say that a pre-existing documentation problem isn't a release blocker, but in this case that procedure is likely to become of interest to a whole lot more people than it was before, thanks to SR/HS. So we need to understand whether there's a problem and fix it. > * Walreceiver crashes in AIX - The person who originally reported this > problem has been slow to get back to us with the information needed to > figure out what is going on. Yes. If he doesn't provide adequate data, and we can't reproduce it elsewhere, we should not consider this a release blocker. > * BUG #5595: Documentation is not installs from VPATH build. - I'm not > sure this is really a release-blocker, but either way, I have been > assuming it's Peter's issue to fix. I believe this is a regression from prior branches, probably caused by the switch away from distributing the prebuilt docs as sub-tarballs. As such, it's at least a candidate release-blocker. > * trace_recovery_messages inconsistency - This one seems like a simple > case of deciding what makes most sense, and doing it. We should > definitely fix this before the wrap. Agreed, it's a matter of getting some consensus. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
"Kevin Grittner" writes: > Nobody responded when I asked about this recently, but shouldn't > that list include "BUG #5607: memmory leak in ecpg"? We have a > patch from Zoltán Böszörményi from before this bug report which > seems to address the issue and which Michael Meskes said "Feel free > to apply". > We don't want to ship 9.0 with known memory leaks, do we? Better a memory leak than broken ecpg ;-). Nobody except Michael is terribly comfortable with that code, so we'd all rather wait for him to review and apply the patch. More generally, pre-existing bugs have never been considered release stoppers. At this point what we would block the release for is *new* bugs in 9.0. (An exception to that general rule is pre-existing bugs that would require an initdb to fix; but this one isn't that either.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Conflicted names of error conditions.
Thanks for you answer, Tom! I've implemented mapping between SQLSTATE codes and C++ exception classes of my library. And of course, I've resolved the conflict of names by giving a proper name to my classes. Regards, Dmitriy 2010/8/16 Tom Lane > Dmitriy Igrishin writes: > > According to > > http://www.postgresql.org/docs/9.0/static/errcodes-appendix.html > > some error conditions has non-unique *names*. There are: > > modifying_sql_data_not_permitted, > > prohibited_sql_statement_attempted, > > reading_sql_data_not_permitted > > from SQL Routine Exception and External Routine Exception classes. > > > It should be? > > Yup, that's what the SQL standard calls them :-(. In practice, either > underlying SQLSTATE will match that name in an EXCEPTION block, so > it doesn't matter a whole lot. If you have a case where you feel it > does matter, you can trap by the SQLSTATE code instead. > >regards, tom lane >
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 20:45, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Aug 16, 2010 at 20:27, Tom Lane wrote: >>> Second, does git offer a way to collate matching log entries across >>> multiple branches? > >> But what really is the usecase there? > > Generating back-branch update release notes, mainly. We usually do that > first for the newest back branch, and then copy paste and edit as needed > into the older ones. It's a lot easier to see what needs to be adjusted > if you're looking at something like > > 2010-08-13 12:27 tgl > > * src/backend/: catalog/namespace.c, utils/cache/plancache.c > (REL9_0_STABLE), catalog/namespace.c, utils/cache/plancache.c > (REL8_3_STABLE), catalog/namespace.c, utils/cache/plancache.c > (REL8_4_STABLE), catalog/namespace.c, utils/cache/plancache.c: Fix > Assert failure in PushOverrideSearchPath when trying to restore a > search path that specifies useTemp, but there is no active temp > schema in the current session. (This can happen if the path was > saved during a transaction that created a temp schema and was later > rolled back.) For existing callers it's sufficient to ignore the > useTemp flag in this case, though we might later want to offer an > option to create a fresh temp schema. So far as I can tell this is > just an Assert failure: in a non-assert build, the code would push > a zero onto the new search path, which is useless but not very > harmful. Per bug report from Heikki. > > than half a dozen independent lists. > > I've also found that answering questions about when some old patch got > added is easier from this format than I think it'd be if I had only > per-branch lists. I do have both the combined log history and > per-branch log histories at hand (from separate cvs2cl runs), but I find > that I hardly ever consult the latter. Hmm. Ok. I don't know if it does, so I'll hope someone else can tell us if it does :-) BTW, you do have things like "git log --grep=foo" to search the log directly, instead of working through cvs2cl output of course, but that doesn't quite solve your problem, I can see that. > It's not a showstopper, but if git can't do it I'll be disappointed. If there's no way to do it offhand, I'm pretty sure we can write a short script that does it for us. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Magnus Hagander writes: > On Mon, Aug 16, 2010 at 20:27, Tom Lane wrote: >> Second, does git offer a way to collate matching log entries across >> multiple branches? > But what really is the usecase there? Generating back-branch update release notes, mainly. We usually do that first for the newest back branch, and then copy paste and edit as needed into the older ones. It's a lot easier to see what needs to be adjusted if you're looking at something like 2010-08-13 12:27 tgl * src/backend/: catalog/namespace.c, utils/cache/plancache.c (REL9_0_STABLE), catalog/namespace.c, utils/cache/plancache.c (REL8_3_STABLE), catalog/namespace.c, utils/cache/plancache.c (REL8_4_STABLE), catalog/namespace.c, utils/cache/plancache.c: Fix Assert failure in PushOverrideSearchPath when trying to restore a search path that specifies useTemp, but there is no active temp schema in the current session. (This can happen if the path was saved during a transaction that created a temp schema and was later rolled back.) For existing callers it's sufficient to ignore the useTemp flag in this case, though we might later want to offer an option to create a fresh temp schema. So far as I can tell this is just an Assert failure: in a non-assert build, the code would push a zero onto the new search path, which is useless but not very harmful. Per bug report from Heikki. than half a dozen independent lists. I've also found that answering questions about when some old patch got added is easier from this format than I think it'd be if I had only per-branch lists. I do have both the combined log history and per-branch log histories at hand (from separate cvs2cl runs), but I find that I hardly ever consult the latter. It's not a showstopper, but if git can't do it I'll be disappointed. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Triggers on VIEWs
On 16 August 2010 18:50, Tom Lane wrote: > Dean Rasheed writes: >> On 15 August 2010 18:38, Dean Rasheed wrote: >>> There are still a number of things left todo: >>> - extend ALTER VIEW with enable/disable trigger commands > >> On further reflection, I wonder if the ability to disable VIEW >> triggers is needed/wanted at all. > > AFAIK the only real use for disabling triggers is in connection with > trigger-based replication systems. A view wouldn't be carrying > replication-related triggers anyway, so I think this could certainly > be left out of a first cut. > > You aren't saying that you can see some reason why this couldn't be > added later, if wanted, correct? > Yes. It should be easy to add later if wanted. I just don't see much use for it, and I don't want to add more to an already quite big patch, if it's not really needed. Regards, Dean -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
Robert Haas wrote: > That sounds good, except for the part about nobody doing anything > about the 9.0 open issues. Those issues are: > > [four issues listed] Nobody responded when I asked about this recently, but shouldn't that list include "BUG #5607: memmory leak in ecpg"? We have a patch from Zoltán Böszörményi from before this bug report which seems to address the issue and which Michael Meskes said "Feel free to apply". We don't want to ship 9.0 with known memory leaks, do we? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 20:27, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Aug 16, 2010 at 20:11, Tom Lane wrote: >>> The other thing that I'd like to see some data on is the commit log >>> entries. Can we produce anything comparable to cvs2cl output from >>> the test repository? > >> For a single branch, just do "git log ", e.g. "git log >> master" or "git log REL8_2_STABLE" on your clone. > >> Is that enough, or do you need one for all branches at once? > > Well, I guess there are two sub-parts to my question then. First, and > most important for the immediate issue, have you done anything to verify > the commit-message data matches between the cvs and git repositories? Not beyond manually looking both using "git log" and the gitweb interface. > Second, does git offer a way to collate matching log entries across > multiple branches? The main advantage of cvs2cl has always been that it > would do that, so that you didn't have to look at five independent log > entries after a commit that fixed the same bug in five branches... I don't know about that - somebody else? If there isn't one, it should be fairly simple to write one, since tracking the changelog is much easier in git than cvs (given that it's global and not per-file). But what really is the usecase there? If you run "git log" on REL8_4_STABLE you get the changelog for REL8_4_STABLE, and if you run it on master, you get it for the tip. In that mode, you'll never end up getting them twice. Are you saying you want a cross-branch changelog, that also groups all the commit messages together? Or am I misunderstanding you? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Magnus Hagander writes: > On Mon, Aug 16, 2010 at 20:11, Tom Lane wrote: >> The other thing that I'd like to see some data on is the commit log >> entries. Can we produce anything comparable to cvs2cl output from >> the test repository? > For a single branch, just do "git log ", e.g. "git log > master" or "git log REL8_2_STABLE" on your clone. > Is that enough, or do you need one for all branches at once? Well, I guess there are two sub-parts to my question then. First, and most important for the immediate issue, have you done anything to verify the commit-message data matches between the cvs and git repositories? Second, does git offer a way to collate matching log entries across multiple branches? The main advantage of cvs2cl has always been that it would do that, so that you didn't have to look at five independent log entries after a commit that fixed the same bug in five branches... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs Desgin Doc on Wiki
On Sun, Aug 15, 2010 at 7:44 AM, Hitoshi Harada wrote: > We (Marko, David Fetter and I) discussed on IRC about design of > writeable CTEs. It does and will contain not only syntax but also > miscellaneous specifications, general rules and restrictions. I hope > this will help the patch reviews and stop dangerous design at the > early stage. If you find something wrong, or have request, please > notify. > > http://wiki.postgresql.org/wiki/WriteableCTEs > > We will keep to add details. Any comments are welcome. There are really two separate features here, and it might be worth giving them separate names and separate designs (and separate patches). Allowing the main query to be an insert, update, or delete seems easier than allowing the toplevel CTEs to contain those constructs, although I might be wrong about that. Under features, what is DCL? There has been previous talk of allowing WITH (COPY ...) and I am personally of the opinion that it would be nice to be able to do WITH (EXPLAIN ...). DDL seems like a poor idea. P.S. Call me a prude, but your choice of shorthand for insert-update-delete may not be the best. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Return of the Solaris vacuum polling problem -- anyone remember this?
All, This is something I'd swear we fixed around 8.3.2. However, I'm seeing it again in production, and was wondering if anyone could remember what the root cause was and how we fixed it. The problem is that sometimes (but not the majority of times) autovaccum with cost_delay is going into a pathological cycle where it polls the system clock after reading every single disk page of a table. On large tables, this results in vacuum not completing within the lifetime of the server. In most cases, killing autovaccuum and restarting it will cause it to behave normally. The below is the truss from the exhibited issue on 8.3.11 on Solaris 10u7, compiled with sun cc: pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 lseek(39, 0x28F88000, SEEK_SET) = 0x28F88000 write(39, "0E10\0\0E0 CB5C101\001\0".., 8192) = 8192 lseek(39, 0x28FCA000, SEEK_SET) = 0x28FCA000 read(39, " q\r\0\0 `9CD2B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " F0E\0\090A888 H01\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " q\r\0\0C819D3B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 lseek(39, 0x28F9, SEEK_SET) = 0x28F9 write(39, "0E10\0\0 0 gB7C101\001\0".., 8192) = 8192 lseek(39, 0x28FD, SEEK_SET) = 0x28FD read(39, " q\r\0\0 X 8D3B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " t0F\0\0 H +8F !01\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " q\r\0\0F0 sD3B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " F0E\0\0 0C888 H01\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 lseek(39, 0x28FDA000, SEEK_SET) = 0x28FDA000 read(39, " q\r\0\0C0D1D3B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " q\r\0\0D8F0D3B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " F0E\0\0800189 H01\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " q0F\0\0D0 ^A9F701\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " F0E\0\010 ?89 H01\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " q\r\0\0 x mD4B001\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " F0E\0\0 X _89 H01\001\0".., 8192)= 8192 pollsys(0xFD7FFFDF9030, 0, 0xFD7FFFDF90C0, 0x) = 0 read(39, " q\r\0\0 @ADD4B001\001\0".., 8192)= 8192 For contrast, this is normal behavior: read(10, " }\0\0\0 X82 >E301\0\0\0".., 8192)= 8192 read(10, " }\0\0\018 4 ME301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0E881 NE301\0\0\0".., 8192)= 8192 semop(16777221, 0xFD7FFFDF8FB8, 1) = 0 read(10, " }\0\0\0 PEE \E301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0\b k ^E301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 8E0 jE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 P07 nE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0D885 xE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 8D }E301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 xD280E301\0\0\0".., 8192)= 8192 read(10, " }\0\0\010DF8CE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0E09E8EE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0C8E29CE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\080889EE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0B0 UADE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0C0E4BCE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 !C0E301\0\0\0".., 8192)= 8192 read(10, " }\0\0\010 UCDE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0F8EBCEE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\08092DDE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0A8 QDFE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 x cEDE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0D8 "EFE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 P15FAE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0C8C0FDE301\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 PFC\tE401\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 8C3\rE401\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 @890FE401\0\0\0".., 8192)= 8192 read(10, " }\0\0\0 r11E401\0\0\0".., 8192)= 8192 -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsq
Re: [HACKERS] Todays git migration results
On Mon, Aug 16, 2010 at 20:11, Tom Lane wrote: > Magnus Hagander writes: >> Attached is a ZIP file with the diffs generated when converting the >> cvs repo to git based off a cvs snapshot from this morning. It >> contains a diff file for every branch and every tag present. (If a >> file is missing, that means there were no diffs for that branch/tag). > >> It's a lot of diffs - 135. But many of those are because the exact sam >> ething is in all tags on a branch. The directory "unique" contains one >> copy of a unique set of diffs (doesn't look at the individual changes, >> just the complete diff file), which is "only" 30 different files. > >> As before, almost everything seems related to the initial import and >> vendor branch. There is nothing in any code. > > I'm curious about the discrepancies in the $Date$ tags in some of the > doc/FAQ_xxx files. It's surely not a showstopper, but I'd feel better > if we understood the cause of that. Everything else seems to be > disagreement about the vendor branch version numbers, which I'm happy > to write off as a conversion artifact. > > The other thing that I'd like to see some data on is the commit log > entries. Can we produce anything comparable to cvs2cl output from > the test repository? For a single branch, just do "git log ", e.g. "git log master" or "git log REL8_2_STABLE" on your clone. Is that enough, or do you need one for all branches at once? (if you don't have a local clone of it, lmk and I can generate that output for you) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
On Mon, Aug 16, 2010 at 1:47 PM, Tom Lane wrote: > I wrote: >> Magnus Hagander writes: >>> According to the decision at the developer meeting, the migration to >>> git should happen 17-20 Aug. Here's my proposed timeline. This will >>> obviously affect development work some, and since the original >>> timeline called for us having already released 9.0 buy then ;) > >> Core is currently talking about exactly when we want to push out >> 9.0beta5 and/or 9.1alpha1, and the scheduling of the git transition has >> to wait on that decision. I don't think we're going to be ready for it >> to begin on Tuesday. > > The current feeling among core seems to be that we should allow the git > conversion to proceed according to Magnus' proposed schedule. This > would mean that we will *not* be able to wrap either a 9.0 update or > 9.1alpha1 releases this week. What we'd probably try to do instead is > wrap 9.1alpha1 early next week, as the first attempt to generate > tarballs from the git repository, and then wrap 9.0beta5 (or rc1) later > in the week. This slips the 9.0 schedule an extra week, but considering > that nobody seems to be doing anything about the open 9.0 issues, that > was likely to happen anyway. > > If anybody's really unhappy with this timeline, speak up now. That sounds good, except for the part about nobody doing anything about the 9.0 open issues. Those issues are: * Backup procedure is wrong? - Nobody's been able to clearly articulate what the problem is, and according to Fujii Masao it's been this way since 8.2. * Walreceiver crashes in AIX - The person who originally reported this problem has been slow to get back to us with the information needed to figure out what is going on. It would be helpful if someone else could test to see if this a general problem with AIX, or something specific to the reporter's installation. Or if someone wants to provide me with a login... * BUG #5595: Documentation is not installs from VPATH build. - I'm not sure this is really a release-blocker, but either way, I have been assuming it's Peter's issue to fix. * trace_recovery_messages inconsistency - This one seems like a simple case of deciding what makes most sense, and doing it. We should definitely fix this before the wrap. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Todays git migration results
Magnus Hagander writes: > Attached is a ZIP file with the diffs generated when converting the > cvs repo to git based off a cvs snapshot from this morning. It > contains a diff file for every branch and every tag present. (If a > file is missing, that means there were no diffs for that branch/tag). > It's a lot of diffs - 135. But many of those are because the exact sam > ething is in all tags on a branch. The directory "unique" contains one > copy of a unique set of diffs (doesn't look at the individual changes, > just the complete diff file), which is "only" 30 different files. > As before, almost everything seems related to the initial import and > vendor branch. There is nothing in any code. I'm curious about the discrepancies in the $Date$ tags in some of the doc/FAQ_xxx files. It's surely not a showstopper, but I'd feel better if we understood the cause of that. Everything else seems to be disagreement about the vendor branch version numbers, which I'm happy to write off as a conversion artifact. The other thing that I'd like to see some data on is the commit log entries. Can we produce anything comparable to cvs2cl output from the test repository? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Triggers on VIEWs
Dean Rasheed writes: > On 15 August 2010 18:38, Dean Rasheed wrote: >> There are still a number of things left todo: >> - extend ALTER VIEW with enable/disable trigger commands > On further reflection, I wonder if the ability to disable VIEW > triggers is needed/wanted at all. AFAIK the only real use for disabling triggers is in connection with trigger-based replication systems. A view wouldn't be carrying replication-related triggers anyway, so I think this could certainly be left out of a first cut. You aren't saying that you can see some reason why this couldn't be added later, if wanted, correct? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
I wrote: > Magnus Hagander writes: >> According to the decision at the developer meeting, the migration to >> git should happen 17-20 Aug. Here's my proposed timeline. This will >> obviously affect development work some, and since the original >> timeline called for us having already released 9.0 buy then ;) > Core is currently talking about exactly when we want to push out > 9.0beta5 and/or 9.1alpha1, and the scheduling of the git transition has > to wait on that decision. I don't think we're going to be ready for it > to begin on Tuesday. The current feeling among core seems to be that we should allow the git conversion to proceed according to Magnus' proposed schedule. This would mean that we will *not* be able to wrap either a 9.0 update or 9.1alpha1 releases this week. What we'd probably try to do instead is wrap 9.1alpha1 early next week, as the first attempt to generate tarballs from the git repository, and then wrap 9.0beta5 (or rc1) later in the week. This slips the 9.0 schedule an extra week, but considering that nobody seems to be doing anything about the open 9.0 issues, that was likely to happen anyway. If anybody's really unhappy with this timeline, speak up now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Triggers on VIEWs
On 15 August 2010 18:38, Dean Rasheed wrote: > There are still a number of things left todo: > - extend ALTER VIEW with enable/disable trigger commands On further reflection, I wonder if the ability to disable VIEW triggers is needed/wanted at all. I just noticed that while it is possible to disable a RULE on a TABLE, it is not possible to do so on VIEW. This certainly makes sense for the _RETURN rule, although possibly some people might have a use for disabling other rules on views. The situation with triggers is similar - disabling an INSTEAD OF trigger would be pointless, and could only lead to errors when updating the view. Some people may have a use case for disabling BEFORE and AFTER statement triggers on views, but I suspect that the number of such cases is small, and I'm tempted to omit this, for now at least. Thoughts? - Dean -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On 16/08/10 20:17, Joseph Adams wrote: Also, an idea would be to make json_send and json_recv (binary JSON send/receive) use BSON rather than JSON-encoded text, as sending/receiving JSON-encoded text is exactly what text send/receive do. The usual reason to use the binary format is performance, so it doesn't make much sense to use BSON for that if the internal storage format is something else. It would most likely be slower, not faster, than sending the string as is. Of course, if you switch to using BSON as the internal storage format, then it's natural to use that for the binary I/O format too. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal / proof of concept: Triggers on VIEWs
On 16 August 2010 14:45, David Fetter wrote: > Please add this to the next commitfest :) > Done. Thanks, Dean > https://commitfest.postgresql.org/action/commitfest_view?id=7 > > Cheers, > David. > -- > David Fetter http://fetter.org/ > Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter > Skype: davidfetter XMPP: david.fet...@gmail.com > iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics > > Remember to vote! > Consider donating to Postgres: http://www.postgresql.org/about/donate > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On Sun, Aug 15, 2010 at 11:47 PM, Andrew Dunstan wrote: > > If BSON is simply in effect an efficient encoding of JSON, then it's not > clear to me that we would want another type at all. Rather, we might want to > consider storing the data in this supposedly more efficient format, and > maybe also some conversion routines. An issue is that the current JSON data type implementation preserves the original text (meaning if you say '[1,2,"\u0020" ]'::JSON, it will yield '[1,2,"\u0020" ]' rather than '[1,2," "]' . I haven't researched BSON much at all, but I seriously doubt that part of its spec includes handling external JSON encoding details like whitespace and superfluous escapes. Even though I spent a long time implementing it, the original text preservation feature should be dropped, in my opinion. Users tend to care more about the data inside of a JSON value rather than how it's encoded, and replacement of values can have funky consequences on indentation when working with indented JSON text (similar to how pasting in a text editor puts pasted content at the wrong indent level). By dropping original text preservation, in the future (or now), JSON could be encoded in BSON (or a more efficient format) in the database rather than in JSON-encoded text. Also, an idea would be to make json_send and json_recv (binary JSON send/receive) use BSON rather than JSON-encoded text, as sending/receiving JSON-encoded text is exactly what text send/receive do. Joey Adams -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Committers info for the git migration - URGENT!
On Mon, Aug 16, 2010 at 17:20, Itagaki Takahiro wrote: > 2010/8/16 Magnus Hagander : >> If you want to *change* your email address from the one in the list >> attached here, please let me know *ASAP*. At the latest I need to know >> this before tuesday evening european time (see separate email about >> migration timeline). > > Could you change my address to itagaki.takah...@gmail.com ? > Thanks. Updated. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PL/pgSQL EXECUTE '..' USING with unknown
On 16/08/10 03:35, Tom Lane wrote: Heikki Linnakangas writes: One approach is to handle the conversion from unknown to the right data type transparently in the backend. Attached patch adds a coerce-param-hook for fixed params that returns a CoerceViaIO node to convert the param to the right type at runtime. That's quite similar to the way unknown constants are handled. The idea of using a coerce_hook instead of inventing several new API layers is attractive, but have you checked that there are no callers for which this would be a bad idea? That code is used in a lot of different contexts, but I can't see any where this could cause a problem. In general, I can't think of a case where we would want to throw an error on an unknown parameter where we accept an unknown constant at the same location. Completely rejecting unknown parameters might make sense in some contexts, but that's not the current behavior either, unknown parameters are accepted in some contexts. Another issue is that this fails to mimic the usual varparams behavior that a Param of unknown type should be resolved to only one type when it is referenced in multiple places. I'm not sure that that's a critical behavior, but I'm definitely not sure that it's not. Yeah, that's exactly what I was referring to when I said: The patch doesn't currently check that a parameter is only resolved to one type in the same query, but that can be added. I'll add that check. Better to be conservative and relax it later if needed, than to be lenient now and regret it later. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On 8/16/10 8:40 AM, Christopher Browne wrote: On Mon, Aug 16, 2010 at 11:21 AM, Robert Haas wrote: On Mon, Aug 16, 2010 at 11:05 AM, Tom Lane wrote: Andrew Dunstan writes: If BSON is simply in effect an efficient encoding of JSON, then it's not clear to me that we would want another type at all. Rather, we might want to consider storing the data in this supposedly more efficient format, and maybe also some conversion routines. Hmm, that's an interesting plan ... It is interesting, but I'm not sure that it will actually work out well in practice. If what we want to do is compress JSON, TOAST will do that for us without any additional code, and probably a lot more efficiently. Of course, until someone tests it, we're just speculating wildly. Yep, that was exactly what struck me. TOAST is quite likely to be a good answer for this. The reason to want some other binary format would be if there were other benefits to be had. An "XML encoding" format could be interesting if it allowed having GIST-ish indexes to search for tags particularly efficiently. I say "XML encoding" because I've not got any reason to think that a JSON/BSON-only format would necessarily be preferable. But "interesting" isn't the same thing as "the right answer." For now, TOAST seems perfectly reasonable. If there's some "wire format for XML" that would allow more efficient data transfer, that would be an improvement. BSON sounds like it's something like that, but only if it's better than "flavour of the week." XML encoding has certainly been investigated within the W3C public docs: http://www.w3.org/2003/08/binary-interchange-workshop/Report.html (discussion) http://www.w3.org/TR/xbc-characterization/ (summary) Leading to the current draft of EXI: http://www.w3.org/XML/EXI/ The spec is a rather large undertaking. It makes sense to add to the XML ToDo wiki page. EXI will certainly be better than TOAST for larger XML docs. ... BSON does not compress text content -- TOAST would still have its advantages. It mainly shortens the representation of JSON data structures. Again, I think the primary benefit of BSON would be data traversal. The benefit is the same for a client receiving BSON, as the server. Data lengths are specified, allowing quick optimizations for things like key_exists and equivalencies. Client's supporting BSON could benefit from a quick pass-through. And I'd imagine a very slight benefit toward indexing, were GIN / hstore processes used. Still, as has been noted on this thread.. We don't have numbers to work with. With json as a core data type; and "bson" as a possible function working with the json type, there's not much of a risk going in either direction (text + TOAST, bson + TOAST). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Committers info for the git migration - URGENT!
On 08/16/2010 06:38 AM, Magnus Hagander wrote: > The current mapping used is the same one as on git.postgresql.org (see > attached file). > > Per discussions earlier on this list, we encourage people to use an > email address that is permanent and stable, and does not for example > change if you change your ISP or if you change employer. But in the > end, the decision of which email address to use is up to you. > > If you want to *change* your email address from the one in the list > attached here, please let me know *ASAP*. At the latest I need to know > this before tuesday evening european time (see separate email about > migration timeline). Interesting list -- some names on there I haven't seen in a long time... Mine is fine. Thanks, Joe -- Joe Conway credativ LLC: http://www.credativ.us Linux, PostgreSQL, and general Open Source Training, Service, Consulting, & 24x7 Support signature.asc Description: OpenPGP digital signature
Re: [HACKERS] Committers info for the git migration - URGENT!
Excerpts from Magnus Hagander's message of lun ago 16 09:38:12 -0400 2010: > If you want to *change* your email address from the one in the list > attached here, please let me know *ASAP*. At the latest I need to know > this before tuesday evening european time (see separate email about > migration timeline). FWIW my address is fine. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "Bogus data in lock file" shouldn't be FATAL?
Alvaro Herrera writes: > In that case, maybe consider fsync'ing it. BTW, I see you already proposed that in the thread at http://archives.postgresql.org/message-id/e3e180dc0905070719q58136caai23fbb777fd3c0...@mail.gmail.com I'm not sure how come the idea fell through the cracks, but we should surely have done it then. I think I was reading the initial complaint as being just logfile corruption, and failed to make the connection to the postmaster.pid file. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] refactoring comment.c
Excerpts from KaiGai Kohei's message of lun ago 16 00:19:54 -0400 2010: > (2010/08/16 11:50), Robert Haas wrote: > When we were developing largeobject access controls, Tom Lane commented > as follows: > > * Re: [HACKERS] [PATCH] Largeobject access controls > http://marc.info/?l=pgsql-hackers&m=125548822906571&w=2 > | I notice that the patch decides to change the pg_description classoid for > | LO comments from pg_largeobject's OID to pg_largeobject_metadata's. This > | will break existing clients that look at pg_description (eg, pg_dump and > | psql, plus any other clients that have any intelligence about comments, > | for instance it probably breaks pgAdmin). And there's just not a lot of > | return that I can see. I agree that using pg_largeobject_metadata would > | be more consistent given the new catalog layout, but I'm inclined to think > | we should stick to the old convention on compatibility grounds. Given > | that choice, for consistency we'd better also use pg_largeobject's OID not > | pg_largeobject_metadata's in pg_shdepend entries for LOs. > > He concerned about existing applications which have knowledge about internal > layout of system catalogs, then I fixed up the patch according to the > suggestion. I think that with this patch we have the return for the change that we didn't have previously. A patch that changes it should also fix pg_dump and psql at the same time, but IMHO it doesn't make sense to keep adding grotty hacks on top of it. Maybe we could do with a grotty hack in obj_description() instead? (...checks...) Oh, it's defined as a SQL function directly in pg_proc.h :-( -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "Bogus data in lock file" shouldn't be FATAL?
Alvaro Herrera writes: > Excerpts from Tom Lane's message of lun ago 16 11:49:51 -0400 2010: >> The bottom line here is that it's not clear to me whether changing this >> would be a net reliability improvement or not. Maybe better to leave >> it alone. > In that case, maybe consider fsync'ing it. Hrm ... I had supposed we did fsync lockfiles, but a look at the code shows not. This seems like a clear oversight. I think we should not only add that, but back-patch it. It seems entirely plausible that the lack of an fsync is exactly what led to the original complaint. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Conflicted names of error conditions.
Dmitriy Igrishin writes: > According to > http://www.postgresql.org/docs/9.0/static/errcodes-appendix.html > some error conditions has non-unique *names*. There are: > modifying_sql_data_not_permitted, > prohibited_sql_statement_attempted, > reading_sql_data_not_permitted > from SQL Routine Exception and External Routine Exception classes. > It should be? Yup, that's what the SQL standard calls them :-(. In practice, either underlying SQLSTATE will match that name in an EXCEPTION block, so it doesn't matter a whole lot. If you have a case where you feel it does matter, you can trap by the SQLSTATE code instead. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "Bogus data in lock file" shouldn't be FATAL?
Excerpts from Tom Lane's message of lun ago 16 11:49:51 -0400 2010: > The bottom line here is that it's not clear to me whether changing this > would be a net reliability improvement or not. Maybe better to leave > it alone. In that case, maybe consider fsync'ing it. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "Bogus data in lock file" shouldn't be FATAL?
Excerpts from Tom Lane's message of lun ago 16 11:18:32 -0400 2010: > We could perhaps address that risk another way: after having written > postmaster.pid, try to read it back to verify that it contains what we > wrote, and abort if not. Then, if we can't read it during startup, > it's okay to assume there is no conflicting postmaster. Makes sense. BTW some past evidence: http://archives.postgresql.org/message-id/e3e180dc0905070719q58136caai23fbb777fd3c0...@mail.gmail.com -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
On Mon, 2010-08-16 at 16:19 +0200, Magnus Hagander wrote: > On Mon, Aug 16, 2010 at 16:15, Tom Lane wrote: > > Magnus Hagander writes: > >> On Mon, Aug 16, 2010 at 15:58, Tom Lane wrote: > >>> I can see the point of wanting to be dead certain the repository isn't > >>> changing under you during the data migration. Perhaps we need an agreed > >>> window between last call for commits and the actual lock-out. > > > >> To prevent that, I just need to shut it down for 2 minutes to rsync > >> the latest changes off it and onto the new git box. Maybe that's how > >> we should do it. > > > > That sounds like a reasonable scheme to me, especially since it leaves > > the cvs repository functional. Dunno about you, but I want a fallback > > plan in case this turns into a disaster ... > > Oh, I had no intention whatsoever of *removing* it. Just disabling login to > it. +1 JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] "Bogus data in lock file" shouldn't be FATAL?
Robert Haas writes: > On Mon, Aug 16, 2010 at 11:18 AM, Tom Lane wrote: >> We could perhaps address that risk another way: after having written >> postmaster.pid, try to read it back to verify that it contains what we >> wrote, and abort if not. Then, if we can't read it during startup, >> it's okay to assume there is no conflicting postmaster. > What if it was readable when written but has since become unreadable? Yup, that's the weak spot in any such assumption. One might also draw an analogy to the case of failing to open postmaster.pid because of permissions change, which seems at least as likely as a data change. And we consider that as fatal, for good reason I think. > My basic feeling on this is that manual intervention to start the > server is really undesirable and we should try hard to avoid needing > it. That having been said, accidentally starting two postmasters at > the same time that are accessing the same data files would be several > orders of magnitude worse. We can't afford to compromise on any > interlock mechanisms that are necessary to prevent that from > happening. Yeah. At the same time, it's really really bad to encourage people to remove postmaster.pid manually as the first attempt to fix anything. That completely destroys whatever interlock you thought you had. So it's not too hard to make a case that avoiding this scenario will really make things safer not less so. The bottom line here is that it's not clear to me whether changing this would be a net reliability improvement or not. Maybe better to leave it alone. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On Mon, 2010-08-16 at 11:40 -0400, Christopher Browne wrote: > On Mon, Aug 16, 2010 at 11:21 AM, Robert Haas wrote: > > On Mon, Aug 16, 2010 at 11:05 AM, Tom Lane wrote: > >> Andrew Dunstan writes: > >>> If BSON is simply in effect an efficient encoding of JSON, then it's not > >>> clear to me that we would want another type at all. Rather, we might > >>> want to consider storing the data in this supposedly more efficient > >>> format, and maybe also some conversion routines. > >> > >> Hmm, that's an interesting plan ... > > > > It is interesting, but I'm not sure that it will actually work out > > well in practice. If what we want to do is compress JSON, TOAST will > > do that for us without any additional code, and probably a lot more > > efficiently. Of course, until someone tests it, we're just > > speculating wildly. > > Yep, that was exactly what struck me. TOAST is quite likely to be a > good answer for this. Except: How much JSON data will actually be TOASTed? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On Mon, Aug 16, 2010 at 11:21 AM, Robert Haas wrote: > On Mon, Aug 16, 2010 at 11:05 AM, Tom Lane wrote: >> Andrew Dunstan writes: >>> If BSON is simply in effect an efficient encoding of JSON, then it's not >>> clear to me that we would want another type at all. Rather, we might >>> want to consider storing the data in this supposedly more efficient >>> format, and maybe also some conversion routines. >> >> Hmm, that's an interesting plan ... > > It is interesting, but I'm not sure that it will actually work out > well in practice. If what we want to do is compress JSON, TOAST will > do that for us without any additional code, and probably a lot more > efficiently. Of course, until someone tests it, we're just > speculating wildly. Yep, that was exactly what struck me. TOAST is quite likely to be a good answer for this. The reason to want some other binary format would be if there were other benefits to be had. An "XML encoding" format could be interesting if it allowed having GIST-ish indexes to search for tags particularly efficiently. I say "XML encoding" because I've not got any reason to think that a JSON/BSON-only format would necessarily be preferable. But "interesting" isn't the same thing as "the right answer." For now, TOAST seems perfectly reasonable. If there's some "wire format for XML" that would allow more efficient data transfer, that would be an improvement. BSON sounds like it's something like that, but only if it's better than "flavour of the week." -- http://linuxfinances.info/info/linuxdistributions.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Conflicted names of error conditions.
Hey all, According to http://www.postgresql.org/docs/9.0/static/errcodes-appendix.html some error conditions has non-unique *names*. There are: modifying_sql_data_not_permitted, prohibited_sql_statement_attempted, reading_sql_data_not_permitted from SQL Routine Exception and External Routine Exception classes. It should be? Regards, Dmitriy
Re: [HACKERS] "Bogus data in lock file" shouldn't be FATAL?
On Mon, Aug 16, 2010 at 11:18 AM, Tom Lane wrote: > This complaint: > http://archives.postgresql.org/pgsql-admin/2010-08/msg00111.php > > seems to suggest that this code in CreateLockFile() is not well-thought-out: > > if (other_pid <= 0) > elog(FATAL, "bogus data in lock file \"%s\": \"%s\"", > filename, buffer); > > as it means that a corrupted (empty, in this case) postmaster.pid file > prevents the server from starting until somebody intervenes manually. > > I think that the original concern was that if we couldn't read valid > data out of postmaster.pid then we couldn't be sure if there were a > conflicting postmaster running. But if that's the plan then > CreateLockFile is violating it further down, where it happily skips the > PGSharedMemoryIsInUse check if it fails to pull shmem ID numbers from > the file. > > We could perhaps address that risk another way: after having written > postmaster.pid, try to read it back to verify that it contains what we > wrote, and abort if not. Then, if we can't read it during startup, > it's okay to assume there is no conflicting postmaster. What if it was readable when written but has since become unreadable? My basic feeling on this is that manual intervention to start the server is really undesirable and we should try hard to avoid needing it. That having been said, accidentally starting two postmasters at the same time that are accessing the same data files would be several orders of magnitude worse. We can't afford to compromise on any interlock mechanisms that are necessary to prevent that from happening. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Committers info for the git migration - URGENT!
2010/8/16 Magnus Hagander : > If you want to *change* your email address from the one in the list > attached here, please let me know *ASAP*. At the latest I need to know > this before tuesday evening european time (see separate email about > migration timeline). Could you change my address to itagaki.takah...@gmail.com ? Thanks. -- Itagaki Takahiro -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
On Mon, Aug 16, 2010 at 11:05 AM, Tom Lane wrote: > Andrew Dunstan writes: >> If BSON is simply in effect an efficient encoding of JSON, then it's not >> clear to me that we would want another type at all. Rather, we might >> want to consider storing the data in this supposedly more efficient >> format, and maybe also some conversion routines. > > Hmm, that's an interesting plan ... It is interesting, but I'm not sure that it will actually work out well in practice. If what we want to do is compress JSON, TOAST will do that for us without any additional code, and probably a lot more efficiently. Of course, until someone tests it, we're just speculating wildly. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] "Bogus data in lock file" shouldn't be FATAL?
This complaint: http://archives.postgresql.org/pgsql-admin/2010-08/msg00111.php seems to suggest that this code in CreateLockFile() is not well-thought-out: if (other_pid <= 0) elog(FATAL, "bogus data in lock file \"%s\": \"%s\"", filename, buffer); as it means that a corrupted (empty, in this case) postmaster.pid file prevents the server from starting until somebody intervenes manually. I think that the original concern was that if we couldn't read valid data out of postmaster.pid then we couldn't be sure if there were a conflicting postmaster running. But if that's the plan then CreateLockFile is violating it further down, where it happily skips the PGSharedMemoryIsInUse check if it fails to pull shmem ID numbers from the file. We could perhaps address that risk another way: after having written postmaster.pid, try to read it back to verify that it contains what we wrote, and abort if not. Then, if we can't read it during startup, it's okay to assume there is no conflicting postmaster. Or, given the infrequency of complaints, maybe it's better not to touch this. Thoughts? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] security label support, part.2
* Kevin Grittner (kevin.gritt...@wicourts.gov) wrote: > Many of the features KaiGai has discussed would fit nicely with > court requirements -- and might even be prerequisites for > considering moving security to the database level. Mandating > identical security for all tables in a hierarchy would be a problem. What you're describing isn't how inheiritance used to work in PG anyway, so it's not really like we've made things worse. What used to happen is that if your query against the parent table happened to hit a table you didn't have access to, it'd fail outright with a permissions error, not just skip over the things you didn't have access to. That certainly wasn't ideal. I think what you're really looking for is RLS (Row-Level Security), which I think we would want to implement independently of the inheiritance system (though it'd have to work with it, of course). That's certainly something that I think would be great to have in PG and would ideally be something which would address both of your "sometimes everything is public except rows which look like X" and "all of these types are non-public" situations. I don't believe it's something that could be addressed *only* by inheiritance though, in any case. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] JSON Patch for PostgreSQL - BSON Support?
Andrew Dunstan writes: > If BSON is simply in effect an efficient encoding of JSON, then it's not > clear to me that we would want another type at all. Rather, we might > want to consider storing the data in this supposedly more efficient > format, and maybe also some conversion routines. Hmm, that's an interesting plan ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] shared_preload_libraries is ignored in single user mode
2010/8/16 KaiGai Kohei : > Although nobody paid an attention, it seems to me a problem to be fixed. > > The attached patch fixes the problem using a simple idea which adds > process_shared_preload_libraries() at PostgresMain() when we launched > it in single-user mode. I have no confidence at all that this is a sane thing to do. I think any enhanced security provider that needs system objects to be labelled should provide a script to label them after the fact. You can't count on everyone who wants to use SE-PostgreSQL having made that decision at initdb time. I think we want to keep single-user mode as lean and mean as possible, so that people can rely on it when they need to fix their broken database. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PL/pgSQL EXECUTE '..' USING with unknown
=?ISO-8859-1?Q?C=E9dric_Villemain?= writes: > Yes, and you point out another thing. EXECUTE is a way to bypass the > named prepare statement, to be sure query is replanned each time. > Unfortunely the current implementation of EXECUTE USING is not working > this way. Uh ... what do you base that statement on? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Committers info for the git migration - URGENT!
On Mon, Aug 16, 2010 at 9:38 AM, Magnus Hagander wrote: > Attention committers!! ;) > > When we migrate to git, we will do a one-time mapping of your old > username to an email address (as was discussed on the developer > meeting in Ottawa earlier this year). This is stamped on every commit > you have ever done, since that's how git works. It's part of the > commit itself, so it *cannot* be changed once this is done. We can of > course change which email address is used for *new* commits, but not > for the existing once once they have gone in. > > The current mapping used is the same one as on git.postgresql.org (see > attached file). > > Per discussions earlier on this list, we encourage people to use an > email address that is permanent and stable, and does not for example > change if you change your ISP or if you change employer. But in the > end, the decision of which email address to use is up to you. > > If you want to *change* your email address from the one in the list > attached here, please let me know *ASAP*. At the latest I need to know > this before tuesday evening european time (see separate email about > migration timeline). > > Since there is now a very tight timeline on this (sorry!), please ping > any other committers you have on your IM/IRC list that you know don't > read their -hackers email daily... Please make me rh...@postgresql.org - thanks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
On Mon, Aug 16, 2010 at 16:15, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Aug 16, 2010 at 15:58, Tom Lane wrote: >>> I can see the point of wanting to be dead certain the repository isn't >>> changing under you during the data migration. Perhaps we need an agreed >>> window between last call for commits and the actual lock-out. > >> To prevent that, I just need to shut it down for 2 minutes to rsync >> the latest changes off it and onto the new git box. Maybe that's how >> we should do it. > > That sounds like a reasonable scheme to me, especially since it leaves > the cvs repository functional. Dunno about you, but I want a fallback > plan in case this turns into a disaster ... Oh, I had no intention whatsoever of *removing* it. Just disabling login to it. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Git migration timeline
On Mon, Aug 16, 2010 at 10:15 AM, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Aug 16, 2010 at 15:58, Tom Lane wrote: >>> I can see the point of wanting to be dead certain the repository isn't >>> changing under you during the data migration. Perhaps we need an agreed >>> window between last call for commits and the actual lock-out. > >> To prevent that, I just need to shut it down for 2 minutes to rsync >> the latest changes off it and onto the new git box. Maybe that's how >> we should do it. > > That sounds like a reasonable scheme to me, especially since it leaves > the cvs repository functional. Dunno about you, but I want a fallback > plan in case this turns into a disaster ... I don't think anybody proposed permanently deleting the CVS repository. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
