Re: [HACKERS] Block-level CRC checks
Simon Riggs wrote: There is no creation of corruption events. This scheme detects corruption events that *have* occurred. Now I understand that we previously would have recovered seamlessly from such events, but they were corruption events nonetheless and I think they need to be reported. (For why, see Conclusion #2, below). No, you're still missing the point. The point is *not* random bit errors affecting hint bits, but the torn page problem. Today, a torn page is a completely valid and expected behavior from the OS and storage subsystem. We handle it with full_page_writes, and by relying on the fact that it's OK for a hint bit set to get lost. With your scheme, a torn page would become a corrupt page. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On 12/1/09, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Nov 30, 2009 at 4:54 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: Le 30 nov. 2009 à 22:38, Robert Haas a écrit : I still don't really understand why we wouldn't want RESET ALL to reset the application name. In what circumstances would you want the application name to stay the same across a RESET ALL? I can't see any use case, but SET/RESET is tied to SESSION whereas application_name is a CONNECTION property. So it's a hard sell that reseting the session will change connection properties. Is there any technical difference between a connection property and a session property? If so, what is it? The point is that every other thing you can set in a libpq connection string is persistent throughout the connection. For the ones that you can change at all, such as client_encoding, *RESET ALL actually resets it to what was specified in the connection string*. It does not satisfy the POLA for application_name to behave differently. +1 This SESSION/CONNECITION terminology is confusing, better would be talk about client connection/session (client-pooler) and server connection/session (pooler-server) if you are talking about pooling. I think the argument about poolers expecting something different is hogwash. A pooler would want RESET ALL to revert the connection state to what it was at establishment. That would include whatever application name the pooler would have specified when it started the connection, I should think. The only reason we're even having this discussion is that libpq isn't able to make application_name work exactly like its other connection parameters because of the backwards-compatibility issue. Maybe we should think a bit harder about that. Or else give up having libpq manage it like a connection parameter. Making it work in session pooling mode (pgpool) is easy - RESET ALL and SET needs to work. The question is whether it should work also in transaction pooling mode (pgbouncer / JDBC). I see 2 variants: 1. Clients are allowed to specify it only in startup packet. But, uh, poolers can set it also in the middle of session. 2. Make it into protocol-tracked variable. The 1) seems inconsistent and backwards-incompatible - client does not know server version yet and old servers dont accept it. I don't see problems with 2). Or we could decide it is not meant for transaction pooling environments. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 10:04 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: There is no creation of corruption events. This scheme detects corruption events that *have* occurred. Now I understand that we previously would have recovered seamlessly from such events, but they were corruption events nonetheless and I think they need to be reported. (For why, see Conclusion #2, below). No, you're still missing the point. The point is *not* random bit errors affecting hint bits, but the torn page problem. Today, a torn page is a completely valid and expected behavior from the OS and storage subsystem. We handle it with full_page_writes, and by relying on the fact that it's OK for a hint bit set to get lost. With your scheme, a torn page would become a corrupt page. Well, its easy to keep going on about how much you think I misunderstand. But I think that's just misdirection. The way we handle torn page corruptions *hides* actual corruptions from us. The frequency of true positives and false positives is important here. If the false positive ratio is very small, then reporting them is not a problem because of the benefit we get from having spotted the true positives. Some convicted murderers didn't do it, but that is not an argument for letting them all go free (without knowing the details). So we need to know what the false positive ratio is before we evaluate the benefit of either reporting or non-reporting possible corruption events. When do you think torn pages happen? Only at crash, or other times also? Do they always happen at crash? Are there ways to re-check a block that has suffered a hint-related torn page issue? Are there ways to isolate and minimise the reporting of false positives? Those are important questions and this is not black and white. If the *only* answer really is we-must-WAL-log everything, then that is the answer, as an option. I suspect that there is a less strict possibility, if we question our assumptions and look at the frequencies. We know that I have no time to work on this; I am just trying to hold open the door to a few possibilities that we have not fully considered in a balanced way. And I myself am guilty of having slammed the door previously. I encourage development of a way forward based upon a balance of utility. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tue, Dec 1, 2009 at 12:26 AM, Andres Freund and...@anarazel.de wrote: Actually I think the poolers make a good case for a SET variant which emulates connection set variables... RESET ALL in a connection pooler does different things than RESET ALL outside of one. Eh? Not sure I follow that, but then I haven't had a coffee yet. I do see the argument that RESET ALL should revert user changes to application_name though, but I maintain they should reset to the value set at connection time, not to null. As has been pointed out already, other values set at connection time cannot be reset, so allowing that for application name does seem like a POLA violation. Upthread, Tom suggested a new 'SET DEFAULT ...' variant of SET which could be used to set the default GUC value that RESET would revert to. This seems to me to be the ideal solution, and I'd somewhat hesitantly volunteer to work on it (hesitantly as it means touching the parser and other areas of the code I currently have no experience of). -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tuesday 01 December 2009 09:59:17 Dave Page wrote: On Tue, Dec 1, 2009 at 12:26 AM, Andres Freund and...@anarazel.de wrote: Actually I think the poolers make a good case for a SET variant which emulates connection set variables... RESET ALL in a connection pooler does different things than RESET ALL outside of one. Eh? Not sure I follow that, but then I haven't had a coffee yet. Well. RESET ALL in a pooler sets values to the initial connection values the pooler had, not the ones of pooled connection. On the same time there are multiple people complaining about such default values being contraproductive to pooling environments because they reset to the wrong values. I dont really get that argument - the pooler should just issue a SET CONNECTION DEFAULT for all connection values. That would make it far more transparent than before... Upthread, Tom suggested a new 'SET DEFAULT ...' variant of SET which could be used to set the default GUC value that RESET would revert to. This seems to me to be the ideal solution, and I'd somewhat hesitantly volunteer to work on it (hesitantly as it means touching the parser and other areas of the code I currently have no experience of). As I had initially suggested something like that I agree here. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
Dave Page wrote: Upthread, Tom suggested a new 'SET DEFAULT ...' variant of SET which could be used to set the default GUC value that RESET would revert to. This seems to me to be the ideal solution, and I'd somewhat hesitantly volunteer to work on it (hesitantly as it means touching the parser and other areas of the code I currently have no experience of). If an application can do SET DEFAULT, how does the connection pooler *really* reset the value back to what it was? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tue, Dec 1, 2009 at 9:16 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Dave Page wrote: Upthread, Tom suggested a new 'SET DEFAULT ...' variant of SET which could be used to set the default GUC value that RESET would revert to. This seems to me to be the ideal solution, and I'd somewhat hesitantly volunteer to work on it (hesitantly as it means touching the parser and other areas of the code I currently have no experience of). If an application can do SET DEFAULT, how does the connection pooler *really* reset the value back to what it was? There has to be some level of trust here :-). As the alternative would involve bumping the fe-be protocol version, it seems like a reasonable approach to me. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tuesday 01 December 2009 10:16:45 Heikki Linnakangas wrote: Dave Page wrote: Upthread, Tom suggested a new 'SET DEFAULT ...' variant of SET which could be used to set the default GUC value that RESET would revert to. This seems to me to be the ideal solution, and I'd somewhat hesitantly volunteer to work on it (hesitantly as it means touching the parser and other areas of the code I currently have no experience of). If an application can do SET DEFAULT, how does the connection pooler *really* reset the value back to what it was? Why does it need to? SET DEFAULT should imho only be allowed for values whcih can be set during connection initiation. For those it can simply issue the sets anyway. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
The point is that every other thing you can set in a libpq connection string is persistent throughout the connection. For the ones that you can change at all, such as client_encoding, *RESET ALL actually resets it to what was specified in the connection string*. It does not satisfy the POLA for application_name to behave differently. I think the argument about poolers expecting something different is hogwash. A pooler would want RESET ALL to revert the connection state to what it was at establishment. That would include whatever application name the pooler would have specified when it started the connection, I should think. +1. Connection poolers shoud be transparent to the clients. If some connection poolers want to behavior differently, then probably they would be better to be called TP monitor or some such. TP monitor has its own API and it is at liberty behave what it wants. Don't get me wrong. I would not say TP monitor is useless, rather it has very usefull use cases I think. However, pushing its semantics about sessions to PostgreSQL side, would be counterproductive for both TP monitor and PostgreSQL. -- Tatsuo Ishii SRA OSS, Inc. Japan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New VACUUM FULL
On Mon, 2009-11-30 at 14:38 +0900, Itagaki Takahiro wrote:
* VACUUM (FULL REPLACE) pg_class should be rejected, not silently
turned into VACUUM (FULL INPLACE) pg_class.
Hmmm, it requires to remember whether REPLACE is specified or not
for the non-INPLACE vacuum, but I don't want to add VACOPT_REPLACE
only for the purpose.
I just removed FULL REPLACE syntax; Only FULL and FULL INPLACE are
available. VACUUM FULL without INPLACE behaves as cluster-like rewrites
for non-system tables. FULL INPLACE is a traditional vacuum full.
System catalogs are always vacuumed with INPLACE version.
- VACUUM FULL / VACUUM (FULL) = rewritten version
- VACUUM (FULL INPLACE) = traditional version
Ok, looks good. It's cleaner now, too.
It might make the code cleaner, but I want vacuumdb in 8.5 to work on older
versions of servers unless we use the new feature. Older servers can only
accept older syntax, so I avoided using the new syntax if not needed.
(The new patch still uses two versions of syntax.)
Good point. I attached a suggestion of how it might look if you detected
the server version explicitly. You don't have to use it, but it's what I
had in mind.
Also, I think the current version fails if -i is passed and it's
connecting to an old server, so explicit server version detection may be
required.
* The patch should be merged with CVS HEAD. The changes required are
minor; but notice that there is a minor style difference in the assert
in vacuum().
Very minor style issue: it looks like Tom specifically changed the order
of the expression in the Assert() from your first vacuum options patch.
I attached a diff to show you what I mean -- the complex boolean
expressions are easier to read if the styles match.
* vacuumdb should reject -i without -f
* The replace or inplace option is a magical default, because VACUUM
FULL defaults to replace for user tables and inplace for system
tables. I tried to make that more clear in my documentation suggestions.
* There are some windows line endings in the patch, which should be
removed.
Great, thank you for the patch!
Marking as ready.
Regards,
Jeff Davis
*** a/src/backend/commands/vacuum.c
--- b/src/backend/commands/vacuum.c
***
*** 303,310 vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
Assert((vacstmt-options VACOPT_VACUUM) ||
!(vacstmt-options (VACOPT_FULL | VACOPT_FREEZE)));
Assert((vacstmt-options VACOPT_ANALYZE) || vacstmt-va_cols == NIL);
! Assert(!(vacstmt-options VACOPT_INPLACE) ||
! (vacstmt-options VACOPT_FULL));
stmttype = (vacstmt-options VACOPT_VACUUM) ? VACUUM : ANALYZE;
--- 303,310
Assert((vacstmt-options VACOPT_VACUUM) ||
!(vacstmt-options (VACOPT_FULL | VACOPT_FREEZE)));
Assert((vacstmt-options VACOPT_ANALYZE) || vacstmt-va_cols == NIL);
! Assert((vacstmt-options VACOPT_FULL) ||
! !(vacstmt-options VACOPT_INPLACE));
stmttype = (vacstmt-options VACOPT_VACUUM) ? VACUUM : ANALYZE;
*** a/src/bin/scripts/vacuumdb.c
--- b/src/bin/scripts/vacuumdb.c
***
*** 203,225 vacuum_one_database(const char *dbname, bool full, bool inplace, bool verbose,
PQExpBufferData sql;
PGconn *conn;
initPQExpBuffer(sql);
appendPQExpBuffer(sql, VACUUM);
! if (inplace)
{
! appendPQExpBuffer(sql, (FULL INPLACE);
if (freeze)
! appendPQExpBuffer(sql, , FREEZE);
if (verbose)
! appendPQExpBuffer(sql, , VERBOSE);
if (analyze)
! appendPQExpBuffer(sql, , ANALYZE);
! appendPQExpBuffer(sql, ));
}
else
{
if (full)
appendPQExpBuffer(sql, FULL);
if (freeze)
--- 203,250
PQExpBufferData sql;
PGconn *conn;
+ int version;
+ bool first_opt = true;
initPQExpBuffer(sql);
+ conn = connectDatabase(dbname, host, port, username, prompt_password, progname);
+ version = PQserverVersion(conn);
+
appendPQExpBuffer(sql, VACUUM);
!
! if (version = 80500)
{
! if (full)
! {
! appendPQExpBuffer(sql, %sFULL%s, first_opt ? ( : , ,
! inplace ? INPLACE : );
! first_opt = false;
! }
if (freeze)
! {
! appendPQExpBuffer(sql, %sFREEZE, first_opt ? ( : , );
! first_opt = false;
! }
if (verbose)
! {
! appendPQExpBuffer(sql, %sVERBOSE, first_opt ? ( : , );
! first_opt = false;
! }
if (analyze)
! {
! appendPQExpBuffer(sql, %sANALYZE, first_opt ? ( : , );
! first_opt = false;
! }
! if (!first_opt)
! appendPQExpBuffer(sql, ));
}
else
{
+ /*
+ * On older servers, VACUUM FULL is equivalent to VACUUM (FULL
+ * INPLACE) on newer servers, so we can ignore 'inplace'.
+ */
if (full)
appendPQExpBuffer(sql, FULL);
if (freeze)
***
*** 229,239 vacuum_one_database(const char *dbname, bool full, bool inplace, bool verbose,
if (analyze)
appendPQExpBuffer(sql, ANALYZE);
}
if (table)
appendPQExpBuffer(sql, %s, table);
Re: [HACKERS] Application name patch - v4
On 12/1/09, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Dave Page wrote: Upthread, Tom suggested a new 'SET DEFAULT ...' variant of SET which could be used to set the default GUC value that RESET would revert to. This seems to me to be the ideal solution, and I'd somewhat hesitantly volunteer to work on it (hesitantly as it means touching the parser and other areas of the code I currently have no experience of). If an application can do SET DEFAULT, how does the connection pooler *really* reset the value back to what it was? By doing SET DEFAULT... There actually *is* a problem that SET DEFAULT would solve: 1) Pooler creates a connection with one default value. 2) Client creates a connection with another default value (giving param in startup pkt) 3) Pooler does SET to apply client's default values. 4) Client does SET to some random value 5) Client does RESET foo/ALL; expecting get default value from 2), instead it gets poolers default value from 1). The inconsistency would be fixed if pooler could do SET DEFAULT in 3). Note - client doing SET DEFAULT itself would not break anything. As long we are talking about protocol-tracked parameters... OTOH, the only app that exhibits the such RESET problem is src/test/regress so I'm not sure it's worth spending effort to fix this. Especially as this open door on app doing SET DEFAULT on non-tracked GUC vars, which seems to be a much bigger problem. I don't see how this SET DEFAULT would fix the appname vs. poolers problem in any way. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enable-thread-safety defaults?
Peter Eisentraut wrote:
On m?n, 2009-11-30 at 12:21 -0500, Bruce Momjian wrote:
! for thread safety; use --disable-thread-safety to disable
threading.])
--disable-thread-safety does not disable threading, it disables thread
safety.
Good point! Patch updated and attached.
What are we going to do for build farm members who don't support
threading? Is someone going to manually update their configure flags?
--
Bruce Momjian br...@momjian.ushttp://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: configure
===
RCS file: /cvsroot/pgsql/configure,v
retrieving revision 1.659
diff -c -c -r1.659 configure
*** configure 30 Nov 2009 16:50:37 - 1.659
--- configure 1 Dec 2009 11:22:36 -
***
*** 823,829
enable_depend
enable_cassert
enable_thread_safety
- enable_thread_safety_force
with_tcl
with_tclconfig
with_perl
--- 823,828
***
*** 1497,1505
--enable-dtrace build with DTrace support
--enable-depend turn on automatic dependency tracking
--enable-cassertenable assertion checks (for debugging)
! --enable-thread-safety make client libraries thread-safe
! --enable-thread-safety-force
! force thread-safety despite thread test failure
--disable-float4-byval disable float4 passed by value
--disable-float8-byval disable float8 passed by value
--disable-largefile omit support for large files
--- 1496,1502
--enable-dtrace build with DTrace support
--enable-depend turn on automatic dependency tracking
--enable-cassertenable assertion checks (for debugging)
! --disable-thread-safety disable thread-safety in client libraries
--disable-float4-byval disable float4 passed by value
--disable-float8-byval disable float8 passed by value
--disable-largefile omit support for large files
***
*** 4859,4892
#
{ $as_echo $as_me:$LINENO: checking allow thread-safe client libraries 5
$as_echo_n checking allow thread-safe client libraries... 6; }
- if test $PORTNAME != win32; then
-
-
- # Check whether --enable-thread-safety was given.
- if test ${enable_thread_safety+set} = set; then
- enableval=$enable_thread_safety;
- case $enableval in
- yes)
- :
- ;;
- no)
- :
- ;;
- *)
- { { $as_echo $as_me:$LINENO: error: no argument expected for --enable-thread-safety option 5
- $as_echo $as_me: error: no argument expected for --enable-thread-safety option 2;}
-{ (exit 1); exit 1; }; }
- ;;
- esac
-
- else
- enable_thread_safety=no
-
- fi
-
-
- else
- # Win32 should always use threads
# Check whether --enable-thread-safety was given.
--- 4856,4861
***
*** 4912,4953
fi
- fi
-
-
-
- # Check whether --enable-thread-safety-force was given.
- if test ${enable_thread_safety_force+set} = set; then
- enableval=$enable_thread_safety_force;
- case $enableval in
- yes)
- :
- ;;
- no)
- :
- ;;
- *)
- { { $as_echo $as_me:$LINENO: error: no argument expected for --enable-thread-safety-force option 5
- $as_echo $as_me: error: no argument expected for --enable-thread-safety-force option 2;}
-{ (exit 1); exit 1; }; }
- ;;
- esac
-
- else
- enable_thread_safety_force=no
-
- fi
-
-
- if test $enable_thread_safety = yes -o \
- $enable_thread_safety_force = yes; then
- enable_thread_safety=yes # for 'force'
-
- cat confdefs.h \_ACEOF
- #define ENABLE_THREAD_SAFETY 1
- _ACEOF
-
- fi
{ $as_echo $as_me:$LINENO: result: $enable_thread_safety 5
$as_echo $enable_thread_safety 6; }
--- 4881,4886
***
*** 21316,21325
if test $PTHREAD_CC != $CC; then
{ { $as_echo $as_me:$LINENO: error:
PostgreSQL does not support platforms that require a special compiler
! for thread safety. 5
$as_echo $as_me: error:
PostgreSQL does not support platforms that require a special compiler
! for thread safety. 2;}
{ (exit 1); exit 1; }; }
fi
--- 21249,21258
if test $PTHREAD_CC != $CC; then
{ { $as_echo $as_me:$LINENO: error:
PostgreSQL does not support platforms that require a special compiler
! for thread safety; use --disable-thread-safety to disable thread safety. 5
$as_echo $as_me: error:
PostgreSQL does not support platforms that require a special compiler
! for thread safety; use --disable-thread-safety to disable thread safety. 2;}
{ (exit 1); exit 1; }; }
fi
***
*** 21465,21472
if test x$ac_cv_header_pthread_h = xyes; then
:
else
! { { $as_echo $as_me:$LINENO: error: pthread.h not found, required for --enable-thread-safety 5
! $as_echo $as_me: error: pthread.h not found, required for --enable-thread-safety
Re: [HACKERS] Block-level CRC checks
Simon Riggs wrote: The way we handle torn page corruptions *hides* actual corruptions from us. The frequency of true positives and false positives is important here. If the false positive ratio is very small, then reporting them is not a problem because of the benefit we get from having spotted the true positives. Some convicted murderers didn't do it, but that is not an argument for letting them all go free (without knowing the details). So we need to know what the false positive ratio is before we evaluate the benefit of either reporting or non-reporting possible corruption events. When do you think torn pages happen? Only at crash, or other times also? Do they always happen at crash? Are there ways to re-check a block that has suffered a hint-related torn page issue? Are there ways to isolate and minimise the reporting of false positives? Those are important questions and this is not black and white. If the *only* answer really is we-must-WAL-log everything, then that is the answer, as an option. I suspect that there is a less strict possibility, if we question our assumptions and look at the frequencies. We know that I have no time to work on this; I am just trying to hold open the door to a few possibilities that we have not fully considered in a balanced way. And I myself am guilty of having slammed the door previously. I encourage development of a way forward based upon a balance of utility. I think the problem boils down to what the user response should be to a corruption report. If it is a torn page, it would be corrected and the user doesn't have to do anything. If it is something that is not correctable, then the user has corruption and/or bad hardware. I think the problem is that the existing proposal can't distinguish between these two cases so the user has no idea how to respond to the report. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
On Mon, Nov 30, 2009 at 10:16 PM, Greg Smith g...@2ndquadrant.com wrote: Considering that one of those was a holiday week with a lot of schedule disruption proceeding it, I don't know how much faster things could have moved. There were large chunks of the reviewer volunteers who wanted only jobs they could finish before the holiday, and others who weren't available at all until afterwards. And I don't know about every else, but I took all four days off and started today behind by several hundred list messages. I was planning to start nagging again tomorrow, hoping that most would be caught up from any holiday time off too by then. Right now, of the 16 patches listed in Needs Review besides the ECPG ones Michael is working through, the breakdown is like this: Not reviewed at all yet: 6 Reviewed once, updated, waiting for re-review: 10 So the bulk of the problem for keeping the pipeline moving is in these re-reviews holding up transitions to Ready for committer. I've had some discussion with Robert about how to better distinguish in the management app when the reviewer has work they're expected to do vs. when they think they're done with things. We're converging toward a more clear set of written guidelines to provide for managing future CommitFests as part of that, right now there's a few too many fuzzy parts for my liking. If the need here is to speed up how fast things are fed to committers, we can certainly do that. The current process still favors having reviewers do as much as possible first, as shown by all the stuff sitting in the re-review queue. The work we're waiting on them for could be done by the committers instead if we want to shorten the whole process a bit. I don't think that's really what you want though. I think the pressure has to be applied all through the process. People who haven't provided a review by now are certainly overdue for a polite poke, Thanksgiving or no Thanksgiving. If the first review doesn't happen until the third week of the CommitFest, how is the patch going to get a fair shake by the end of the fourth one? I mean, if that happens to a small number of patches, OK, but when it's 20% of what's in the CommitFest, it seems like it's bound to lead to a huge crunch at the end. In any case, unlike last CommitFest, where the problem was a lack of adequate committer activity, it's pretty clear that the the problem this CommitFest has been a combination of slow reviews and slow updates by patch authors. I've been keeping a loose eye on the patch queue and my impression is that there has rarely been more than 1 patch other than HS + SR in the Ready for Committer state, and many times zero. That's means the pipeline is stalling, and that's bad. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tue, 1 Dec 2009 09:59:17 +0100, Dave Page dp...@pgadmin.org wrote: I do see the argument that RESET ALL should revert user changes to application_name though, but I maintain they should reset to the value set at connection time, not to null. As has been pointed out already, other values set at connection time cannot be reset, so allowing that for application name does seem like a POLA violation. I'd like to support this Argument. As I understand this patch from http://archives.postgresql.org/pgsql-hackers/2009-10/msg00711.php it is intended to support some kind of feature like the SQL Server ...;Application Name=MyApp;... connection string value, making the name of the user level (or whatever) application name available at the Database/SQL level. I don't know about pgpool but as far as I know, some client side connection pooling implementations use one pool per connection string/url (.Net Data Providers, JDBC). They would probably want set the application_name in the startup message and will expect it to fall back to this value when calling RESET ALL (or what ever you like to be the command to go back to the values that were requested on connection startup) on recycling a connection from the pool. Any other solution would greatly complicate recycling of connections for per connection string pooling szenarios. Regards, Brar -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] [PATCH] Windows x64
Hello.
The following patches support Windows x64.
1) use intptr_t for Datum and pointer macros. (to support Windows LLP64)
almost the same as that post before.
http://archives.postgresql.org/pgsql-hackers/2009-06/threads.php#01364
2) use appropriate macro and datatypes for Windows API.
enables more than 32bits shared memory.
3) Build scripts for MSVC, this came from
http://archives.postgresql.org/pgsql-hackers/2008-07/msg00440.php
add new parameters to config.pl.
You need define platform to x64 for 64bit programs.
-
Windows x64 binary that applied patch and build with MSVS2005 can pass
all regression tests (vcregress.bat).
I was checked where the string converted with %ld is used.
An especially fatal part is not found excluding one of plperl.
But there is still a possibility that elog messages output a incorrect value.
(I thought it is not fatal, ignored these for the present.)
(eg) src/backend/port/win32_shmem.c, line 167
'size' is 'size_t' = 64bit value.
| ereport(FATAL,
| (errmsg(could not create shared memory segment: %lu, GetLastError()),
| errdetail(Failed system call was CreateFileMapping(size=%lu, name=%s).,
| (unsigned long) size, szShareMem)));
The code that becomes a problem of plperl is the following.
The address is converted into the string, and it is used as hash key.
However, there is really little possibility that two address values
become the same low word, and the problem will not occur.
(Of course, it is necessary to fix though the problem doesn't occur.)
--- src/pl/plperl/plperl.c 2009-11-30 18:56:30.0 +0900
+++ /tmp/plperl.c 2009-12-01 18:46:43.0 +0900
@@ -95,7 +95,7 @@
**/
typedef struct plperl_query_desc
{
- charqname[sizeof(long) * 2 + 1];
+ charqname[sizeof(void *) * 2 + 1];
void *plan;
int nargs;
Oid*argtypes;
@@ -2343,7 +2343,8 @@
/
qdesc = (plperl_query_desc *) malloc(sizeof(plperl_query_desc));
MemSet(qdesc, 0, sizeof(plperl_query_desc));
- snprintf(qdesc-qname, sizeof(qdesc-qname), %lx, (long) qdesc);
+ /* XXX: for LLP64, use %p or %ll */
+ snprintf(qdesc-qname, sizeof(qdesc-qname), %p, qdesc);
qdesc-nargs = argc;
qdesc-argtypes = (Oid *) malloc(argc * sizeof(Oid));
qdesc-arginfuncs = (FmgrInfo *) malloc(argc * sizeof(FmgrInfo));
--
Tsutomu Yamada
SRA OSS, Inc. Japan
From b6e51007eb3bfb9dd4333dfa1bfd479a049d70f0 Mon Sep 17 00:00:00 2001
From: Tsutomu Yamada tsut...@sraoss.co.jp
Date: Wed, 26 Aug 2009 21:39:32 +0900
Subject: [PATCH 1/3] use uintptr_t for Datum
fix typedef, and pointer arithmetics.
for LLP64 (Windows x64)
---
src/backend/access/common/heaptuple.c |6 +++---
src/backend/access/hash/hashfunc.c|2 +-
src/backend/storage/lmgr/lwlock.c |2 +-
src/include/access/tupmacs.h |2 +-
src/include/c.h | 11 +++
src/include/pg_config.h.win32 |7 +++
src/include/postgres.h|4 ++--
src/interfaces/ecpg/ecpglib/data.c|8
8 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index a86716e..f429f68 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -192,7 +192,7 @@ heap_fill_tuple(TupleDesc tupleDesc,
if (att[i]-attbyval)
{
/* pass-by-value */
- data = (char *) att_align_nominal((long) data, att[i]-attalign);
+ data = (char *) att_align_nominal(data, att[i]-attalign);
store_att_byval(data, values[i], att[i]-attlen);
data_length = att[i]-attlen;
}
@@ -226,7 +226,7 @@ heap_fill_tuple(TupleDesc tupleDesc,
else
{
/* full 4-byte header varlena */
-data = (char *) att_align_nominal((long) data,
+data = (char *) att_align_nominal(data,
att[i]-attalign);
data_length = VARSIZE(val);
memcpy(data, val, data_length);
@@ -243,7 +243,7 @@ heap_fill_tuple(TupleDesc tupleDesc,
else
{
/* fixed-length pass-by-reference */
- data = (char *) att_align_nominal((long) data, att[i]-attalign);
+ data = (char *) att_align_nominal(data, att[i]-attalign);
Assert(att[i]-attlen 0);
data_length = att[i]-attlen;
memcpy(data, DatumGetPointer(values[i]), data_length);
diff --git a/src/backend/access/hash/hashfunc.c b/src/backend/access/hash/hashfunc.c
index a38103e..d3efb29 100644
--- a/src/backend/access/hash/hashfunc.c
+++ b/src/backend/access/hash/hashfunc.c
@@ -319,7 +319,7 @@ hash_any(register const unsigned char *k, register int keylen)
a = b = c = 0x9e3779b9 + len + 3923095;
/* If the source pointer is word-aligned, we use word-wide fetches */
- if (((long) k
Re: [HACKERS] ProcessUtility_hook
Tom Lane escreveu: Bruce Momjian br...@momjian.us writes: So, if someone writes a patch, and it is reviewed, and the patch author updates the patch and replies, it still should be reviewed again before being committed? Well, that's for the reviewer to say --- if the update satisfies his concerns, he should sign off on it, if not not. I've tried to avoid pre-empting that process. That's correct. I didn't have time to review the new patch yet. :( I'll do it later today. IIRC Tom had some objections (during the last CF) the way the patch was proposed and suggested changes. Let's see if the Takahiro-san did everything that was suggested. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 06:35 -0500, Bruce Momjian wrote: Simon Riggs wrote: The way we handle torn page corruptions *hides* actual corruptions from us. The frequency of true positives and false positives is important here. If the false positive ratio is very small, then reporting them is not a problem because of the benefit we get from having spotted the true positives. Some convicted murderers didn't do it, but that is not an argument for letting them all go free (without knowing the details). So we need to know what the false positive ratio is before we evaluate the benefit of either reporting or non-reporting possible corruption events. When do you think torn pages happen? Only at crash, or other times also? Do they always happen at crash? Are there ways to re-check a block that has suffered a hint-related torn page issue? Are there ways to isolate and minimise the reporting of false positives? Those are important questions and this is not black and white. If the *only* answer really is we-must-WAL-log everything, then that is the answer, as an option. I suspect that there is a less strict possibility, if we question our assumptions and look at the frequencies. We know that I have no time to work on this; I am just trying to hold open the door to a few possibilities that we have not fully considered in a balanced way. And I myself am guilty of having slammed the door previously. I encourage development of a way forward based upon a balance of utility. I think the problem boils down to what the user response should be to a corruption report. If it is a torn page, it would be corrected and the user doesn't have to do anything. If it is something that is not correctable, then the user has corruption and/or bad hardware. I think the problem is that the existing proposal can't distinguish between these two cases so the user has no idea how to respond to the report. If 99.5% of cases are real corruption then there is little need to distinguish between the cases, nor much value in doing so. The prevalence of the different error types is critical to understanding how to respond. If a man pulls a gun on you, your first thought isn't some people remove guns from their jacket to polish them, so perhaps he intends to polish it now because the prevalence of shootings is high, when faced by people with guns, and the risk of dying is also high. You make a judgement based upon the prevalence and the risk. That is all I am asking for us to do here, make a balanced call. These recent comments are a change in my own position, based upon evaluating the prevalence and the risk. I ask others to consider the same line of thought rather than a black/white assessment. All useful detection mechanisms have non-zero false positives because we would rather sometimes ring the bell for no reason than to let bad things through silently, as we do now. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
On Mon, Nov 30, 2009 at 11:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: As I have observed before, I think we need some infrastructure to help committers claim items, so we don't duplicate work. Robert acknowledged the need for a claimed by committer field in the fest application, but he hasn't got round to it yet. In the meantime I've been adding a Taking this one... type of comment to an entry I want to claim. Sorry I haven't gotten around to this. Beyond being a little burned out a little bit, I have been a little bit under the weather and a little occupied with life apart from PostgreSQL, as if there were such a thing. Anyway, one of the concerns I have about this is that adding another field to the commitfest_view page seems as though it will create some layout issues - the leftmost column will get squished. I could (a) go ahead and do it anyway or (b) do it, but modify the layout in some unspecified way so that it doesn't impact the format as much or of course (c) not do it. Any thoughts? It would also like to clarify the use case for this a little bit more. Is this just to track patches which committers are in the process of committing (or have committed)? Or would a committer potentially set this on some patch that was still being reviewed, and if so would that mean don't review this any more because I'm taking over or I'm planning to pick this up when the review process completes or something else? Thanks, ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Simon Riggs wrote: I think the problem is that the existing proposal can't distinguish between these two cases so the user has no idea how to respond to the report. If 99.5% of cases are real corruption then there is little need to distinguish between the cases, nor much value in doing so. The prevalence of the different error types is critical to understanding how to respond. If a man pulls a gun on you, your first thought isn't some people remove guns from their jacket to polish them, so perhaps he intends to polish it now because the prevalence of shootings is high, when faced by people with guns, and the risk of dying is also high. You make a judgement based upon the prevalence and the risk. That is all I am asking for us to do here, make a balanced call. These recent comments are a change in my own position, based upon evaluating the prevalence and the risk. I ask others to consider the same line of thought rather than a black/white assessment. All useful detection mechanisms have non-zero false positives because we would rather sometimes ring the bell for no reason than to let bad things through silently, as we do now. OK, but what happens if someone gets the failure report, assumes their hardware is faulty and replaces it, and then gets a failure report again? I assume torn pages are 99% of the reported problem, which are expected and are fixed, and bad hardware 1%, quite the opposite of your numbers above. What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Mon, Nov 30, 2009 at 3:27 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: Proposal * We reserve enough space on a disk block for a CRC check. When a dirty block is written to disk we calculate and annotate the CRC value, though this is *not* WAL logged. Imagine this: 1. A hint bit is set. It is not WAL-logged, but the page is dirtied. 2. The buffer is flushed out of the buffer cache to the OS. A new CRC is calculated and stored on the page. 3. Half of the page is flushed to disk (aka torn page problem). The CRC made it to disk but the flipped hint bit didn't. You now have a page with incorrect CRC on disk. This is probably a stupid question, but why doesn't the other half of the page make it to disk? Somebody pulls the plug first? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
Robert Haas wrote: On Mon, Nov 30, 2009 at 11:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: As I have observed before, I think we need some infrastructure to help committers claim items, so we don't duplicate work. Robert acknowledged the need for a claimed by committer field in the fest application, but he hasn't got round to it yet. ?In the meantime I've been adding a Taking this one... type of comment to an entry I want to claim. Sorry I haven't gotten around to this. Beyond being a little burned out a little bit, I have been a little bit under the weather and a little occupied with life apart from PostgreSQL, as if there were such a thing. Anyway, one of the concerns I have about this is that adding another field to the commitfest_view page seems as though it will create some layout issues - the leftmost column will get squished. I could (a) go ahead and do it anyway or (b) do it, but modify the layout in some unspecified way so that it doesn't impact the format as much or of course (c) not do it. Any thoughts? It would also like to clarify the use case for this a little bit more. Is this just to track patches which committers are in the process of committing (or have committed)? Or would a committer potentially set this on some patch that was still being reviewed, and if so would that mean don't review this any more because I'm taking over or I'm planning to pick this up when the review process completes or something else? I am thinking we can just add a new status, claimed by committer and not bother about adding a new column with the committer name. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Robert Haas wrote: On Mon, Nov 30, 2009 at 3:27 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: Proposal * We reserve enough space on a disk block for a CRC check. When a dirty block is written to disk we calculate and annotate the CRC value, though this is *not* WAL logged. Imagine this: 1. A hint bit is set. It is not WAL-logged, but the page is dirtied. 2. The buffer is flushed out of the buffer cache to the OS. A new CRC is calculated and stored on the page. 3. Half of the page is flushed to disk (aka torn page problem). The CRC made it to disk but the flipped hint bit didn't. You now have a page with incorrect CRC on disk. This is probably a stupid question, but why doesn't the other half of the page make it to disk? Somebody pulls the plug first? Yep, the pages are 512 bytes on disk, so you might get only some of the 16 512-byte blocks to disk, or the 512-byte block might be partially written. Full page writes fix these on recovery. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 07:05 -0500, Bruce Momjian wrote: I assume torn pages are 99% of the reported problem, which are expected and are fixed, and bad hardware 1%, quite the opposite of your numbers above. On what basis do you make that assumption? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Simon Riggs wrote: On Tue, 2009-12-01 at 07:05 -0500, Bruce Momjian wrote: I assume torn pages are 99% of the reported problem, which are expected and are fixed, and bad hardware 1%, quite the opposite of your numbers above. On what basis do you make that assumption? Because we added full page write protection to fix the reported problem of torn pages, which we had on occasion; now we don't. Bad hardware reports are less frequent. And we know we can reproduce torn pages by shutting of power to a server without battery-backed cache. We don't know how to produce I/O failures on demand. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
bruce wrote: What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. Sorry, stupid idea on my part. We don't WAL log hit bit changes so there is no guarantee the page is in WAL on recovery. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 07:58 -0500, Bruce Momjian wrote: bruce wrote: What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. Sorry, stupid idea on my part. We don't WAL log hit bit changes so there is no guarantee the page is in WAL on recovery. I thought it was a reasonable idea. We would need to re-check CRCs after a crash and zero any that mismatched. Then we can start checking them again as we run. In any case, it seems strange to do nothing to protect the database in normal running just because there is one type of problem that occurs when we crash. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 07:42 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Tue, 2009-12-01 at 07:05 -0500, Bruce Momjian wrote: I assume torn pages are 99% of the reported problem, which are expected and are fixed, and bad hardware 1%, quite the opposite of your numbers above. On what basis do you make that assumption? Because we added full page write protection to fix the reported problem of torn pages, which we had on occasion; now we don't. Bad hardware reports are less frequent. Bad hardware reports are infrequent because we lack a detection system for them, which is the topic of this thread. It would be circular to argue that as a case against. It's also an argument that only effects crashes. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
Bruce Momjian wrote: It would also like to clarify the use case for this a little bit more. Is this just to track patches which committers are in the process of committing (or have committed)? Or would a committer potentially set this on some patch that was still being reviewed, and if so would that mean don't review this any more because I'm taking over or I'm planning to pick this up when the review process completes or something else? I am thinking we can just add a new status, claimed by committer and not bother about adding a new column with the committer name. I think it would be more flexible and useful to be able to claim it before the review is finished. It would mean in effect I am following this and intend to commit it when it's ready. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Bruce Momjian wrote: What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. Unfortunately that's not true. You can crash, leading to a torn page, and then start up the database and shut it down cleanly. The torn page is still there, even though the last shutdown was a clean one. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Mon, Nov 30, 2009 at 9:27 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: Proposal * We reserve enough space on a disk block for a CRC check. When a dirty block is written to disk we calculate and annotate the CRC value, though this is *not* WAL logged. Imagine this: 1. A hint bit is set. It is not WAL-logged, but the page is dirtied. 2. The buffer is flushed out of the buffer cache to the OS. A new CRC is calculated and stored on the page. 3. Half of the page is flushed to disk (aka torn page problem). The CRC made it to disk but the flipped hint bit didn't. You now have a page with incorrect CRC on disk. What if we treated the hint bits as all-zeros for the purpose of CRC calculation? This would exclude them from the checksum. Greetings Marcin Mańk -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tuesday 01 December 2009 14:38:26 marcin mank wrote: On Mon, Nov 30, 2009 at 9:27 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: Proposal * We reserve enough space on a disk block for a CRC check. When a dirty block is written to disk we calculate and annotate the CRC value, though this is *not* WAL logged. Imagine this: 1. A hint bit is set. It is not WAL-logged, but the page is dirtied. 2. The buffer is flushed out of the buffer cache to the OS. A new CRC is calculated and stored on the page. 3. Half of the page is flushed to disk (aka torn page problem). The CRC made it to disk but the flipped hint bit didn't. You now have a page with incorrect CRC on disk. What if we treated the hint bits as all-zeros for the purpose of CRC calculation? This would exclude them from the checksum. That sounds like doing a complete copy of the wal page zeroing specific fields and then doing wal - rather expensive I would say. Both, during computing the checksum and checking it... Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature request: permissions change history for auditing
Thom Brown escreveu: As far as I am aware, there is no way to tell when a user/role was granted permissions or had permissions revoked, or who made these changes. I'm wondering if it would be useful for security auditing to maintain a history of permissions changes only accessible to superusers? If the utility command hook patch is approved, it will be possible to track commands rather than DML ones. In that case, it would be trivial to do some extension that covers your audit concerns. [1] https://commitfest.postgresql.org/action/patch_view?id=196 -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
* Andres Freund and...@anarazel.de [091201 08:42]: On Tuesday 01 December 2009 14:38:26 marcin mank wrote: On Mon, Nov 30, 2009 at 9:27 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: Proposal * We reserve enough space on a disk block for a CRC check. When a dirty block is written to disk we calculate and annotate the CRC value, though this is *not* WAL logged. Imagine this: 1. A hint bit is set. It is not WAL-logged, but the page is dirtied. 2. The buffer is flushed out of the buffer cache to the OS. A new CRC is calculated and stored on the page. 3. Half of the page is flushed to disk (aka torn page problem). The CRC made it to disk but the flipped hint bit didn't. You now have a page with incorrect CRC on disk. What if we treated the hint bits as all-zeros for the purpose of CRC calculation? This would exclude them from the checksum. That sounds like doing a complete copy of the wal page zeroing specific fields and then doing wal - rather expensive I would say. Both, during computing the checksum and checking it... No, it has nothing to do with WAL, it has to do with when writing pages out... You already double-buffer them (to avoid the page changing while you checksum it) before calling write, but the code writing (and then reading) pages doesn't currently have to know all the internal stuff needed decide what's a hint bit and what's not... And adding that information into the buffer in/out would be a huge wart on the modularity of the PG code... a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] Block-level CRC checks
On Tue, Dec 1, 2009 at 8:30 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Bruce Momjian wrote: What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. Unfortunately that's not true. You can crash, leading to a torn page, and then start up the database and shut it down cleanly. The torn page is still there, even though the last shutdown was a clean one. Thinking through this, as I understand it, in order to prevent this problem, you'd need to be able to predict at recovery time which pages might have been torn by the unclean shutdown. In order to do that, you'd need to know which pages were waiting to be written to disk at the time of the shutdown. For ordinary page modifications, that's not a problem, because there will be WAL records for those pages that need to be replayed, and we could recompute the CRC at the same time. But for hint bit changes, there's no persistent state that would tell us which hint bits were in the midst of being flipped when the system went down, so the only way to make sure all the CRCs are correct would be to rescan every page in the entire cluster and recompute every CRC. Is that right? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
On Tue, Dec 1, 2009 at 8:27 AM, Andrew Dunstan and...@dunslane.net wrote: Bruce Momjian wrote: It would also like to clarify the use case for this a little bit more. Is this just to track patches which committers are in the process of committing (or have committed)? Or would a committer potentially set this on some patch that was still being reviewed, and if so would that mean don't review this any more because I'm taking over or I'm planning to pick this up when the review process completes or something else? I am thinking we can just add a new status, claimed by committer and not bother about adding a new column with the committer name. I think it would be more flexible and useful to be able to claim it before the review is finished. It would mean in effect I am following this and intend to commit it when it's ready. If we went with Bruce's interpretation, we could have a committer field that only appears when the status is Claimed by Committer or Committed and the contents of that field could be displayed in parentheses in the status column, like this: Claimed by Committer (Tom Lane). If we went with Andrew's interpretation, we would need a completely separate column, because there wouldn't be any logical relationship between the status field and the committer field. Any other votes? Tom? On a possibly related note, I am not totally sure that we want to enshrine the principle that committers categorically won't touch patches that are not yet marked Ready for Committer. For major patches like SE-PostgreSQL or the partitioning stuff, early committer involvement is an extremely important ingredient for success. And, I have an uncomfortable feeling about having Tom, Bruce, and Andrew all intentionally sitting on the bench waiting for reviews to complete while the days tick away. On the other hand, I also agree with Tom's point that, if completing reviews doesn't affect whether the patch gets in, there's less incentive for people to review. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
Robert Haas robertmh...@gmail.com writes: On Mon, Nov 30, 2009 at 11:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert acknowledged the need for a claimed by committer field in the fest application, but he hasn't got round to it yet. Sorry I haven't gotten around to this. Beyond being a little burned out a little bit, I have been a little bit under the weather and a little occupied with life apart from PostgreSQL, as if there were such a thing. Anyway, one of the concerns I have about this is that adding another field to the commitfest_view page seems as though it will create some layout issues - the leftmost column will get squished. I could (a) go ahead and do it anyway or (b) do it, but modify the layout in some unspecified way so that it doesn't impact the format as much or of course (c) not do it. Any thoughts? I would be satisfied if there were a claimed by field in the per-patch detail page, which is where you'd have to go to set it anyway. If you want you could add an additional status value claimed by committer so it'd be visible in the main page. It would also like to clarify the use case for this a little bit more. It's to keep committers from treading on each others' toes. Right now, if say Andrew is working over a patch with intent to commit, there's no visibility of that fact in the fest status. I would imagine that a patch should not normally get into this state until it's been marked ready for committer by the reviewer. Except perhaps in cases where the reviewer and committer are the same person. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tuesday 01 December 2009 15:26:21 Aidan Van Dyk wrote: * Andres Freund and...@anarazel.de [091201 08:42]: On Tuesday 01 December 2009 14:38:26 marcin mank wrote: On Mon, Nov 30, 2009 at 9:27 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: Proposal * We reserve enough space on a disk block for a CRC check. When a dirty block is written to disk we calculate and annotate the CRC value, though this is *not* WAL logged. Imagine this: 1. A hint bit is set. It is not WAL-logged, but the page is dirtied. 2. The buffer is flushed out of the buffer cache to the OS. A new CRC is calculated and stored on the page. 3. Half of the page is flushed to disk (aka torn page problem). The CRC made it to disk but the flipped hint bit didn't. You now have a page with incorrect CRC on disk. What if we treated the hint bits as all-zeros for the purpose of CRC calculation? This would exclude them from the checksum. That sounds like doing a complete copy of the wal page zeroing specific fields and then doing wal - rather expensive I would say. Both, during computing the checksum and checking it... No, it has nothing to do with WAL, it has to do with when writing pages out... You already double-buffer them (to avoid the page changing while you checksum it) before calling write, but the code writing (and then reading) pages doesn't currently have to know all the internal stuff needed decide what's a hint bit and what's not... err, yes. That WAL slipped in, sorry. But it would still either mean a third copy of the page or a rather complex jumping around on the page... Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Robert Haas wrote: On Tue, Dec 1, 2009 at 8:30 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Bruce Momjian wrote: What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. Unfortunately that's not true. You can crash, leading to a torn page, and then start up the database and shut it down cleanly. The torn page is still there, even though the last shutdown was a clean one. Thinking through this, as I understand it, in order to prevent this problem, you'd need to be able to predict at recovery time which pages might have been torn by the unclean shutdown. In order to do that, you'd need to know which pages were waiting to be written to disk at the time of the shutdown. For ordinary page modifications, that's not a problem, because there will be WAL records for those pages that need to be replayed, and we could recompute the CRC at the same time. But for hint bit changes, there's no persistent state that would tell us which hint bits were in the midst of being flipped when the system went down, so the only way to make sure all the CRCs are correct would be to rescan every page in the entire cluster and recompute every CRC. Is that right? Yep. Even if rescanning every page in the cluster was feasible from a performance point-of-view, it would make the CRC checking a lot less useful. It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
Robert Haas robertmh...@gmail.com writes: If we went with Bruce's interpretation, we could have a committer field that only appears when the status is Claimed by Committer or Committed and the contents of that field could be displayed in parentheses in the status column, like this: Claimed by Committer (Tom Lane). If we went with Andrew's interpretation, we would need a completely separate column, because there wouldn't be any logical relationship between the status field and the committer field. Any other votes? Tom? I'm happy with Andrew's interpretation --- I just want a separate text field for inserting a committer's name. I don't want any magic behavior of that field. On a possibly related note, I am not totally sure that we want to enshrine the principle that committers categorically won't touch patches that are not yet marked Ready for Committer. No, but I think that should be the default assumption once a reviewer has been assigned. If the reviewer doesn't totally fall down on the job, he/she should be allowed to finish reviewing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
On Tue, Dec 1, 2009 at 9:42 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: If we went with Bruce's interpretation, we could have a committer field that only appears when the status is Claimed by Committer or Committed and the contents of that field could be displayed in parentheses in the status column, like this: Claimed by Committer (Tom Lane). If we went with Andrew's interpretation, we would need a completely separate column, because there wouldn't be any logical relationship between the status field and the committer field. Any other votes? Tom? I'm happy with Andrew's interpretation --- I just want a separate text field for inserting a committer's name. I don't want any magic behavior of that field. OK, I'll add a separate text field for the committer's name, but for now it won't display on the summary page, just the detail page. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 15:30 +0200, Heikki Linnakangas wrote: Bruce Momjian wrote: What might be interesting is to report CRC mismatches if the database was shut down cleanly previously; I think in those cases we shouldn't have torn pages. Unfortunately that's not true. You can crash, leading to a torn page, and then start up the database and shut it down cleanly. The torn page is still there, even though the last shutdown was a clean one. There seems to be two ways forwards: journalling or fsck. We can either * WAL-log all changes to a page (journalling) (8-byte overhead) * After a crash disable CRC checks until a full database scan has either re-checked CRC or found CRC mismatch, report it in the LOG and then reset the CRC. (fsck) (8-byte overhead) Both of which can be optimised in various ways. Also, we might * Put all hint bits in the block header to allow them to be excluded more easily from CRC checking. If we used 3 more bits from ItemIdData.lp_len (limiting tuple length to 4096) then we could store some hints in the item pointer. HEAP_XMIN_INVALID can be stored as LP_DEAD, since that will happen very quickly anyway. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, Dec 1, 2009 at 9:40 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Even if rescanning every page in the cluster was feasible from a performance point-of-view, it would make the CRC checking a lot less useful. It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. Yeah. Thanks for the explanation - I think I understand the problem now. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
BTW, if you have time for any purely cosmetic details ... the way the CommitFest field on a patch detail page works has always struck me as weird. It's a data field, and so if it has any behavior at all that behavior ought to involve modifying its value. But what it actually is is a navigation link. I think you ought to drop it down to a plain text field and add a Back to CommitFest link to the page header, similar to the way navigation works on other dependent pages. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
* Simon Riggs: * Put all hint bits in the block header to allow them to be excluded more easily from CRC checking. If we used 3 more bits from ItemIdData.lp_len (limiting tuple length to 4096) then we could store some hints in the item pointer. HEAP_XMIN_INVALID can be stored as LP_DEAD, since that will happen very quickly anyway. What about putting the whole visibility information out-of-line, into its own B-tree, indexed by page number? -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deleted WAL files held open by backends in Linux
Tom Lane t...@sss.pgh.pa.us wrote: How old were the sessions you were looking at? Days to months old. If we think this is worth doing something about (I'm not convinced yet) Once one knows about the issue, it's only a minor annoyance, and that infrequently, so it's not worth anything that would take significant effort or would cause any measurable performance hit. then the answer would be to forcibly close a backend's open WAL file in some reasonably seldom-used code path. One possibility that comes to mind is to do it in ProcessCatchupEvent(), which will be invoked in approximately the right circumstances: a backend that is sitting idle for a long time within an otherwise busy system. That wouldn't be a 100% solution, because if the backend is handling a steady stream of queries it will likely never run ProcessCatchupEvent(). But the places that would be 100% (like transaction commit) would probably entail too much of a performance hit from repeatedly opening and closing WAL files. In our case there would tend to be bursts of activity on some of these connections, and they would usually have a dribble of activity from the monitoring system. This dribble would consist of a couple sub-millisecond SELECT statements once every five or ten seconds. (I have no clue whether that's frequent enough to suppress the ProcessCatchupEvent() call.) Is there a reasonably cheap way to check whether the backend has a WAL file open and whether that one is the current append target? As you point out, the need to actually close such a file would be infrequent, and it seems as though when the need exists, it wouldn't be a matter of *whether* it would need to be closed, just *when*. Currently it sounds like we would often wind up doing it the next time we try to run a query which has to flush dirty buffers, or when the connection is being closed. It seems like those might not be the ideal time to add the overhead of the close, so we would just need to find someplace which isn't worse. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest status/management
Robert Haas wrote: OK, I'll add a separate text field for the committer's name, but for now it won't display on the summary page, just the detail page. Perhaps it could go underneath the reviewer names, maybe in a different color. (And yes, like many of us I suck at GUI design, so feel free to discount any suggestions I make in that area). cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Windows x64
On Tue, Dec 1, 2009 at 6:25 AM, Tsutomu Yamada tsut...@sraoss.co.jp wrote: Hello. The following patches support Windows x64. 1) use intptr_t for Datum and pointer macros. (to support Windows LLP64) almost the same as that post before. http://archives.postgresql.org/pgsql-hackers/2009-06/threads.php#01364 2) use appropriate macro and datatypes for Windows API. enables more than 32bits shared memory. 3) Build scripts for MSVC, this came from http://archives.postgresql.org/pgsql-hackers/2008-07/msg00440.php add new parameters to config.pl. You need define platform to x64 for 64bit programs. You should add your patch to the currently open commitfest here: https://commitfest.postgresql.org/action/commitfest_view/open And perhaps also review the patch submission guidelines here: http://wiki.postgresql.org/wiki/Submitting_a_Patch Thanks, ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deleted WAL files held open by backends in Linux
Kevin Grittner kevin.gritt...@wicourts.gov writes: Is there a reasonably cheap way to check whether the backend has a WAL file open and whether that one is the current append target? Detecting whether we have a WAL file open is trivial (just look at the static variable holding the file descriptor). Determining whether it's still the current append target is not so cheap though; it would require examining shared-memory status which means taking a lock on that status (and it's a high-traffic lock already). We could have the open WAL file dropped if stale as a side-effect anytime we have occasion to examine that shared state anyway. But in a nearly-read-only session such as your example I'm not sure that would happen often enough to fix the problem. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Florian Weimer fwei...@bfk.de writes: What about putting the whole visibility information out-of-line, into its own B-tree, indexed by page number? Hint bits need to be *cheap* to examine. Otherwise there's little point in having them at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Fwd: psql+krb5
2009/11/30 rahimeh khodadadi rahimeh.khodad...@gmail.com: -- Forwarded message -- From: rahimeh khodadadi rahimeh.khodad...@gmail.com Date: 2009/11/29 Subject: Re: psql+krb5 To: Denis Feklushkin denis.feklush...@gmail.com Please review the guidelines for reporting a problem, which you can find here: http://wiki.postgresql.org/wiki/Guide_to_reporting_problems It seems to me that you've done the exact opposite of nearly everything suggested there, starting with cross-posting your email to four mailing lists at least three of which are irrelevant to the problem that you're attempting to solve. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 16:40 +0200, Heikki Linnakangas wrote: It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. They are already masked from us, so continuing to mask those errors would not put us in a worse position. If we are saying that 99% of page corruptions are caused at crash time because of torn pages on hint bits, then only WAL logging can help us find the 1%. I'm not convinced that is an accurate or safe assumption and I'd at least like to see LOG entries showing what happened. ISTM we could go for two levels of protection. CRC checks and scanner for Level 1 protection, then full WAL logging for Level 2 protection. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2009-12-01 at 16:40 +0200, Heikki Linnakangas wrote: It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. They are already masked from us, so continuing to mask those errors would not put us in a worse position. No, it would just destroy a large part of the argument for why this is worth doing. We detect disk errors ... except for ones that happen during a database crash. Say what? The fundamental problem with this is the same as it's been all along: the tradeoff between implementation work expended, performance overhead added, and net number of real problems detected (with a suitably large demerit for actually *introducing* problems) just doesn't look attractive. You can make various compromises that improve one or two of these factors at the cost of making the others worse, but at the end of the day I've still not seen a combination that seems worth doing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, Dec 1, 2009 at 10:35 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2009-12-01 at 16:40 +0200, Heikki Linnakangas wrote: It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. They are already masked from us, so continuing to mask those errors would not put us in a worse position. If we are saying that 99% of page corruptions are caused at crash time because of torn pages on hint bits, then only WAL logging can help us find the 1%. I'm not convinced that is an accurate or safe assumption and I'd at least like to see LOG entries showing what happened. It may or may not be true that most page corruptions happen at crash time, but it's certainly false that they are caused at crash time *because of torn pages on hint bits*. If only part of a block is written to disk and the unwritten parts contain hint-bit changes - that's not corruption. That's design behavior. Any CRC system needs to avoid complaining about errors when that happens because otherwise people will think that their database is corrupted and their hardware is faulty when in reality it is not. If we could find a way to put the hint bits in the same 512-byte block as the CRC, that might do it, but I'm not sure whether that is possible. Ignoring CRC errors after a crash until we've re-CRC'd the entire database will certainly eliminate the bogus error reports, but it seems likely to mask a large percentage of legitimate errors. For example, suppose that I write 1MB of data out to disk and then don't access it for a year. During that time the data is corrupted. Then the system crashes. Upon recovery, since there's no way of knowing whether hint bits on those pages were being updated at the time of the crash, so the system re-CRC's the corrupted data and declares it known good. Six months later, I try to access the data and find out that it's bad. Sucks to be me. Now consider the following alternative scenario: I write the block to disk. Five minutes later, without an intervening crash, I read it back in and it's bad. Yeah, the system detects it. Which is more likely? I'm not an expert on disk failure modes, but my intuition is that the first one will happen often enough to make us look silly. Is it 10%? 20%? 50%? I don't know. But ISTM that a CRC system that has no ability to determine whether a system is still ok post-crash is not a compelling proposition, even though it might still be able to detect some problems. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
Dave Page dp...@pgadmin.org writes: On Tue, Dec 1, 2009 at 9:16 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: If an application can do SET DEFAULT, how does the connection pooler *really* reset the value back to what it was? There has to be some level of trust here :-). As the alternative would involve bumping the fe-be protocol version, it seems like a reasonable approach to me. I don't think that we need to bump the protocol version. The real alternative here would be that libpq sends a startup packet that includes application_name, and if it gets an error back from that, it starts over without the app name. The main disadvantage would be that you'd get a double connection attempt == more overhead anytime you use an 8.5+ libpq to connect to 8.4- server. People never complained that hard about the similar double connection attempt when 7.4+ libpq connected to 7.3- servers, so maybe we should just go that way. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EOL for 7.4?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I wrote: No, it should be longer. June is practically around the corner as far as business planning is concerned. Make it a year. Since it's mid November, why not just say 2011? This thread never got resolved. I think we can all agree that EOL for 7.4 is a when, not an if? Can we get -core to take a stance here and pick a date? I like the clean smooth lines of January 2011, and thus saying that 2010 is the last year in which we'll backpatch things to the 7.4 branch. But I'll stick to whatever core thinks is best. Just let the advocacy team know so we can start work on it. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation PGP Key: 0x14964AC8 200912011122 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAksVQtgACgkQvJuQZxSWSsj26ACgr/QRKytEc9dWYar0gY6HJZ0C YYsAni96hrCF0AmBIjY/Fg5vHS+LauKT =ELLh -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tue, Dec 1, 2009 at 4:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't think that we need to bump the protocol version. The real alternative here would be that libpq sends a startup packet that includes application_name, and if it gets an error back from that, it starts over without the app name. The main disadvantage would be that you'd get a double connection attempt == more overhead anytime you use an 8.5+ libpq to connect to 8.4- server. People never complained that hard about the similar double connection attempt when 7.4+ libpq connected to 7.3- servers, so maybe we should just go that way. I looked (briefly) at doing that when we first ran into this suggestion. As you pointed out at the time, it seemed like that would require some fairly ugly hackery in fe-connect.c -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [GENERAL] [HACKERS] Fwd: psql+krb5
Except that he posted a month ago and got no answers... On Tue, Dec 1, 2009 at 8:22 AM, Robert Haas robertmh...@gmail.com wrote: 2009/11/30 rahimeh khodadadi rahimeh.khodad...@gmail.com: -- Forwarded message -- From: rahimeh khodadadi rahimeh.khodad...@gmail.com Date: 2009/11/29 Subject: Re: psql+krb5 To: Denis Feklushkin denis.feklush...@gmail.com Please review the guidelines for reporting a problem, which you can find here: http://wiki.postgresql.org/wiki/Guide_to_reporting_problems It seems to me that you've done the exact opposite of nearly everything suggested there, starting with cross-posting your email to four mailing lists at least three of which are irrelevant to the problem that you're attempting to solve. ...Robert -- Sent via pgsql-general mailing list (pgsql-gene...@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general -- When fascism comes to America, it will be intolerance sold as diversity. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
Dave Page dp...@pgadmin.org writes: On Tue, Dec 1, 2009 at 4:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't think that we need to bump the protocol version. The real alternative here would be that libpq sends a startup packet that includes application_name, and if it gets an error back from that, it starts over without the app name. I looked (briefly) at doing that when we first ran into this suggestion. As you pointed out at the time, it seemed like that would require some fairly ugly hackery in fe-connect.c Perhaps, but at the time it wasn't apparent that issuing a separate SET would create user-visible behavioral inconsistencies. Now that we've realized that, I think we should reconsider. If people are agreed that double connect is a better alternative I'm willing to go look at how to make it happen. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On Tue, Dec 1, 2009 at 4:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: If people are agreed that double connect is a better alternative I still kinda like 'SET DEFAULT', but I'm far from wed to it. A double connect certainly seems like it would be better than the inconsistency. I'm willing to go look at how to make it happen. That's good, 'cos I'm sure it'll end up being a whole lot less ugly than if I did it :-) -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Rewrite GEQO`s gimme_tree function so that it always finds a
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Playing around with this a bit, I was easily able to get 2-second planing times on 15 table join, 6-second planning times on a 16 table join and 30-second planning times on a 17 table join. This makes it hard to support raising the GEQO threshold, as most recently suggested by Greg Sabino Mullane here (and previously by me on an earlier thread): What about 14? Could we at least raise it to 14? 1/2 :) I'm worried this is going to get bogged down like so many of our threads, where we worry about verified test cases and getting things exactly right and end up not making any changes at all (see also: random_page_cost). Robert, any ideas on a way to fix this overall process problem, outside of this particular geqo issue? (If we had a bug tracker, this would certainly be a place to stick something like this). - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation PGP Key: 0x14964AC8 200912011139 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAksVRroACgkQvJuQZxSWSsjXKgCgk1LEtvDr1mIfUjN9ez/lw60/ HcAAoPSGyqzAXL6hE1YSMb2bQoOm+oKL =eAYb -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
Greg Sabino Mullane g...@turnstep.com writes: This thread never got resolved. I think we can all agree that EOL for 7.4 is a when, not an if? Can we get -core to take a stance here and pick a date? I like the clean smooth lines of January 2011, and thus saying that 2010 is the last year in which we'll backpatch things to the 7.4 branch. But I'll stick to whatever core thinks is best. Just let the advocacy team know so we can start work on it. If we're going to set the date that far off, I'd be inclined to EOL 8.0 at the same time. It'll be six years old by then. You could make a good argument for nuking 8.1 at the same time --- it'll turn five in November 2010. Personally I'll still be on the hook for maintaining 8.1 in RHEL5 so I'd be just as happy to keep it alive a bit longer, but if the community doesn't want to deal with it that makes perfect sense. I have no personal commitment to 8.0 at all because Red Hat never shipped that in a RHEL release ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
On Tue, Dec 1, 2009 at 4:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: Greg Sabino Mullane g...@turnstep.com writes: This thread never got resolved. I think we can all agree that EOL for 7.4 is a when, not an if? Can we get -core to take a stance here and pick a date? I like the clean smooth lines of January 2011, and thus saying that 2010 is the last year in which we'll backpatch things to the 7.4 branch. But I'll stick to whatever core thinks is best. Just let the advocacy team know so we can start work on it. If we're going to set the date that far off, I'd be inclined to EOL 8.0 at the same time. It'll be six years old by then. You could make a good argument for nuking 8.1 at the same time --- it'll turn five in November 2010. Personally I'll still be on the hook for maintaining 8.1 in RHEL5 so I'd be just as happy to keep it alive a bit longer, but if the community doesn't want to deal with it that makes perfect sense. I have no personal commitment to 8.0 at all because Red Hat never shipped that in a RHEL release ... Presumably you'll be on the hook until 2014 for 8.1 security patches for RHEL 5 (looking at http://www.redhat.com/security/updates/errata/)? I can't see the community wanting to support it for that long, so I'd suggest we can (respectfully) ignore your commitments in that regard. I'm for EOLing *at least* 7.4 and 8.0 by January 2011, and I'm certainly not going to argue against doing the same for 8.1. Frankly, I think we could do 7.4 and maybe 8.0 six months earlier. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
On 12/1/09, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: On Tue, Dec 1, 2009 at 4:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't think that we need to bump the protocol version. The real alternative here would be that libpq sends a startup packet that includes application_name, and if it gets an error back from that, it starts over without the app name. I looked (briefly) at doing that when we first ran into this suggestion. As you pointed out at the time, it seemed like that would require some fairly ugly hackery in fe-connect.c Perhaps, but at the time it wasn't apparent that issuing a separate SET would create user-visible behavioral inconsistencies. Now that we've realized that, I think we should reconsider. If people are agreed that double connect is a better alternative I'm willing to go look at how to make it happen. Is it supposed to work with pooling or not? If the pooler gets new connection with same username:database as some existing connection, but with different appname, what it is supposed to do? -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Empty dictionary file when creating text search dictionary
Found this a couple of weeks back and just re-tested against head: CREATE TEXT SEARCH DICTIONARY with an empty thesaurus file will crasch the backend. To reproduce: $ echo $(pg_config --sharedir)/tsearch_data/thesaurus_empty.ths Then use this thesaurus to create a text search dictionary: CREATE TEXT SEARCH DICTIONARY thesaurus_astro ( TEMPLATE = thesaurus, DictFile = thesaurus_empty, Dictionary = english_stem ); It doesn't matter if the file is compleatly empty or just have comments. The result is the same. Add a word: $ echo foo: bar $(pg_config --sharedir)/tsearch_data/thesaurus_empty.ths Creating the dictionary will now work. Sorry I have no patch to attach. Regards, roppert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] 答复: [HACKERS] is isolation level 'Serializa ble' in pg not same as 'serializable' in S QL-92?
Thank you very much! -邮件原件- 发件人: Robert Haas [mailto:robertmh...@gmail.com] 发送时间: 2009年12月1日 1:37 收件人: 张中 抄送: pgsql-hackers@postgresql.org 主题: Re: [HACKERS] is isolation level 'Serializable' in pg not same as 'serializable' in SQL-92? 2009/11/30 张茂森 maosen.zh...@alibaba-inc.com: pgsql-hackers is not the right place for user questions; try pgsql-general or pgsql-novice. The answer to your question is in the documentation. You can find it here: http://www.postgresql.org/docs/8.4/static/transaction-iso.html#MVCC-SERIALIZABILITY ...Robert This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you. 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
On Tue, 1 Dec 2009, Tom Lane wrote: Greg Sabino Mullane g...@turnstep.com writes: This thread never got resolved. I think we can all agree that EOL for 7.4 is a when, not an if? Can we get -core to take a stance here and pick a date? I like the clean smooth lines of January 2011, and thus saying that 2010 is the last year in which we'll backpatch things to the 7.4 branch. But I'll stick to whatever core thinks is best. Just let the advocacy team know so we can start work on it. If we're going to set the date that far off, I'd be inclined to EOL 8.0 at the same time. It'll be six years old by then. You could make a good argument for nuking 8.1 at the same time --- it'll turn five in November 2010. Personally I'll still be on the hook for maintaining 8.1 in RHEL5 so I'd be just as happy to keep it alive a bit longer, but if the community doesn't want to deal with it that makes perfect sense. I have no personal commitment to 8.0 at all because Red Hat never shipped that in a RHEL release ... Just curious, but since you do all the back patching as it is, and building the source tarballs is simple enough ... What are RedHats EOL dates for the various releases? Doesn't mean that packagers have to make new packages ... I personally think new packages shouldn't be made for anything older then *maybe* 3 releases (8.2, 8.3 and 8.4), but even that I think tends to be a bit excessive ... but doing source tar balls is easy enough ... Marc G. FournierHub.Org Hosting Solutions S.A. scra...@hub.org http://www.hub.org Yahoo:yscrappySkype: hub.orgICQ:7615664MSN:scra...@hub.org -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [GENERAL] [HACKERS] Fwd: psql+krb5
On Tue, Dec 1, 2009 at 11:26 AM, Scott Marlowe scott.marl...@gmail.com wrote: Except that he posted a month ago and got no answers... Gee, I wonder why. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 07:05 -0500, Bruce Momjian wrote: All useful detection mechanisms have non-zero false positives because we would rather sometimes ring the bell for no reason than to let bad things through silently, as we do now. OK, but what happens if someone gets the failure report, assumes their hardware is faulty and replaces it, and then gets a failure report again? They are stupid? Nobody just replaces hardware. You test it. We can't fix stupid. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 10:55 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2009-12-01 at 16:40 +0200, Heikki Linnakangas wrote: It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. They are already masked from us, so continuing to mask those errors would not put us in a worse position. No, it would just destroy a large part of the argument for why this is worth doing. We detect disk errors ... except for ones that happen during a database crash. Say what? I know what I said sounds ridiculous, I'm just trying to keep my mind open about the tradeoffs. The way to detect 100% of corruptions is to WAL-log 100% of writes to blocks and we know that sucks performance - twas me that said it in the original discussion. I'm trying to explore whether we can detect 100% of other errors at some intermediate percentage of WAL-logging. If we decide that there isn't an intermediate position worth taking, I'm happy, as long it was a fact-based decision. The fundamental problem with this is the same as it's been all along: the tradeoff between implementation work expended, performance overhead added, and net number of real problems detected (with a suitably large demerit for actually *introducing* problems) just doesn't look attractive. You can make various compromises that improve one or two of these factors at the cost of making the others worse, but at the end of the day I've still not seen a combination that seems worth doing. I agree. But also I do believe there are people that care enough about this to absorb a performance hit and the new features in 8.5 will bring in a new crop of people that care about those things very much. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Empty dictionary file when creating text search dictionary
This is a duplicate report and has been fixed by Tom already. --- Robert Gravsj? wrote: Found this a couple of weeks back and just re-tested against head: CREATE TEXT SEARCH DICTIONARY with an empty thesaurus file will crasch the backend. To reproduce: $ echo $(pg_config --sharedir)/tsearch_data/thesaurus_empty.ths Then use this thesaurus to create a text search dictionary: CREATE TEXT SEARCH DICTIONARY thesaurus_astro ( TEMPLATE = thesaurus, DictFile = thesaurus_empty, Dictionary = english_stem ); It doesn't matter if the file is compleatly empty or just have comments. The result is the same. Add a word: $ echo foo: bar $(pg_config --sharedir)/tsearch_data/thesaurus_empty.ths Creating the dictionary will now work. Sorry I have no patch to attach. Regards, roppert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 10:55 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2009-12-01 at 16:40 +0200, Heikki Linnakangas wrote: It's not hard to imagine that when a hardware glitch happens causing corruption, it also causes the system to crash. Recalculating the CRCs after crash would mask the corruption. They are already masked from us, so continuing to mask those errors would not put us in a worse position. No, it would just destroy a large part of the argument for why this is worth doing. We detect disk errors ... except for ones that happen during a database crash. Say what? The fundamental problem with this is the same as it's been all along: the tradeoff between implementation work expended, performance overhead added, and net number of real problems detected (with a suitably large demerit for actually *introducing* problems) just doesn't look attractive. You can make various compromises that improve one or two of these factors at the cost of making the others worse, but at the end of the day I've still not seen a combination that seems worth doing. Let me try a different but similar perspective. The problem we are trying to solve here, only matters to a very small subset of the people actually using PostgreSQL. Specifically, a percentage that is using PostgreSQL in a situation where they can lose many thousands of dollars per minute or hour should an outage occur. On the other hand it is those very people that are *paying* people to try and implement these features. Kind of a catch-22. The hard core reality is this. *IF* it is one of the goals of this project to insure that the software can be safely, effectively, and responsibly operated in a manner that is acceptable to C* level people in a Fortune level company then we *must* solve this problem. If it is not the goal of the project, leave it to EDB/CMD/2ndQuandrant to fork it because it will eventually happen. Our customers are demanding these features. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Simon Riggs wrote: Also, we might * Put all hint bits in the block header to allow them to be excluded more easily from CRC checking. If we used 3 more bits from ItemIdData.lp_len (limiting tuple length to 4096) then we could store some hints in the item pointer. HEAP_XMIN_INVALID can be stored as LP_DEAD, since that will happen very quickly anyway. OK, here is another idea, maybe crazy: When we read in a page that has an invalid CRC, we check the page to see which hint bits are _not_ set, and we try setting them to see if can get a matching CRC. If there no missing hint bits and the CRC doesn't match, we know the page is corrupt. If two hint bits are missing, we can try setting one and both of them and see if can get a matching CRC. If we can, the page is OK, if not, it is corrupt. Now if 32 hint bits are missing, but could be based on transaction status, then we would need 2^32 possible hint bit combinations, so we can't do the test and we just assume the page is valid. I have no idea what percentage of corruption this would detect, but it might have minimal overhead because the overhead only happens when we detect a non-matching CRC due to a crash of some sort. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deleted WAL files held open by backends in Linux
Tom Lane t...@sss.pgh.pa.us wrote: Determining whether it's still the current append target is not so cheap though; it would require examining shared-memory status which means taking a lock on that status (and it's a high-traffic lock already). I haven't reviewed the internal locking techniques, so this may well be a dumb question, but... Since we only care whether the value is equal, and an occasional false report of equality wouldn't hurt anything, couldn't we bypass the lock in this particular case? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Rewrite GEQO`s gimme_tree function so that it always finds a
On Tue, Dec 1, 2009 at 11:39 AM, Greg Sabino Mullane g...@turnstep.com wrote: Playing around with this a bit, I was easily able to get 2-second planing times on 15 table join, 6-second planning times on a 16 table join and 30-second planning times on a 17 table join. This makes it hard to support raising the GEQO threshold, as most recently suggested by Greg Sabino Mullane here (and previously by me on an earlier thread): What about 14? Could we at least raise it to 14? 1/2 :) I doubt we can raise it at all without lying to ourselves about the likely results of so doing. The GEQO planning times are in the low double digits of milliseconds. My apps typically have a budget of at most ~200 ms to plan and execute the query, and I'm not always operating on empty tables. I'm worried this is going to get bogged down like so many of our threads, where we worry about verified test cases and getting things exactly right and end up not making any changes at all (see also: random_page_cost). Robert, any ideas on a way to fix this overall process problem, outside of this particular geqo issue? (If we had a bug tracker, this would certainly be a place to stick something like this). I'm not sure I agree with the premise that there is a problem in need of fixing. I think we're usually pretty good about fixing things when there is a simple, straightforward fix. When the only real fixes involve writing a lot of code, we tend to be good about fixing them when - and only when - someone is willing and able to write that code. Often that's not the case, but that's an economic problem more than a process problem. And then there are cases (like CRCs) where we can't even figure out what the code would look like, and then we tend to do nothing, but what's the other choice? Obviously you see this issue differently so I'd like to hear more of your thoughts. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, Dec 1, 2009 at 1:02 PM, Joshua D. Drake j...@commandprompt.com wrote: The hard core reality is this. *IF* it is one of the goals of this project to insure that the software can be safely, effectively, and responsibly operated in a manner that is acceptable to C* level people in a Fortune level company then we *must* solve this problem. If it is not the goal of the project, leave it to EDB/CMD/2ndQuandrant to fork it because it will eventually happen. Our customers are demanding these features. OK, and when you fork it, how do you plan to implement it? The problem AFAICS is not that anyone hugely dislikes the feature; it's that nobody is really clear on how to implement it in a way that's actually useful. So far the only somewhat reasonable suggestions I've seen seem to be: 1. WAL-log setting the hint bits. If you don't like the resulting performance, shut off the feature. 2. Rearrange the page so that all the hint bits are in the first 512 bytes along with the CRC, so that there can be no torn pages. AFAICS, no one has rendered judgment on whether this is a feasible solution. Does $COMPETITOR offer this feature? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 13:05 -0500, Bruce Momjian wrote: Simon Riggs wrote: Also, we might * Put all hint bits in the block header to allow them to be excluded more easily from CRC checking. If we used 3 more bits from ItemIdData.lp_len (limiting tuple length to 4096) then we could store some hints in the item pointer. HEAP_XMIN_INVALID can be stored as LP_DEAD, since that will happen very quickly anyway. OK, here is another idea, maybe crazy: When there's nothing else left, crazy wins. When we read in a page that has an invalid CRC, we check the page to see which hint bits are _not_ set, and we try setting them to see if can get a matching CRC. If there no missing hint bits and the CRC doesn't match, we know the page is corrupt. If two hint bits are missing, we can try setting one and both of them and see if can get a matching CRC. If we can, the page is OK, if not, it is corrupt. Now if 32 hint bits are missing, but could be based on transaction status, then we would need 2^32 possible hint bit combinations, so we can't do the test and we just assume the page is valid. I have no idea what percentage of corruption this would detect, but it might have minimal overhead because the overhead only happens when we detect a non-matching CRC due to a crash of some sort. Perhaps we could store a sector-based parity bit each 512 bytes in the block. If there are an even number of hint bits set, if odd we unset the parity bit. So whenever we set a hint bit we flip the parity bit for that sector. That way we could detect which sectors are potentially missing in an effort to minimize the number of combinations we need to test. That would require only 16 bits for an 8192 byte block; we store it next to the CRC, so we know that was never altered separately. So total 6 byte overhead. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v4
Marko Kreen mark...@gmail.com writes: If the pooler gets new connection with same username:database as some existing connection, but with different appname, what it is supposed to do? Whatever it wants to. People seem to be imagining that the appname isn't under the control of the pooler. It's a connection property, remember? It won't be set at all unless the pooler explicitly sets it or allows it to be set. I would imagine that typically a pooler would consider the whole connection string as defining connection properties and so appname would work the same as username or anything else, ie, you get shunted into a different connection pool if you ask for a different appname. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby remaining issues
Simon Riggs wrote: commit 02c3eadb766201db084b668daa271db4a900adc9 Author: Simon Riggs sri...@ebony.(none) Date: Sat Nov 28 06:23:33 2009 + Added wal_standby_info GUC to turn RM_STANDBY_ID messages on/off. Various comments added also. This patch makes it unsafe to start hot standby mode from a shutdown checkpoint, because we don't know if wal_standby_info was enabled in the master. I still don't think we need the GUC. But for future-proofing, perhaps we should add a flag to shutdown checkpoint records, indicating whether it's safe to start hot standby from it. That way, if we decide to add a GUC like that at a later stage, we don't need to change the on-disk format. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, 2009-12-01 at 13:20 -0500, Robert Haas wrote: On Tue, Dec 1, 2009 at 1:02 PM, Joshua D. Drake j...@commandprompt.com wrote: The hard core reality is this. *IF* it is one of the goals of this project to insure that the software can be safely, effectively, and responsibly operated in a manner that is acceptable to C* level people in a Fortune level company then we *must* solve this problem. If it is not the goal of the project, leave it to EDB/CMD/2ndQuandrant to fork it because it will eventually happen. Our customers are demanding these features. OK, and when you fork it, how do you plan to implement it? Hey man, I am not an engineer :P. You know that. I am just speaking the pressures that some of us are having in the marketplace about these types of features. red judgment on whether this is a feasible solution. Does $COMPETITOR offer this feature? My understanding is that MSSQL does. I am not sure about Oracle. Those are the only two I run into (I don't run into MySQL at all). I know others likely compete in the DB2 space. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
Marc G. Fournier scra...@hub.org writes: What are RedHats EOL dates for the various releases? Dave already mentioned a public page for that: http://www.redhat.com/security/updates/errata/ Based on track record so far, Red Hat isn't going to care about anything but high-priority security issues towards the end of the life cycle, but theoretically I'm on the hook till 2014 for 8.1.x. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Bruce Momjian br...@momjian.us writes: OK, here is another idea, maybe crazy: When we read in a page that has an invalid CRC, we check the page to see which hint bits are _not_ set, and we try setting them to see if can get a matching CRC. If there no missing hint bits and the CRC doesn't match, we know the page is corrupt. If two hint bits are missing, we can try setting one and both of them and see if can get a matching CRC. If we can, the page is OK, if not, it is corrupt. Now if 32 hint bits are missing, but could be based on transaction status, then we would need 2^32 possible hint bit combinations, so we can't do the test and we just assume the page is valid. A typical page is going to have something like 100 tuples, so potentially 2^400 combinations to try. I don't see this being realistic from that standpoint. What's much worse is that to even find the potentially missing hint bits, you need to make very strong assumptions about the validity of the rest of the page. The suggestions that were made upthread about moving the hint bits could resolve the second objection, but once you do that you might as well just exclude them from the CRC and eliminate the guessing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2009-12-01 at 13:05 -0500, Bruce Momjian wrote: When we read in a page that has an invalid CRC, we check the page to see which hint bits are _not_ set, and we try setting them to see if can get a matching CRC. Perhaps we could store a sector-based parity bit each 512 bytes in the block. If there are an even number of hint bits set, if odd we unset the parity bit. So whenever we set a hint bit we flip the parity bit for that sector. That way we could detect which sectors are potentially missing in an effort to minimize the number of combinations we need to test. Actually, the killer problem with *any* scheme involving guessing is that each bit you guess translates directly to removing one bit of confidence from the CRC value. If you try to guess at as many as 32 bits, it is practically guaranteed that you will find a combination that makes a 32-bit CRC appear to match. Well before that, you have degraded the reliability of the error detection to the point that there's no point. The bottom line here seems to be that the only practical way to do anything like this is to move the hint bits into their own area of the page, and then exclude them from the CRC. Are we prepared to once again blow off any hope of in-place update for another release cycle? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deleted WAL files held open by backends in Linux
Kevin Grittner kevin.gritt...@wicourts.gov writes: Tom Lane t...@sss.pgh.pa.us wrote: Determining whether it's still the current append target is not so cheap though; it would require examining shared-memory status which means taking a lock on that status (and it's a high-traffic lock already). I haven't reviewed the internal locking techniques, so this may well be a dumb question, but... Since we only care whether the value is equal, and an occasional false report of equality wouldn't hurt anything, couldn't we bypass the lock in this particular case? Perhaps, if you didn't mind sometimes getting a wrong answer. I guess the cost of that would be pretty small in this particular usage. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
* Tom Lane t...@sss.pgh.pa.us [091201 13:58]: Actually, the killer problem with *any* scheme involving guessing is that each bit you guess translates directly to removing one bit of confidence from the CRC value. If you try to guess at as many as 32 bits, it is practically guaranteed that you will find a combination that makes a 32-bit CRC appear to match. Well before that, you have degraded the reliability of the error detection to the point that there's no point. Exactly. The bottom line here seems to be that the only practical way to do anything like this is to move the hint bits into their own area of the page, and then exclude them from the CRC. Are we prepared to once again blow off any hope of in-place update for another release cycle? Well, *I* think if we're ever going to have really reliable in-place upgrades that we can expect to function release after release, we're going to need to be able to read in old version pages, and convert them to current version pages, for some set of old version (I'ld be happy with $VERSION-1)... But I don't see that happening any time soon... But I'm not loading TB of data either, my largest clusters are a couple of gigs, so I acknowledge my priorities are probably quite different then some of the companies driving a lot of the heavy development. a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] Re: [COMMITTERS] pgsql: Rewrite GEQO`s gimme_tree function so that it always finds a
Robert Haas robertmh...@gmail.com writes: On Tue, Dec 1, 2009 at 11:39 AM, Greg Sabino Mullane g...@turnstep.com wrote: What about 14? Could we at least raise it to 14? 1/2 :) I doubt we can raise it at all without lying to ourselves about the likely results of so doing. The GEQO planning times are in the low double digits of milliseconds. My apps typically have a budget of at most ~200 ms to plan and execute the query, and I'm not always operating on empty tables. The reason this is a configurable parameter is so that people can tune it to their own needs. I think the current default fits all right with our usual policy of being conservative about hardware requirements. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, Dec 1, 2009 at 6:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: OK, here is another idea, maybe crazy: When we read in a page that has an invalid CRC, we check the page to see which hint bits are _not_ set, and we try setting them to see if can get a matching CRC. Unfortunately you would also have to try *unsetting* every hint bit as well since the updated hint bits might have made it to disk but not the CRC leaving the old CRC for the block with the unset bits. I actually independently had the same thought today that Simon had of moving the hint bits to the line pointer. We can obtain more free bits in the line pointers by dividing the item offsets and sizes by maxalign if we need it. That should give at least 4 spare bits which is all we need for the four VALID/INVALID hint bits. It should be relatively cheap to skip the hint bits in the line pointers since they'll be the same bits of every 16-bit value for a whole range. Alternatively we could just CRC the tuples and assume a corrupted line pointer will show itself quickly. That would actually make it faster than a straight CRC of the whole block -- making lemonade out of lemons as it were. There's still the all-tuples-in-page-are-visible hint bit and the hint bits in btree pages. I'm not sure if those are easier or harder to solve. We might be able to assume the all-visible flag will not be torn from the crc as long as they're within the same 512 byte sector. And iirc the btree hint bits are in the line pointers themselves as well? Another thought is that would could use the MSSQL-style torn page detection of including a counter (or even a bit?) in every 512-byte chunk which gets incremented every time the page is written. If they don't all match when read in then the page was torn and we can't check the CRC. That gets us the advantage that we can inform the user that a torn page was detected so they know that they must absolutely use full_page_writes on their system. Currently users are in the dark whether their system is susceptible to them or not and have now idea with what frequency. Even here there are quite divergent opinions about their frequency and which systems are susceptible to them or immune. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deleted WAL files held open by backends in Linux
Tom Lane t...@sss.pgh.pa.us wrote: Perhaps, if you didn't mind sometimes getting a wrong answer. Well, it would be a heuristic which would close the deleted file *almost* all the time. When it didn't, the next check would probably catch it. Assuming that you would never get an indication that it was different when it hadn't changed, the worst that could happen is that you infrequently get the current behavior. That all *seems* sane to me, if low priroity. Any objections to my putting it on the TODO list? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Greg Stark gsst...@mit.edu writes: Another thought is that would could use the MSSQL-style torn page detection of including a counter (or even a bit?) in every 512-byte chunk which gets incremented every time the page is written. I think we can dismiss that idea, or any idea involving a per-page status value, out of hand. The implications for tuple layout are just too messy. I'm not especially thrilled with making assumptions about the underlying device block size anyway. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
is there a reason why we can't follow a similar 4+3 life cycle? packagers r produced for the first 4y after .0 release and only source updates for year 5 thru 7? if we could advertise such on the web, there would be no question as to when bug reports are accepted (n+4y) and when only security ... and after y7, it's just not supported at all ... that would kill packager requirements on 8.0, 8.1 (as of last month) and totally kill 7.4 as of nov '10 Sent from my iPhone On 2009-12-01, at 14:33, Tom Lane t...@sss.pgh.pa.us wrote: Marc G. Fournier scra...@hub.org writes: What are RedHats EOL dates for the various releases? Dave already mentioned a public page for that: http://www.redhat.com/security/updates/errata/ Based on track record so far, Red Hat isn't going to care about anything but high-priority security issues towards the end of the life cycle, but theoretically I'm on the hook till 2014 for 8.1.x. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
All, I feel strongly that we should be verifying pages on write, or at least providing the option to do so, because hardware is simply not reliable. And a lot of our biggest users are having issues; it seems pretty much guarenteed that if you have more than 20 postgres servers, at least one of them will have bad memory, bad RAID and/or a bad driver. (and yes, InnoDB, DB2 and Oracle do provide tools to detect hardware corruption when it happens. Oracle even provides correction tools. We are *way* behind them in this regard) There are two primary conditions we are testing for: (a) bad RAM, which happens as frequently as 8% of the time on commodity servers, and given a sufficient amount of RAM happens 99% of the time due to quantum effects, and (b) bad I/O, in the form of bad drivers, bad RAID, and/or bad disks. Our users want to potentially take two degrees of action on this: 1. detect the corruption immediately when it happens, so that they can effectively troubleshoot the cause of the corruption, and potentially shut down the database before further corruption occurs and while they still have clean backups. 2. make an attempt to fix the corrupted page before/immediately after it is written. Further, based on talking to some of these users who are having chronic and not-debuggable issues on their sets of 100's of PostgreSQL servers, there are some other specs: -- Many users would be willing to sacrifice significant performance (up to 20%) as a start-time option in order to be corruption-proof. -- Even more users would only be interested in using the anti-corruption options after they know they have a problem to troubleshoot it, and then turn the corruption detection back off. So, based on my conversations with users, what we really want is a solution which does (1) for both (a) and (b) as a start-time option, and having siginificant performance overhead for this option is OK. Now, does block-level CRCs qualify? The problem I have with CRC checks is that it only detects bad I/O, and is completely unable to detect data corruption due to bad memory. This means that really we want a different solution which can detect both bad RAM and bad I/O, and should only fall back on CRC checks if we're unable to devise one. One of the things Simon and I talked about in Japan is that most of the time, data corruption makes the data page and/or tuple unreadable. So, checking data format for readable pages and tuples (and index nodes) both before and after write to disk (the latter would presumably be handled by the bgwriter and/or checkpointer) would catch a lot of kinds of corruption before they had a chance to spread. However, that solution would not detect subtle corruption, like single-bit-flipping issues caused by quantum errors. Also, it would require reading back each page as it's written to disk, which is OK for a bunch of single-row writes, but for bulk data loads a significant problem. So, what I'm saying is that I think we really want a better solution, and am throwing this out there to see if anyone is clever enough. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enable-thread-safety defaults?
2009/12/1 Bruce Momjian br...@momjian.us: Peter Eisentraut wrote: On m?n, 2009-11-30 at 12:21 -0500, Bruce Momjian wrote: ! for thread safety; use --disable-thread-safety to disable threading.]) --disable-thread-safety does not disable threading, it disables thread safety. Good point! Patch updated and attached. What are we going to do for build farm members who don't support threading? Is someone going to manually update their configure flags? Yeah, I think so. Unless there's a whole lot of them, in which case we revert the patch. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PgSQL patch review
This is totally separate from the really important question of whether SE-Linux has a future, and another about whether, if SE-Linux has a future, PostgreSQL needs to go there. If the hooks are generic enough that the could potentially be adapted to other security frameworks, yes. The need to have cohesive centralized systems permissions management hasn't gone away, whatever anyone thinks of the SE-linux implementation. That's why I was hoping to have the TrustedSolaris folks working on this, but we've pretty much lost access to them. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
Scrappy wrote: is there a reason why we can't follow a similar 4+3 life cycle? packagers r produced for the first 4y after .0 release and only source updates for year 5 thru 7? if we could advertise such on the web, there would be no question as to when bug reports are accepted (n+4y) and when only security ... and after y7, it's just not supported at all ... that would kill packager requirements on 8.0, 8.1 (as of last month) and totally kill 7.4 as of nov '10 What packagers produce is surely up to them. If RedHat or Devrim or Dave want to produce a package that's their prerogative. And IMNSHO 4 years is too short a period for non-security bugs. We have seen odd behaviour issues past those dates. The time between these periodic debates seems to be getting shorter and shorter. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PgSQL patch review
On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: This is totally separate from the really important question of whether SE-Linux has a future, and another about whether, if SE-Linux has a future, PostgreSQL needs to go there. Why would we think that it doesn't? Maybe I haven't been following as much as I should but as I understand it, SE-Linux is pretty much the defacto policy framework for Linux. Has that changed? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enable-thread-safety defaults?
Magnus Hagander mag...@hagander.net writes: 2009/12/1 Bruce Momjian br...@momjian.us: What are we going to do for build farm members who don't support threading? Is someone going to manually update their configure flags? Yeah, I think so. Unless there's a whole lot of them, in which case we revert the patch. It would seem like we ought to try the one-liner patch Magnus proposed (ie flip the default) and see what the effects are, before we go with the much larger patch Bruce wrote. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [CORE] [HACKERS] EOL for 7.4?
Andrew Dunstan and...@dunslane.net writes: The time between these periodic debates seems to be getting shorter and shorter. No, this is just a continuation of the unresolved thread from a month or so ago. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
Josh Berkus j...@agliodbs.com wrote: And a lot of our biggest users are having issues; it seems pretty much guarenteed that if you have more than 20 postgres servers, at least one of them will have bad memory, bad RAID and/or a bad driver. Huh?!? We have about 200 clusters running on about 100 boxes, and we see that very rarely. On about 100 older boxes, relegated to less critical tasks, we see a failure maybe three or four times per year. It's usually not subtle, and a sane backup and redundant server policy has kept us from suffering much pain from these. I'm not questioning the value of adding features to detect corruption, but your numbers are hard to believe. The problem I have with CRC checks is that it only detects bad I/O, and is completely unable to detect data corruption due to bad memory. This means that really we want a different solution which can detect both bad RAM and bad I/O, and should only fall back on CRC checks if we're unable to devise one. md5sum of each tuple? As an optional system column (a la oid)? checking data format for readable pages and tuples (and index nodes) both before and after write to disk Given that PostgreSQL goes through the OS, and many of us are using RAID controllers with BBU RAM, how do you do a read with any confidence that it came from the disk? (I mean, I know how to do that for a performance test, but as a routine step during production use?) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Block-level CRC checks
On Tue, Dec 1, 2009 at 7:19 PM, Josh Berkus j...@agliodbs.com wrote: However, that solution would not detect subtle corruption, like single-bit-flipping issues caused by quantum errors. Well there is a solution for this, ECC RAM. There's *no* software solution for it. The corruption can just as easily happen the moment you write the value before you calculate any checksum or in the register holding the value before you even write it. Or it could occur the moment after you finish checking the checksum. Also you're not going to be able to be sure you're checking the actual dram and not the L2 cache or the processor's L1/L0 caches. ECC RAM solves this problem properly and it does work. There's not much point in paying a much bigger cost for an ineffective solution. Also, it would require reading back each page as it's written to disk, which is OK for a bunch of single-row writes, but for bulk data loads a significant problem. Not sure what that really means for Postgres. It would just mean reading back the same page of memory from the filesystem cache that we just read. It sounds like you're describing fsyncing every single page to disk and then wait 1min/7200 or even 1min/15k to do a direct read for every single page -- that's not a 20% performance hit though. We would have to change our mascot from the elephant to a snail I think. You could imagine a more complex solution where you have a separate process wait until the next checkpoint then do direct reads for all the blocks written since the previous checkpoint (which have now been fsynced) and verify that the block on disk has a verifiable CRC. I'm not sure even direct reads let you get the block on disk if someone else has written the block into cache though. If you could then this sounds like it could be made to work efficiently (with sequential bitmap-style scans) and could be quite handy. What I like about that is you could deprioritize this process's i/o so that it didn't impact the main processing. As things stand this wouldn't detect pages written because they were dirtied by hint bit updates but that could be addressed a few different ways. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SE-PgSQL patch review
Joshua D. Drake j...@commandprompt.com writes: On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: This is totally separate from the really important question of whether SE-Linux has a future, and another about whether, if SE-Linux has a future, PostgreSQL needs to go there. Why would we think that it doesn't? Have you noticed anyone except Red Hat taking it seriously? I work for Red Hat and have drunk a reasonable amount of the SELinux koolaid, but I can't help observing that it's had very limited uptake outside Red Hat. It's not clear that there are many people who find it a cost-effective solution to their problems. As for the number of people prepared to write custom policy for it --- which would be required to use it effectively for almost any PG application --- I could probably hold a house party for all of them and not break a sweat serving drinks. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
