Re: [PATCHES] Testing pg_terminate_backend()
Bruce Momjian wrote: bruce wrote: Magnus Hagander wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? That was me. As long as the script runs properly on linux, I can get that started as soon as I'm fed instructions on how to do it :-) Do I just fix the paths and set it running, or do I need to prepare something else? Nothing special to prepare. Compile with asserts enabled, and run the script. The comment at the top explains how to analyze the log for interesting error messages. Oh, you need to set a variable in the script indicating the average number of seconds it takes to run the regression test on your system. Done that. Also, I needed to replace gmake with make, since I'm on linux... Anyway. It's been running for about 12 hours now, and I have *nothing* in the output file. That tells me that the script doesn't appear to be working - correct? It should output *something* there, right? (It's obviously running, because I've got about 400,000 lines in nohup.out..) Argh. So here I am looking at it now for details, and it seems the script should be run from src/test/regress, and I ran it from the root directory.. Oops. Also, I needed to place psql in the path - it failed to find it, but hid the error message. Just a hint if someone else is running it ;-) //Magnus -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Magnus Hagander wrote: Done that. Also, I needed to replace gmake with make, since I'm on linux... Anyway. It's been running for about 12 hours now, and I have *nothing* in the output file. That tells me that the script doesn't appear to be working - correct? It should output *something* there, right? (It's obviously running, because I've got about 400,000 lines in nohup.out..) Argh. So here I am looking at it now for details, and it seems the script should be run from src/test/regress, and I ran it from the root directory.. Oops. Also, I needed to place psql in the path - it failed to find it, but hid the error message. Just a hint if someone else is running it ;-) Yea, basically you need to rewrite my script. :-( -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Bruce Momjian wrote: Magnus Hagander wrote: Done that. Also, I needed to replace gmake with make, since I'm on linux... Anyway. It's been running for about 12 hours now, and I have *nothing* in the output file. That tells me that the script doesn't appear to be working - correct? It should output *something* there, right? (It's obviously running, because I've got about 400,000 lines in nohup.out..) Argh. So here I am looking at it now for details, and it seems the script should be run from src/test/regress, and I ran it from the root directory.. Oops. Also, I needed to place psql in the path - it failed to find it, but hid the error message. Just a hint if someone else is running it ;-) Yea, basically you need to rewrite my script. :-( Not really, but it did need a couple of adjustments :-) It's been running fine now for a number of hours, with output that looks similar to the stuff you posted. I'll leave it running.. //Magnus -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Magnus Hagander wrote: It's been running fine now for a number of hours, with output that looks similar to the stuff you posted. I'll leave it running.. Perhaps it would be a good idea to leave it running on code with some bugs on it, just to check if the problems show up. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
On Mon, Apr 21, 2008 at 7:25 PM, Magnus Hagander [EMAIL PROTECTED] wrote: Not really, but it did need a couple of adjustments :-) It's been running fine now for a number of hours, with output that looks similar to the stuff you posted. I'll leave it running.. If you can come up with an easily installable tarball, I can dedicate 1 or 2 boxes to run it 24/7. -- Guillaume -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Guillaume Smet wrote: On Mon, Apr 21, 2008 at 7:25 PM, Magnus Hagander [EMAIL PROTECTED] wrote: Not really, but it did need a couple of adjustments :-) It's been running fine now for a number of hours, with output that looks similar to the stuff you posted. I'll leave it running.. If you can come up with an easily installable tarball, I can dedicate 1 or 2 boxes to run it 24/7. Sure. I updated the script so it will be clearer what you have to modify. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + #!/bin/bash REGRESSION_DURATION=# average duration of regression test, in seconds OUTFILE= [ $REGRESSION_DURATION -o ! $OUTFILE ] echo Must set REGRESSION_DURATION and OUTFILE in the script 12 exit 1 [ ! -f parallel_schedule ] echo This must be from the Postgres src/test/regress directory 12 exit 1 if gmake -h /dev/null 21 thenMAKE=gmake elseMAKE=make fi # To analyze output, use: echo Running ... echo To analyze the output log file, use: echo grep '^\+ *[A-Z][A-Z]*:' /rtmp/regression.sigterm | sort | uniq | less while : do ( SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767` echo Sleeping $SLEEP seconds sleep $SLEEP echo Trying kill # send up to 7 kill signals for X in 1 2 3 4 5 6 7 do psql -p 55432 -qt -c SELECT pg_terminate_backend(stat.procpid) FROM (SELECT procpid FROM pg_stat_activity ORDER BY random() LIMIT 1) AS stat template1 2 /dev/null if [ $? -eq 0 ] thenecho Kill sent fi sleep 5 done ) $MAKE check wait [ -s regression.diffs ] cat regression.diffs $OUTFILE done -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
[PATCHES] Testing pg_terminate_backend()
bruce wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: The closest thing I can think of to an automated test is to run repeated sets of the parallel regression tests, and each time SIGTERM a randomly chosen backend at a randomly chosen time. Then see if anything funny Yep, that was my plan, plus running the parallel regression tests you get the possibility of 2 backends. I was intentionally suggesting only one kill per test cycle. Multiple kills will probably create an O(N^2) explosion in the set of possible downstream-failure deltas. I doubt you'd really get any improvement in testing coverage to justify the much larger amount of hand validation needed. It also strikes me that you could make some simple alterations to the regression tests to reduce the set of observable downstream deltas. For example, anyplace where a test loads a table with successive INSERTs and that table is used by later tests, wrap the INSERT sequence with BEGIN/END. Then there is only one possible downstream delta (empty table) and not N different possibilities for an N-row table. I have added pg_terminate_backend() to use SIGTERM and will start running tests as discussed with Tom. I will post my scripts too. Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. I have processed the combined regression.diffs files by pickouting out all the new error messages. I don't see anything unusual in there. Should I run it differently? -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + #!/bin/bash REGRESSION_DURATION=80 # average duration of regression test in seconds OUTFILE=/rtmp/regression.sigterm # To analyze output, use: # grep '^\+ *[A-Z][A-Z]*:' /rtmp/regression.sigterm | sort | uniq | less cd /pg/test/regress while : do ( SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767` echo Sleeping $SLEEP seconds sleep $SLEEP echo Trying kill # send up to 7 kill signals for X in 1 2 3 4 5 6 7 do psql -p 55432 -qt -c SELECT pg_terminate_backend(stat.procpid) FROM (SELECT procpid FROM pg_stat_activity ORDER BY random() LIMIT 1) AS stat template1 2 /dev/null if [ $? -eq 0 ] thenecho Kill sent fi sleep 5 done ) gmake check wait [ -s regression.diffs ] cat regression.diffs $OUTFILE done + CONTEXT: COPY bool_test, line 1: TRUE null FALSE null + CONTEXT: COPY create_table_test, line 1: 5 10 + CONTEXT: COPY x, line 0: + CONTEXT: COPY x, line 1: 1 test_1 + CONTEXT: COPY x, line 1: 4000:\X:C:\X:\X + CONTEXT: SQL function declares_cursor + CONTEXT: SQL function max_xacttest + ERROR: INSERT has more expressions than target columns + ERROR: cannot drop table test1 because other objects depend on it + ERROR: column a of relation xacttest does not exist + ERROR: column f1 does not exist + ERROR: column fooid does not exist + ERROR: current transaction is aborted, commands ignored until end of transaction block + ERROR: cursor foo25 does not exist + ERROR: function getfoo(integer) does not exist + ERROR: index onek_nulltest does not exist + ERROR: index wowidx does not exist + ERROR: no such savepoint + ERROR: prepared statement q5 does not exist + ERROR: relation a_star does not exist + ERROR: relation aggtest does not exist + ERROR: relation array_index_op_test does not exist + ERROR: relation array_op_test does not exist + ERROR: relation b_star does not exist + ERROR: relation bt_f8_heap does not exist + ERROR: relation bt_i4_heap does not exist + ERROR: relation bt_name_heap does not exist + ERROR: relation bt_txt_heap does not exist + ERROR: relation c_star does not exist + ERROR: relation d_star does not exist + ERROR: relation e_star does not exist + ERROR: relation emp does not exist + ERROR: relation equipment_r does not exist + ERROR: relation f_star does not exist + ERROR: relation fast_emp4000 does not exist + ERROR: relation foo already exists + ERROR: relation gcircle_tbl does not exist + ERROR: relation gpolygon_tbl does not exist + ERROR: relation hash_f8_heap does not exist + ERROR: relation hash_i4_heap does not exist + ERROR: relation hash_name_heap does not exist + ERROR: relation hash_txt_heap does not exist + ERROR: relation hobbies_r does not exist + ERROR: relation ihighway does not exist + ERROR: relation int2_tbl does not exist + ERROR:
Re: [PATCHES] Testing pg_terminate_backend()
Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767` Uh, where's the randomness coming from? regards, tom lane -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? That was me. As long as the script runs properly on linux, I can get that started as soon as I'm fed instructions on how to do it :-) Do I just fix the paths and set it running, or do I need to prepare something else? SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767` Uh, where's the randomness coming from? ... but I should probably wait until that one is answered or fixed, I guess :-) //Magnus -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Magnus Hagander wrote: Tom Lane wrote: SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767` Uh, where's the randomness coming from? ... but I should probably wait until that one is answered or fixed, I guess :-) bash. RANDOM Each time this parameter is referenced, a random integer between 0 and 32767 is generated. The sequence of random numbers may be initialized by assigning a value to RANDOM. If RANDOM is unset, it loses its special properties, even if it is subsequently reset. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? Yes, that is what it needs. SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767` Uh, where's the randomness coming from? In bash $RANDOM returns a random number from 0-32k every time; #!/bin/bash is specified in the top line. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
Magnus Hagander wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? That was me. As long as the script runs properly on linux, I can get that started as soon as I'm fed instructions on how to do it :-) Do I just fix the paths and set it running, or do I need to prepare something else? Nothing special to prepare. Compile with asserts enabled, and run the script. The comment at the top explains how to analyze the log for interesting error messages. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
bruce wrote: Magnus Hagander wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? That was me. As long as the script runs properly on linux, I can get that started as soon as I'm fed instructions on how to do it :-) Do I just fix the paths and set it running, or do I need to prepare something else? Nothing special to prepare. Compile with asserts enabled, and run the script. The comment at the top explains how to analyze the log for interesting error messages. Oh, you need to set a variable in the script indicating the average number of seconds it takes to run the regression test on your system. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
Re: [PATCHES] Testing pg_terminate_backend()
bruce wrote: bruce wrote: Magnus Hagander wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Attached is my test script. I ran it for 14 hours (asserts on), running 450 regression tests, with up to seven backends killed per regression test. Hmm, there are something on the order of 1 SQL commands in our regression tests, so even assuming perfect randomness you've exercised SIGTERM on maybe 10% of them --- and of course there's multiple places in a complex DDL command where SIGTERM might conceivably be a problem. Who was volunteering to run this 24x7 for awhile? That was me. As long as the script runs properly on linux, I can get that started as soon as I'm fed instructions on how to do it :-) Do I just fix the paths and set it running, or do I need to prepare something else? Nothing special to prepare. Compile with asserts enabled, and run the script. The comment at the top explains how to analyze the log for interesting error messages. Oh, you need to set a variable in the script indicating the average number of seconds it takes to run the regression test on your system. And you have to set the location of the output file and where your regression test directory is located. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches
