Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: I just got to the point of having what appears to be a working but poorly optimized version of serializable transactions, Cool. so it is critical that I create a good set of tests to confirm correct behavior and monitor for regressions as I optimize. I see that you've been working on dtester recently -- should I grab what you've got here, stick with 0.1, or do you want to package something? I'm preparing for a 0.1 release, yes. However, I'm trying to keep up with the postgres-dtests branch, so that should continue to work with the latest dtester code. If I should pull from your git, any hints on the best git statements to merge that in are welcome, I'm still rather new to git, and I tend not to get things right on a first try without some hints. :-/ I'd recommend merging postgres-dtests into your branch, so you'll get all of the adjustments from there, if I change dtester itself. Once 0.1 is out, I'm *trying* to remain backwards compatible in future releases. However, that holds me from releasing 0.1 in the first place :-) Independent of versions and releases: if you build your tests on top of dtester, I'm more than willing to support you with that. If you keep your code in a git repository, I could even provide patches, in case I need (or want) to change the dtester interface. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
[off-list to avoid distracting from the 9.0 wrap-up effort] Markus Wanner mar...@bluegap.ch wrote: Quoting Kevin Grittner kevin.gritt...@wicourts.gov: I strongly encourage you to set that up on git.postgresql.org. I'm about to provide git repositories for Postgres-R anyway, so I've setup two projects on git.postgres-r.org: dtester: that's the driver/harness code postgres-dtests: a Postgres clone with the dtester patch applied - this is based on the Postgres git repository, so you can easily switch between Postgres branches. I just got to the point of having what appears to be a working but poorly optimized version of serializable transactions, so it is critical that I create a good set of tests to confirm correct behavior and monitor for regressions as I optimize. I see that you've been working on dtester recently -- should I grab what you've got here, stick with 0.1, or do you want to package something? If I should pull from your git, any hints on the best git statements to merge that in are welcome, I'm still rather new to git, and I tend not to get things right on a first try without some hints. :-/ Thanks again for this tool! -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
I wrote: [off-list to avoid distracting from the 9.0 wrap-up effort] Arg. I really didn't mean that to go to the list. :-( -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Jan Urbański wrote: sorry to butt in to the conversation, but I have spent some time wrapping/refining the concepts in dtester, and the results are here: http://git.wulczer.org/?p=twisted-psql.git;a=summary That seems to cover the concurrent psql part of dtester. But I don't see how that's wrapping or refining dtester. I borrowed the idea of wrapping a psql in a Twisted protocol and added a Deferred interface around it, which made it possible to run tests with trial: the Twisted unit testing framework. dtester works pretty much the same way, except that it doesn't use trial. But the Deferred interface is about the same. As a developer of a failry large Python system based on Twisted, that sports hundreds of trial-based tests, I very strongly recommend trial for asynchronous unit testing. It handles lots of boring details, is well maintained and Twisted itself is simply designed to do asynchronous programming. As an added bonus, the runnning and reporting infrastructure is already there, you just write the tests. I'm using trial for two other projects as well and I've started with it for Postgres. IMO trial is very good for unit tests, but fails horribly for anything that involves complex setups and watch-guarding of multiple processes. That's where dtester shines. My code is very rough and lacks good error reporting, for instance failed tests will probably result in a test hung and the need to Ctrl+C, but that can be easily improved. Don't underestimate that. There are lots of sources of errors. Not having a sentinel over the postmaster itself, unit tests from trial simply fail to notice errors there. (And coding for Postgres, these are the ones you are interested in). Now combine all of that and error handling and (useful) reporting is *the* major challenge in automating testing. A thing that would help tremendously would be a real Twisted protocol that talks to PG on the protocol level, not by parsing psql output (which is very clumsy and error prone IMHO). I agree. (Well, now I do). I found one such project: http://www.jamwt.com/pgasync/ but it had some issues with committing Yeah, I didn't like that one, either. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Quoting Kevin Grittner kevin.gritt...@wicourts.gov: I strongly encourage you to set that up on git.postgresql.org. I'm about to provide git repositories for Postgres-R anyway, so I've setup two projects on git.postgres-r.org: dtester: that's the driver/harness code postgres-dtests: a Postgres clone with the dtester patch applied - this is based on the Postgres git repository, so you can easily switch between Postgres branches. I'd like to clean postgres-dtests in the sense that all tests included there are expected to succeed on Postgres HEAD. Those (like the initial SSI ones) that are expected to fail should get marked as such in there. If you want to add SSI specific tests, which your are expecting to succeed on your branch, I'd recommend you create your own branch (or merge into your branch from postgres-dtests). Git makes that simple enough. I've barely scratched the surface on git in the last few weeks, and already I'm a big fan. I was never convinced that subversion was an improvement over cvs -- they each had advantages over the other which seemed a wash for me -- but git takes everything to a whole new level. I agree, and would like to extend that to DVCSes in general. Having started with monotone, I'm used to a different level of convenience, especially WRT network exchange and merging. To be fair, I'm excited about how fast git is (especially WRT network exchange, where monotone just plain sucks). I see your point. Even with a general solution, probably best to leave the pset there for psql, though. Agreed, that's fixed in the new git repository. I'll look those over. Any caveats beyond what you already mentioned of which I should be particularly careful? Uh.. no, there's no difference other than that. It's a paradigm question. Ones like it that way, others the other. And then there are applications that are a better fit than others... Now, I tend towards the event based approach, because it basically relieves you from having to deal with concurrent threads and all their issues. You need to get a single ordering of events anyway, if you want to check ordering constraints. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner wrote: I do want to expand the tests quite a bit -- do I work them all into this same file, or how would I proceed? I think I'll need about 20 more tests, but I don't want to get in the way of your work on the framework which runs them. Well, first of all, another piece of the missing manual: there are BaseTest and SyncTest classes. Those based on BaseTest runs within the event loop of the twisted framework, thus need to be written in the very same asynchronous fashion. Mostly calling async methods that return a Deferred object, on which you may addCallback() or addErrback(). See the fine twisted documentation, especially the part about Low-Level Networking and Event Loop here: http://twistedmatrix.com/documents/current/core/howto/index.html I'm not sure about how to organize the tests and ongoing development of the framework. I've already broken the Postgres-R tests with dtester-0.0. Hi, sorry to butt in to the conversation, but I have spent some time wrapping/refining the concepts in dtester, and the results are here: http://git.wulczer.org/?p=twisted-psql.git;a=summary It reqires Twisted and has been tested on Python 2.5 (should work on 2.6, no idea about 3.0). The program you use to run it - trial - should come with your distro's Twisted packages. The tests don't start a server or anything, so you need to have a PG instance running. To try it: git clone git://wulczer.org/twisted-psql.git cd twisted-psql # this is important, or Python won't find the modules $EDITOR config.py # set the path to psql and connection details for PG trial test.test_serialization_error trial test.test_true_serialization Both tests should pass, the latter being marked as an expectedFailure. You can then look at test/*.py to see my (puny) attempt at having some abstraction layer over the asynchronocity of the tests. I borrowed the idea of wrapping a psql in a Twisted protocol and added a Deferred interface around it, which made it possible to run tests with trial: the Twisted unit testing framework. As a developer of a failry large Python system based on Twisted, that sports hundreds of trial-based tests, I very strongly recommend trial for asynchronous unit testing. It handles lots of boring details, is well maintained and Twisted itself is simply designed to do asynchronous programming. As an added bonus, the runnning and reporting infrastructure is already there, you just write the tests. My code is very rough and lacks good error reporting, for instance failed tests will probably result in a test hung and the need to Ctrl+C, but that can be easily improved. A thing that would help tremendously would be a real Twisted protocol that talks to PG on the protocol level, not by parsing psql output (which is very clumsy and error prone IMHO). I found one such project: http://www.jamwt.com/pgasync/ but it had some issues with committing (all my test programs were exiting before PG got the final COMMIT, which resulted in the impossibility to do anything) and it does too much things that Python PG drivers like to do (like declaring a CURSOR for each query, bleah). A good implementation would hugely improve the quality and robustness of any such testsuite. Cheers, Jan -- Jan Urbanski GPG key ID: E583D7D2 ouden estin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner wrote: Kevin Grittner wrote: I differentiate tests and test suites. Tests mainly have a run method, while test suites have setUp and tearDown ones. I hadn't caught on to that distinction yet. That should help. uses means that the referenced task has complimentary setUp and tearDown methods, and the dependent task may only run after a successful invocation of the referenced task's setUp method, and the referenced task will wait for completion of all dependent tasks before invoking tearDown. Absolutely correct (may I just copy that para for documentation)? ;-) Use any of my language that you like, but I'm not generally known for my word-smithing ability, so use at your own risk. ;-) Two additional things: tests and test suites may have requirements (in the form of interfaces). The used test suites are passed to the dependent task and it may call the referenced tasks's methods, for example to get the database directory or to run a certain SQL command. Makes sense. Second, if the referenced task fails, any running dependent task is getting aborted as well. That might be obvious, though. I figured that, although it's good to have it confirmed. depends means that the tearDown method of the referenced task doesn't undo the work of its setUp, at least for purposes of the dependent task. The dependent task can only start after successful completion of the referenced class's work (*just* setUp, or all the way to tearDown?), but the referenced task doesn't need to wait for the dependent task. Hm.. no, not quite. The fact that not all suites clean up after them has nothing to do with how they are referenced (uses or depends). So far, it's entirely up to the test suite. I dislike that, but it works. (I've been thinking about some separate resource allocation handling and what not, but..) The only difference between depends and uses is the requirements fulfilling. uses does that, while depends only adds the timing and functional dependencies, but doesn't pass the referenced task as an argument to the dependent task. OK, that accounts for most of the differences between what they sounded like to me and what I saw in the code. That's workable, now that I understand it. onlyAfter means that the dependent task must wait for completion of the referenced task, but doesn't care whether or not the referenced class completed successfully. That's how I think it *should* be. ATM onlyAfter requires successful completion of the dependent task. That accounts for the rest of the differences. I'd like to change that to support onlyAfter, onlyAfterSuccessOf and onlyAfterFailureOf. Plus onlyBefore for convenience. onlyAfterSuccessOf would be the same as depends with an empty tearDown method? So it would effectively be syntactic sugar, for convenience? An onlyBefore reference from a to b would be semantically identical to an onlyAfter reference from b to a? If so, that one seems to me like it would muddy the waters more than it would help. Thank you for thinking through all of this. I'm sure you understand now, why it's not a version 0.1, yet :-) Thank you for putting it together! I was afraid I was going to have to go off-task on serializable implementation to write something so I could test it. I'm more than happy to help stabilize your tool instead! -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner wrote: Kevin Grittner wrote: args=['psql', '-A', '--pset=pager=off', That looks like a correct fix for psql, yes. Other processes might be confused by (or at least act differently with) a PAGER env variable, so that still needs to be cleared in general. I see your point. Even with a general solution, probably best to leave the pset there for psql, though. [discussion of BaseTest vs SyncTest] Both have their own set of caveats, IMO. I'll look those over. Any caveats beyond what you already mentioned of which I should be particularly careful? Maybe we put up a git branch with the dtester patches included? So whenever I want to change the framework, I can check if and how it affects your tests. I strongly encourage you to set that up on git.postgresql.org. If for some reason that's not practicable, I can give you write permissions to my repository there and set up a dtester branch for this. I've barely scratched the surface on git in the last few weeks, and already I'm a big fan. I was never convinced that subversion was an improvement over cvs -- they each had advantages over the other which seemed a wash for me -- but git takes everything to a whole new level. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: I must admit that I haven't ever tested on python 2.6 before. I'll try that (especially as it's the staircase to 3.0, IIUC). I don't use python much, so I can't comment on that. I do see that my system has these two versions on it, with a symlink that makes 2.6 the default. Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2 Python 3.0.1+ (r301:69556, Apr 15 2009, 15:59:22) [GCC 4.3.3] on linux2 I haven't quite gotten it to work yet; I'll start over with 3.0 and see how it goes. I'll also attach the results of the 2.6 attempt. Try a CursesReporter() instead, it gives much nicer output! Thanks, I'll try it. A few other issues in testing so far: (1) I see that a 'make dcheck' does a 'make install'. That's not right. For one thing I usually install in a location where I need to sudo to install; but more importantly, I want to do all checks *before* I install. It's easy enough to work around that for now, but I don't think it's acceptable long-term. (2) After a 'make dcheck' failure, the cluster created for the testing is left running. (3) If the install could check dependencies, report problems, and refuse to install without required packages, that would be less confusing for python novices (like me). Perhaps some of these problems will go away with python 3.0, but I figured I should pass the info along. Thanks again for this. It should help me a lot. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
I wrote: I'll also attach the results of the 2.6 attempt. Let's try that again. -Kevin kgri...@kgrittn-desktop:~/git/postgresql/kgrittn$ make dcheck make -C src/test dcheck make[1]: Entering directory `/home/kgrittn/git/postgresql/kgrittn/src/test' make -C regress dcheck make[2]: Entering directory `/home/kgrittn/git/postgresql/kgrittn/src/test/regress' ./pg_dtester.py --temp-install --top-builddir=../../.. \ --multibyte=SQL_ASCII Postgres dtester suiteCopyright (c) 2004-2010, by Markus Wanner temp_install: creating temporary installation initdb-0: initializing database system 0 pg-0: starting database system 0 testdb: creating database testdb at server 0 conn-0A: connecting to database testdb at server 0 conn-0B: connecting to database testdb at server 0 conn-0C: connecting to database testdb at server 0 test-conn-0A: test started test-conn-0B: test started test-conn-0C: test started startup of test test-conn-0A failed, skipping. Traceback (most recent call last): File /usr/lib/python2.6/dist-packages/twisted/internet/base.py, line 757, in runUntilCurrent call.func(*call.args, **call.kw) File /usr/local/lib/python2.6/dist-packages/dtester/runner.py, line 62, in checkTimeout TimeoutError(TIMEOUT: %s! % self.msg))) File /usr/lib/python2.6/dist-packages/twisted/internet/defer.py, line 269, in errback self._startRunCallbacks(fail) File /usr/lib/python2.6/dist-packages/twisted/internet/defer.py, line 312, in _startRunCallbacks self._runCallbacks() --- exception caught here --- File /usr/lib/python2.6/dist-packages/twisted/internet/defer.py, line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File /usr/local/lib/python2.6/dist-packages/dtester/runner.py, line 186, in cbTestFailed self.reporter.stopTest(tname, test, False, failure) File /usr/local/lib/python2.6/dist-packages/dtester/reporter.py, line 92, in stopTest row = tb.pop() exceptions.IndexError: pop from empty list startup of test test-conn-0B failed, skipping. Traceback (most recent call last): File /usr/lib/python2.6/dist-packages/twisted/internet/base.py, line 757, in runUntilCurrent call.func(*call.args, **call.kw) File /usr/local/lib/python2.6/dist-packages/dtester/runner.py, line 62, in checkTimeout TimeoutError(TIMEOUT: %s! % self.msg))) File /usr/lib/python2.6/dist-packages/twisted/internet/defer.py, line 269, in errback self._startRunCallbacks(fail) File /usr/lib/python2.6/dist-packages/twisted/internet/defer.py, line 312, in _startRunCallbacks self._runCallbacks() --- exception caught here --- File /usr/lib/python2.6/dist-packages/twisted/internet/defer.py, line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File /usr/local/lib/python2.6/dist-packages/dtester/runner.py, line 186, in cbTestFailed self.reporter.stopTest(tname, test, False, failure) File /usr/local/lib/python2.6/dist-packages/dtester/reporter.py, line 92, in stopTest row = tb.pop() exceptions.IndexError: pop from empty list startup of test test-conn-0C failed, skipping. Traceback (most recent call last): File /usr/lib/python2.6/dist-packages/twisted/internet/base.py, line 757, in runUntilCurrent call.func(*call.args, **call.kw) File
Re: [HACKERS] Testing with concurrent sessions
Hi, Quoting Kevin Grittner kevin.gritt...@wicourts.gov: I haven't quite gotten it to work yet; I'll start over with 3.0 and see how it goes. Let's stick to 2.x versions, first... I'll also attach the results of the 2.6 attempt. Thanks, that looks already pretty promising. ;-) A few other issues in testing so far: (1) I see that a 'make dcheck' does a 'make install'. That's not right. For one thing I usually install in a location where I need to sudo to install; but more importantly, I want to do all checks *before* I install. It's easy enough to work around that for now, but I don't think it's acceptable long-term. It does: temp_install: creating temporary installation means it's running make install in the background. (2) After a 'make dcheck' failure, the cluster created for the testing is left running. That counts as a bug. I also get that from time to time (and with Postgres-R testing on 3+ instances, it's even more annoying). Note that the error just before that is, that a psql process it starts cannot connect to its postmaster (startup of test test-conn-0A failed, skipping.) Please check the log (src/test/regress/dtester.log) for why that failed in the first place. Can you connect manually to the database (that's still running after a make dcheck)? (3) If the install could check dependencies, report problems, and refuse to install without required packages, that would be less confusing for python novices (like me). I'm not exactly a distutils hacker... Anybody else got any clue here? Perhaps some of these problems will go away with python 3.0, but I figured I should pass the info along. I'd rather suspect that more of them will arise. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Quoting Kevin Grittner kevin.gritt...@wicourts.gov: I haven't quite gotten it to work yet; I'll start over with 3.0 and see how it goes. Let's stick to 2.x versions, first... OK It does: temp_install: creating temporary installation means it's running make install in the background. OK, sorry for misreading that. (2) After a 'make dcheck' failure, the cluster created for the testing is left running. That counts as a bug. I also get that from time to time (and with Postgres-R testing on 3+ instances, it's even more annoying). Note that the error just before that is, that a psql process it starts cannot connect to its postmaster (startup of test test-conn-0A failed, skipping.) Please check the log (src/test/regress/dtester.log) for why that failed in the first place. Not sure what's relevant there. Entire file tarball attached. Can you connect manually to the database (that's still running after a make dcheck)? Yes I can. Any queries you'd like me to run in there? -Kevin dtester.log.tar.gz Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Second: at the very end of pg_dtester.py, you find the line: reporter = StreamReporter() Try a CursesReporter() instead, it gives much nicer output! When I try to do that, Kate complains (I'm even copying their typo): You are trying to save a python file as non ASCII, without specifiying a correct source encoding line for encoding utf-8 It offers these options: Insert: # -*- coding: utf-8 -*- Save Nevertheless Cancel Should that coding line be in there? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Dead or not, it still works, even against 8.4. I have many programs that use it. It's simply a wrapper around the libpq interface and as long as the libpq interface remains stable (which we go to great pains to do), so will this module. Well, I stand corrected. Good to know. Given the talk of importing some perl module into the postgresql tree it just seemed more logical to me to take something that was close to libpq and had no external dependancies than taking a module with an external dependancy (namely DBI). Yes, I could see that. Actually, I just came across another one by Hiroyuki OYAMA and Aaron Crane. This was last updated January 10, 2010! : http://search.cpan.org/~arc/DBD-PgPP-0.08/ Still requires DBI of course, but no Perl library or compiling required as DBD::Pg does. So we've not got three valid options. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001151129 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktQmI0ACgkQvJuQZxSWSsgNugCgjwkT9QwGpvhcIXCNYhRcTwSW JZcAnjvrsjwpO/QvJ1LzU+cUZ4UqajxV =bu4q -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: Not sure what's relevant there. Entire file tarball attached. Due to reasons mentioned in this thread as well, I've decided to use psql to connect to the database. dtester is parsing its output and checks that against expectations. Hawever, that has its own pitfalls, so in the end I'm almost about to change back to using libpq or implementing the bare minimum protocol (that might have its own merits within the twisted world, if implemented in the required async fashion). Strangely, your log has escape codes in it, which I'm assuming makes the parsing choke. Is that something special to your installation? My psql never colors its outputs... However, the quickest way forward probably is to filter out escape sequences. Turning off tty is not really an option, because dtester doesn't have a chance to capture all necessary events, in that mode. Yes I can. Any queries you'd like me to run in there? It looks like psql can connect, too. It's just the parsing of outputs which fails. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: You are trying to save a python file as non ASCII, without specifiying a correct source encoding line for encoding utf-8 I wasn't aware I had non-ascii characters in there. Inserting an encoding line seems fine. I'll fix that for the upcoming version 0.1. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: I wasn't aware I had non-ascii characters in there. Inserting an encoding line seems fine. I'll fix that for the upcoming version 0.1. Yeah, I couldn't find any, either. I just tried creating a minimal python file in Kate, and it gave me that even though I *know* it was all ASCII characters right off my keyboard. I guess Kate is being overly picky. On the other hand, if it silences an annoying message sometimes, maybe that's reason enough to add it. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Strangely, your log has escape codes in it, which I'm assuming makes the parsing choke. Is that something special to your installation? My psql never colors its outputs... I haven't configured anything like that intentionally. I don't *see* any colors when I use psql. Can you think of anywhere I should check something which might be causing this? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: I haven't configured anything like that intentionally. I don't *see* any colors when I use psql. Can you think of anywhere I should check something which might be causing this? No idea ATM. However, just to make sure that has absolutely nothing to do with the curses reporter I've written: is that dtester.log you just sent the log from a run with the StreamReporter or the CursesReporter? (Should not have any influence for the log, but you never know). Please recheck with the StreamReporter and try to grep the lines starting with [psql0], [psql1] and [psql2]. Dtester simply logs all and any output of all 3rd party processes started. Alternatively, you may want to filter out all lines that start with [postmaster0], that might already reduce what we can consider noise in this case. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Strangely, your log has escape codes in it, which I'm assuming makes the parsing choke. Is that something special to your installation? My pager is less; could that cause it? Could the twisted environment look like one where the pager should kick in? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: My pager is less; could that cause it? Could the twisted environment look like one where the pager should kick in? Yes, that could be it. At least it fails here, too, if I set PAGER=less. Try: PAGER=more make dcheck So, the solution probably lies in adjusting the environment, before starting psql. (Maybe even dropping all existing environment variables for better control of the situation). Will add that for dtester 0.1. (Also note that I plan to move most of what's currently in the patch to the dtester package itself. However, that requires it to be (even more) generic.) Thank you for testing the tester ;-) Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner wrote: Hi, Kevin Grittner wrote: My pager is less; could that cause it? Could the twisted environment look like one where the pager should kick in? Yes, that could be it. At least it fails here, too, if I set PAGER=less. Try: PAGER=more make dcheck Surely for automated use you want the psql pager off altogether. psql --pset pager=off or some such invocation should do it. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: So, the solution probably lies in adjusting the environment, before starting psql. (Maybe even dropping all existing environment variables for better control of the situation). Will add that for dtester 0.1. Based on Andrew's suggestion, I changed line 276 to: args=['psql', '-A', '--pset=pager=off', I now get 5 of 6 tests succeeded (83.3%), processed in 18.5 seconds. I'm not clear on what you want to see from the run or whether it might be better sent off-list. Also, in looking closer at how you have the tests defined, it doesn't look to me like you're carefully interleaving specific sequences of statements on specific connections so much as opening multiple connections and then for each statement saying run this on all connections. That's certainly a valid test to include, but I need the more controlled format, too. It does appear that that's pretty straightforward to code; you just haven't chosen to do so in the particular tests here, correct? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Kevin Grittner kevin.gritt...@wicourts.gov wrote: Also, in looking closer at how you have the tests defined, it doesn't look to me like you're carefully interleaving specific sequences of statements on specific connections so much as opening multiple connections and then for each statement saying run this on all connections. I take it back; you've got both. I do want to expand the tests quite a bit -- do I work them all into this same file, or how would I proceed? I think I'll need about 20 more tests, but I don't want to get in the way of your work on the framework which runs them. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Please recheck with the StreamReporter and try to grep the lines starting with [psql0], [psql1] and [psql2]. Dtester simply logs all and any output of all 3rd party processes started. For me, all psql output seems to be [psql0]; no [psql1] or [psql2]. Bug? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Go try it, read the code and simply ask, if you get stuck. I'll try to come up with some more documentation and such... I'm a little unclear about the differences between uses, depends, and onlyAfter. Here's what they *sound* like they mean, to me; although I don't think the code isn't entirely consistent with this interpretation. uses means that the referenced task has complimentary setUp and tearDown methods, and the dependent task may only run after a successful invocation of the referenced task's setUp method, and the referenced task will wait for completion of all dependent tasks before invoking tearDown. depends means that the tearDown method of the referenced task doesn't undo the work of its setUp, at least for purposes of the dependent task. The dependent task can only start after successful completion of the referenced class's work (*just* setUp, or all the way to tearDown?), but the referenced task doesn't need to wait for the dependent task. onlyAfter means that the dependent task must wait for completion of the referenced task, but doesn't care whether or not the referenced class completed successfully. How close am I? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: I'm a little unclear about the differences between uses, depends, and onlyAfter. Here's what they *sound* like they mean, to me; although I don't think the code isn't entirely consistent with this interpretation. Wow, you are way ahead of me. I intended to write some documentation about that, but... I differentiate tests and test suites. Tests mainly have a run method, while test suites have setUp and tearDown ones. uses means that the referenced task has complimentary setUp and tearDown methods, and the dependent task may only run after a successful invocation of the referenced task's setUp method, and the referenced task will wait for completion of all dependent tasks before invoking tearDown. Absolutely correct (may I just copy that para for documentation)? ;-) Two additional things: tests and test suites may have requirements (in the form of interfaces). The used test suites are passed to the dependent task and it may call the referenced tasks's methods, for example to get the database directory or to run a certain SQL command. Second, if the referenced task fails, any running dependent task is getting aborted as well. That might be obvious, though. depends means that the tearDown method of the referenced task doesn't undo the work of its setUp, at least for purposes of the dependent task. The dependent task can only start after successful completion of the referenced class's work (*just* setUp, or all the way to tearDown?), but the referenced task doesn't need to wait for the dependent task. Hm.. no, not quite. The fact that not all suites clean up after them has nothing to do with how they are referenced (uses or depends). So far, it's entirely up to the test suite. I dislike that, but it works. (I've been thinking about some separate resource allocation handling and what not, but..) The only difference between depends and uses is the requirements fulfilling. uses does that, while depends only adds the timing and functional dependencies, but doesn't pass the referenced task as an argument to the dependent task. onlyAfter means that the dependent task must wait for completion of the referenced task, but doesn't care whether or not the referenced class completed successfully. That's how I think it *should* be. ATM onlyAfter requires successful completion of the dependent task. I'd like to change that to support onlyAfter, onlyAfterSuccessOf and onlyAfterFailureOf. Plus onlyBefore for convenience. This is all work in progress and I'm open to suggestions and requests. Thank you for thinking through all of this. I'm sure you understand now, why it's not a version 0.1, yet :-) Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: Based on Andrew's suggestion, I changed line 276 to: args=['psql', '-A', '--pset=pager=off', That looks like a correct fix for psql, yes. Thanks for pointing that out Andrew. Other processes might be confused by (or at least act differently with) a PAGER env variable, so that still needs to be cleared in general. I now get 5 of 6 tests succeeded (83.3%), processed in 18.5 seconds. That's perfect. The one test that fails is expected to fail (another thing dtester doesn't support, yet). The serialization code you write should finally make that test pass ;-) I do want to expand the tests quite a bit -- do I work them all into this same file, or how would I proceed? I think I'll need about 20 more tests, but I don't want to get in the way of your work on the framework which runs them. Well, first of all, another piece of the missing manual: there are BaseTest and SyncTest classes. Those based on BaseTest runs within the event loop of the twisted framework, thus need to be written in the very same asynchronous fashion. Mostly calling async methods that return a Deferred object, on which you may addCallback() or addErrback(). See the fine twisted documentation, especially the part about Low-Level Networking and Event Loop here: http://twistedmatrix.com/documents/current/core/howto/index.html The SyncTest is based on BaseTest, but a new thread is created to run its run method, passing back its results to the main event loop when done. That allows you to call blocking methods without having to care about blocking the entire event loop. However, it makes interacting between the two models a bit complicated. To call an async function from a SyncTest, you need to call the syncCall method. The separate thread then waits for some callback in the main event loop. Both have their own set of caveats, IMO. I'm not sure about how to organize the tests and ongoing development of the framework. I've already broken the Postgres-R tests with dtester-0.0. Maybe we put up a git branch with the dtester patches included? So whenever I want to change the framework, I can check if and how it affects your tests. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi,
Markus Wanner wrote:
Sorry, if that didn't get clear. I'm trying to put together something I
can release real soon now (tm). I'll keep you informed.
Okay, here we go: dtester version 0.0.
This emerged out of Postgres-R, where I don't just need to test multiple
client connections, but multiple postmasters interacting with each
other. None the less, it may be suitable for other needs as well,
especially testing with concurrent sessions.
I've decided to release this as a separate project named dtester, as
proposed by Michael Tan (thanks for your inspiration).
It's certainly missing lots of things, mainly documentation. However,
I've attached a patch which integrates nicely into the Postgres
Makefiles, so you just need to say: make dcheck.
That very same patch includes a test case with three concurrent
transactions with circular dependencies, where the current SERIALIZABLE
isolation level fails to provide serializability.
Installing dtester itself is as simple as 'python setup.py install' in
the extracted archive's directory.
Go try it, read the code and simply ask, if you get stuck. I'll try to
come up with some more documentation and such...
Regards
Markus Wanner
dtester-0.0.tar.bz2
Description: application/bzip
#
# old_revision [651e6b451b14a926bbca5ac0b308bc5f979c8c8e]
#
# add_file src/test/regress/pg_dtester.py.in
# content [edcf9857ee4d884b6120b72bf319774126241ee3]
#
# patch GNUmakefile.in
# from [da68d137cee6cec6458a1d9e877c5b623a11415a]
#to [b605a66cee8f77978da09c326cf9dbaeba464bd2]
#
# patch src/test/regress/GNUmakefile
# from [ed08fe1be025ede31fa2dba35f81f653809a2096]
#to [55ae74cf991ae07b0d5b082882ff5ba1f1ef4036]
#
--- src/test/regress/pg_dtester.py.in edcf9857ee4d884b6120b72bf319774126241ee3
+++ src/test/regress/pg_dtester.py.in edcf9857ee4d884b6120b72bf319774126241ee3
@@ -0,0 +1,762 @@
+#!/usr/bin/python
+
+#-
+#
+# dtester.py.in
+#
+# Sample test suite running two concurrent transactions, showing
+#off some capabilities of dtester.
+#
+# Copyright (c) 2006-2010, Markus Wanner
+#
+#-
+
+import re, os, sys, getopt
+from twisted.internet import defer, reactor
+
+from dtester.events import EventMatcher, EventSource, Event, \
+ ProcessOutputEvent, ProcessErrorEvent, ProcessEndedEvent
+from dtester.exceptions import TestAborted
+from dtester.test import TestSuite, BaseTest, SyncTest
+from dtester.reporter import StreamReporter
+from dtester.runner import Runner, Timeout
+
+# ** definition of tests and suites ***
+
+class InstallationSuite(TestSuite):
+
+ setUpDescription = creating temporary installation
+ tearDownDescription = removing temporary installation
+
+ needs = (('shell', IShell or something),)
+
+ def setUp(self):
+ # inherit getConfig from the shell
+ setattr(self, 'getConfig', self.shell.getConfig)
+ setattr(self, 'runCommand', self.shell.runCommand)
+ setattr(self, 'recursive_remove', self.shell.recursive_remove)
+
+ # (re) create an installation directory
+ self.pg_inst_dir = self.shell.getConfig('inst_dir')
+ if os.path.exists(self.pg_inst_dir):
+ self.shell.recursive_remove(self.pg_inst_dir)
+ os.mkdir(self.pg_inst_dir)
+
+ # install into that directory
+ proc = self.shell.runCommand('make', 'make',
+ args=['make', '-C', self.shell.getConfig('top-builddir'),
+ 'DESTDIR=%s' % self.pg_inst_dir, 'install',
+ 'with_perl=no', 'with_python=no'],
+ lineBasedOutput=True)
+
+ d = self.waitFor(proc, EventMatcher(ProcessEndedEvent))
+ d.addCallback(self.makeTerminated)
+ proc.start()
+
+ # FIXME: how to properly handle these?
+ self.shell.addEnvPath(self.shell.getConfig('bindir'))
+ self.shell.addEnvLibraryPath(self.shell.getConfig('libdir'))
+ return d
+
+ def makeTerminated(self, event):
+ if event.exitCode != 0:
+ raise Exception(Initdb returned %d % event.exitCode)
+ else:
+ return True
+
+ def tearDown(self):
+ # The installation procedure should be able to simply override any
+ # formerly installed files, so we save the time to clean up the
+ # installation directory.
+ return
+
+
+class InitdbSuite(TestSuite):
+
+ args = (('number', int), )
+ needs = (('shell', IShell or something),)
+
+ def setUpDescription(self):
+ return initializing database system %d % self.number
+
+ def tearDownDescription(self):
+ return removing database system %d % self.number
+
+ def getNumber(self):
+ return self.number
+
+ def getDir(self):
+ return self.dbdir
+
+ def setUp(self):
+ self.dbdir = %s%d % \
+ (self.shell.getConfig('pgdata_prefix'), self.number)
+ proc = self.shell.runCommand(
+'initdb-%d' % self.number,
+'initdb', args = [
+'initdb', '-D', self.dbdir,
+'-A', 'trust', '--noclean'],
+lineBasedOutput=True)
+
+ d = defer.Deferred()
+
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Okay, here we go: dtester version 0.0. C'mon, you could have tried to inspire a *bit* more confidence by calling it version 0.1 or something! ;-) It's certainly missing lots of things, mainly documentation. However, I've attached a patch which integrates nicely into the Postgres Makefiles, so you just need to say: make dcheck. That sounds very cool. That very same patch includes a test case with three concurrent transactions with circular dependencies, where the current SERIALIZABLE isolation level fails to provide serializability. Fantastic! I'll expand that a bit Installing dtester itself is as simple as 'python setup.py install' in the extracted archive's directory. Go try it, read the code and simply ask, if you get stuck. I'll try to come up with some more documentation and such... I'm reading through it all now. Expect feedback soon! And THANK YOU VERY MUCH! -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: C'mon, you could have tried to inspire a *bit* more confidence by calling it version 0.1 or something! ;-) LOL As David used to say: JFDI I found that I just needed to ask for python-twisted. Oh, sorry, yes, requirements: python, twisted. I must admit that I haven't ever tested on python 2.6 before. I'll try that (especially as it's the staircase to 3.0, IIUC). Two more things: the concurrent update test (in the patch part) is complete, while the second one is just a skeleton, ATM. (Just does a concurrent COMMIT without actually doing anything). Second: at the very end of pg_dtester.py, you find the line: reporter = StreamReporter() Try a CursesReporter() instead, it gives much nicer output! Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Mon, Jan 11, 2010 at 04:17:42AM -, Greg Sabino Mullane wrote: Because you'd have to build DBD::Pg against the new libpq, as you do psql. That means you need DBD::Pg sources and the build environment for Perl (headers etc) not just a working Perl runtime. Big difference. Yes, but that is what I was envisioning. As you point out, that's the only sane way to make sure we have a good version of DBD::Pg with which to test. As a side effect, it put libpq through some extra paces as well. :) Is there a reason why you're suggesting using DBI? There is also the Pg perl module which works as well and is one tenth of the size. It also doesn't have external dependancies. It's just a plain wrapper around libpq, which for the purposes of testing may be better. http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm Have a nice day, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Please line up in a tree and maintain the heap invariant while boarding. Thank you for flying nlogn airlines. signature.asc Description: Digital signature
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Is there a reason why you're suggesting using DBI? There is also the Pg perl module which works as well and is one tenth of the size. It also doesn't have external dependancies. It's just a plain wrapper around libpq, which for the purposes of testing may be better. http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm Works as well? Did you take a look at that link? The last update was early 2000, which should give you an indication of just how dead it is. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 20100241 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktLYtwACgkQvJuQZxSWSsjNJgCg1IZBNIZsUMZ2V83/NjgJnrGO 3NEAoK/w0byO45zmni/i9lnhliD4UpkU =ndmb -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Mon, Jan 11, 2010 at 05:42:08PM -, Greg Sabino Mullane wrote: Is there a reason why you're suggesting using DBI? There is also the Pg perl module which works as well and is one tenth of the size. It also doesn't have external dependancies. It's just a plain wrapper around libpq, which for the purposes of testing may be better. http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm Works as well? Did you take a look at that link? The last update was early 2000, which should give you an indication of just how dead it is. Dead or not, it still works, even against 8.4. I have many programs that use it. It's simply a wrapper around the libpq interface and as long as the libpq interface remains stable (which we go to great pains to do), so will this module. Given the talk of importing some perl module into the postgresql tree it just seemed more logical to me to take something that was close to libpq and had no external dependancies than taking a module with an external dependancy (namely DBI). Have a nice day, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Please line up in a tree and maintain the heap invariant while boarding. Thank you for flying nlogn airlines. signature.asc Description: Digital signature
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. Er...why not? That's what psql uses. Because you'd have to build DBD::Pg against the new libpq, as you do psql. That means you need DBD::Pg sources and the build environment for Perl (headers etc) not just a working Perl runtime. Big difference. Yes, but that is what I was envisioning. As you point out, that's the only sane way to make sure we have a good version of DBD::Pg with which to test. As a side effect, it put libpq through some extra paces as well. :) Is bundling a Perl module in the source tree and building it as part of the Pg build a reasonable choice? Personally, I don't think so. *shrug* It's different, but it's the best solution to the problem at hand. It wouldn't be built as part of Pg, only as part of the tests. Additionally, a dedicated testing tool like some folks have been talking about would be really handy for users who want to test their schema. I've had to write my own (in Java, or I'd be offering it) for this purpose, as psql is completely unsuitable for concurrent-run testing and I needed to show that my locking was safe and deadlock-free in some of the more complex stored procs I have. Sure, but it's the difference between waiting for someone to write something (and then dealing with the invevitable bugs, tweaks, and enhancements), or using a solid, known quantity (DBI + Test::More). - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001102316 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktKplIACgkQvJuQZxSWSsj7TQCeMOXWS+uLIZE9QbeBWPxYv/rg HhEAn0QZUzE2/8uyg5Oi+K8qL/oTeDSO =R8Az -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 8/01/2010 1:39 AM, Greg Sabino Mullane wrote: -BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. Er...why not? That's what psql uses. Because you'd have to build DBD::Pg against the new libpq, as you do psql. That means you need DBD::Pg sources and the build environment for Perl (headers etc) not just a working Perl runtime. Big difference. There's no guarantee the user's machine has the same major version of libpq and thus no guarantee that DBD::Pq can be redirected to use your custom libpq by LD_LIBRARY_PATH. It might also override the library search path with rpath linking. Building your own would be pretty much unavoidable unless you're prepared to either require the user to provide a matching version of DBD::Pg or have the tests running with whatever random version happens to be lying around. Using whatever DBD::Pg version happens to be present on the machine would be a bit of a nightmare for reproducibility of test results, and would be really unattractive for use in the standard tests. make check fails on my some-random-distro would become painfully common on the GENERAL list... Is bundling a Perl module in the source tree and building it as part of the Pg build a reasonable choice? Personally, I don't think so. Additionally, a dedicated testing tool like some folks have been talking about would be really handy for users who want to test their schema. I've had to write my own (in Java, or I'd be offering it) for this purpose, as psql is completely unsuitable for concurrent-run testing and I needed to show that my locking was safe and deadlock-free in some of the more complex stored procs I have. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 7/01/2010 9:15 AM, Robert Haas wrote: On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: David E. Wheelerda...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? At this point, I'm personally wondering if it's worth putting together a simple (ish) C program that reads a file describing command interleavings on n connections. It fires up one thread per connection required, then begins queuing commands up for the threads to execute in per-thread fifo queues. The input file may contain synchronization points where two or more explicitly specified threads (or just all threads) must finish all their queued work before they may be given more. Yes, it requires wrangling low-level threading ( pthreads, or the practically equivalent for simple purposes but differently spelled win32 threading ) so it's not going to be beautiful. But it'd permit a declarative form for tests and a single, probably fairly maintainable, test runner. I reluctantly suspect that XML would be a good way to describe the tests - first a block declaring your connections and their conn strings, then a sequence of statements (each of which is associated with a named connection) and synchronization points. Though, come to think of it, a custom plaintext format would be pretty trivial too. CONN conn1: dbname=regress, user=regress CONN conn2: dbname=regress, user=regress STMT conn1: SELECT blah blah; STMT conn2: UPDATE blah blah; SYNC conn1, conn2 etc. Or alternately one-file-per-connection (which would be handy if one connection has *lots* of commands and others only occasional ones) - the only trouble there being how to conveniently specify synchronization points. Anyway: If Java were acceptable I'd put one together now - but somehow I don't think requiring Java would be popular if Perl is an issue ;-) My C/pthreads is more than a little bit rusty (ie: practially nonexistent) and mostly confined to exception-controlled C++ code with RAII for lock management. If C++ is OK, I can write and post a tool for evaluation, but if it must be plain C ... well, I'll avoid scarring you with the sight of what I'd produce. I suspect that a custom tool for this job could actually be fairly simple. A lot simpler than a proper, clean and usable parallel psql, anyway. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that it should do what you would do yourself by hand. And adding DBI as a requirement for running a buildfarm member would be a significant extra barrier to entry, ISTM. (I am very fond of DBI, and use it frequently, BTW) What about something less than a requirement then? If you have it great, you can run these extra tests. If you don't have it, no harm, no foul. We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. All the prerequisites should be in place for 99% of the machines: a C compiler and Perl are the biggies, and I can't see any buildfarm members running without those. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071014 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktF+ucACgkQvJuQZxSWSsjYOgCglyLIyGCr60og+iQSnyRgkowd +lYAnRDjPe/XxC7gb9OBPdpZlqU0wncK =kPIR -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 2010-01-07 11:50 +0200, Craig Ringer wrote: On 7/01/2010 9:15 AM, Robert Haas wrote: Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? At this point, I'm personally wondering if it's worth putting together a simple (ish) C program that reads a file describing command interleavings on n connections. It fires up one thread per connection required, then begins queuing commands up for the threads to execute in per-thread fifo queues. The input file may contain synchronization points where two or more explicitly specified threads (or just all threads) must finish all their queued work before they may be given more. CONN conn1: dbname=regress, user=regress CONN conn2: dbname=regress, user=regress STMT conn1: SELECT blah blah; STMT conn2: UPDATE blah blah; SYNC conn1, conn2 etc. Or alternately one-file-per-connection (which would be handy if one connection has *lots* of commands and others only occasional ones) - the only trouble there being how to conveniently specify synchronization points. I had a similar syntax in mind, but instead of using threads, just execute the file in order using asynchronous connections. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 2010-01-07 18:13 +0200, Marko Tiikkaja wrote: I had a similar syntax in mind, but instead of using threads, just execute the file in order using asynchronous connections. I completely failed to make the point here which was to somehow mark which statements will (or, should) block. So here we go: A= BEGIN; B= BEGIN; A= UPDATE foo ..; B= UPDATE foo ..; -- this will block A= COMMIT; B= SELECT * FROM foo; B= COMMIT; [expected output here] Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 6, 2010, at 6:26 PM, Tom Lane wrote: We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Greg Sabino Mullane g...@turnstep.com writes: We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. As a packager, my reaction to that is over my dead body. We have enough trouble keeping our own software up to date, and pretty much every external component that we've started to bundle has been a disaster from a maintenance standpoint. (Examples: the zic database is constant work and yet almost never up to date; the snowball stemmer never gets updated.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
David E. Wheeler wrote: On Jan 6, 2010, at 6:26 PM, Tom Lane wrote: We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it. Unless I am mistaken, Perl is required in any case to build from CVS, although not from a tarball. DBI/DBD::pg is quite another issue, however. I have been deliberately very conservative about what modules to require for the buildfarm, and we have similarly (and I think wisely) been conservative about what modules to require for Perl programs in the build process. Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. cheers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Thu, Jan 7, 2010 at 11:46 AM, Andrew Dunstan and...@dunslane.net wrote: Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. I completely agree. As between those two options, count me as +1 for the thin veneer. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Wed, Jan 06, 2010 at 08:40:28PM -0500, Andrew Dunstan wrote: A parallel psql seems to me a better way to go. We talked about that a while ago, but I don't recall what happened to it. Greg Stark had a patch a couple of years ago. Dunno what happened to it since then. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 6, 2010, at 6:31 PM, Kevin Grittner wrote:
As far as I've been able to determine so far, to call psql in a
relatively portable way would require something like this:
http://perldoc.perl.org/perlfork.html
Here's an example using IPC::Open3:
#!/usr/local/bin/perl -w
use strict;
use warnings;
use IPC::Open3;
use Symbol 'gensym';
use constant EOC = __DONE__\n;
my ($in1, $out1, $err1) = (gensym, gensym, gensym);
my ($in2, $out2, $err2) = (gensym, gensym, gensym);
my $pid1 = open3 $in1, $out1, $err1, 'bash';
my $pid2 = open3 $in2, $out2, $err2, 'bash';
print $in1 cd ~/dev/postgresql\n;
print $in1 ls doc\n;
print $in1 echo , EOC;
while ((my $line = $out1)) {
last if $line eq EOC;
print LS: $line;
}
print Finished file listing\n\n;
print $in2 cd ~/dev/postgresql\n;
print $in2 head -4 README\n;
print $in2 echo , EOC;
while (defined( my $line = $out2 )) {
last if $line eq EOC;
print HEAD: $line;
}
print Finished reading README\n\n;
print $in1 exit\n;
print $in2 exit\n;
waitpid $pid2, 0;
print All done!\n;
With that, I get:
LS: KNOWN_BUGS
LS: MISSING_FEATURES
LS: Makefile
LS: README.mb.big5
LS: README.mb.jp
LS: TODO
LS: bug.template
LS: src
Finished file listing
HEAD: PostgreSQL Database Management System
HEAD: =
HEAD:
HEAD: This directory contains the source code distribution of the
PostgreSQL
Finished reading README
All done!
I could easily write a very simple module to abstract all that stuff for you,
then you could just do something like:
my $psql1 = Shell::Pipe-new(qw(psql -U postgres));
my $psql2 = Shell::Pipe-new(qw(psql -U postgres));
$psql1-print('SELECT * from pg_class');
while (my $line = $psql1-readln) { print Output: $line\n }
$psql1-close;
All I'd need is some more reliable way than echo DONE__\n to be able to tell
when a particular command has finished outputting.
Thoughts?
Best,
David
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Andrew Dunstan and...@dunslane.net writes: Unless I am mistaken, Perl is required in any case to build from CVS, although not from a tarball. Right, but to my mind building from a tarball needs to include the ability to run the regression tests on what you built. So injecting Perl into that is moving the goalposts on build requirements. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 7, 2010, at 9:08 AM, Tom Lane wrote: Right, but to my mind building from a tarball needs to include the ability to run the regression tests on what you built. So injecting Perl into that is moving the goalposts on build requirements. In that case, there's nothing for it except concurrent psql. Or else some sort of shell environment that's available on all platforms. do we require bash on Windows? Oh, wait, the Windows build requires Perl… Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
David E. Wheeler da...@kineticode.com writes: On Jan 7, 2010, at 9:08 AM, Tom Lane wrote: Right, but to my mind building from a tarball needs to include the ability to run the regression tests on what you built. So injecting Perl into that is moving the goalposts on build requirements. In that case, there's nothing for it except concurrent psql. Unless we are prepared to define concurrency testing as something separate from the basic regression tests. Which is kind of annoying but perhaps less so than the alternatives. It certainly seems to me to be the kind of thing you wouldn't need to test in order to have confidence in a local build. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 7, 2010, at 9:19 AM, Tom Lane wrote: In that case, there's nothing for it except concurrent psql. Unless we are prepared to define concurrency testing as something separate from the basic regression tests. Which is kind of annoying but perhaps less so than the alternatives. It certainly seems to me to be the kind of thing you wouldn't need to test in order to have confidence in a local build. I was rather assuming that was what we were talking about here, since we have in the past discussed testing things like dump and restore, which would require something like Perl to handle multiple processes, and wouldn't work very well for a regular release. I think if we have the ability to add tests that are not distributed, it gives us a lot more freedom and opportunity to test things that are not currently well-tested. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Unless we are prepared to define concurrency testing as something separate from the basic regression tests. Which is kind of annoying but perhaps less so than the alternatives. It certainly seems to me to be the kind of thing you wouldn't need to test in order to have confidence in a local build. I thought we were leaning towards something separate. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071234 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGGxYACgkQvJuQZxSWSsgG0gCfZ4eTpXd/97p3zSJpLqGhKMh6 8nMAoJ2lQUaCWNVeSPDU8fq7VnkO0s4C =xBOo -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. Er...why not? That's what psql uses. As for those advocating using a custom C program written using libpq - that's basically what DBI/DBD::Pg ends up being! Only with a shiny Perl outside and years of real-world testing and usage. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. I'm still *very* interested in making a libpq-less pure perl driver, if anyone feels like funding it, let me know! :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071236 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGHF0ACgkQvJuQZxSWSsiFmACfUJbRDUJGvDTJNjgj/dyQKVCA tZwAn2fiXKNWbWzYXobrHZjeE8aSSiVv =sGzK -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. As a packager, my reaction to that is over my dead body. We have enough trouble keeping our own software up to date, and pretty much every external component that we've started to bundle has been a disaster from a maintenance standpoint. (Examples: the zic database is constant work and yet almost never up to date; the snowball stemmer never gets updated.) As a counterargument, I'll point out that this won't be as critical as zic, especially if we're talking about an additional/optional set of tests. Also, Tim Bunce and I are right here, so the maintenance should not be that bad (and I'd hazard that a lot more people in the community know Perl/DBI than zic or stemmers). - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071315 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGJNQACgkQvJuQZxSWSshoHwCg9urTTT19m55soiIjuYKKuB5L PjYAoJbDAe7j4xsxsSFfVEkYDFpTjKE9 =48oW -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Alvaro Herrera alvhe...@commandprompt.com wrote: Kevin Grittner escribió: Are we anywhere close to an agreement on what the multi-session psql implementation would look like? See http://archives.postgresql.org/message-id/8204.1207689...@sss.pgh.pa.us and followups. Thanks, I had missed or forgotten this thread. It seems like it drifted more-or-less to a consensus. Does everyone agree that this thread represents the will of the community? I see the thread started at the point where Greg Stark had a patch, but by the end it was a start over, stealing code as appropriate situation. Has anything happened with that? Any plans? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote: I'm still *very* interested in making a libpq-less pure perl driver, if anyone feels like funding it, let me know! :) You mean this one: http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm ? Cheers, M -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
A.M. wrote: On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote: I'm still *very* interested in making a libpq-less pure perl driver, if anyone feels like funding it, let me know! :) You mean this one: http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm ? It has a list of limitations as long as your arm. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
David E. Wheeler da...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote: David E. Wheeler da...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 6, 2010, at 1:52 PM, Kevin Grittner wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Probably the simplest way is to use the core IPC::Open3 module: http://search.cpan.org/perldoc?IPC::Open3 IPC::Run might be easier to use if it's available, but it's not in Perl core, so YMMV. Really it's up to andrew what modules he requires test servers to have. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote: Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script If you're more comfortable with shell, then yes. Although then it won't run on Windows, will it? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 2010-01-07 00:08 +0200, Peter Eisentraut wrote: On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote: David E. Wheelerda...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script. I don't see how that would work, but I might have misunderstood what we're reaching for here. What I think would be most useful would be to interleave statements between transactions, not just randomly fire psql sessions and hope for race conditions. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: On 2010-01-07 00:08 +0200, Peter Eisentraut wrote: Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script. I don't see how that would work, but I might have misunderstood what we're reaching for here. What I think would be most useful would be to interleave statements between transactions, not just randomly fire psql sessions and hope for race conditions. Yeah, I want to test specific interleavings of statements on concurrent connections. There may *also* be some tests which throw a lot at the server concurrently in a more random fashion, but it is important to be able to have some very controlled tests where we don't count on randomly creating the desired conflicts. It would be valuable to be able to include some of these tests with controlled and predicatable statement interleavings in the make check tests. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Marko Tiikkaja wrote: On 2010-01-07 00:08 +0200, Peter Eisentraut wrote: Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script. I don't see how that would work, but I might have misunderstood what we're reaching for here. What I think would be most useful would be to interleave statements between transactions, not just randomly fire psql sessions and hope for race conditions. Open a few psql with -f pointing to a pipe, and from the shell write into the pipe? I don't think it's straightforward, but it should be possible. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Alvaro Herrera alvhe...@commandprompt.com wrote: Open a few psql with -f pointing to a pipe, and from the shell write into the pipe? I don't think it's straightforward, but it should be possible. I'll play with it and see what I can do. Thanks, -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: David E. Wheeler da...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Robert Haas wrote: On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: David E. Wheeler da...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that it should do what you would do yourself by hand. And adding DBI as a requirement for running a buildfarm member would be a significant extra barrier to entry, ISTM. (I am very fond of DBI, and use it frequently, BTW) I'm persuadable on most things, but this one would take a bit of doing. A parallel psql seems to me a better way to go. We talked about that a while ago, but I don't recall what happened to it. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Wed, Jan 6, 2010 at 8:40 PM, Andrew Dunstan and...@dunslane.net wrote: Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that it should do what you would do yourself by hand. It just seems crazy to me to try to test anything without proper language bindings. Opening a psql session and parsing the results seems extraordinarily painful. I wonder if it would make sense write a small wrapper program that uses libpq and dumps out the results in a format that is easy for Perl to parse. Another idea would be to make a set of Perl libpq bindings that is simpler than DBD::Pg and don't go through DBI. If we put those in the main source tree (perhaps as a contrib module) they would be available wherever we need them. A parallel psql seems to me a better way to go. We talked about that a while ago, but I don't recall what happened to it. That seems like a dead-end to me. It's hard for me to imagine it's ever going to be more than a toy. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Robert Haas robertmh...@gmail.com writes: It just seems crazy to me to try to test anything without proper language bindings. Opening a psql session and parsing the results seems extraordinarily painful. I wonder if it would make sense write a small wrapper program that uses libpq and dumps out the results in a format that is easy for Perl to parse. Another idea would be to make a set of Perl libpq bindings that is simpler than DBD::Pg and don't go through DBI. If we put those in the main source tree (perhaps as a contrib module) they would be available wherever we need them. We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. A parallel psql seems to me a better way to go. We talked about that a while ago, but I don't recall what happened to it. That seems like a dead-end to me. It's hard for me to imagine it's ever going to be more than a toy. Well, the argument there is that it might be useful for actual use, not only testing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Tom Lane wrote: We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. Well, if that's the consensus, I have to choose between trying to implement multi-session psql and using testing which can't carry over to long-term regular use. Are we anywhere close to an agreement on what the multi-session psql implementation would look like? (If not I can put something forward.) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Wed, Jan 6, 2010 at 9:26 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: It just seems crazy to me to try to test anything without proper language bindings. Opening a psql session and parsing the results seems extraordinarily painful. I wonder if it would make sense write a small wrapper program that uses libpq and dumps out the results in a format that is easy for Perl to parse. Another idea would be to make a set of Perl libpq bindings that is simpler than DBD::Pg and don't go through DBI. If we put those in the main source tree (perhaps as a contrib module) they would be available wherever we need them. We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. I suppose that depends on the context. I'm not exactly sure what Kevin's goal is here. For basic regression tests, yeah, we'd probably like to keep that Perl-free. For more complex testing, I think using Perl makes sense. Or to put the shoe on the other foot, if we DON'T allow the use of Perl for more complex testing, then we're probably not going to have any more complex tests. If we use a hypothetical concurrent psql implementation to run the tests, how will we analyze the results? It's no secret that the current regression tests are fairly limited, in part because the only thing we can do with them is diff the output against one or two known good results. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Robert Haas wrote: I'm not exactly sure what Kevin's goal is here. I think it would be insane to try to do something like serializable isolation mode without having regression tests. I would want more tests than could reasonably go into the 'make check' suite to support development, but it would be very good to have some go in there. For basic regression tests, yeah, we'd probably like to keep that Perl-free. OK. Is parallel psql the only reasonable option? For more complex testing, I think using Perl makes sense. Or to put the shoe on the other foot, if we DON'T allow the use of Perl for more complex testing, then we're probably not going to have any more complex tests. Do you envision some test suite committed to CVS beyond the 'make check' tests, for on demand testing at a more rigorous level? Am I missing something that's already there? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Kevin Grittner escribió: Well, if that's the consensus, I have to choose between trying to implement multi-session psql and using testing which can't carry over to long-term regular use. Are we anywhere close to an agreement on what the multi-session psql implementation would look like? (If not I can put something forward.) See http://archives.postgresql.org/message-id/8204.1207689...@sss.pgh.pa.us and followups. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Wed, Jan 6, 2010 at 10:00 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: For basic regression tests, yeah, we'd probably like to keep that Perl-free. OK. Is parallel psql the only reasonable option? It seems so, assuming you're willing to concede that it is a reasonable option in the first place. For more complex testing, I think using Perl makes sense. Or to put the shoe on the other foot, if we DON'T allow the use of Perl for more complex testing, then we're probably not going to have any more complex tests. Do you envision some test suite committed to CVS beyond the 'make check' tests, for on demand testing at a more rigorous level? Am I missing something that's already there? Personally, I tend to think that to test this well you are going to need a test suite written in a scripting language. Whether or not that gets committed to CVS is a political question, but I would be in favor of it (assuming it's good, of course). Maybe you will find that you can do it all with concurrent psql, but (1) I'm not convinced and (2) if that's your plan, does that mean you're going to do nothing until someone implements concurrent psql? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Robert Haas wrote: It just seems crazy to me to try to test anything without proper language bindings. Opening a psql session and parsing the results seems extraordinarily painful. I've written a Python based program that spawns a captive psql and talks to it--twice for different people--that ultimately uses the same sort of open3() spawning David mentioned is available via IPC::Open3. You can throw together a prototype that works well enough for some purposes in a couple of hours. I don't know that it would ever reach really robust though. The main problem with that whole approach is that you have to be extremely careful in how you deal with the situation where the captive program is spewing an unknown amount of information back at you. How do you know when it's done? Easy for the child and its master to deadlock if you're not careful. In the past I worked around that issue by just waiting for the process to end and then returning everything it had written until that time. I don't know that this would be flexible enough for what's needed for concurrent testing, where people are probably going to want more of a send a command, get some lines back again approach that keeps the session open. If I thought a captive psql would work well in this context I'd have written a prototype already. I'm not sure if it's actually possible to do this well enough to meet expectations. Parsing psql output is completely viable for trivial purposes though, and if the requirements were constrained enough it might work well enough for simple concurrent testing. While both concurrent psql and the libpq shim you suggested would take more work, I feel a bit more confident those would result in something that really worked as expected on every platform when finished. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On ons, 2010-01-06 at 14:10 -0800, David E. Wheeler wrote: On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote: Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script If you're more comfortable with shell, then yes. Although then it won't run on Windows, will it? Well, I'm not saying that this is necessarily the better alternative. But you might have to compromise somewhere. Otherwise this project will take two years to complete and will be an unmaintainable mess (see pg_regress). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On ons, 2010-01-06 at 20:49 -0500, Robert Haas wrote: Another idea would be to make a set of Perl libpq bindings that is simpler than DBD::Pg and don't go through DBI. If we put those in the main source tree (perhaps as a contrib module) they would be available wherever we need them. http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/perl5/?hideattic=0 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: It's very soon going to be critical that I be able to test particular interleavings of statements in particular concurrent transaction sets to be able to make meaningful progress on the serializable transaction work. I've something in place for Postgres-R, as I also need to test concurrent transactions there. It's based on python/twisted and is able to start multiple Postgres instances (as required for testing replication) and query them concurrently (as you seem to need as well). It uses an asynchronous event loop (from twisted) and basically controls processes, issues queries and checks results and ordering constraints (e.g. transaction X must commit and return a result before transaction Y). I'm still under the impression that this testing framework needs cleanup. However, others already showed interest as well... Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Markus Wanner mar...@bluegap.ch wrote: Kevin Grittner wrote: It's very soon going to be critical that I be able to test particular interleavings of statements in particular concurrent transaction sets to be able to make meaningful progress on the serializable transaction work. I've something in place for Postgres-R, as I also need to test concurrent transactions there. It's based on python/twisted and is able to start multiple Postgres instances (as required for testing replication) and query them concurrently (as you seem to need as well). It uses an asynchronous event loop (from twisted) and basically controls processes, issues queries and checks results and ordering constraints (e.g. transaction X must commit and return a result before transaction Y). Where would I find this (and any related documentation)? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Hi, Kevin Grittner wrote: Where would I find this (and any related documentation)? Sorry, if that didn't get clear. I'm trying to put together something I can release real soon now (tm). I'll keep you informed. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
David E. Wheeler da...@kineticode.com wrote: I think the consensus was, failing support for concurrent sessions in psql, to use a Perl script to control multiple psql sessions and perhaps use Test::More to do the testing. Are there any examples of that? While I can hack my way through regular expressions when I need them, perl as a language is something I don't know at all; with an example I might be able to come up to speed quickly, though. Although pgTAP might make sense, too, if the tests ought to run in the database. I need to run statements against a database; I don't particularly need any special features of psql for this. Can anyone confirm that pgTAP can let you interleave specific statements against specific connections in a specific sequence? (The answer to that didn't leap out at me in a quick scan of the docs.) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On mån, 2010-01-04 at 17:10 -0600, Kevin Grittner wrote: David E. Wheeler da...@kineticode.com wrote: I think the consensus was, failing support for concurrent sessions in psql, to use a Perl script to control multiple psql sessions and perhaps use Test::More to do the testing. Are there any examples of that? While I can hack my way through regular expressions when I need them, perl as a language is something I don't know at all; with an example I might be able to come up to speed quickly, though. If you're not deep into Perl, perhaps ignore the Test::More comment for the moment and just use DBI to connect to several database sessions, execute your queries and check if the results are what you want. Once you have gotten somewhere with that, wrapping a test harness around it is something others will be able to help with. Although pgTAP might make sense, too, if the tests ought to run in the database. I need to run statements against a database; I don't particularly need any special features of psql for this. Can anyone confirm that pgTAP can let you interleave specific statements against specific connections in a specific sequence? (The answer to that didn't leap out at me in a quick scan of the docs.) pgTAP isn't really going to help you here, as it runs with *one* database session, and its main functionality is to format the result of SQL functions into TAP output, which is not very much like what you ought to be doing. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 4, 2010, at 3:29 PM, Peter Eisentraut wrote: If you're not deep into Perl, perhaps ignore the Test::More comment for the moment and just use DBI to connect to several database sessions, execute your queries and check if the results are what you want. Once you have gotten somewhere with that, wrapping a test harness around it is something others will be able to help with. Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). pgTAP isn't really going to help you here, as it runs with *one* database session, and its main functionality is to format the result of SQL functions into TAP output, which is not very much like what you ought to be doing. Right, exactly. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 1, 2010, at 6:01 PM, Kevin Grittner wrote: It's very soon going to be critical that I be able to test particular interleavings of statements in particular concurrent transaction sets to be able to make meaningful progress on the serializable transaction work. It would be wonderful if some of these scripts could be integrated into the PostgreSQL 'make check' scripts, although that's not an absolute requirement. I'm not really concerned about performance tests for a while, just testing the behavior of particular interleavings of statements in multiple sessions. If psql isn't expected to support that soon, any suggestions? Is pgTAP suited to this? We've discussed it a bit in the past with regard to testing replication and such. I think the consensus was, failing support for concurrent sessions in psql, to use a Perl script to control multiple psql sessions and perhaps use Test::More to do the testing. Although pgTAP might make sense, too, if the tests ought to run in the database. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Testing with concurrent sessions
There has been periodic discussion here about allowing psql to deal with multiple sessions, or possibly creating another tool to allow this sort of test. Is anyone working on this? It's very soon going to be critical that I be able to test particular interleavings of statements in particular concurrent transaction sets to be able to make meaningful progress on the serializable transaction work. It would be wonderful if some of these scripts could be integrated into the PostgreSQL 'make check' scripts, although that's not an absolute requirement. I'm not really concerned about performance tests for a while, just testing the behavior of particular interleavings of statements in multiple sessions. If psql isn't expected to support that soon, any suggestions? Is pgTAP suited to this? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
