Re: [HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier
I was already starting to consider making the chroot logic optional, based on the resistence expressed by folks on this thread. How about the following: During the configure phase, it checks for chroot and setuid and friends that it will need. The regression suite has config parameters to specify where the chroot directories are to live, defaulting to something sensible. We have two almost identical make targets, called something like 'replicationcheck' and 'sudofullreplicationcheck', and only do the chroot stuff if uid=0, the directories exist, and the bind mounts exist, and the make target was the 'sudofullreplicationcheck'. The tablespace tests would have to be optional, only running in the full test and not the non-full test, and that makes some complications with having two different expectations (in the sense of the results/ vs. expected/ directories). I'm inclined to change the name of the tests from 'replicationtest' or 'replicationcheck' to something broader like 'clustercheck', owing to the expectation that more than replication could be tested in this framework. The "sudofull" prefix is just a placefiller -- I don't like that naming convention. Not sure about the name to use. mark On Monday, January 6, 2014 10:17 AM, David Johnston wrote: Andres Freund-3 wrote > On 2014-01-06 09:12:03 -0800, Mark Dilger wrote: >> The reason I was going to all the trouble of creating >> chrooted environments was to be able to replicate >> clusters that have tablespaces. Not doing so makes >> the test code simpler at the expense of reducing >> test coverage. > >> I am using the same binaries. The chroot directories >> are not "chroot jails". I'm intentionally bind mounting >> out to all the other directories on the system, except >> the other clusters' data directories and tablespace >> directories. The purpose of the chroot is to make the >> paths the same on all clusters without the clusters >> clobbering each other. > > I don't think the benefit of being able to test tablespaces without > restarts comes even close to offsetting the cost of requiring sudo > permissions and introducing OS dependencies. E.g. there's pretty much no > hope of making this work sensibly on windows. > > So I'd just leave out that part. Only skimming this thread but even if only a handful of buildfarm animals can run this extended test bundle because of the restrictive requirements it is likely better than discarding them altogether. The main thing in this case is to segregate out this routine so that it has to be invoked explicitly and ideally in a "ignore if pre-reqs are missing" manner. Increasing the likelihood and frequency of test runs in what is a fairly popular platform and that covers non-OS specific code as well has benefits. As long at it doesn't poison anything else I don't see that much harm coming of it. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Re-In-core-regression-tests-for-replication-cascading-archiving-PITR-etc-Michael-Paquier-tp5785400p578.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier
Andres Freund-3 wrote > On 2014-01-06 09:12:03 -0800, Mark Dilger wrote: >> The reason I was going to all the trouble of creating >> chrooted environments was to be able to replicate >> clusters that have tablespaces. Not doing so makes >> the test code simpler at the expense of reducing >> test coverage. > >> I am using the same binaries. The chroot directories >> are not "chroot jails". I'm intentionally bind mounting >> out to all the other directories on the system, except >> the other clusters' data directories and tablespace >> directories. The purpose of the chroot is to make the >> paths the same on all clusters without the clusters >> clobbering each other. > > I don't think the benefit of being able to test tablespaces without > restarts comes even close to offsetting the cost of requiring sudo > permissions and introducing OS dependencies. E.g. there's pretty much no > hope of making this work sensibly on windows. > > So I'd just leave out that part. Only skimming this thread but even if only a handful of buildfarm animals can run this extended test bundle because of the restrictive requirements it is likely better than discarding them altogether. The main thing in this case is to segregate out this routine so that it has to be invoked explicitly and ideally in a "ignore if pre-reqs are missing" manner. Increasing the likelihood and frequency of test runs in what is a fairly popular platform and that covers non-OS specific code as well has benefits. As long at it doesn't poison anything else I don't see that much harm coming of it. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Re-In-core-regression-tests-for-replication-cascading-archiving-PITR-etc-Michael-Paquier-tp5785400p578.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier
On 2014-01-06 01:25:57 +, Greg Stark wrote: > -- > greg > On 5 Jan 2014 14:54, "Mark Dilger" wrote: > > > > I am building a regression test system for replication and came across > > this email thread. I have gotten pretty far into my implementation, but > > would be happy to make modifications if folks have improvements to > > suggest. If the community likes my design, or a modified version based > > on your feedback, I'd be happy to submit a patch. > > This sounds pretty cool. The real trick will be in testing concurrent > behaviour -- I.e. queries on the slave when it's replaying logs at a > certain point. But right now we have nothing so anything would be an > improvement. Abhijit Menon-Sen (CCed) has prototyped an isolationtester version that can connect to multiple nodes. Once we've got automated setup of multiple nodes, pursuing that makes sense again. > > This is possible all on one system because the database clusters > > are chroot'ed to see their own /data directory and not the /data directory > > of the other chroot'ed clusters, although the rest of the system, like > /bin > > and /etc and /dev are all bind mounted and visible to each cluster. > > This isn't necessary. You can use the same binaries and run initdb with a > different location just fine. Then start up the database with -D to specify > the directory. Very emphathically seconded. It should absolutely not be necessary to use different chroots. Pretty much the only case that will require that is tablespaces unless you do some pretty ugly hackery... In almost all scenarios you'll have to change either unix_socket_directory or port (recommended) in addition to the datadir - but that's not a problem. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier
-- greg On 5 Jan 2014 14:54, "Mark Dilger" wrote: > > I am building a regression test system for replication and came across > this email thread. I have gotten pretty far into my implementation, but > would be happy to make modifications if folks have improvements to > suggest. If the community likes my design, or a modified version based > on your feedback, I'd be happy to submit a patch. This sounds pretty cool. The real trick will be in testing concurrent behaviour -- I.e. queries on the slave when it's replaying logs at a certain point. But right now we have nothing so anything would be an improvement. > This is possible all on one system because the database clusters > are chroot'ed to see their own /data directory and not the /data directory > of the other chroot'ed clusters, although the rest of the system, like /bin > and /etc and /dev are all bind mounted and visible to each cluster. This isn't necessary. You can use the same binaries and run initdb with a different location just fine. Then start up the database with -D to specify the directory.
