Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Mikael Kjellström]

On 2017-05-01 21:19, Mikael Kjellström wrote:


Thanks.  Will try it out.


Just wanted to report that I've tried it and it works as expected.

Thanks for the really fast fixes.

/Mikael


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Mikael Kjellström]

On 2017-05-01 21:10, Andrew Dunstan wrote:

Not sure I understand what "rerun a branch" from scratch means. If you
zap the branch directory you lose all its state. That's generally a bad
thing.


I mean like the first time when you set up the buildfarm client / 
branch.  And it's not something I recomend doing all the time but some 
times when the buildfarm client doesn't clean up after it self and there 
are a lot of crap left it's "easier" to just zap everything and start 
over to get it running again.




Anyway, this patch should fix it for all uses. It creates the directory
if it doesn't exist.

diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
index 91c1362..175eaa7 100644
--- a/PGBuild/Utils.pm
+++ b/PGBuild/Utils.pm
@@ -14,6 +14,8 @@ See accompanying License file for license details
 use strict;
 use warnings;

+use File::Path;
+
 use Exporter   ();
 our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);

@@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
 sub run_log
 {
 my $command = shift;
-my $file=
-
"$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
+   my $filedir =
"$main::branch_root/$main::st_prefix$main::logdirname";
+   mkpath($filedir);
+   my $file= "$filedir/lastcomand.log";
 unlink $file;
 system("$command > $file 2>&1");
 my @loglines;


Ok.

Thanks.  Will try it out.

/Mikael


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Andrew Dunstan]

On 05/01/2017 03:00 PM, Mikael Kjellström wrote:
>
> On 2017-05-01 20:56, Andrew Dunstan wrote:
>> OK, coming up with a more comprehensive fix.
>
> Ok.
>
>> The obvious workaround for now is to create the directory and dont zap
>> it or its parents. You should only have to do it once (per branch)
>
> Yes, I know.  That is what I have been doing so far.  But if you want
> to rerun a branch from scratch it's much easier to just do:
>
> rm -rf buildroot/HEAD
> or
> rm -rf buildroot/REL9_6_STABLE
>
> and then the buildfarm script should do the right thing and create all
> the directories that it needs.
>


Not sure I understand what "rerun a branch" from scratch means. If you
zap the branch directory you lose all its state. That's generally a bad
thing.

Anyway, this patch should fix it for all uses. It creates the directory
if it doesn't exist.

diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
index 91c1362..175eaa7 100644
--- a/PGBuild/Utils.pm
+++ b/PGBuild/Utils.pm
@@ -14,6 +14,8 @@ See accompanying License file for license details
 use strict;
 use warnings;
 
+use File::Path;
+
 use Exporter   ();
 our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
 
@@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
 sub run_log
 {
 my $command = shift;
-my $file=
- 
"$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
+   my $filedir =
"$main::branch_root/$main::st_prefix$main::logdirname";
+   mkpath($filedir);
+   my $file= "$filedir/lastcomand.log";
 unlink $file;
 system("$command > $file 2>&1");
 my @loglines;


cheers

andrew


-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Mikael Kjellström]

On 2017-05-01 20:56, Andrew Dunstan wrote:

OK, coming up with a more comprehensive fix.


Ok.


The obvious workaround for now is to create the directory and dont zap
it or its parents. You should only have to do it once (per branch)


Yes, I know.  That is what I have been doing so far.  But if you want to 
rerun a branch from scratch it's much easier to just do:


rm -rf buildroot/HEAD
or
rm -rf buildroot/REL9_6_STABLE

and then the buildfarm script should do the right thing and create all 
the directories that it needs.


/Mikael


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Andrew Dunstan]

On 05/01/2017 02:46 PM, Mikael Kjellström wrote:
>
>
> On 2017-05-01 20:44, Mikael Kjellström wrote:
>> Nope, that didn't do it.
>
> Or well.  It fixed the check_make bug but not the other bug with that
> the loach.lastrun-logs-directory isn't created before trying to write
> to it.
>


OK, coming up with a more comprehensive fix.

The obvious workaround for now is to create the directory and dont zap
it or its parents. You should only have to do it once (per branch)

cheers

andrew

-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Mikael Kjellström]

On 2017-05-01 20:44, Mikael Kjellström wrote:

Nope, that didn't do it.


Or well.  It fixed the check_make bug but not the other bug with that 
the loach.lastrun-logs-directory isn't created before trying to write to it.


/Mikael


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Mikael Kjellström]

On 2017-05-01 20:25, Andrew Dunstan wrote:

OK, that's a bug. Mea culpa.

the quick fix is this patch:


diff --git a/run_build.pl b/run_build.pl
index aeb8966..822b4de 100755
--- a/run_build.pl
+++ b/run_build.pl
@@ -1008,7 +1008,8 @@ sub writelog

 sub check_make
 {
-my @out = run_log("$make -v");
+   # don't use run_log here - it's too early in the process
+my @out = `$make -v 2>&1`;
 return undef unless ($? == 0 && grep {/GNU Make/} @out);
 return 'OK';
 }


Nope, that didn't do it.

If I zap REL9_6_STABLE and then run:

./run_build.pl REL9_6_STABLE

I get the following output:

cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory
cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory
cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory

Buildfarm member loach failed on REL9_6_STABLE stage pgsql-Git

/Mikael



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Andrew Dunstan]

On 05/01/2017 02:13 PM, Mikael Kjellström wrote:
> On 2017-04-19 15:59, Andrew Dunstan wrote:
>
>
>> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
>> be downloaded from
>> 
>>
>
> I don't know if it's only me or if others have noticed this also but I
> have the buildfarm client set up like this:
>
> /home/pgbf
> /home/pgbf/build-farm-4
> /home/pgbf/buildfarm@ -> build-farm-4.19/
> /home/pgbf/buildroot
>
> and then I have a cron job that does this:
>
> x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE
>
> and when it tries to run I get the following error:
>
> cannot create
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
> No such file or directory
> gmake is not GNU Make - please fix config file at ./run_build.pl line
> 343.
>
> if I manually create the directory loach.lastrun-logs the script then
> continues normally.
>
> I am no perl guru so I can't figure out really what is going on here. 
> I think that the directory
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be
> created before the PGBuild/Utils.pm sub run_log tries to write to it.
>
>


OK, that's a bug. Mea culpa.

the quick fix is this patch:


diff --git a/run_build.pl b/run_build.pl
index aeb8966..822b4de 100755
--- a/run_build.pl
+++ b/run_build.pl
@@ -1008,7 +1008,8 @@ sub writelog
 
 sub check_make
 {
-my @out = run_log("$make -v");
+   # don't use run_log here - it's too early in the process
+my @out = `$make -v 2>&1`;
 return undef unless ($? == 0 && grep {/GNU Make/} @out);
 return 'OK';
 }



cheers

andrew




-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Andrew Dunstan]

On 05/01/2017 02:13 PM, Mikael Kjellström wrote:
> On 2017-04-19 15:59, Andrew Dunstan wrote:
>
>
>> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
>> be downloaded from
>> 
>>
>
> I don't know if it's only me or if others have noticed this also but I
> have the buildfarm client set up like this:
>
> /home/pgbf
> /home/pgbf/build-farm-4
> /home/pgbf/buildfarm@ -> build-farm-4.19/
> /home/pgbf/buildroot
>
> and then I have a cron job that does this:
>
> x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE
>
> and when it tries to run I get the following error:
>
> cannot create
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
> No such file or directory
> gmake is not GNU Make - please fix config file at ./run_build.pl line
> 343.
>
> if I manually create the directory loach.lastrun-logs the script then
> continues normally.
>
> I am no perl guru so I can't figure out really what is going on here. 
> I think that the directory
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be
> created before the PGBuild/Utils.pm sub run_log tries to write to it.
>


I will check on it. Thanks for the report.

cheers

andrew

-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Mikael Kjellström]

On 2017-04-19 15:59, Andrew Dunstan wrote:



I have released version 4.19 of the PostgreSQL Buildfarm client. It can
be downloaded from



I don't know if it's only me or if others have noticed this also but I 
have the buildfarm client set up like this:


/home/pgbf
/home/pgbf/build-farm-4
/home/pgbf/buildfarm@ -> build-farm-4.19/
/home/pgbf/buildroot

and then I have a cron job that does this:

x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE

and when it tries to run I get the following error:

cannot create 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: 
No such file or directory

gmake is not GNU Make - please fix config file at ./run_build.pl line 343.

if I manually create the directory loach.lastrun-logs the script then 
continues normally.


I am no perl guru so I can't figure out really what is going on here.  I 
think that the directory 
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be 
created before the PGBuild/Utils.pm sub run_log tries to write to it.


/Mikael


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Andrew Dunstan]

On 04/19/2017 01:38 PM, Tom Lane wrote:
> Andrew Dunstan  writes:
>> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
>> be downloaded from
>> 
> Nice work!
>


Thank you.


>>   * Improvements to TAP tests logging and coverage. Each test set (i.e
>> each /t directory) is now logged separately. Coverage is extended to
>> remaining test sets in src/test except SSL tests.
> I was going to plead with buildfarm owners who use --enable-tap-tests to
> update sooner rather than later, because this separate-logging behavior
> is going to make it a lot less painful to decipher failures during the
> TAP tests.  However, in view of my results in
> https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us
> I can't really recommend that owners of slower machines update just yet.
> Hopefully we can do something to trim down the added runtime.


As I mentioned in reply to that thread, you can disable the extra TAP
tests with this addition to the command line:

--skip-steps=misc-check


>
>> These changes mean that the client is more useful for testing
>> development code, and also that testing config settings is much simpler.
>> An initial test run on a fresh installation is now as simple as:
>> ...
>> To test development code, the from-source option is now much more
>> flexible and friendly. For example, one might do something like:
> These things are pretty cool, but maybe they need to be documented at
> https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto
> ?
>
>   



Indeed. I am planning to.

cheers

andrew


-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [buildfarm-members] BuildFarm client release 4.19

[Tom Lane]

Andrew Dunstan  writes:
> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
> be downloaded from
> 

Nice work!

>   * Improvements to TAP tests logging and coverage. Each test set (i.e
> each /t directory) is now logged separately. Coverage is extended to
> remaining test sets in src/test except SSL tests.

I was going to plead with buildfarm owners who use --enable-tap-tests to
update sooner rather than later, because this separate-logging behavior
is going to make it a lot less painful to decipher failures during the
TAP tests.  However, in view of my results in
https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us
I can't really recommend that owners of slower machines update just yet.
Hopefully we can do something to trim down the added runtime.

> These changes mean that the client is more useful for testing
> development code, and also that testing config settings is much simpler.
> An initial test run on a fresh installation is now as simple as:
> ...
> To test development code, the from-source option is now much more
> flexible and friendly. For example, one might do something like:

These things are pretty cool, but maybe they need to be documented at
https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto
?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers