Re: [152630] contrib/buildbot-test

[Mojca Miklavec]

On 14 September 2016 at 21:15, Ryan Schmidt wrote:
>> On Sep 14, 2016, at 2:01 PM, Rainer Müller wrote:
>> On 2016-09-14 16:10, Ryan Schmidt wrote:
>>>
>>> I had to remove these lines when deploying because I don't have a "docs" 
>>> worker set up yet.
>>>
>>> Configuration Errors:
>>>  builder 'docs-www' uses unknown slaves 'docs'
>>>  builder 'docs-guide' uses unknown slaves 'docs'
>>
>> I agree this will be more than just documentation. Maybe give it a very
>> generic name such as "jobs"?
>
> I like that. Or we could call it "steve".

I like the idea :) :) :)

Anyway, can someone put an additional line into slaves.json.sample for
that worker?

It doesn't hurt to have a worker listed even if the actual machine
hasn't been set up yet. I got the same error as above (builder ...
uses unknown slaves 'docs'.)

Mojca
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [152630] contrib/buildbot-test

[Ryan Schmidt]

> On Sep 14, 2016, at 2:44 PM, Rainer Müller  wrote:
> 
> On 2016-09-14 21:15, Ryan Schmidt wrote:
>> The master uses MacPorts in /opt/local for its own software (nginx, 
>> buildbot, rsync, etc.). I can manually install anything else there that we 
>> need. It's probably not best to automate selfupdating that MacPorts install, 
>> but it can have its own install in another prefix that it selfupdates. 
>> Giving the buildbot user sudoers access to that port command seems fine.
> 
> Currently the only job that would require software would be the
> generation of the guide with the ports: docbook-xsl docbook-xml libxslt
> 
> For www, we could add a syntax check of the PHP files to avoid deploying
> anything obviously broken.

I wouldn't bother with that now. It hasn't been a problem so far, and I would 
prefer if the existing web site didn't stick around too much longer. Let's talk 
later about replacing it.

> Later I have plans to also add a job for generating HTML versions of the
> new man pages (requires asciidoc) and maybe another one for the Doxygen
> documentation of base (requires doxygen).
> 
> The need for additional ports would usually only arise when we add a new
> builder to master.cfg, which also needs to be deployed by you anyway.
> The more I think about it, I come to the conclusion that we probably do
> not even need that automation and could just keep a list of ports that
> should be installed on this worker next to master.cfg.

I'll just add them to the list of dependencies in the infrastructure portfile 
that I use to install the buildbot master configuration.


___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [152630] contrib/buildbot-test

[Rainer Müller]

On 2016-09-14 21:15, Ryan Schmidt wrote:
> The master uses MacPorts in /opt/local for its own software (nginx, buildbot, 
> rsync, etc.). I can manually install anything else there that we need. It's 
> probably not best to automate selfupdating that MacPorts install, but it can 
> have its own install in another prefix that it selfupdates. Giving the 
> buildbot user sudoers access to that port command seems fine.

Currently the only job that would require software would be the
generation of the guide with the ports: docbook-xsl docbook-xml libxslt

For www, we could add a syntax check of the PHP files to avoid deploying
anything obviously broken.

Later I have plans to also add a job for generating HTML versions of the
new man pages (requires asciidoc) and maybe another one for the Doxygen
documentation of base (requires doxygen).

The need for additional ports would usually only arise when we add a new
builder to master.cfg, which also needs to be deployed by you anyway.
The more I think about it, I come to the conclusion that we probably do
not even need that automation and could just keep a list of ports that
should be installed on this worker next to master.cfg.

Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [152630] contrib/buildbot-test

[Ryan Schmidt]

> On Sep 14, 2016, at 2:01 PM, Rainer Müller  wrote:
> 
> On 2016-09-14 16:10, Ryan Schmidt wrote:
>>> +"deploy": {
>>> +"www": {
>>> +"host": "",
>>> +"user": "",
>>> +"sshkeyfile": "",
>>> +"sshknownhostsfile": "ssh_known_hosts",
>>> +"destpath": ""
>>> +},
>>> +"guide": {
>>> +"host": "",
>>> +"user": "",
>>> +"sshkeyfile": "",
>>> +"sshknownhostsfile": "ssh_known_hosts",
>>> +"destpath": ""
>>> +}
>>> +}
>>> 
>>> }
>> 
>> I had to remove these lines when deploying because I don't have a "docs" 
>> worker set up yet.
>> 
>> Configuration Errors:
>>  builder 'docs-www' uses unknown slaves 'docs'
>>  builder 'docs-guide' uses unknown slaves 'docs'
> 
> Yes, that is the right solution. The previous logic based it on the
> existence of the "docs" worker, now it is controlled by the entries in
> the "deploy" dict.
> 
>> This worker will run on the same server as the master. We have other tasks 
>> we need to run on the master as well: mirroring distfiles on port changes; 
>> copying ports to the rsync server on port changes; creating ports.tar on 
>> port changes; creating base.tar on base changes. Will these builders use the 
>> same worker? If so, what name could we pick (not "docs") that would be 
>> sufficiently all-encompassing?
> 
> Yes, I would put all of these tasks on the same set of workers
> (let's start with just one, but if we ever get too many of these tasks,
> we could add more for load balancing).
> 
> I agree this will be more than just documentation. Maybe give it a very
> generic name such as "jobs"?

I like that. Or we could call it "steve".


> This worker would need the required software installed, preferably from
> ports. If we want to automate port install/upgrade, we would have to run
> the worker as root and drop privileges with sudo/su for the actual jobs.
> An alternative solution would be to provide an existing port install and
> only run 'sudo port ...' with sudoers allowing the user to run this
> command without password.

The master uses MacPorts in /opt/local for its own software (nginx, buildbot, 
rsync, etc.). I can manually install anything else there that we need. It's 
probably not best to automate selfupdating that MacPorts install, but it can 
have its own install in another prefix that it selfupdates. Giving the buildbot 
user sudoers access to that port command seems fine.


___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [152630] contrib/buildbot-test

[Rainer Müller]

On 2016-09-14 16:10, Ryan Schmidt wrote:
>> +"deploy": {
>> +"www": {
>> +"host": "",
>> +"user": "",
>> +"sshkeyfile": "",
>> +"sshknownhostsfile": "ssh_known_hosts",
>> +"destpath": ""
>> +},
>> +"guide": {
>> +"host": "",
>> +"user": "",
>> +"sshkeyfile": "",
>> +"sshknownhostsfile": "ssh_known_hosts",
>> +"destpath": ""
>> +}
>> +}
>>
>>  }
> 
> I had to remove these lines when deploying because I don't have a "docs" 
> worker set up yet.
>
> Configuration Errors:
>   builder 'docs-www' uses unknown slaves 'docs'
>   builder 'docs-guide' uses unknown slaves 'docs'

Yes, that is the right solution. The previous logic based it on the
existence of the "docs" worker, now it is controlled by the entries in
the "deploy" dict.

> This worker will run on the same server as the master. We have other tasks we 
> need to run on the master as well: mirroring distfiles on port changes; 
> copying ports to the rsync server on port changes; creating ports.tar on port 
> changes; creating base.tar on base changes. Will these builders use the same 
> worker? If so, what name could we pick (not "docs") that would be 
> sufficiently all-encompassing?

Yes, I would put all of these tasks on the same set of workers
(let's start with just one, but if we ever get too many of these tasks,
we could add more for load balancing).

I agree this will be more than just documentation. Maybe give it a very
generic name such as "jobs"?

This worker would need the required software installed, preferably from
ports. If we want to automate port install/upgrade, we would have to run
the worker as root and drop privileges with sudo/su for the actual jobs.
An alternative solution would be to provide an existing port install and
only run 'sudo port ...' with sudoers allowing the user to run this
command without password.

Rainer
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [152630] contrib/buildbot-test

[Ryan Schmidt]

> On Sep 13, 2016, at 6:52 PM, rai...@macports.org wrote:
> 
> Revision
> 152630
> Author
> rai...@macports.org
> Date
> 2016-09-13 16:52:14 -0700 (Tue, 13 Sep 2016)
> Log Message
> 
> buildbot: add deploy config for www/guide
> 
> This adds preliminary support for deploying generated static files with rsync
> to the host running the web server. The builders and schedulers will only be
> added if the deploy configuration is filled.
> 
> Modified Paths
> 
>   • contrib/buildbot-test/config.json.sample
>   • contrib/buildbot-test/master.cfg
> Diff
> 
> Modified: contrib/buildbot-test/config.json.sample (152629 => 152630)
> 
> --- contrib/buildbot-test/config.json.sample  2016-09-13 20:48:39 UTC (rev 
> 152629)
> +++ contrib/buildbot-test/config.json.sample  2016-09-13 23:52:14 UTC (rev 
> 152630)
> 
> @@ -9,5 +9,21 @@
> 
>  "svnurl":   "https://svn.macports.org/repository/macports/trunk
> ",
> 
>  "archivesite":  "https://packages.macports.org
> ",
> 
>  "slaveprefix":  "/opt/local",
> 
> -"toolsprefix":  "/opt/mports"
> 
> +"toolsprefix":  "/opt/mports",
> +"deploy": {
> +"www": {
> +"host": "",
> +"user": "",
> +"sshkeyfile": "",
> +"sshknownhostsfile": "ssh_known_hosts",
> +"destpath": ""
> +},
> +"guide": {
> +"host": "",
> +"user": "",
> +"sshkeyfile": "",
> +"sshknownhostsfile": "ssh_known_hosts",
> +"destpath": ""
> +}
> +}
> 
>  }

I had to remove these lines when deploying because I don't have a "docs" worker 
set up yet.


Configuration Errors:
  builder 'docs-www' uses unknown slaves 'docs'
  builder 'docs-guide' uses unknown slaves 'docs'


This worker will run on the same server as the master. We have other tasks we 
need to run on the master as well: mirroring distfiles on port changes; copying 
ports to the rsync server on port changes; creating ports.tar on port changes; 
creating base.tar on base changes. Will these builders use the same worker? If 
so, what name could we pick (not "docs") that would be sufficiently 
all-encompassing?



___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev