In general I agree on this proposal (as it's similar to what I and others have talked about implementing in another Zuul installation). I can't speak too much on the OpenStack use side as I don't have the domain knowledge, but I agree it's far easier to train people to "plunk files here that you want archived".
- jlk On Tue, Oct 10, 2017 at 3:42 PM, Monty Taylor <[email protected]> wrote: > Hey everybody! > > I'd like to make a proposal for changing how we do logs/artifacts/docs > collection based on the last few weeks/months of writing things- and of > having to explain to people how to structure build and publish jobs over > the last couple of weeks. > > tl;dr - I'd like to change the publication interface to be directories on > the remote node, not directories on the executor > > Rationale > ========= > > If jobs have to copy files back to the executor as part of the publication > interface, then the zuul admins can't change the mechanism of how > artifacts, logs or docs are published without touching a ton of potentially > in-tree job content. > > Doing so should also allow us to stop having a second copy of build logic > in the artifact release jobs. > > Implementation > ============== > > Define a root 'output' dir on the remote nodes. Different types of output > can be collected by putting them into subdirectories of that dir on the > remote nodes and expecting that base jobs will take care of them. > > People using jobs defined in zuul-jobs should define a variable > "zuul_output_dir", either in site-variables or in their base job. Jobs in > zuul-jobs can and will depend on that variable existing - it will be > considered part of the base job interface zuul-jobs expects. > > Jobs in zuul-jobs will recognize three specific types of job output: > * logs > * artifacts > * docs > > Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir }}/logs", > "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir }}/docs" as > appropriate. > > A role will be added to zuul-jobs that can be used in base jobs that will > ensure those directories all exist. > > Compression > ----------- > > Deployers may choose to have their base job compress items in {{ > zuul_output_dir }} as part of processing them, or may prefer not to > depending on whether CPU or network is more precious. Jobs in zuul-jobs > should just move/copy things into {{ zuul_output_dir }} on the node and > leave compression, transfer and publication as a base-job operation. > > Easy Output Copying > ------------------- > > A role will also be added to zuul-jobs to facilitate easy/declarative > output copying. > > It will take as input a dictionary of files/folders named > 'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir }} > on the remote node and is intended to be used before output fetching in a > base job's post-playook. > > The input will be a dictionary so that zuul variable merging can be used > for accumulating. > > Keys of the dictionary will be things to copy. Valid values will be the > type of output to copy: > > * logs > * artifacts > * docs > * null # ansible null, not the string null > > null will be used for 'a parent job said to copy this, but this job wants > to not do so' > > The simple content copying role will not be flexible or powerful. People > wanting more expressive output copying have all of the normal tools for > moving files around at their disposal. It will obey the following rules: > > * All files/folders will be copied if they exist, or ignored if they don't > * Items will be copied as-if 'cp -a' had been used. > * Order of files is undefined > * Conflicts between declared files are an error. > > Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output }} > variable for content they need copied in place on a remote node. Jobs > defined in zuul-jobs should instead copy their output to {{ zuul_output_dir > }} This prevents zuul deployers from being required to put the easy output > copying role into their base jobs. Jobs defined in zuul-jobs may use the > role behind the scenes. > > Filter Plugin > ------------- > > Since the pattern of using a dictionary in job variables to take advantage > of variable merging is bound to come up more than once, we'll define a > filter plugin in zuul called 'zuul_list_from_value' (or some better name) > that will return the list of keys that match a given value. So that given > the following in a job defintion: > > vars: > zuul_copy_output: > foo/bar.html: logs > other/logs/here.html: logs > foo/bar.tgz: artifacts > > Corresponding Ansible could do: > > - copy: > src: {{ item }} > dest: {{ zuul_log_dir }} > with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }} > > For OpenStack Today > =================== > > We will define zuul_output_dir to be "{{ ansible_user_dir }}/zuul-output" > in our site-variables.yaml file. > > Implement the following in OpenStack's base job: > > We will have the base job will include the simple copying role. > > Logs > ---- > > Base job post playbook will always grab {{ zuul_output_dir }}/logs from > nodes, same as today: > * If there are more than one node, grab it into {{ zuul.executor.log_dir > }}/{{ inventory_hostname }}. > * If only one node, grab into {{ zuul.executor.log_dir }} directly > > This will mean moving things like 'fetch-tox-output' logic into the 'tox' > role itself, so after it's run the appropriate tox logs will be copied to > {{ zuul_output_dir }}/logs > > Artifacts and docs > ------------------ > > Base job will always ... > > * Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to > {{ zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir }}/docs > * Publish docs content to logs.openstack.org/{{ > <http://logs.openstack.org/%7B%7B> zuul_log_path }}/docs - allows for > easy setting of success-url to 'docs/html" in base jobs regardless of the > subdir the job puts docs in to (see docs vs. releasenotes) > * rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents > of {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap > * Sign each artifact > * Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{ > suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix }} > depending on if {{ zuul.tag exists }} or not > * If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path > }}/artifacts - to allow people to iterate on artifact upload jobs in check > and verify that they work > * If {{ zuul.change }} is not set, upload artifacts to > tarballs.openstack.org/{{ <http://tarballs.openstack.org/%7B%7B> > zuul.project.short_name }, same as today. > Communicate artifact and docs upload paths to child jobs using: > https://review.openstack.org/#/c/504808/ > > Artifact Release jobs > ===================== > > The release jobs become similar to v2 release jobs, but can be much more > generalized and reusable. They can also mostly be nodeless jobs (javascript > npm publication is a notable exception - it will need a node) > > Artifacts > --------- > > * Fetch artifacts and signatures from tarballs.openstack.org/{{ > <http://tarballs.openstack.org/%7B%7B> zuul.project.short_name }} - based > on values set by parent job > Verify fetched signature is correct > * Upload artifact to appropriate destination(s) (pypi, npm, maven, etc) > > Docs > ---- > > * Rsync docs from logs.openstack.org/{{ <http://logs.openstack.org/%7B%7B> > zuul_log_path }}/docs - based on values set by parent job > * Publish to release-job-specific destination as the current jobs do > > This should allow us to make job/project/content specific publication jobs > without needing those jobs to duplicate content in the build jobs. > > It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs without > needing a second copy in openstack-zuul-jobs because openstack happens to > builds tarballs differently. The pypi publication step is the same no > matter how the tarball is produced. > > - jobs: > - build-openstack-sphinx-docs > - publish-sphinx-docs-to-afs: > dependencies: > - build-openstack-sphinx-docs > > Rollout > ======= > > The steps should be fairly straightforward (and are fine to do post > rollout) > > * Add variable to site-variables > * Add support to base job for creating directories and then copying > content back > * Add easy copy helper role > * Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using > the new interface - can be done one at a time > * Update usage > > If people are ok with the direction generally, we should be able to get > the enablement pieces in pretty quickly. > > Monty > > _______________________________________________ > OpenStack-Infra mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
