On Tue, Oct 10, 2017 at 05:42:12PM -0500, Monty Taylor wrote: > Hey everybody! > > I'd like to make a proposal for changing how we do logs/artifacts/docs > collection based on the last few weeks/months of writing things- and of > having to explain to people how to structure build and publish jobs over the > last couple of weeks. > > tl;dr - I'd like to change the publication interface to be directories on > the remote node, not directories on the executor > > Rationale > ========= > > If jobs have to copy files back to the executor as part of the publication > interface, then the zuul admins can't change the mechanism of how artifacts, > logs or docs are published without touching a ton of potentially in-tree job > content. > > Doing so should also allow us to stop having a second copy of build logic in > the artifact release jobs. > > Implementation > ============== > > Define a root 'output' dir on the remote nodes. Different types of output > can be collected by putting them into subdirectories of that dir on the > remote nodes and expecting that base jobs will take care of them. > > People using jobs defined in zuul-jobs should define a variable > "zuul_output_dir", either in site-variables or in their base job. Jobs in > zuul-jobs can and will depend on that variable existing - it will be > considered part of the base job interface zuul-jobs expects. > > Jobs in zuul-jobs will recognize three specific types of job output: > * logs > * artifacts > * docs > > Jobs in zuul-jobs will put job outputs into "{{ zuul_output_dir }}/logs", > "{{ zuul_ouptut_dir }}/artifacts" and "{{ zuul_output_dir }}/docs" as > appropriate. > > A role will be added to zuul-jobs that can be used in base jobs that will > ensure those directories all exist. > > Compression > ----------- > > Deployers may choose to have their base job compress items in {{ > zuul_output_dir }} as part of processing them, or may prefer not to > depending on whether CPU or network is more precious. Jobs in zuul-jobs > should just move/copy things into {{ zuul_output_dir }} on the node and > leave compression, transfer and publication as a base-job operation. > > Easy Output Copying > ------------------- > > A role will also be added to zuul-jobs to facilitate easy/declarative output > copying. > > It will take as input a dictionary of files/folders named > 'zuul_copy_output'. The role will copy contents into {{ zuul_output_dir }} > on the remote node and is intended to be used before output fetching in a > base job's post-playook. > > The input will be a dictionary so that zuul variable merging can be used for > accumulating. > > Keys of the dictionary will be things to copy. Valid values will be the type > of output to copy: > > * logs > * artifacts > * docs > * null # ansible null, not the string null > > null will be used for 'a parent job said to copy this, but this job wants to > not do so' > > The simple content copying role will not be flexible or powerful. People > wanting more expressive output copying have all of the normal tools for > moving files around at their disposal. It will obey the following rules: > > * All files/folders will be copied if they exist, or ignored if they don't > * Items will be copied as-if 'cp -a' had been used. > * Order of files is undefined > * Conflicts between declared files are an error. > > Jobs defined in zuul-jobs should not depend on the {{ zuul_copy_output }} > variable for content they need copied in place on a remote node. Jobs > defined in zuul-jobs should instead copy their output to {{ zuul_output_dir > }} This prevents zuul deployers from being required to put the easy output > copying role into their base jobs. Jobs defined in zuul-jobs may use the > role behind the scenes. > > Filter Plugin > ------------- > > Since the pattern of using a dictionary in job variables to take advantage > of variable merging is bound to come up more than once, we'll define a > filter plugin in zuul called 'zuul_list_from_value' (or some better name) > that will return the list of keys that match a given value. So that given > the following in a job defintion: > > vars: > zuul_copy_output: > foo/bar.html: logs > other/logs/here.html: logs > foo/bar.tgz: artifacts > > Corresponding Ansible could do: > > - copy: > src: {{ item }} > dest: {{ zuul_log_dir }} > with_items: {{ zuul_copy_output | zuul_list_from_value(logs) }} > > For OpenStack Today > =================== > > We will define zuul_output_dir to be "{{ ansible_user_dir }}/zuul-output" in > our site-variables.yaml file. > > Implement the following in OpenStack's base job: > > We will have the base job will include the simple copying role. > > Logs > ---- > > Base job post playbook will always grab {{ zuul_output_dir }}/logs from > nodes, same as today: > * If there are more than one node, grab it into {{ zuul.executor.log_dir > }}/{{ inventory_hostname }}. > * If only one node, grab into {{ zuul.executor.log_dir }} directly > > This will mean moving things like 'fetch-tox-output' logic into the 'tox' > role itself, so after it's run the appropriate tox logs will be copied to {{ > zuul_output_dir }}/logs > > Artifacts and docs > ------------------ > > Base job will always ... > > * Grab {{ zuul_output_dir }}/artifacts and {{ zuul_output_dir }}/docs to {{ > zuul.executor.work_dir }}/artifacts and {{ zuul.executor.work_dir }}/docs > * Publish docs content to logs.openstack.org/{{ zuul_log_path }}/docs - > allows for easy setting of success-url to 'docs/html" in base jobs > regardless of the subdir the job puts docs in to (see docs vs. releasenotes) > * rsync docs content to logs.o.o/{{ zuul_log_path }}/docs after contents of > {{ zuul_output_dir }}/logs so that explicit docs win if there is overlap > * Sign each artifact > * Copy each artifact to {{ zuul.project.short_name }}.{{ zuul.tag }}.{{ > suffix }} OR {{ zuul.project.short_name }}.{{ zuul.branch }}.{{ suffix }} > depending on if {{ zuul.tag exists }} or not > * If {{ zuul.change }} is set, upload artifacts to {{ zuul_log_path > }}/artifacts - to allow people to iterate on artifact upload jobs in check > and verify that they work > * If {{ zuul.change }} is not set, upload artifacts to > tarballs.openstack.org/{{ zuul.project.short_name }, same as today. > Communicate artifact and docs upload paths to child jobs using: > https://review.openstack.org/#/c/504808/ > ++
I've made one small change your current logic, by creating new fetch-zuul-log[2] and fetch-zuul-docs[3] roles. Only because I think it is much easier to read then adding everything into a single fetch-output role. We'd do the same for artifacts, and any new folder a zuul user would create. This also allows an operator the ability to choose which role to also include for their base jobs. [2] https://review.openstack.org/583346/ [3] https://review.openstack.org/583350/ > Artifact Release jobs > ===================== > > The release jobs become similar to v2 release jobs, but can be much more > generalized and reusable. They can also mostly be nodeless jobs (javascript > npm publication is a notable exception - it will need a node) > > Artifacts > --------- > > * Fetch artifacts and signatures from tarballs.openstack.org/{{ > zuul.project.short_name }} - based on values set by parent job > Verify fetched signature is correct > * Upload artifact to appropriate destination(s) (pypi, npm, maven, etc) > > Docs > ---- > > * Rsync docs from logs.openstack.org/{{ zuul_log_path }}/docs - based on > values set by parent job > * Publish to release-job-specific destination as the current jobs do > > This should allow us to make job/project/content specific publication jobs > without needing those jobs to duplicate content in the build jobs. > > It means we can ALSO put jobs like 'publish-to-pypi' in zuul-jobs without > needing a second copy in openstack-zuul-jobs because openstack happens to > builds tarballs differently. The pypi publication step is the same no matter > how the tarball is produced. > > - jobs: > - build-openstack-sphinx-docs > - publish-sphinx-docs-to-afs: > dependencies: > - build-openstack-sphinx-docs > > Rollout > ======= > > The steps should be fairly straightforward (and are fine to do post rollout) > > * Add variable to site-variables > * Add support to base job for creating directories and then copying content > back > * Add easy copy helper role > * Start transitioning jobs in zuul-jobs and openstack-zuul-jobs to using the > new interface - can be done one at a time > * Update usage > > If people are ok with the direction generally, we should be able to get the > enablement pieces in pretty quickly. > > Monty > I'd like to first land our base-minimal[3] (trusted) / base (untrusted) changes then this. But I've already done some testing of new log publishing[4] and works as expected. - Paul [3] https://review.openstack.org/#/q/topic:base-minimal-jobs [4] https://review.openstack.org/584614/ _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra