On 09/16/2013 07:22 PM, Joshua Hesketh wrote: > So if zuul dictates where a log goes and we place the objects in swift > with that path (change / patchset / pipeline / job / run) then zuul > could also handle placing indexes as it should know which objects to > expect. > > That said, if the path is deterministic (such as that) and the workers > provide the index for a run then I'm not sure how useful an index for > patchsets would be. I'd be interested to know if anybody uses the link > http://logs.openstack.org/34/45334/ without having come from gerrit or > another source where it is published.
https://pypi.python.org/pypi/git-os-job > Because of its deterministic > nature perhaps the use case where it is needed could be served otherhow? > > Cheers, > Josh > > -- > Rackspace Australia > > On 9/13/13 2:49 AM, James E. Blair wrote: >> Joshua Hesketh <[email protected]> writes: >> >>> We could then use either psuedo folders[0] or have the worker generate >>> an index. For example, why not create an index object with links to >>> the other objects (using the known serving application URL prepended)? >>> In fact, the reporter can choose whether to generate an index file or >>> just send the psuedo folder to be served up. >> This is one of the main reasons we don't use swift today. Consider this >> directory: >> >> http://logs.openstack.org/34/45334/ >> >> It contains all of the runs of all of the jobs for all of the patchsets >> for change 45334. That's very useful for discoverability; the >> alternative is to read the comments in gerrit and examine the links >> one-by-one. A full-depth example: >> >> http://logs.openstack.org/34/45334/7/check/gate-zuul-python27/7c48ee3/ >> >> (That's change / patchset / pipeline / job / run.) >> >> Each individual job is concerned with only the last component of that >> hierarchy, and has no knowledge of what other related jobs may have run >> before or will run after, so creating an index under those circumstances >> is difficult. Moreover, if you consider that in the new system, we >> won't be able to trust a job with access to any pseudo-directory level >> higher than its individual run, there is actually no way for it to >> create any of the higher-level indexes. >> >> If we want to maintain that level of discoverability, then I think we >> need something outside of the job to create indexes (in my earlier >> email, the artifact-serving component does this). If we are okay losing >> that, then yes, we can just sort of shove everything related to a run >> into a certain arbitrary location whose path won't be important anymore. >> Within the area written to by a single run, however, we may still have >> subdirectories. Whether and how to create swift directory markers for >> those is still an issue (see my other email). But perhaps they are not >> necessary, and yes, certainly _within the directory for a run_, we could >> create index files for as needed. >> >> Note the following implementation quirks we have observed: >> >> * Rackspace does not perform autoindex-like functionality for directory >> markers unless you are using the CDN (which has its own complications >> related to cache timeouts, dns hostnames, etc). >> >> * HPCloud does not recognize directory markers when generating index >> pages for the public view of containers. >> >> We may want and indeed be able to use the staticweb feature, along with >> the CDN -- but there's enough complication here that we'll need to get >> fairly detailed in the design and validate our assumptions. >> >> -Jim >> >> _______________________________________________ >> OpenStack-Infra mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > > _______________________________________________ > OpenStack-Infra mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
