On Wed, 29 May 2013 17:11:11 +0100 James Tunnicliffe <[email protected]> wrote:
> Issues raised in > http://bazaar.launchpad.net/~linaro-automation/linaro-android-build-tools/trunk/view/head:/utils/new-publish/README > > Getting a token: > I see this as the service that starts the job has some secret allowing > it to request the token, which is then passed onto the job. Yes, just like that: we have fixed (and thus trusted) job scheduler, which can reliably authenticate itself to a token server, get a token and pass that token to a much less trusted builder. > This may > require a Jenkins plugin. Yes, that's one possibility, as line 94 of the doc you quote says. Another possibility is to have extra frontend on top of Jenkins, which would request a token and inject it into Jenkins job (as a paremeter during start). That's possible right away with android-build and would be possible with other Jenkins with original ci-frontend plan. > > Open Embedded Builds/rsync: > This is not required. OE has a list of package sources and versions > associated with each build so we should be able to write a simple > "upload packages if that version doesn't exist" script. This probably > requires us to host files in openembedded/sources/<package > name>/<version>/<package file>, but that is a simple enough change. James, for me, what you said translates into something "let's dig into adhoc peculiarities of one usecase, because it seems that it's easy to handle that, as if we don't know that it the end it still won't be that easy, but the result will not be reusable". Nope, let's shoot for general solution of problem of publishing arbitrary file sets. That's also "easy". The hardest part of that is to accept that we'll duplicate best-practice solution (rsync) and by duplicating it, we'll lose extra features like intra-file optimizations (because *that* we for sure don't want to duplicate). That's why I'd like who anyone has strong reasons for going with rsync is come up with them before we decide on architecture (keeping in mind that rsync comes with friends like ssh, LDAP, PAM, Kerberos, etc.) > > publish --token=<token> --type=<build_type> --strip=<strip> <build_id> > <glob_pattern>... > > This seems like a reasonable starting point. Lets make sure that it > uses a configuration file to specify what to do with those build types > etc. Preferably one that it can update from a public location so we > don't have to re-spin the tool to add a new build type (though I guess > we normally check it out of VCS as we go, so that works too). Well, on client side, it's ideally just a single file which just handle obvious filtering options (like <glob_pattern> or --strip=) locally and passes the rest to API/service. Server-side can handle the options in any way it wants, note that options above don't require much "configuration", for example --type= just maps to top-level download dir. Except for, well, we already have adhoc behavioral idiosyncrasies, like Android builds flattening happening on server. You hardly can "configure" that (though lambdas in YAML sounds cool (for debugging :-D)). Better approach would be to move that stuff to client side and have simple well-defined publishing semantics. > > James > > On 29 May 2013 16:57, James Tunnicliffe > <[email protected]> wrote: > > Hi Paul, > > > > Thanks for this. I need a mechanism for publishing from CI runtime > > jobs so this is important to me. I did look into using SSH/SFTP and > > it is simple to do in a reasonably insecure way, it would be much > > better to have an HTTP[S] based solution that uses temporary > > authentication tokens. > > > > I was looking at this today because I woke up early worrying about > > it. Clearly I need more interesting stuff to think about! (and now, > > more sleep). > > > > Anyway, it should be possible to do in Django: > > > > https://docs.djangoproject.com/en/dev/topics/http/file-uploads/ > > https://pypi.python.org/pypi/django-transfer/0.2-2 > > http://wiki.nginx.org/HttpUploadModule > > > > My own notes are more focused on an internal server that would > > upload files to releases/snapshots but could retain files until > > disk space was needed to act as a cache. I was going to look at > > extending linaro-licenese-protection for this so there was no way > > to use the cache to avoid licenses. I was also going to have > > completely private files that you could only access if you had the > > authentication token that a job was given. > > > > https://docs.google.com/a/linaro.org/document/d/1_ewb-xFDJc8Adk7AijV95XthGMvYVtlSLozbJxsyI4o/edit# > > > > Feel free to add comments or insert more information and thoughts. > > > > Note that for high performance uploads we probably want to hand off > > the upload to a web server. That django-transfer module doesn't > > support any Apache related upload stuff, which may mean that it > > doesn't exist. Moving to an nginx based solution would be easy > > enough if we needed to (we could replace the mod-xsendfile with the > > equivalent nginx call). > > > > I think a prototype branch is ~1 day of engineering effort (no > > nginx, token based upload call in place, probably some kind of > > token request call, probably limited security, no > > proxy-publishing). Adding the rest of the features, testing etc > > probably takes it to more like 1 week. > > > > James > > > > > > On 29 May 2013 16:26, Paul Sokolovsky <[email protected]> > > wrote: > >> > >> > >> Begin forwarded message: > >> > >> Date: Wed, 29 May 2013 17:19:31 +0300 > >> From: Paul Sokolovsky <[email protected]> > >> To: Tyler Baker <[email protected]>, Alan Bennett > >> <[email protected]> Cc: Senthil Kumaran > >> <[email protected]>, Fathi Boudra > >> <[email protected]> Subject: Re: New publishing infra > >> prototype report > >> > >> > >> Hello Tyler, > >> > >> As brought up today on IRC, it's a month since the below report and > >> proposal for further steps, and I don't remember any reply. This > >> whole publishing thing is peculiar, as it sits still when its not > >> needed, but it's actually in ambush there to cause havoc at any > >> time. > >> > >> For example, today Senthil came up with the question on how to > >> publish to snapshots. Fortunately, it turned out that it was > >> request for publishing a single file manually. But I know the guys > >> are working on Fedora builds, and that definitely will need > >> automated publishing (unless initial requirements as provided by > >> Fathi changed). And it's definitely needed for CBuild migration, > >> which I assume will be worked on next month. > >> > >> Btw, I discovered that a BP for that was submitted yet by Danilo: > >> https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/file-publishing-api > >> > >> > >> Thanks, > >> Paul > >> > >> > >> On Mon, 29 Apr 2013 18:58:39 +0300 > >> Paul Sokolovsky <[email protected]> wrote: > >> > >>> Hello, > >>> > >>> Last month I worked on a blueprint > >>> https://blueprints.launchpad.net/linaro-android-infrastructure/+spec/prototype-new-publishing > >>> to prototype an implementation of publishing framework which > >>> wouldn't depend on particular Jenkins features (and misfeatures) > >>> and could be reused for other services across Linaro CI > >>> infrastructure. Among these other projects are: > >>> > >>> 1. OpenEmbedded builds - efficient ("fresh only") publishing of > >>> source tarballs and cache files. > >>> 2. CBuild - publishing of toolchain build artifacts and logs. > >>> 3. Fedora/Lava - publishing of build artifacts and logs. > >>> > >>> So, the good news is that was possible to implement a publishing > >>> system whose interface is a single script which hides all the > >>> publishing complexity underneath. Implementation was a cumbersome, > >>> because existing publishing backend was reused, but it already > >>> opens possibility for better logging, debugging, profiling, etc. > >>> > >>> With proof-of-concept client-side available, the main complexity > >>> still lies in server-side backend. It's clear that current "SFTP > >>> + SSH trigger script" approach doesn't scale well in terms of > >>> ease of setup and security. I added my considerations to address > >>> that topic in " Conclusions and Future Work" section of > >>> http://bazaar.launchpad.net/~linaro-automation/linaro-android-build-tools/trunk/view/head:/utils/new-publish/README > >>> > >>> So, action items I suggest based on this report: > >>> > >>> 1. Tyler to consult with Fathi (Fedora), Marcin (OE) and me > >>> (CBuild) and prepare architecture/spec for the general publishing > >>> system. It would be nice to BP this task to start in 13.05. > >>> 2. Depending on the time required to prepare spec, implementation > >>> can be scheduled right away, or postponed until LCE13, so we had > >>> another chance to discuss it face to face (as adhoc meeting, or > >>> as a session, if it's really worth it). > >>> > >>> > >>> Thanks, > >>> Paul > >>> > >>> Linaro.org | Open source software for ARM SoCe > >>> Follow Linaro: http://www.facebook.com/pages/Linaro > >>> http://twitter.com/#!/linaroorg - > >>> http://www.linaro.org/linaro-blog > >> > >> > >> > >> -- > >> Best Regards, > >> Paul > >> > >> Linaro.org | Open source software for ARM SoCs > >> Follow Linaro: http://www.facebook.com/pages/Linaro > >> http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > >> > >> > >> -- > >> Best Regards, > >> Paul > >> > >> Linaro.org | Open source software for ARM SoCs > >> Follow Linaro: http://www.facebook.com/pages/Linaro > >> http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > > > > > > > > -- > > James Tunnicliffe > > > -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog _______________________________________________ linaro-validation mailing list [email protected] http://lists.linaro.org/mailman/listinfo/linaro-validation
