Hello James, On Wed, 29 May 2013 16:57:50 +0100 James Tunnicliffe <james.tunnicli...@linaro.org> wrote:
> Hi Paul, > > Thanks for this. I need a mechanism for publishing from CI runtime > jobs so this is important to me. I did look into using SSH/SFTP and it > is simple to do in a reasonably insecure way, it would be much better > to have an HTTP[S] based solution that uses temporary authentication > tokens. > > I was looking at this today because I woke up early worrying about it. > Clearly I need more interesting stuff to think about! (and now, more > sleep). Well, I guess each of us (individual engineers) was hit by publishing issues already, so that's clearly a thing to worry about and I share your concerns. So, thanks for joining the discussion (and let's nag Tyler for spec together ;-) ). > > Anyway, it should be possible to do in Django: > > https://docs.djangoproject.com/en/dev/topics/http/file-uploads/ > https://pypi.python.org/pypi/django-transfer/0.2-2 > http://wiki.nginx.org/HttpUploadModule > > My own notes are more focused on an internal server that would upload > files to releases/snapshots but could retain files until disk space > was needed to act as a cache. I was going to look at extending > linaro-licenese-protection for this so there was no way to use the > cache to avoid licenses. I was also going to have completely private > files that you could only access if you had the authentication token > that a job was given. > > https://docs.google.com/a/linaro.org/document/d/1_ewb-xFDJc8Adk7AijV95XthGMvYVtlSLozbJxsyI4o/edit# > > Feel free to add comments or insert more information and thoughts. I'll look in more detail later. Just having a small window before restarting work on Crowd API auth ;-). > > Note that for high performance uploads we probably want to hand off > the upload to a web server. That django-transfer module doesn't > support any Apache related upload stuff, which may mean that it > doesn't exist. Moving to an nginx based solution would be easy enough > if we needed to (we could replace the mod-xsendfile with the > equivalent nginx call). Yeah, I had experience with all that stuff at SourceForge.net, where high availability and extra features (like progress indicator on web page) were a must. I'd think that we can start with simple Django-level PUT handler, but yes, we should be prepared that it'll have performance issues and for need to upgrade architecture. > I think a prototype branch is ~1 day of engineering effort (no nginx, > token based upload call in place, probably some kind of token request > call, probably limited security, no proxy-publishing). Adding the rest > of the features, testing etc probably takes it to more like 1 week. Well, I'd really like us (all) to first consider architecture well enough. I also lean towards having that as a web service, but I'd like someone who has other opinion to have their say too. > > James > > > On 29 May 2013 16:26, Paul Sokolovsky <paul.sokolov...@linaro.org> > wrote: > > > > > > Begin forwarded message: > > > > Date: Wed, 29 May 2013 17:19:31 +0300 > > From: Paul Sokolovsky <paul.sokolov...@linaro.org> > > To: Tyler Baker <tyler.ba...@linaro.org>, Alan Bennett > > <alan.benn...@linaro.org> Cc: Senthil Kumaran > > <senthil.kuma...@linaro.org>, Fathi Boudra <fathi.bou...@linaro.org> > > Subject: Re: New publishing infra prototype report > > > > > > Hello Tyler, > > > > As brought up today on IRC, it's a month since the below report and > > proposal for further steps, and I don't remember any reply. This > > whole publishing thing is peculiar, as it sits still when its not > > needed, but it's actually in ambush there to cause havoc at any > > time. > > > > For example, today Senthil came up with the question on how to > > publish to snapshots. Fortunately, it turned out that it was > > request for publishing a single file manually. But I know the guys > > are working on Fedora builds, and that definitely will need > > automated publishing (unless initial requirements as provided by > > Fathi changed). And it's definitely needed for CBuild migration, > > which I assume will be worked on next month. > > > > Btw, I discovered that a BP for that was submitted yet by Danilo: > > https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/file-publishing-api > > > > > > Thanks, > > Paul > > > > > > On Mon, 29 Apr 2013 18:58:39 +0300 > > Paul Sokolovsky <paul.sokolov...@linaro.org> wrote: > > > >> Hello, > >> > >> Last month I worked on a blueprint > >> https://blueprints.launchpad.net/linaro-android-infrastructure/+spec/prototype-new-publishing > >> to prototype an implementation of publishing framework which > >> wouldn't depend on particular Jenkins features (and misfeatures) > >> and could be reused for other services across Linaro CI > >> infrastructure. Among these other projects are: > >> > >> 1. OpenEmbedded builds - efficient ("fresh only") publishing of > >> source tarballs and cache files. > >> 2. CBuild - publishing of toolchain build artifacts and logs. > >> 3. Fedora/Lava - publishing of build artifacts and logs. > >> > >> So, the good news is that was possible to implement a publishing > >> system whose interface is a single script which hides all the > >> publishing complexity underneath. Implementation was a cumbersome, > >> because existing publishing backend was reused, but it already > >> opens possibility for better logging, debugging, profiling, etc. > >> > >> With proof-of-concept client-side available, the main complexity > >> still lies in server-side backend. It's clear that current "SFTP + > >> SSH trigger script" approach doesn't scale well in terms of ease > >> of setup and security. I added my considerations to address that > >> topic in " Conclusions and Future Work" section of > >> http://bazaar.launchpad.net/~linaro-automation/linaro-android-build-tools/trunk/view/head:/utils/new-publish/README > >> > >> So, action items I suggest based on this report: > >> > >> 1. Tyler to consult with Fathi (Fedora), Marcin (OE) and me > >> (CBuild) and prepare architecture/spec for the general publishing > >> system. It would be nice to BP this task to start in 13.05. > >> 2. Depending on the time required to prepare spec, implementation > >> can be scheduled right away, or postponed until LCE13, so we had > >> another chance to discuss it face to face (as adhoc meeting, or as > >> a session, if it's really worth it). > >> > >> > >> Thanks, > >> Paul > >> > >> Linaro.org | Open source software for ARM SoCe > >> Follow Linaro: http://www.facebook.com/pages/Linaro > >> http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > > > > > > > > -- > > Best Regards, > > Paul > > > > Linaro.org | Open source software for ARM SoCs > > Follow Linaro: http://www.facebook.com/pages/Linaro > > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > > > > > > -- > > Best Regards, > > Paul > > > > Linaro.org | Open source software for ARM SoCs > > Follow Linaro: http://www.facebook.com/pages/Linaro > > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > > > -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog _______________________________________________ linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
