Re: P2P Packaging/Koji Cloud
On 12/07/2011 06:20 PM, Denis Arnaud wrote: As a side note, rather than using Snap (and Augeas, and...), we (in my department) tend to prefer Chef (http://www.opscode.com/chef/), which has got a broader scope, and allows much more complex configurations and automation tasks. Denis Chef, like puppet, is a great tool. But it is somewhat different than Snap, Augeas, etc. With chef, puppet, etc. The end user sets up centralized recipes which describe what systems should look like, how they behave, etc. With Snap, you have an existing system which you would like to use as the basis to replicate at some later point (either locally or elsewhere). In any case, slightly off topic, but Snap is now in Fedora (well updates-testing, will be pushed to updates at the end of this week, sooner if I can get +1 karma). Plus alot more features have gone in / are being pushed, including much expanded documentation, a test harness, and the start of the ability to migrate snapshots between hosts (eg backup an Ubuntu system, restore on Fedora). Stay tuned for more updates! -Mo -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 07 Dec 2011 15:25:18 -0800 Adam Williamson awill...@redhat.com wrote: Well, yes, but only because you shifted the entire terms of the thread without telling anyone else. All of the above - about how the idea was to build packages with untrusted build dependencies in trustworthy places - may have been perfectly clear _to you_, but it was not the idea that started this thread. This thread was specifically about creating a 'cloud', containing random contributor's machines, for doing builds, and several people suggested that might be 'safe' for scratch builds but not for 'release' builds. I didn't shift the terms. I've been consistent. Random machine instances from volunteers are a non-starter. You seem to have changed the terms of debate rather considerably with your _own_ idea, which you introduced into the thread, but which clearly was not actually very similar at all to the idea of the person who actually started the thread, Denis Arnaud. The extent of the difference between what _he_ was thinking/talking about and what _you_ are thinking/talking about has only been made clear in this last post of yours. I answered Denis immediately at the first post and he concurred it was a problem. I like to think the conversation evolved as we discussed. The other thing that's confusing is your use of 'wherever' and 'anywhere' when talking about where the builds take place: ec2 or rax or wherever scratch or personal chainbuilds could be built in ec2 or rax or anywhere w/o an issue ec2 or rax, I'm fine with. 'anywhere' or 'wherever', I'm not fine with. I'm guessing, now you've explained things a bit better, that by 'anywhere' or 'wherever' you really mean 'in any reasonably trustworthy cloud', but that's not what you actually _said_, nor was that meaning particularly apparent. I actually said in any cloud instance where you have a contractual (read financial) relationship. I made that pretty clear. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Thu, 2011-12-08 at 12:33 -0500, seth vidal wrote: I answered Denis immediately at the first post and he concurred it was a problem. I like to think the conversation evolved as we discussed. It's probably not worth pursuing this much further, but I'd just note that the specific sub-thread I replied to was not along these lines. It was specifically about the use of 'random VMs' to form a cloud. The very first line of quoted text in my mail is: On 12/07/2011 01:25 PM, seth vidal wrote: If I were going to use random vm's I'd want to: In general, I'd suggest that if someone 'misunderstands' what you're talking about, it's worth considering the possibility that perhaps the discussion was not as straightforward as you thought, and the 'misunderstanding' is as much on your part as the other person's, instead of firing off a mail filled with lines like I don't think you understand, I don't think you understand where the insecurity is in the system, No one has EVER seriously considered a random person's VM. ever. (clearly not accurate, if you follow the context of my post), and do you understand now?, essentially telling me I'm being an idiot and to pay closer attention. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
Date: Thu, 8 Dec 2011 12:33:38 -0500 From: seth vidal skvi...@fedoraproject.org I answered Denis immediately at the first post and he concurred it was a problem. I like to think the conversation evolved as we discussed. Yes, the conversation has shifted... Indeed, I mixed two distinct ideas: 1. One corresponding to an actual use case: using a private/trusted cloud service to do scratch, chain or even massive builds. 2. Another one corresponding to a dream, really: a P2P cloud, just because it sounds exciting, and would certainly be a USP (unique selling point) for Fedora. From the answers, it seems that both ideas ring a bell. And, of course, the dream does not seem doable, mainly for security reason. Or it could be implemented as a proof-of-concept of all the technologies and skills, which Fedora gathers under its umbrella? Nevertheless, for now, I am really interested by the item #1 above. For the #2, we never know. Maybe a GSoC, or something like that, could make it happen... Thanks for having shared your thoughts! Regards Denis -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, Dec 7, 2011 at 8:46 AM, Denis Arnaud denis.arnaud_fed...@m4x.org wrote: Hello, RedHat-hosted Koji servers offer an invaluable service by allowing all of us, package maintainers, to build all of our Fedora packages. I guess that that infrastructure is not cost-less for RedHat and and the quality of service is great (for instance, the wait in the queues, before Koji actually builds the packages submitted via the command-line client, is not so long). As Fedora is pretty advanced in the cloud/virtualisation arena, we could imagine a Koji Cloud, hosted on VMs offered by volunteers. For instance, I could contribute a few VMs in Europe (hosted on http://www.ovh.co.uk/). Our Cloud SIG (https://fedoraproject.org/wiki/Cloud_SIG) and/or Virt ML (https://admin.fedoraproject.org/mailman/listinfo/virt and https://fedoraproject.org/wiki/Getting_started_with_virtualization)/RedHat ET (http://et.redhat.com/) colleagues could help designing and implementing the following infrastructure: I believe Seth Vidal looked at auto-starting koji builder instances in the cloud a while ago. Hopefully he'll chime in. The one thing you don't address is security. If we're going to push stuff to a public cloud, how do you ensure that the builder VMs aren't compromised? josh -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On 12/07/2011 02:46 PM, Denis Arnaud wrote: Hello, RedHat-hosted Koji servers offer an invaluable service by allowing all of us, package maintainers, to build all of our Fedora packages. I guess that that infrastructure is not cost-less for RedHat and and the quality of service is great (for instance, the wait in the queues, before Koji actually builds the packages submitted via the command-line client, is not so long). As Fedora is pretty advanced in the cloud/virtualisation arena, we could imagine a Koji Cloud, hosted on VMs offered by volunteers. For instance, I could contribute a few VMs in Europe (hosted on http://www.ovh.co.uk/). Our Cloud SIG (https://fedoraproject.org/wiki/Cloud_SIG) and/or Virt ML (https://admin.fedoraproject.org/mailman/listinfo/virt and https://fedoraproject.org/wiki/Getting_started_with_virtualization)/RedHat ET (http://et.redhat.com/) colleagues could help designing and implementing the following infrastructure: * VM template/images, ready to be started on the volunteer's servers everywhere in the world, 24x7. - SSH public keys of Koji administrators would be part of the images, so that they can have an easy access to them, just in case. - Those VMs would update themselves automatically. - The containers could be standardised as well. For instance, ProxMox/OpenVZ or Fedora/CentOS with libvirt. * A directory (LDAP, or something less centralised, like the address book of Skype, for instance), keeping track of all those VMs: - with the corresponding last known status; - with the VM configurations (Fedora/CentOS release, CPU, memory, disk usage, etc); - with some rating corresponding to their quality of service (build duration, reliability of the VM, MTBF, etc). * A dispatcher system: - which would route the Koji build requests to available VMs; - collect the outcome of the builds (logs, RPM packages, statistics, QoS, etc) and store them in the current (centralised) Koji infrastructure. As I am not a specialist of all those technologies, I may have forgotten a lot of things, but you get the idea. Doesn't it sound great? Does it sound realisable? Am I crazy to dream to such an infrastructure? I'm currently writing a proposal of similar architecture for testing purposes. Looks like the core -- community provided virtual machines is the common component for all this stuff so if designed correctly it can be shared for testing/koji/whatever. I will let you know when the proposal is done so we can discuss the details. Regards, -- Richard Marko -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 7 Dec 2011 14:46:18 +0100 Denis Arnaud denis.arnaud_fed...@m4x.org wrote: Hello, RedHat-hosted Koji servers offer an invaluable service by allowing all of us, package maintainers, to build all of our Fedora packages. I guess that that infrastructure is not cost-less for RedHat and and the quality of service is great (for instance, the wait in the queues, before Koji actually builds the packages submitted via the command-line client, is not so long). As Fedora is pretty advanced in the cloud/virtualisation arena, we could imagine a Koji Cloud, hosted on VMs offered by volunteers. For instance, I could contribute a few VMs in Europe (hosted on http://www.ovh.co.uk/). Our Cloud SIG (https://fedoraproject.org/wiki/Cloud_SIG) and/or Virt ML ( https://admin.fedoraproject.org/mailman/listinfo/virt and https://fedoraproject.org/wiki/Getting_started_with_virtualization)/RedHat ET (http://et.redhat.com/) colleagues could help designing and implementing the following infrastructure: * VM template/images, ready to be started on the volunteer's servers everywhere in the world, 24x7. - SSH public keys of Koji administrators would be part of the images, so that they can have an easy access to them, just in case. - Those VMs would update themselves automatically. - The containers could be standardised as well. For instance, ProxMox/OpenVZ or Fedora/CentOS with libvirt. * A directory (LDAP, or something less centralised, like the address book of Skype, for instance), keeping track of all those VMs: - with the corresponding last known status; - with the VM configurations (Fedora/CentOS release, CPU, memory, disk usage, etc); - with some rating corresponding to their quality of service (build duration, reliability of the VM, MTBF, etc). * A dispatcher system: - which would route the Koji build requests to available VMs; - collect the outcome of the builds (logs, RPM packages, statistics, QoS, etc) and store them in the current (centralised) Koji infrastructure. As I am not a specialist of all those technologies, I may have forgotten a lot of things, but you get the idea. Doesn't it sound great? Does it sound realisable? Am I crazy to dream to such an infrastructure? I've looked into spawning virt instances to do building and it is pretty doable. The problem with them being offered by volunteers is trust: 1. how do we trust the initial installation hasn't been poisoned unless we ship all the bits over ourselves. 2. how do we trust the in-flight build isn't molested 3. how do the people providing the trust insure against tainted/dangerous builds doing $bad_things on their systems. this is why I concluded that the idea of donated/volunteered VM was not going to work - additionally b/c the bandwidth requirements are non-trivial for many builds. However, building on environments where we have a contractual (read: financial) relationship will work better and where the remote end has protected themselves against attacks from the VMs. I'm speaking of cloud hosting providers like amazon ec2 and rackspace cloud servers. I've worked on some code to spawn off an instance, submit jobs + packages, build them (a chain-build so you don't have to keep respawning them) then collect all the results back to your local machine. It works - it requires setting up trusted images at those cloud providers but that's not very hard to do and keep current. Right now I'm porting the code to use a different cloud-communication API than I was using before. The problems still persist with bandwidth consumption and to some extent with trust but trust is mitigated b/c the relationship with the provider is more standardized and less haphazard. I have a couple of systems inside the red hat colo that I had planned on reinstalling to f16 and setting up openstack on them to play with the same idea but on a local cloud instance. Is all this inline with the problems you've thought about? -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
2011/12/7 seth vidal skvi...@fedoraproject.org I've looked into spawning virt instances to do building and it is pretty doable. The problem with them being offered by volunteers is trust [...] You are right. I had not thought at that... how naive of me :( The volunteers/trustees would sign the builds with their own private keys, for instance with their FAS keys. Then, we could have some trustworthiness ratings for all the submitters, like we have today for the packagers (new packager, proven-packager, sponsor). While the submitter is still not trusted, the centralised Koji infrastructure can duplicate the build, and check that it gives the same results. And even when the submitter is trusted, some random duplicate builds can occur. If the submitter taints the builds, it will be flagged as a potential fraud. A human being would have to have a look at it then. Or, the VMs could do scratch builds (only). When those builds are successful, the VMs then just act as a standard clients to the central Koji servers, and the packages are re-built in that safe infrastructure. Overall, the central Koji infrastructure would be off-loaded from all the scratch builds, as well as from the failed builds. Which is already not so bad, is it? I've worked on some code to spawn off an instance, submit jobs + packages, build them (a chain-build so you don't have to keep respawning them) then collect all the results back to your local machine. It works - it requires setting up trusted images at those cloud providers but that's not very hard to do and keep current. Right now I'm porting the code to use a different cloud-communication API than I was using before. That would be very cool. Do you intend to use DeltaCloud ( http://deltacloud.apache.org/), or something like that? I have a couple of systems inside the red hat colo that I had planned on reinstalling to f16 and setting up openstack on them to play with the same idea but on a local cloud instance. For sure, I would like to set up something like that for my own usage. Is all this inline with the problems you've thought about? Yes, that is fully in-line, and very interesting! Denis PS: why isn't there a virtualisation SIG? As there is already a mailing list, it may be just a question of adding the corresponding Wiki page? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 7 Dec 2011 18:31:27 +0100 Denis Arnaud denis.arnaud_fed...@m4x.org wrote: 2011/12/7 seth vidal skvi...@fedoraproject.org I've looked into spawning virt instances to do building and it is pretty doable. The problem with them being offered by volunteers is trust [...] You are right. I had not thought at that... how naive of me :( The volunteers/trustees would sign the builds with their own private keys, for instance with their FAS keys. Then, we could have some trustworthiness ratings for all the submitters, like we have today for the packagers (new packager, proven-packager, sponsor). While the submitter is still not trusted, the centralised Koji infrastructure can duplicate the build, and check that it gives the same results. And even when the submitter is trusted, some random duplicate builds can occur. If the submitter taints the builds, it will be flagged as a potential fraud. A human being would have to have a look at it then. Well it's beyond that - we still can't trust the build host isn't compromised in such a way to also add trojans to a mock chroot we then build inside of. In short, if people are volunteering VMs for us - we have to be incredibly suspicious of those VMs and from what they are built. If I were going to use random vm's I'd want to: 1. connect using ssh 2. push over my own rpm/python/etc binaries 3. checksum all the rest of the installed (and running) software 4. verify those checksums versus my known good set 5. THEN push over the pkgs to be built If we run this on ec2 or cloudservers then we can build up our own image and use that for the initial trusted environment. ec2 has the benefit of letting you share an image with others and from there we can build. my goal is a very short stack of code such that anyone can run it to build pkgs - if only b/c they don't want to run mock locally b/c of limited local bandwidth or cpu time. Or, the VMs could do scratch builds (only). When those builds are successful, the VMs then just act as a standard clients to the central Koji servers, and the packages are re-built in that safe infrastructure. Overall, the central Koji infrastructure would be off-loaded from all the scratch builds, as well as from the failed builds. Which is already not so bad, is it? I think a reasonable goal is: - make it trivial to build pkgs in a remote cloud instance from arbitrary yum repository sources - implement the requirements to make this trustworthy enough for koji to do it, too. That would be very cool. Do you intend to use DeltaCloud ( http://deltacloud.apache.org/), or something like that? I'm using libcloud, actually. I'm interested in pursuing this in python, not ruby. Yes, that is fully in-line, and very interesting! PS: why isn't there a virtualisation SIG? As there is already a mailing list, it may be just a question of adding the corresponding Wiki page? I thought there was a cloud sig or something. To be honest the sigs are mind-numbingly confusing to me as to 1. what they do and 2. which ones actually still exist/do something. So I tend to not pay much attention to them. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 07 Dec 2011 13:35:03 -0500 Mo Morsi mmo...@redhat.com wrote: On 12/07/2011 01:25 PM, seth vidal wrote: That would be very cool. Do you intend to use DeltaCloud ( http://deltacloud.apache.org/), or something like that? I'm using libcloud, actually. I'm interested in pursuing this in python, not ruby. Deltacloud's primary interface is REST based so you can use it from any language. Additionally there are python specific bindings to the REST interface that are pretty simple to use. https://github.com/mifo/deltacloud-python The infrastructure to deal with deltacloud is much higher than just using libcloud. Having to have a daemon running just to do communication is, imo, not acceptable for something being spawned off like this. deltacloudd makes deltacloud a non-starter for me. In any case, another nifty idea would be to use Snap to take snapshots of the build environments on the cloud for local replication on different Koji systems and/or local vms: https://github.com/movitto/snap sapping them is going to involve a lot of bandwidth which costs money. Since we have a mirror inside the amazon ec2 network space snapping is likely to be MORE expensive than just installing from there. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
Le mercredi 07 décembre 2011 à 10:36 -0500, seth vidal a écrit : I've looked into spawning virt instances to do building and it is pretty doable. The problem with them being offered by volunteers is trust: 1. how do we trust the initial installation hasn't been poisoned unless we ship all the bits over ourselves. 2. how do we trust the in-flight build isn't molested 3. how do the people providing the trust insure against tainted/dangerous builds doing $bad_things on their systems. this is why I concluded that the idea of donated/volunteered VM was not going to work - additionally b/c the bandwidth requirements are non-trivial for many builds. Concerning trust, the classic way it has been solved before (by seti…) is to farm the same build to several independant nodes, cheksum results and make sure they all agree Of course that supposes builds are strictly reproductible (centos folks would love this) and that makes the system a lot less efficient. But then, trust has a price too -- Nicolas Mailhot -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On 12/07/2011 01:40 PM, seth vidal wrote: On Wed, 07 Dec 2011 13:35:03 -0500 Mo Morsimmo...@redhat.com wrote: On 12/07/2011 01:25 PM, seth vidal wrote: That would be very cool. Do you intend to use DeltaCloud ( http://deltacloud.apache.org/), or something like that? I'm using libcloud, actually. I'm interested in pursuing this in python, not ruby. Deltacloud's primary interface is REST based so you can use it from any language. Additionally there are python specific bindings to the REST interface that are pretty simple to use. https://github.com/mifo/deltacloud-python The infrastructure to deal with deltacloud is much higher than just using libcloud. Having to have a daemon running just to do communication is, imo, not acceptable for something being spawned off like this. deltacloudd makes deltacloud a non-starter for me. Yes this is being worked on. It'll be soon possible to run deltacloud like any other library, in process. Plus with the added benefit that you're using a project whose developers are highly involved with the Fedora community. In any case, another nifty idea would be to use Snap to take snapshots of the build environments on the cloud for local replication on different Koji systems and/or local vms: https://github.com/movitto/snap sapping them is going to involve a lot of bandwidth which costs money. Since we have a mirror inside the amazon ec2 network space snapping is likely to be MORE expensive than just installing from there. -sv Not really, the snapshots are extremely optimized. Only the package list is saved plus any files modified outside of the package system. Plus you can restrict what exactly is backed up, only backing up your build root for example. -Mo -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On 12/07/2011 01:25 PM, seth vidal wrote: That would be very cool. Do you intend to use DeltaCloud ( http://deltacloud.apache.org/), or something like that? I'm using libcloud, actually. I'm interested in pursuing this in python, not ruby. Deltacloud's primary interface is REST based so you can use it from any language. Additionally there are python specific bindings to the REST interface that are pretty simple to use. https://github.com/mifo/deltacloud-python In any case, another nifty idea would be to use Snap to take snapshots of the build environments on the cloud for local replication on different Koji systems and/or local vms: https://github.com/movitto/snap -Mo -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Thu, 08 Dec 2011 04:34:57 +0900
夜神 岩男 supergiantpot...@yahoo.co.jp wrote:
An idea just struck me that may work.
If the system is made light enough that it is utterly painless for
anyone to contribute processing time then cross-checking of hashes
could be made statistically secure, save for a widespread compromise
of the entire Fedora userbase.
For example, if I just yum install skyline[1] and then set
chkconfig skyline on or whatever the systemd equivalent is (sorry,
haven't kept up lately) and my system just started pulling packages
to build/rebuild constantly in the background with a priority level
low enough I'd never notice it, then overnight Fedora as a project
would have FAR more build time than it needs.
So how is this secure?
Each time a build is made, the building system makes a hash of the
set of RPMs in $wherever/mock/result/{foo,bar} and sends the
completed data back to the Fedora build system.
Now the Fedora build system checks the hash to make sure it is
correct. Of course it is, because we've only got one sample.
But then we collect the other builds of the same RPM from, say, 10
other random systems and compare their hashes to what was received
from the initial build system.
A has that doesn't match whatever arises as the common hash is either
an error (likely considering the amount of transmission involved) or
poisoned. These will stick out like a sore thumb amongst the forest
of correct builds.
So how would I compromise this system? I would have to poison every
single SRPM departing the Fedora build infrastructure. Hard, maybe,
but possible.
How do we prevent that? Use SSL to transfer the packages in the first
place, and now that avenue is not available in the time allotted for
the attack to proceed.
This system depends chiefly on one thing: Having a boatload of
contributors. I think we could easily expect 1,000+ active
contributors, particularly if we make this a happy competition
complete with a stats tracker the way the BOINC and SETI@Home
trackers work. People who send more than 1 bad hash could be notified
by their system that things aren't working out which could be an
early warning in identifying whether the system is indeed being
attacked (or the individual's system has been compromised). A
throttle based on such indicators could let us know to halt the
distributed build and switch back to old koji.
Just some ideas.
1. Sorry, silly package name idea from the image of city construction
koji + cloud
Bandwidth is the big concern for the end user here and then the other
issue is - is all of this worth it for building pkgs? I don't think it
is, personally, pkg building is not that huge of a hit, afaict to
getting things done.
I mean the sum total of the steps were talking about, even now is more
or less:
1. init host
2. stuff some files onto it
3. start up a process
4. communicate to that process
5. build pkg
6. stuff pkgs into a local repo
7. go to 5 until no more pkgs
8. download all pkgs back to original client
9. destroy host.
you can do it now if you're willing to do steps 5-8 manually.
-sv
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 07 Dec 2011 15:02:42 -0500 Przemek Klosowski przemek.klosow...@nist.gov wrote: On 12/07/2011 01:25 PM, seth vidal wrote: If I were going to use random vm's I'd want to: 1. connect using ssh 2. push over my own rpm/python/etc binaries 3. checksum all the rest of the installed (and running) software 4. verify those checksums versus my known good set 5. THEN push over the pkgs to be built I'd say we need to be paranoid on this one and consider a tainted kernel where your own binaries would report A-OK on a rigged gcc because kernel has a special case for it. Test builds and QA might be OK, but nothing that results in shipped bits. So I have two thoughts on this: 1. scratch or personal chainbuilds could be built in ec2 or rax or anywhere w/o an issue 2. for our shipping pkgs building them in the existing koji buildsystems and/or in a dedicated private cloud instance makes sense - if only for resource allocation and control. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Thu, 08 Dec 2011 05:35:02 +0900 夜神 岩男 supergiantpot...@yahoo.co.jp wrote: On 12/08/2011 05:12 AM, seth vidal wrote: Bandwidth is the big concern for the end user here and then the other issue is - is all of this worth it for building pkgs? I don't think it is, personally, pkg building is not that huge of a hit, afaict to getting things done. I mean the sum total of the steps were talking about, even now is more or less: 1. init host 2. stuff some files onto it 3. start up a process 4. communicate to that process 5. build pkg 6. stuff pkgs into a local repo 7. go to 5 until no more pkgs 8. download all pkgs back to original client 9. destroy host. you can do it now if you're willing to do steps 5-8 manually. I think you are correct in essentially asking if this is a solution in search of a problem. And on the practical side... This all goes back to the first sentence in this response. While it is very possible to do something like this, and the idea is exciting because it is something new, I've never heard of anyone kicking and screaming about a wait que bottleneck or insufficient resources in the Fedora infrastructure. I've also not heard anything like RH is going to drop Fedora so we need to look for a new home, either. I was simply addressing the how to make it secure element. This is a workable method which relies 100% on volunteers (i.e. community resource use, not a paid solution) VS going to some overwrought cloud solution for which there really isn't any backstop for integrity checking compared to the distributed build crosscheck I've described above. So - there is no shortage of resources for building stuff. There is no threat of anyone dropping anything. There is however the following: We want to be able to build pkgs from people we do not trust and using sources of BuildRequirements we do not trust. We cannot do that on our existing infrastructure b/c it is FAR too trivial for a malicious buildrequirement to walk its way out of a chroot. that's the driving force. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On 12/08/2011 05:12 AM, seth vidal wrote: Bandwidth is the big concern for the end user here and then the other issue is - is all of this worth it for building pkgs? I don't think it is, personally, pkg building is not that huge of a hit, afaict to getting things done. I mean the sum total of the steps were talking about, even now is more or less: 1. init host 2. stuff some files onto it 3. start up a process 4. communicate to that process 5. build pkg 6. stuff pkgs into a local repo 7. go to 5 until no more pkgs 8. download all pkgs back to original client 9. destroy host. you can do it now if you're willing to do steps 5-8 manually. I think you are correct in essentially asking if this is a solution in search of a problem. But on the technical side... I don't think bandwidth would be much of an issue, for one thing it could be throttleable and with a huge number of available systems this still results in an overabundance of available systems for build. And anyway most Fedora users have no problem sucking down a DVD-sized spin each time they want to try something other than the default desktop. If this were implemented the entire cycle should be per package or perhaps per some limit of packages that is user-configurable (like do blocks of 10 rpms or do blocks of packages 500Mb 1Gb). With parceled building like that the iterations are smaller but more frequent, and I think less of a burden for someone who just wants to try things out. And on the practical side... This all goes back to the first sentence in this response. While it is very possible to do something like this, and the idea is exciting because it is something new, I've never heard of anyone kicking and screaming about a wait que bottleneck or insufficient resources in the Fedora infrastructure. I've also not heard anything like RH is going to drop Fedora so we need to look for a new home, either. I was simply addressing the how to make it secure element. This is a workable method which relies 100% on volunteers (i.e. community resource use, not a paid solution) VS going to some overwrought cloud solution for which there really isn't any backstop for integrity checking compared to the distributed build crosscheck I've described above. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
An idea just struck me that may work.
If the system is made light enough that it is utterly painless for
anyone to contribute processing time then cross-checking of hashes could
be made statistically secure, save for a widespread compromise of the
entire Fedora userbase.
For example, if I just yum install skyline[1] and then set chkconfig
skyline on or whatever the systemd equivalent is (sorry, haven't kept
up lately) and my system just started pulling packages to build/rebuild
constantly in the background with a priority level low enough I'd never
notice it, then overnight Fedora as a project would have FAR more build
time than it needs.
So how is this secure?
Each time a build is made, the building system makes a hash of the set
of RPMs in $wherever/mock/result/{foo,bar} and sends the completed data
back to the Fedora build system.
Now the Fedora build system checks the hash to make sure it is correct.
Of course it is, because we've only got one sample.
But then we collect the other builds of the same RPM from, say, 10 other
random systems and compare their hashes to what was received from the
initial build system.
A has that doesn't match whatever arises as the common hash is either an
error (likely considering the amount of transmission involved) or
poisoned. These will stick out like a sore thumb amongst the forest of
correct builds.
So how would I compromise this system? I would have to poison every
single SRPM departing the Fedora build infrastructure. Hard, maybe, but
possible.
How do we prevent that? Use SSL to transfer the packages in the first
place, and now that avenue is not available in the time allotted for the
attack to proceed.
This system depends chiefly on one thing: Having a boatload of
contributors. I think we could easily expect 1,000+ active contributors,
particularly if we make this a happy competition complete with a stats
tracker the way the BOINC and SETI@Home trackers work. People who send
more than 1 bad hash could be notified by their system that things
aren't working out which could be an early warning in identifying
whether the system is indeed being attacked (or the individual's system
has been compromised). A throttle based on such indicators could let us
know to halt the distributed build and switch back to old koji.
Just some ideas.
1. Sorry, silly package name idea from the image of city construction
koji + cloud
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On 12/07/2011 01:25 PM, seth vidal wrote: If I were going to use random vm's I'd want to: 1. connect using ssh 2. push over my own rpm/python/etc binaries 3. checksum all the rest of the installed (and running) software 4. verify those checksums versus my known good set 5. THEN push over the pkgs to be built I'd say we need to be paranoid on this one and consider a tainted kernel where your own binaries would report A-OK on a rigged gcc because kernel has a special case for it. Test builds and QA might be OK, but nothing that results in shipped bits. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
2011/12/7 Nicolas Mailhot nicolas.mail...@laposte.net Concerning trust, the classic way it has been solved before (by seti…) is to farm the same build to several independant nodes, cheksum results and make sure they all agree Again, we could use that P2P build system just to alleviate the centralised Koji servers from the scratch builds, on one hand, and from the failing builds on the other hand. That way, the centralised Koji servers would need no alteration. As a side note, rather than using Snap (and Augeas, and...), we (in my department) tend to prefer Chef (http://www.opscode.com/chef/), which has got a broader scope, and allows much more complex configurations and automation tasks. Denis -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 2011-12-07 at 18:12 -0500, seth vidal wrote: On Wed, 07 Dec 2011 13:25:28 -0800 Adam Williamson awill...@redhat.com wrote: I'm not sure we can treat scratch / personal builds with *quite* so much abandon. They're still valuable targets for anyone trying to compromise Fedora, after all. I don't think you understand - we need to be able to reliably reproduce them- sure - but we cannot count on them anymore than we do now. Ie: someone sends an arbitrary pkg with arbitrary repos to supply its buildreqs - we cannot trust the pkg at all. That's ALWAYS true. but we definitely cannot allow the above to build on our existing build boxes. Who uses scratch builds the most? Well, probably Fedora packagers, right? And we probably wind up deploying them on our own systems after we build them. That's what scratch builds are _for_ - testing your stuff before pushing it out more widely. And again - if you are testing your own pkgs - you'll be fine - there's no insecurity there. You trust you. and the trust of the images you're building from is up to which cloud service provider you have a contractual relationship with. So it occurs to me that if we have a hilariously insecure system for doing scratch builds, and someone really wants to do evil things to Fedora, it's going to make their lives a lot easier. I don't think you understand where the insecurity is in the system. All they have to do is compromise a provenpackager's scratch build to include some kind of trojan, then when the provenpackager installs the scratch build they just fired off, hey presto, the attacker has now effectively gained provenpackager privileges. They can just hack into the provenpackager's system using the back door they just trojaned in there and go about making their nefarious changes to Fedora just as if they were the trusted packager; they don't need to attack 'important' builds in-flight any more. Let's put it this way - if we put such a system in place I'd damn well be doing my scratch builds locally from then on. I wouldn't trust them to Joe Q. Random's VM. No one has EVER seriously considered a random person's VM. ever. but I do think a vm you create at ec2 or rax or wherever is just fine. b/c YOU create it with a known good/trusted img as the base. do you understand now? Well, yes, but only because you shifted the entire terms of the thread without telling anyone else. All of the above - about how the idea was to build packages with untrusted build dependencies in trustworthy places - may have been perfectly clear _to you_, but it was not the idea that started this thread. This thread was specifically about creating a 'cloud', containing random contributor's machines, for doing builds, and several people suggested that might be 'safe' for scratch builds but not for 'release' builds. You seem to have changed the terms of debate rather considerably with your _own_ idea, which you introduced into the thread, but which clearly was not actually very similar at all to the idea of the person who actually started the thread, Denis Arnaud. The extent of the difference between what _he_ was thinking/talking about and what _you_ are thinking/talking about has only been made clear in this last post of yours. The other thing that's confusing is your use of 'wherever' and 'anywhere' when talking about where the builds take place: ec2 or rax or wherever scratch or personal chainbuilds could be built in ec2 or rax or anywhere w/o an issue ec2 or rax, I'm fine with. 'anywhere' or 'wherever', I'm not fine with. I'm guessing, now you've explained things a bit better, that by 'anywhere' or 'wherever' you really mean 'in any reasonably trustworthy cloud', but that's not what you actually _said_, nor was that meaning particularly apparent. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 07 Dec 2011 13:25:28 -0800 Adam Williamson awill...@redhat.com wrote: I'm not sure we can treat scratch / personal builds with *quite* so much abandon. They're still valuable targets for anyone trying to compromise Fedora, after all. I don't think you understand - we need to be able to reliably reproduce them- sure - but we cannot count on them anymore than we do now. Ie: someone sends an arbitrary pkg with arbitrary repos to supply its buildreqs - we cannot trust the pkg at all. That's ALWAYS true. but we definitely cannot allow the above to build on our existing build boxes. Who uses scratch builds the most? Well, probably Fedora packagers, right? And we probably wind up deploying them on our own systems after we build them. That's what scratch builds are _for_ - testing your stuff before pushing it out more widely. And again - if you are testing your own pkgs - you'll be fine - there's no insecurity there. You trust you. and the trust of the images you're building from is up to which cloud service provider you have a contractual relationship with. So it occurs to me that if we have a hilariously insecure system for doing scratch builds, and someone really wants to do evil things to Fedora, it's going to make their lives a lot easier. I don't think you understand where the insecurity is in the system. All they have to do is compromise a provenpackager's scratch build to include some kind of trojan, then when the provenpackager installs the scratch build they just fired off, hey presto, the attacker has now effectively gained provenpackager privileges. They can just hack into the provenpackager's system using the back door they just trojaned in there and go about making their nefarious changes to Fedora just as if they were the trusted packager; they don't need to attack 'important' builds in-flight any more. Let's put it this way - if we put such a system in place I'd damn well be doing my scratch builds locally from then on. I wouldn't trust them to Joe Q. Random's VM. No one has EVER seriously considered a random person's VM. ever. but I do think a vm you create at ec2 or rax or wherever is just fine. b/c YOU create it with a known good/trusted img as the base. do you understand now? -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, 2011-12-07 at 16:15 -0500, seth vidal wrote: On Wed, 07 Dec 2011 15:02:42 -0500 Przemek Klosowski przemek.klosow...@nist.gov wrote: On 12/07/2011 01:25 PM, seth vidal wrote: If I were going to use random vm's I'd want to: 1. connect using ssh 2. push over my own rpm/python/etc binaries 3. checksum all the rest of the installed (and running) software 4. verify those checksums versus my known good set 5. THEN push over the pkgs to be built I'd say we need to be paranoid on this one and consider a tainted kernel where your own binaries would report A-OK on a rigged gcc because kernel has a special case for it. Test builds and QA might be OK, but nothing that results in shipped bits. So I have two thoughts on this: 1. scratch or personal chainbuilds could be built in ec2 or rax or anywhere w/o an issue I'm not sure we can treat scratch / personal builds with *quite* so much abandon. They're still valuable targets for anyone trying to compromise Fedora, after all. Who uses scratch builds the most? Well, probably Fedora packagers, right? And we probably wind up deploying them on our own systems after we build them. That's what scratch builds are _for_ - testing your stuff before pushing it out more widely. So it occurs to me that if we have a hilariously insecure system for doing scratch builds, and someone really wants to do evil things to Fedora, it's going to make their lives a lot easier. All they have to do is compromise a provenpackager's scratch build to include some kind of trojan, then when the provenpackager installs the scratch build they just fired off, hey presto, the attacker has now effectively gained provenpackager privileges. They can just hack into the provenpackager's system using the back door they just trojaned in there and go about making their nefarious changes to Fedora just as if they were the trusted packager; they don't need to attack 'important' builds in-flight any more. Let's put it this way - if we put such a system in place I'd damn well be doing my scratch builds locally from then on. I wouldn't trust them to Joe Q. Random's VM. Yes, there are of course many other vectors an evil person could use right now to try and attack Fedora via this indirect method, but I see no pressing reason to make it any easier. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
Date: Wed, 07 Dec 2011 16:01:06 +0100 From: Richard Marko rma...@redhat.com I'm currently writing a proposal of similar architecture for testing purposes. Looks like the core -- community provided virtual machines is the common component for all this stuff so if designed correctly it can be shared for testing/koji/whatever. I will let you know when the proposal is done so we can discuss the details. That would be great, thank you! Regards Denis -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, Dec 7, 2011 at 11:12 AM, seth vidal skvi...@fedoraproject.org wrote: Bandwidth is the big concern for the end user here and then the other issue is - is all of this worth it for building pkgs? I don't think it is, personally, pkg building is not that huge of a hit, afaict to getting things done. +1 as a contributing packager. I really don't want to wait on a network of residential band limited donated cpu cycles (similar to my own) to do a test build and sanity check. If I could wait for that I'd just do it locally. I'd also like to see if we can use utility cloud resources to efficiently scale out a mass rebuild on demand and perform self-hosting test runs or failure to build runs on a regular scheduled basis. Something we can do in a well managed utility cloud I would think without slowing down day to day packaging work. -jefKeeping rawhide synced locally using a combination of packet carrying pneumatic tubes and sleddog teams is hardspaleta -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: P2P Packaging/Koji Cloud
On Wed, Dec 07, 2011 at 03:25:18PM -0800, Adam Williamson wrote: On Wed, 2011-12-07 at 18:12 -0500, seth vidal wrote: On Wed, 07 Dec 2011 13:25:28 -0800 Adam Williamson awill...@redhat.com wrote: I'm not sure we can treat scratch / personal builds with *quite* so much abandon. They're still valuable targets for anyone trying to compromise Fedora, after all. I don't think you understand - we need to be able to reliably reproduce them- sure - but we cannot count on them anymore than we do now. Ie: someone sends an arbitrary pkg with arbitrary repos to supply its buildreqs - we cannot trust the pkg at all. That's ALWAYS true. but we definitely cannot allow the above to build on our existing build boxes. Who uses scratch builds the most? Well, probably Fedora packagers, right? And we probably wind up deploying them on our own systems after we build them. That's what scratch builds are _for_ - testing your stuff before pushing it out more widely. And again - if you are testing your own pkgs - you'll be fine - there's no insecurity there. You trust you. and the trust of the images you're building from is up to which cloud service provider you have a contractual relationship with. So it occurs to me that if we have a hilariously insecure system for doing scratch builds, and someone really wants to do evil things to Fedora, it's going to make their lives a lot easier. I don't think you understand where the insecurity is in the system. All they have to do is compromise a provenpackager's scratch build to include some kind of trojan, then when the provenpackager installs the scratch build they just fired off, hey presto, the attacker has now effectively gained provenpackager privileges. They can just hack into the provenpackager's system using the back door they just trojaned in there and go about making their nefarious changes to Fedora just as if they were the trusted packager; they don't need to attack 'important' builds in-flight any more. Let's put it this way - if we put such a system in place I'd damn well be doing my scratch builds locally from then on. I wouldn't trust them to Joe Q. Random's VM. No one has EVER seriously considered a random person's VM. ever. but I do think a vm you create at ec2 or rax or wherever is just fine. b/c YOU create it with a known good/trusted img as the base. do you understand now? Well, yes, but only because you shifted the entire terms of the thread without telling anyone else. All of the above - about how the idea was to build packages with untrusted build dependencies in trustworthy places - may have been perfectly clear _to you_, but it was not the idea that started this thread. This thread was specifically about creating a 'cloud', containing random contributor's machines, for doing builds, and several people suggested that might be 'safe' for scratch builds but not for 'release' builds. To be fair, if you read the thread again, I think you'll see that Seth *is* consistently arguing against using random contributor's machines and for building on trusted clouds. And as the other major participant in the thread (the original poster being the first; seth being the other due to having replied to pretty much all branches of the thread) he forms the counterweight to the original position. Perhaps it's just that I work with Seth and so I read threads where he participates with an eye towards what he says that that seemed clear to me so take it for what it's worth but that's how I've perceived the thread so far. -Toshio pgpT0AcScaXcz.pgp Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
