Re: Google Summer of Code 2014 question
Hi John, On Wed, Mar 19, 2014 at 2:36 PM, John Matty jcm...@pitt.edu wrote: Hello everybody, My name is John, and I was interested in doing Google Summer of Code for FreeBSD this Summer. I am currently interested in writing a virtual machine execution language. A rough draft of my proposal can be found on the website for GSoC 2014 (I am not exactly sure how to get to it from the outside though, sorry), however the brief description of my project is to specify and implement a domain specific language to start a virtual machine, execute given commands, and store the results of the given command. Then, using a text comparison tool (maybe something similar to `diff'), compare the obtained output to the expected output. I was currently thinking that this would mainly be used for doing repeated testing of software on FreeBSD running in a virtual machine, however I was interested to see if anyone had any ideas for other uses or applications for this language so that I could focus on making it as useful as possible for what people would actually be using it for, or if anyone had any other comments or suggestions? I was also wondering if anyone would be interested in mentoring me for my project this summer? Finally, as I am new to this mailing list, if this is not the correct place to ask this question or anything else like that please let me know so I can move my question to the correct place. Thank you! I could certainly make use of this in bhyve development. For e.g., whenever we make changes to bhyve there is a large test matrix that needs to be addressed: - various guest OSes: FreeBSD (various releases), Linux (various flavors), OpenBSD etc. - 32-bit or 64-bit virtual machine - number of vcpus and guest memory sizes - multiple device emulations (virtio, ahci, passthru) - timer modes (PIT, LAPIC, HPET, ACPI) - interrupt delivery (8259, legacy, MSI, MSI-X) - shutdown types (reboot, halt, acpi power off) A tool like the one you are proposing would be very useful to catch regressions. best Neel --John ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Google Summer of Code 2014 question
On Thu, Mar 20, 2014 at 8:24 AM, John Baldwin j...@freebsd.org wrote: This isn't a bad place to ask this question. I think a person you might want to talk to is Craig Rodrigues (cc'd) who has recently setup Jenkins with the goal of doing automated testing using virtual machines. I'm not sure exactly what design he is considering, but he is the person to ask. Last year I signed up to be a GSoC mentor, but this year I unfortunately do not have time, so haven't done so. Unfortunately, since I did not sign up as a mentor, I can't view your proposal on the google-melange web site. If you look at the following link, you can see the March 13 presentation which describes some work I did setting up Jenkins and BHyve for continuous integration and testing on FreeBSD: https://wiki.freebsd.org/Jenkins#Presentations_and_Working_Groups If you look at this link, item 22: https://wiki.freebsd.org/Jenkins#Jenkins_for_FreeBSD_status you will see that we are trying to work at integrating Jenkins with the Kyua framework, so that Jenkins can run Kyua, and report the results in the Jenkins web UI. Jenkins is used by hundreds of companies, not just in the FreeBSD world. Kyua is gaining popularity in the FreeBSD community, and is already used by major companies (EMC/Isilon and Spectralogic are the ones that I know of). There are already many systems out there for doing automation, starting/stopping VM's, and running tests. I'm focusing on integrating as many existing technologies and getting them to run under FreeBSD to put together a more modern continuous integration and testing pipeline for the FreeBSD project. I cannot comment on the specifics your proposal without reading it. However, I don't see a lot of value in coming up with a new Domain Specific Language for starting/stopping VM's and sending commands to VM's. There are already many people doing this already with other frameworks and automation API's. For example, libvirt ( http://libvirt.org ) is one popular API specifically for controlling VM's. In terms of frameworks that can send remote commands to VM's, there are already plenty of automation systems to choose from: Puppet, Chef, SaltStack, CFEngine are some examples. Creating a new Domain Specific Language for VM control sounds like a more academic/research exercise, so if you can get someone to mentor you for such a project, maybe you can target the results of your work for publication, if that is of interest to you. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Google Summer of Code 2014 question
On Thu, Mar 20, 2014 at 01:54:53PM -0700, Craig Rodrigues wrote:
On Thu, Mar 20, 2014 at 8:24 AM, John Baldwin j...@freebsd.org wrote:
This isn't a bad place to ask this question. I think a person you might
want to talk to is Craig Rodrigues (cc'd) who has recently setup Jenkins
with the goal of doing automated testing using virtual machines. I'm not
sure exactly what design he is considering, but he is the person to ask.
Last year I signed up to be a GSoC mentor, but this year I unfortunately
do not have time, so haven't done so. Unfortunately, since I did not
sign up as a mentor, I can't view your proposal on the google-melange web
site.
If you look at the following link, you can see the March 13
presentation which describes
some work I did setting up Jenkins and BHyve for continuous
integration and testing on FreeBSD:
https://wiki.freebsd.org/Jenkins#Presentations_and_Working_Groups
If you look at this link, item 22:
https://wiki.freebsd.org/Jenkins#Jenkins_for_FreeBSD_status
you will see that we are trying to work at integrating Jenkins with the Kyua
framework, so that Jenkins can run Kyua, and report the results in the
Jenkins web UI.
Jenkins is used by hundreds of companies, not just in the FreeBSD world.
Kyua is gaining popularity in the FreeBSD community, and is already used by
major companies (EMC/Isilon and Spectralogic are the ones that I know of).
There are already many systems out there for doing automation,
starting/stopping VM's,
and running tests. I'm focusing on integrating as many existing
technologies and
getting them to run under FreeBSD to put together a more modern
continuous integration and testing
pipeline for the FreeBSD project.
I cannot comment on the specifics your proposal without reading it.
However, I don't see a lot of value in coming up with a new Domain
Specific Language
for starting/stopping VM's and sending commands to VM's. There are
already many people
doing this already with other frameworks and automation API's. For
example, libvirt ( http://libvirt.org )
is one popular API specifically for controlling VM's.
In terms of frameworks that can send remote commands to VM's, there
are already plenty
of automation systems to choose from: Puppet, Chef, SaltStack,
CFEngine are some examples.
Creating a new Domain Specific Language for VM control sounds like a
more academic/research exercise,
so if you can get someone to mentor you for such a project, maybe you
can target the results
of your work for publication, if that is of interest to you.
John, Craig,
When I initially proposed the idea:
https://wiki.freebsd.org/SummerOfCode2014#TEST-o-steron_for_FreeBSD_.28node.js.2FLua.29
it was meant to be mainly a regression testing tool for FreeBSD. Something
that would let me build and run continuous builds for FreeBSD ARM (ZEDBOARD
kernel), and then make a SD image out of it, which I can boot in QEMU/real
hardware.
It would be nice if we payed attention to BHyVE in this project, but not
limit ourselves to it. E.g.: ARM testing was my original drive towards
proposing this.
For such a system I'd like to be able to recognize what is being printed on
UART at the boot time. Then register the output in a file and later compare
it. And I'd like enough expressiveness to say: if there's no login screen
after 30s of bot time, fail and let me know what the screen content at the
time of failure was.
DSL isn't a must here, I agree, but you'll be limited in testing otherwise.
Unless some technology already has what I want, I suggested getting this
functionality:
vm0 = vm.start(vm0);
vm1 = vm.start(vm1);
vm0.power(1);
vm1.power(1);
vm0.login();
vm1.login();
vm0.cmd(ifconfig .);
vm1.cmd(ifconfig );
vm0.cmd(ping vm1);
vm0.register(function(data) {
if (data =~ /*Not*/) {
error();
}
});
Once this would work for VMs, I'd be interesting in proposing the creation
of the physical infrastructure for continuous FreeBSD testing. We could
start this from Atom-based PC and Raspberry PI netbooting latest FreeBSD.
If John sees it in a similar way and he'd like to explore this path, I'd be
very interested in mentoring this idea. But once again: only if there isn't
something of a similar functionality.
Thanks,
--
Wojciech A. Koszek
wkos...@freebsd.czest.pl
http://FreeBSD.czest.pl/~wkoszek/
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: Google Summer of Code 2014 question
On Thu, Mar 20, 2014 at 2:38 PM, Wojciech A. Koszek wkos...@freebsd.org wrote:
DSL isn't a must here, I agree, but you'll be limited in testing otherwise.
Unless some technology already has what I want, I suggested getting this
functionality:
vm0.cmd(ifconfig .);
vm1.cmd(ifconfig );
vm0.cmd(ping vm1);
vm0.register(function(data) {
if (data =~ /*Not*/) {
error();
}
});
For the above pseudo-code which you listed,
you've pretty much got that functionality in
existing automation tools like Puppet, SaltStack, Chef, etc.
For unit testing, Kyua is quite good, and people are writing lots of unit
tests for it, and even committing some of them directly to FreeBSD.
Once this would work for VMs, I'd be interesting in proposing the creation
of the physical infrastructure for continuous FreeBSD testing.
For the above item, we've already started putting that together. See:
http://wiki.freebsd.org/Jenkins and the presentation I gave on March 13.
At the next Devsummit in Ottawa, we want to talk about this further
and get people
who are interested in this topic to participate in the conversation:
https://wiki.freebsd.org/201405DevSummit/Jenkins
It would be interesting to get Raspberry PI hardware being tested
regularly as part of
continuous integration. We don't have Raspberry PI hardware available
in the FreeBSD
cluster, so someone who is interested would have to take the action item
to acquire the hardware and set things up so the hardware is integrated with
continuous integration/testing.
Looking at the SummerOfCode2014 ideas page, my opinion is that for this item:
https://wiki.freebsd.org/SummerOfCode2014#TEST-o-steron_for_FreeBSD_.28node.js.2FLua.29
- has already been done with things like libvirt, which is being
ported to FreeBSD
by Roman Bogorodskiy. See:
http://empt1e.blogspot.com/2014/03/bhyve-in-libvirt.html
- existing automation frameworks can already be used to send
remote commands to VM's
and for this item:
https://wiki.freebsd.org/SummerOfCode2014#CON-tinuous_INtegration_for_FreeBSD_aka_improved_TinderBox_.28node.js.2FLua.29
- better Tinderbox systems already exist. Jenkins and Buildbot
are two commonly
available open source implementations.
- the jenkins-admin team set up Jenkins (
http://jenkins.freebsd.org ) because we were familiar
it, and so far it seems to work well
My focus is to get existing software that people in the non-FreeBSD world have
developed to solve these types of problems, and get them up and
running in FreeBSD.
--
Craig
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
