<elopio> ping ted. <elopio> do you have some time to talk again about a temp home dir? <elopio> https://bugs.launchpad.net/ubuntu-app-launch/+bug/1376423 <ubot5> Ubuntu bug 1376423 in Ubuntu Application Launcher "There is no easy and future-proof way of starting an app in a clean environment" [Undecided,New] <ted> pong elopio <elopio> ubuntu-qa, thomi and balloons: ^ feel free to also comment that bug. <ted> I kinda disagree with your statement there. <ted> "In the past we were able to get a reasonably clean environment setting the value of $HOME to a temporary directory." <ted> That was always a bad idea. <ted> elopio, Why can't you just wipe the device for a clean environment? <elopio> ted: that's clear for me now. <ted> That's really the only way it's really "clean" <elopio> ted: we are not always testing on phones, we also need to test on dev machines. <dobey> hmm <elopio> ted: and some of the devices we are using are also the ones we dogfood. <ted> elopio, Okay, in a container. <elopio> it's bad to have to reset them every time. <ted> elopio, dog fooded devices can't ever be clean. <elopio> ted: a container can be a good solution. <ted> I'm hoping people don't use that for real testing. <elopio> ted: not for something like running the whole suite to decide if a version is promoted or not. <elopio> but we need to use the dogfooding devices to check that a new test we added works. <elopio> or to debug a test that started failing. <elopio> otherwise we would need two devices. Of every model. <ted> But, you'll need to wipe it to land the silo with the new test, no? -*- balloons settles in <ted> So you kinda need a device you can wipe of every model you care about. <elopio> ted: that's for the silo testers. <ted> Don't we silo MRs for tests? <elopio> they are not necessarily the same as the dogfooders or automation devs. <balloons> ted, we do need a way to run tests sanely as a developer or test writer too <ted> Sure, but those folks don't need a clean test environment either. <elopio> ted: I didn't get that last question. But I think you would agree that any solution we find, needs to be easily runnable from a dev machine. Do we agree there? <balloons> but you can argue about whether the test or the test harness should prep the environment <ted> I'm just saying that we're not going to get "clean" on a device that isn't wiped. <ted> It's always going to be sullied in some way. <elopio> ted: maybe we could get a container on a device that is not wiped <ted> Or there's enough of a chance that it could be that it's not a "result" just a checkup. <balloons> at the very least the test shouldn't be polluting, while still allow us to setup at least the parts we are responsible for correctly <elopio> and generally we will run on an unwiped device just a small subset of tests. <elopio> maybe for that case we don't need a perfectly clean environment. Just something that doesn't delete all the photos I took yesterday. <ted> Sure, the testing shouldn't be destructive. <dobey> elopio: can you not run the tests in a schroot? <elopio> ted: but we need to test the behavior of the gallery app without existing tests. So we need to have an environment that temporarily doesn't see any photos. <balloons> right.. I feel like for the things the app reads and writes, we should be able to control and initialize and clean up well <elopio> dobey: I think we could. Here we just want to start the discussion of which is the right way. <elopio> we need to take into account how slow it is, because if it takes 10 minutes to run two tests, tests are not going to be run often enough. <ted> Sure, so I think that a chroot is an option. <ted> I like creating another user for the tests. <balloons> right.. if the test harness creates the environment, the setup is not easy <ted> That creates a cleanish environment for that user. <elopio> but if we find a way to do it in a reasonable amount of time, it works for me. <ted> If it's a system image based device, then the new user should be very clean. <dobey> ted: or just run them as "guest" user <dobey> ted: and everything gets destroyed on logout <balloons> elopio, well technically adt-run is more or less there, as it sets up the env for it's tests <elopio> ted: so if the solution is creating another user for the tests, we need to start working on reducing the amount of time it takes unity to start or on launching apps without unity. <ted> dobey, Yeah it'd be roughly the same, but I dont' think we'd want automatic delete like guest does. <elopio> but that's also a perfectly valid option. <balloons> there was issues with just jumping to another user, but the apparmor stuff too would be easier <balloons> I know jamie preferred that route <dobey> ted: then a scrhoot is probably the best thing (and i think that's what adt-run uses anyway) <ted> elopio, I can't think of anyone that doesn't want U8 to start faster, but I don't think that blocks working on a new user based solution. <elopio> dobey: adt-run can use anything. lxc, schroot, ssh into a real or emulated device, or even just run on the local machine. <dobey> an lxc/chroot with nested apparmor support would be more in the right direction <elopio> dobey: that's something I like. <elopio> then we need to talk about a virtual framebuffer for MIR, and an easy command that fires up the lxc with all the app armor rules and upstart vars that we will find on ubuntu touch. <jdstrand> you can't run click apps confined in lxc <jdstrand> not yet anyway. that needs the apparmor stacking work to be completed, which we are working on <elopio> jdstrand: welcome to the party :) I'm glad you are here, sorry for not pinging you before. <jjohansen> well you can but it takes some work, and won't be generally supported yet <balloons> jdstrand, welcome indeed :-) <jjohansen> you have to turn off the existing apparmor mediation on lxc, and setup an apparmor policy namespace for the lxc container <elopio> all we are asking is for a blessed way to start apps for testing. And get this blessed way under automated tests so we can rely on it working forever. <jjohansen> then you can in fact do it <elopio> I don't know if the solution should live in ubuntu-app-launch, upstart, autopilot, phablet-tools, or something else. <jdstrand> elopio: for it to work forever in lxc, you need to wait for our 15.04 work. maybe something could be done like jjohansen mentioned, but thinking that wouldn't work fantastic <dobey> elopio: emulator? :) <balloons> so this does seem to push everything into the harness. Should the tests then be 'dumb'? <jjohansen> elopio: sorry I can only offer you a shim atm, I won't promise it will work forever <jdstrand> what about the emulator? <ted> elopio, I don't think you need anything in Mir really, it already supports nesting for the system compositor case. <jjohansen> jdstrand: it works fine, I use it all the time, there just isn't any tooling around it so it is a pita <dobey> elopio: can you not create/destroy emulator instances and run the tests inside the emulator? <elopio> emulator is also an option. But the apps will eventually work on desktop too <jdstrand> jjohansen: sure, that is what I meant be not working fantastic-- would have to do the tooling etc and then it would be tossed out down the road <elopio> it would feel weird to launch an emulator that emulates your actual development device. But if it's fast and reliable, I'm also ok with that. <dobey> elopio: well, that's all a chroot is <jjohansen> jdstrand: ah okay, yep <dobey> elopio: when we get to a converged state, i would expect the emulator to be able to have layouts for tablet and other things as well as phone. <jdstrand> so, I've been welcomed to the party, but what is the topic of the discussion at this party exactly? <dobey> if it doesn't then the emulator itself loses a lot of usefulness <elopio> dobey: I guess you would be able to choose if you want to run on a brand new and clean chroot, or on your real machine with the risk of affecting your contests. <elopio> *contents <ted> jdstrand, Creating a way to run tests, with mock data, in a semi-clean fashion on a dogfooding device. <elopio> that's also fine by me. <balloons> I feel like the tests should be a bit better behaved and not offload everything to the runner, imho. I'm somewhat concerned about the idea of only being able to run tests in an isolated enviornment <jdstrand> we are supposed to also gate on the emulator working correctly. we don't now, but that is planned aiui <dobey> elopio: we can already do that though :) <balloons> jdstrand, https://bugs.launchpad.net/ubuntu-app-launch/+bug/1376423 <ubot5> Ubuntu bug 1376423 in Ubuntu Application Launcher "There is no easy and future-proof way of starting an app in a clean environment" [Undecided,New] <jdstrand> balloons: ok, so this is a continuation of our discussion <balloons> jdstrand, it's a continuation and evolution of Matla discussions yep <jdstrand> (longtime discussion) <elopio> dobey: but we don't have a way now to get the same environment you would get on ubuntu-touch from a chroot. <balloons> and before matla :-) <elopio> at least not easily. <jdstrand> I maintain that creating a new user is the likely the easiest short term solution <elopio> we need to launch the apps with ubuntu-app-launch, and that's not currently possible without unity, I think. <dobey> elopio: well, that's what the emulator is; or you could download the image and unpack it, and use that as the chroot <elopio> and if we need to start unity, then it's slow. <dobey> ubuntu-app-launch works fine without unity <jdstrand> once we have lxc stacking, we should be able to leverage lxc for this more easily <elopio> dobey: it doesn't work on an xvfb. We need to do some extra work to get it set up. <elopio> which is fine. As long as that extra work is encapsulated on a simple API call, with the API being maintained by the devs that will eventually break it. <dobey> elopio: not sure what you're missing there, but it works fine inside my lxc while redirecting DISPLAY to my host <dobey> oh, well it used to. maybe it doesn't any more <balloons> jdstrand, I too am curious about blockers for the new user route <elopio> dobey: so there are two more or less related problems. Run tests in a virtual X, and then run the tests with a clean home. <dobey> "unable to setup cgroup" <dobey> which i guess is the apparmor nesting problem <balloons> if we created a new user, could we simply start the app outside of unity running? <elopio> dobey: and what you are finding is what has brought us to an unmanageable mess. There are simple solutions that work at some point. <elopio> we put them on the test set up, and then they stop working. <elopio> the solution to get them back to green is again simple, so we do it. <dobey> elopio: welcome to the joy of developing things on top of a moving target <elopio> and that has happened for a long time, so the solution is now not simple, nor clean, and will keep breaking. <balloons> ^^ that is the primary concern.. the complexity only continues to grow <elopio> dobey: it wouldn't be hard at all to make sure the solution keeps working always. We just need to make sure that it's tested when releasing new app armor rules, new upstart versions, new container versions. All the weird details will be encapsulated behind a single call. <elopio> we just need to find which is the team that has to implement that testability API. <elopio> currently I don't know. I think that ted and jdstrand agree that a clean environment means a new user. <ted> So I'm with jdstrand on creating a new user <balloons> right.. if there's one proper way to do it, we can test it and keep it working <elopio> if that's the case, where do we put a script that gives us that new user with an env as close as what we will see on ubuntu-touch as possible? <ted> I think you can steal the code from the guest account setup, and do basically the same thing. <elopio> ted: but I don't want to steal code and duplicate it on every test suite set up. <dobey> elopio: it is hard to make a solution that always works, because requirements change, and we get better at doing things. as that happens, stuff will stop working in the same way. <ted> Basically you create the new user and then poke lightdm to autologin as that user the next time it runs. <balloons> elopio, related, would we take the new user concept to the desktop and other platforms as well (instead of mocking)? <ted> Basically a new device at that point. <elopio> I want to just make a single call, and that call to be the same on all the set ups. <dobey> dwim() <balloons> dobey, right.. we have helpers that elopio has maintained for some time to improve the backend and keep things working without breaking tests, all while letting us do simple things <balloons> I would invision this to be more or less the same <elopio> ted: I like how it sounds, and on the test clean up we can reset lightdm to the real behaviour and delete the new user. <dobey> right. it's a matter of getting all the brains in the same room and coming up with something that is implemetable, maintainable, and pushes things further in the right direction <ted> elopio, Correct and you can also suck up the whole home directory as test artifacts on failure. <ted> You'll get the upstart logs, etc all for free without having to know which to grab. <balloons> all good stuff <elopio> if we all agree that's the best solution, then we can start thinking of where to maintain that fixture. And the next problems we will find, like restarting unity. <ted> To be clear, you won't be restarting unity. You'll be logging out the current user and logging in the new one. <elopio> I guess we will be able to call the same fixture from an LXC, schroot or emulator. So it sounds independent from the test runner. <ted> Which, in effect, restarts unity. <elopio> ted: oh, but then if I run it on the machine where I'm writing the tests, the IDE and everything will be closed. <elopio> can't we start an additional user session? <ted> In theory, but you'll be testing Mir in a new way there today. <ted> In the long run that should work, but I'd imagine you find interesting bugs today. <balloons> ahh that was the issue.. I couldn't remember the ! with a new user <elopio> well, I'm not too concerned about how hard would it be. If that's the right(tm) solution, then we look for resources to implement it and while it's done we can keep maintaing our hacks. <balloons> it needs to happen for MIR anyway.. right, we can be a catalyst <elopio> well, I think that it's an essential feature for upstart, ubuntu-app-launch, app-armor, mir and unity to provide these testability features we are requesting. <dobey> ted, elopio: well, you should be able to just run a second X session as the new user too. you don't need to log out the current user. you just need to run the session for the new user somewhere <elopio> it doesn't sound to me like we are spoiled and want extra easy things. I might be wrong, because I'm spoiled :) <dobey> not sure how you do that with mir though <ted> elopio, Everyone thinks their features are essential :-) <ted> dobey, Ubuntu system compositor <elopio> ted: yes, but we have a better argument. If we don't test everybody else's features, they will break :D <dobey> we had to drink from a rolled up newspaper! <ted> elopio, I think of it the other way, if you don't test I don't have more bugs to fix! ;-) <elopio> funny guy. <elopio> ;) <elopio> ok, I'll put all this discussion on a paste and link it to the bug. <elopio> it will be open I guess for a couple of weeks before we decide a course of action. <balloons> ted, lol.. no bugs found, no bugs to fix ;-) <dobey> if you're going to paste it, just paste it in the bug report <dobey> don't link to pastebins in bug reports <elopio> I guess kgunn and the ci-team would also like to be involved in the discusssion. <dobey> pastebins can expire <elopio> dobey: ack. <ted> elopio, Probably a session for the next sprint, seems to align with your timelines. <elopio> thanks everybody for your time. And don't worry, we will keep bothering you <balloons> ted, most certainly it will be discussed at the sprint ;-) live and in person means I hope we have a solution by the end eh? -*- ted looks to see what weapons he can pack in his suitcase <ted> :-) <ted> "solution" <dobey> balloons: if solution is "drink more beer" then maybe <balloons> a possible solution! <ted> Honestly, I think what we have is reasonable, it's more a matter of working out the details at this point.
-- You received this bug notification because you are a member of Unity API bugs, which is subscribed to Ubuntu Application Launcher. https://bugs.launchpad.net/bugs/1376423 Title: There is no easy and future-proof way of starting an app in a clean environment Status in Ubuntu Application Launcher: New Bug description: For manual and automated testing we generally need to be able to start the applications in a clean environment, in a way that the tests are not affected by existing files and configurations. And as the tests should be easily runnable on the devs development machines, the changes made while running the tests should not affect the real environment. In the past we were able to get a reasonably clean environment setting the value of $HOME to a temporary directory. Now this is not enough by a variety of reasons, like a hierarchy of dirs in HOME needed by app armor, apps being launched by upstart, and services that keep listening to the previous value of HOME. Every test suite has solved this on it's own why, some use the real environment but backup the existing files (which is really bad), some patch the HOME and keep adding details every time something changes, and some others just pollute the real environment and can't be run on a dev machine. This is what has been causing failures on the majority of autopilot tests for the last month. Instead of keep doing these multiple solutions, and keep adjusting them every time they stop working; I think we need to find one single way that works for all the apps. And we need to add automated tests for this single way to make sure that no new releases will break it. I'm currently thinking of a testability helper in ubuntu-app-launch that we call to launch an app and it will make sure that it will start all the needed XDG vars pointing to a clean temporary environment. We might still need to restart some services, and we might hit some other problems, but this is just an idea. Feel free to propose alternatives. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-app-launch/+bug/1376423/+subscriptions -- Mailing list: https://launchpad.net/~unity-api-bugs Post to : unity-api-bugs@lists.launchpad.net Unsubscribe : https://launchpad.net/~unity-api-bugs More help : https://help.launchpad.net/ListHelp
