On 20/08/2012, at 5:36 AM, Adam Murdoch wrote: > > On 18/08/2012, at 1:15 AM, Luke Daley wrote: > >> Howdy, >> >> There's an aspect of the migration comparison/verification work that could >> benefit from some focused discussion. It's mostly an implementation level >> issue so this conversation can run in parallel to other work in this space. >> >> At a basic level what we need to do is run two builds and compare _things_ >> about this. What I want to try and lock down is exactly what these “things” >> can be. There's two prevailing schools of thought; these things are always >> files, or they are much more abstract and could be anything. A component of >> this work will be delivering an API that allows the user to assemble their >> own comparison process (for non trivial cases). If everything is a file the >> API could be simpler, *maybe* in a significant way. >> >> In the migration from tool X (Ant or Maven) to Gradle, tool X is more or >> less an opaque box that we poke and then it does something. Our options for >> understanding what it did are its process exit value, its stdio output and >> what it leaves on the filesystem. In a Gradle to Gradle migration (i.e. >> upgrade), we have much richer information exchange potential. >> >> If we decide that _all_ we want to do with this feature is compare the files >> that builds generate then the issue is moot. We can obviously then design >> the entire API around comparing files. >> >> I'm not sure we want to do make this restriction though. I think there's a >> case for comparing other things. >> >> Consider something like a classpath. In the case of a migration from a >> different system, it might be useful to understand how the compile classpath >> is different for an artifact during the development of the Gradle build >> being migrated to. Unless we did this automatically (which we probably could >> do for Maven but not Ant) I doubt many users would bother with this. It >> seems more compelling in the Gradle upgrade case though. That's definitely >> something I'm going to want to be aware of changing if it does. >> >> There are some other Gradle specific things we could compare: >> >> * All of the tasks in a project (i.e. be told that a Gradle upgrade >> introduces or removes tasks that were in the build before) >> * All of the DSL extensions in a project (this one is pretty dubious I think) >> * The set of configurations/archives etc. >> >> Upgrade verification isn't just about giving a binary yes/no answer. Part of >> it will be understanding differences introduced by new versions, which means >> deeper analysis. When (if?) we start using this feature to help people test >> speculative changes to their build, this becomes even more important. >> >> We also may use this “comparison” toolkit in our own QA, which is another >> argument for being more general/abstract. >> >> >> My current thinking is that at the base of the API we need to stay general >> and be able to compare just about anything (by plugging in different >> strategies), and then layer a file oriented set of strategies on top. > > I think anything beyond comparing files and text files is unnecessary, and > overcooking it a bit. Here's why: > > What we're doing here is adding a smoke test that verifies that a change to > the build system is just as likely to not break things, as any other change > (changing the code, tweaking a configuration file, upgrading a dependency, > etc). We don't have to do anything more than say 'according to your current > quality checks, this build works as well as that build'. The goal isn't to > solve the world here, or implement some general purpose comparison engine. > > Instead, we need to do 2 things: > > 1. Encourage people to write tests for the build outcomes. Then, when we want > to verify a change from one build to another, we run the test for both > builds. If it passes in both builds, then the outcome is the same. > > This way, you continue to verify that your build works beyond the lifetime of > any migration you might undertake. If it's important enough to compare during > migration, it's important enough to verify every time something else changes.
Can you give some examples of the kinds of tests you are thinking of here please. I'm not sure I'm on your wavelength. > 2. Be able to compare certain key types of files. In particular, various > archives and text files. > > This way, if you've something about the build you need to verify, you > generate a text file from each build and we can do a text diff on the > contents and tell you if they are the same. I think this the key thing. If people can easily enough represent important internal structures as text files (and I can't see why not), then this strategy should be fine. > So, we have 2 extensible mechanisms here: making use of tests, and comparing > text files. I think in practise this will be plenty. > > Of course, the implementation will have some kind of pluggable strategies, or > abstractions, or what ever. This is fine, but we should keep it internal. We > shouldn't expose this until we know we need it. It is essential that we keep > the public API as focused, and concrete, as possible. For system X to Gradle migration I think this is right, which means that our public API toolkit can focus on a file based API. Let's consider this issue resolved. As I think you are saying, it makes sense to layer this on top of a more general abstraction though. For the Gradle to Gradle case I would hope over time that our comparison inputs become much richer to give people greater confidence about upgrades and for this _having_ to go to a file would be unnecessary indirection. So it might be necessary to effectively have two APIs here and adapt the public API to the internal, more general, API (as opposed to the public component being a subset). We'll see as it shakes out though, as we are only doing the Gradle 2 gradle case right now. -- Luke Daley Principal Engineer, Gradleware http://gradleware.com
