So the proposal I heard was: - We create a branch for replication and begin development work on it there. - We leave trunk for incremental fixes and non-replication feature development. - We do monthly releases off trunk with whatever we have. - We will number accordingly, so if we only have some small fixes then the release will just be 0.7.1 or 0.7.01 or whatever; replication or larger features will bump us up to 0.8.
That seems reasonable to me. What this does mean, though is that LinkedIn may or may not end up running the releases in production before they go out, so this does put more testing/validation burden on the open source process. -Jay On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <jun...@gmail.com> wrote: > Hi, Chris, > > Yes, the changes related to replication are significant. It's actually a > bit hard to keep what we have now as the degenerated case with R=1. > Currently, the partions are numbered locally within each broker. This is > problematic with replication since a partition can have multiple replicas > that physically exist on multiple brokers. So, in our replication design, > partitions will be numbered globally within the cluster. This implies > potential on disk layout changes (see kafka-47) and wire protocol changes. > > You brought up a good question on migration path from 0.7 to 0.8. Since > replication changes are significant, it's going to be hard to make those > changes while keeping backward compatibility. My current feeling is that we > will make 0.8 a non-backward compatible release. To migrate to 0.8, > one possibility is to first create a new cluster of Kafka brokers using 0.8 > (can potentially overlay on existing hardware). Then upgrade all producers > and point them to the new brokers. Drain all old consumers. Upgrade all > consumers and point them to the new brokers. > > What do people feel about this? > > Thanks, > > Jun > > On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs > <chris.burrou...@gmail.com>wrote: > > > On 11/11/2011 06:01 PM, Jay Kreps wrote: > > > 2. Our current focus for linkedin folks was going to be on replication > > > which will be a really disruptive set of features that need to be > > released > > > together. > > > > I think there are some differing assumptions on the impact of > > replication. My assumption has been that replication would be a new > > feature that was 'optional' in that R=1 would not be significantly from > > what we are doing now. Thus replication work could continue in trunk -- > > and even be present in releases -- without being disruptive, the lack > > change would just be to allow and document values of R > 1. > > > > However, both Jay and Jun's description of the work make it sound like > > more of a backwards incompatible no-rolling-restart every > > broker/consumer/producer needs to be upgraded at the same time sort of > > change. > > >
