This sounds good to me and restates what I've heard - I assume that the OSS community will understand that new releases are bleeding edge, it might help to make this crystal clear on the releases page. One thought I had here is that x.0.0 might be considered not hammered on by LinkedIn, however x.1.0 could be?
On Sun, Nov 13, 2011 at 9:49 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > So the proposal I heard was: > > - We create a branch for replication and begin development work on it > there. > - We leave trunk for incremental fixes and non-replication feature > development. > - We do monthly releases off trunk with whatever we have. > - We will number accordingly, so if we only have some small fixes then > the release will just be 0.7.1 or 0.7.01 or whatever; replication or > larger > features will bump us up to 0.8. > > That seems reasonable to me. What this does mean, though is that LinkedIn > may or may not end up running the releases in production before they go > out, so this does put more testing/validation burden on the open source > process. > > -Jay > > On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <jun...@gmail.com> wrote: > > > Hi, Chris, > > > > Yes, the changes related to replication are significant. It's actually a > > bit hard to keep what we have now as the degenerated case with R=1. > > Currently, the partions are numbered locally within each broker. This is > > problematic with replication since a partition can have multiple replicas > > that physically exist on multiple brokers. So, in our replication design, > > partitions will be numbered globally within the cluster. This implies > > potential on disk layout changes (see kafka-47) and wire protocol > changes. > > > > You brought up a good question on migration path from 0.7 to 0.8. Since > > replication changes are significant, it's going to be hard to make those > > changes while keeping backward compatibility. My current feeling is that > we > > will make 0.8 a non-backward compatible release. To migrate to 0.8, > > one possibility is to first create a new cluster of Kafka brokers using > 0.8 > > (can potentially overlay on existing hardware). Then upgrade all > producers > > and point them to the new brokers. Drain all old consumers. Upgrade all > > consumers and point them to the new brokers. > > > > What do people feel about this? > > > > Thanks, > > > > Jun > > > > On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs > > <chris.burrou...@gmail.com>wrote: > > > > > On 11/11/2011 06:01 PM, Jay Kreps wrote: > > > > 2. Our current focus for linkedin folks was going to be on > replication > > > > which will be a really disruptive set of features that need to be > > > released > > > > together. > > > > > > I think there are some differing assumptions on the impact of > > > replication. My assumption has been that replication would be a new > > > feature that was 'optional' in that R=1 would not be significantly from > > > what we are doing now. Thus replication work could continue in trunk > -- > > > and even be present in releases -- without being disruptive, the lack > > > change would just be to allow and document values of R > 1. > > > > > > However, both Jay and Jun's description of the work make it sound like > > > more of a backwards incompatible no-rolling-restart every > > > broker/consumer/producer needs to be upgraded at the same time sort of > > > change. > > > > > >
