I agree this sounds reasonable. In particular KAFKA-172, KAFKA-134, and especially KAFKA-169 are examples of the kind of things I think are worthwhile to get shipped into releases sooner rather than latter.
Thanks everyone. On 11/14/2011 12:49 AM, Jay Kreps wrote: > So the proposal I heard was: > > - We create a branch for replication and begin development work on it > there. > - We leave trunk for incremental fixes and non-replication feature > development. > - We do monthly releases off trunk with whatever we have. > - We will number accordingly, so if we only have some small fixes then > the release will just be 0.7.1 or 0.7.01 or whatever; replication or larger > features will bump us up to 0.8. > > That seems reasonable to me. What this does mean, though is that LinkedIn > may or may not end up running the releases in production before they go > out, so this does put more testing/validation burden on the open source > process. > > -Jay > > On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <jun...@gmail.com> wrote: > >> Hi, Chris, >> >> Yes, the changes related to replication are significant. It's actually a >> bit hard to keep what we have now as the degenerated case with R=1. >> Currently, the partions are numbered locally within each broker. This is >> problematic with replication since a partition can have multiple replicas >> that physically exist on multiple brokers. So, in our replication design, >> partitions will be numbered globally within the cluster. This implies >> potential on disk layout changes (see kafka-47) and wire protocol changes. >> >> You brought up a good question on migration path from 0.7 to 0.8. Since >> replication changes are significant, it's going to be hard to make those >> changes while keeping backward compatibility. My current feeling is that we >> will make 0.8 a non-backward compatible release. To migrate to 0.8, >> one possibility is to first create a new cluster of Kafka brokers using 0.8 >> (can potentially overlay on existing hardware). Then upgrade all producers >> and point them to the new brokers. Drain all old consumers. Upgrade all >> consumers and point them to the new brokers. >> >> What do people feel about this? >> >> Thanks, >> >> Jun >> >> On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs >> <chris.burrou...@gmail.com>wrote: >> >>> On 11/11/2011 06:01 PM, Jay Kreps wrote: >>>> 2. Our current focus for linkedin folks was going to be on replication >>>> which will be a really disruptive set of features that need to be >>> released >>>> together. >>> >>> I think there are some differing assumptions on the impact of >>> replication. My assumption has been that replication would be a new >>> feature that was 'optional' in that R=1 would not be significantly from >>> what we are doing now. Thus replication work could continue in trunk -- >>> and even be present in releases -- without being disruptive, the lack >>> change would just be to allow and document values of R > 1. >>> >>> However, both Jay and Jun's description of the work make it sound like >>> more of a backwards incompatible no-rolling-restart every >>> broker/consumer/producer needs to be upgraded at the same time sort of >>> change. >>> >> >
