One of the "Rejected Alternatives" was to do something "smarter" by automatically reducing the replication factor when the cluster size is smaller than the replication factor. However, this is extremely unintuitive, and in rare cases (e.g., during a partial outage) might even result in internal topics being created with too small of a replication factor. And defaulting to 1 is certainly bad for production use cases, so that's not an option, either.
While defaulting to 3 and failing if the cluster doesn't have 3 nodes is a bit harsher than I'd like, it does appear to be the safer option: an error message (with instructions on how to correct) is better than inadvertently setting the replication factor too small and not knowing about it until it is too late. On Mon, May 8, 2017 at 6:12 PM, BigData dev <bigdatadev...@gmail.com> wrote: > Hi, > I liked the KIP, as it will avoid so many errors which user can make during > setup. > I have 1 questions here. > 1. As default replication factor is set to 3, but if Kafka cluster is setup > for one node, then the user needs to override the default configuraion, > till then topics will not be created. > So, this is the behavior we want to give? > > On Mon, May 8, 2017 at 2:25 PM, Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > Thanks a lot for the KIP Randall. This improvement should simplify both > > regular deployments and testing! > > > > A minor comment. Maybe it would be nice to add a note about why there's > no > > need for the property: config.storage.partitions > > I'm mentioning this for the sake of completeness, in case someone notices > > this slight asymmetry with respect to the newly introduced config > > properties. > > > > This is by no means a blocking comment. > > > > Thanks, > > Konstantine > > > > On Fri, May 5, 2017 at 7:18 PM, Randall Hauch <rha...@gmail.com> wrote: > > > > > Thanks, Gwen. > > > > > > Switching to low-priority is a great idea. > > > > > > The default value for the replication factor configuration is 3, since > > > that makes sense and is safe for production. Using the default values > in > > > the example would mean it could only be run against a Kafka cluster > with > > a > > > minimum of 3 nodes. I propose overriding the example's replication > factor > > > configurations to be 1 so that the examples could be run on any sized > > > cluster. > > > > > > The rejected alternatives mentions why the implementation doesn't try > to > > > be too smart by calculating the replication factor. > > > > > > Best regards, > > > > > > Randall > > > > > > > On May 5, 2017, at 8:02 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > > > > > Looks great to me :) > > > > > > > > Just one note - configurations have levels (which reflect in the > docs) > > - > > > I > > > > suggest putting the whole thing as LOW. Most users will never need to > > > worry > > > > about these. For same reason I recommend leaving them out of the > > example > > > > config files - we already have issues with users playing with configs > > > > without understanding what they are doing and not liking the results. > > > > > > > >> On Fri, May 5, 2017 at 3:42 PM, Randall Hauch <rha...@gmail.com> > > wrote: > > > >> > > > >> Hi, all. > > > >> > > > >> I've been working on KAFKA-4667 to change the distributed worker of > > > Kafka > > > >> Connect to look for the topics used to store connector and task > > > >> configurations, offsets, and status, and if those tasks do not exist > > to > > > >> create them using the new AdminClient. To make this as useful as > > > possible > > > >> and to minimize the need to still manually create the topics, I > > propose > > > >> adding several new distributed worker configurations to specify the > > > >> partitions and replication factor for these topics, and have > outlined > > > them > > > >> in "KIP-154 Add Kafka Connect configuration properties for creating > > > >> internal topics". > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > >> 154+Add+Kafka+Connect+configuration+properties+for+ > > > >> creating+internal+topics > > > >> > > > >> Please take a look and provide feedback. Thanks! > > > >> > > > >> Best regards, > > > >> > > > >> Randall > > > >> > > > > > > > > > > > > > > > > -- > > > > *Gwen Shapira* > > > > Product Manager | Confluent > > > > 650.450.2760 | @gwenshap > > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > > > <http://www.confluent.io/blog> > > > > > >
