Jeremiah, We can certainly enable the "auto.offset.reset" to be configurable. Not sure how making the "auto.commit.enable" configurable would work. Are you thinking that another property would be added to indicate how often to commit? Or would it work completely differently? Just need that fleshed out a bit more.
I do like the suggestion of exposing the config properties as user-defined properties. I have created a ticket to track this information: https://issues.apache.org/jira/browse/NIFI-791 Please feel free to update the ticket with any relevant information as you think of it. Thanks! -Mark ---------------------------------------- > Date: Mon, 27 Jul 2015 15:42:37 -0600 > Subject: GetKafka Processor and Hardcoded Kafka Consumer Configs > From: [email protected] > To: [email protected] > > The GetKafka processor has a couple of Kafka Consumer Config values that > are hard-coded. > > props.setProperty("auto.commit.enable", "true"); // just be explicit > props.setProperty("auto.offset.reset", "smallest"); > > These should be configurable property values in the Processor. Most > notable for me is the "auto.offset.reset". Smallest vs. Largest has some > implications concerning fault tolerance strategies. > > It would be best to expose all of the available Kafka Consumer Config > properties. If these change though between kafka versions it would create > maintenance work for the Processors. > > Another option would be to allow ad-hoc property values and end-user just > supply the kafka config values they want to override. > > > *Jeremiah Adams* > > Senior Software Developer > Pearson > > 2154 East Commons Ave. > Suite 400 > Centennial, CO 80122 > > > Always Learning > Learn more at www.pearson.com
