I think fetchSizeBytes is related to Kafka, not Storm. So if you look at the Kafka documentation http://kafka.apache.org/08/configuration.html you'll see that both producers and consumers have max mesage size in bytes. Your KafkaConfig max fetch size should match the max message size of the producers.
On Tue, Sep 30, 2014 at 3:00 PM, Raphael Hsieh <[email protected]> wrote: > I'm confused about what the values fetchSizeBytes and bufferSizeBytes > relate to in my storm topology. > > My TridentKafkaConfig seems to be completely separated from my StormConfig. > In my TridentKafkaConfig I can set my fetchSizeBytes and bufferSizeBytes > (what's the difference?) to a value, but in my StormConfig, I can set my > max batch size as well. > > if I do: > > tridentKafkaConfig.fetchSizeBytes = 1024; > > then later do > > config.put(RichSpoutBatchExecutor.MAX_BATCH_SIZE_CONF, 1024*1024*10) > > doesn't that mean that my maximum batch size will be 1024 regardless ? > > -- > Raphael Hsieh > > > >
