Hi Suchi, In 0.7 a producer would send to one broker within a session, hence you may want to send enough messages to see them shown up on both brokers.
Guozhang On Sat, Sep 21, 2013 at 11:27 AM, Philip O'Toole <[email protected]> wrote: > It may something with your configuration. Try it with a simpler setup of > one broker, see that both topics appear on the broker, and build on your > understanding and experimentation with that setup. > > Philip > > On Sep 20, 2013, at 11:06 AM, Suchi Amalapurapu <[email protected]> > wrote: > > > Thx for your replies. Can u please clarify this further for me? > > > > In 0.7, if the partition count is per broker, shouldn't each broker have > a > > log directory for every topic? > > I don't see that though... Only some topics have logs in different > brokers. > > > > How is the consumer expected to handle this? My start sequence is as > follows > > - start zookeeper > > - start producer > > - start broker1 > > - start broker2 > > - send message to topic1 > > - send message to topic 2 > > > > log directory for topic1 is created on broker1 and broker2 > > log directory for topic2 is created only on broker1 > > > > The consumer(kafka-storm spout) with a parallelism of one should ideally > > discover the brokers via zookeeper and in this case should read the > > messages in the correct sequence from both? > > Suchi > > > > > > On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu < > [email protected]>wrote: > > > >> Yes I understand that. I am letting the producer/consumer use zookeeper > to > >> discover brokers. > >> I can clearly see in the logs(brokers) that both the brokers create a > new > >> topic log for the same topic. > >> > >> The brokers are in different availability zones. Does that matter? > >> Suchi > >> > >> > >> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <[email protected]> > wrote: > >> > >>> Seems to me you are confusing partitions and brokers. Partition count > has > >>> nothing to do with the number of brokers to which a message a sent -- > just > >>> the number of partitions into which that message will be split when it > >>> gets > >>> to a broker. > >>> > >>> You need to explicitly set the destination brokers in the Producer, not > >>> via > >>> setting "partition" count. > >>> > >>> Philip > >>> > >>> > >>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu < > [email protected] > >>>> wrote: > >>> > >>>> My current setup has 2 brokers, multiple topics, each topic with 1 > >>>> partition, kafka 0.7.1 > >>>> Some topics are logged onto multiple brokers inspite of partitions set > >>> to > >>>> 1. Kafka 0.7.1 does not have replication turned on. > >>>> Why are the messages being logged onto multiple brokers when the > number > >>> of > >>>> partitions is set to 1? > >>>> I have a consumer with parallelism set to 1 and its missing some of > >>> these > >>>> messages. > >>>> Suchi > >> > >> > -- -- Guozhang
