It may something with your configuration. Try it with a simpler setup of one broker, see that both topics appear on the broker, and build on your understanding and experimentation with that setup.
Philip On Sep 20, 2013, at 11:06 AM, Suchi Amalapurapu <[email protected]> wrote: > Thx for your replies. Can u please clarify this further for me? > > In 0.7, if the partition count is per broker, shouldn't each broker have a > log directory for every topic? > I don't see that though... Only some topics have logs in different brokers. > > How is the consumer expected to handle this? My start sequence is as follows > - start zookeeper > - start producer > - start broker1 > - start broker2 > - send message to topic1 > - send message to topic 2 > > log directory for topic1 is created on broker1 and broker2 > log directory for topic2 is created only on broker1 > > The consumer(kafka-storm spout) with a parallelism of one should ideally > discover the brokers via zookeeper and in this case should read the > messages in the correct sequence from both? > Suchi > > > On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu > <[email protected]>wrote: > >> Yes I understand that. I am letting the producer/consumer use zookeeper to >> discover brokers. >> I can clearly see in the logs(brokers) that both the brokers create a new >> topic log for the same topic. >> >> The brokers are in different availability zones. Does that matter? >> Suchi >> >> >> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <[email protected]> wrote: >> >>> Seems to me you are confusing partitions and brokers. Partition count has >>> nothing to do with the number of brokers to which a message a sent -- just >>> the number of partitions into which that message will be split when it >>> gets >>> to a broker. >>> >>> You need to explicitly set the destination brokers in the Producer, not >>> via >>> setting "partition" count. >>> >>> Philip >>> >>> >>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <[email protected] >>>> wrote: >>> >>>> My current setup has 2 brokers, multiple topics, each topic with 1 >>>> partition, kafka 0.7.1 >>>> Some topics are logged onto multiple brokers inspite of partitions set >>> to >>>> 1. Kafka 0.7.1 does not have replication turned on. >>>> Why are the messages being logged onto multiple brokers when the number >>> of >>>> partitions is set to 1? >>>> I have a consumer with parallelism set to 1 and its missing some of >>> these >>>> messages. >>>> Suchi >> >>
