Hey Jeremiah, That's great to hear that you've root-caused it!
Please let us know should you run into further issues :-) Best, Jagadish On Mon, Mar 25, 2019 at 1:16 PM Jeremiah Adams <jad...@helixeducation.com> wrote: > Thanks for the input Jagadish. This makes sense now. I'm seeing one of our > input topic partitions empty. One of our upstream jobs is likely configured > incorrectly. > > > > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > ________________________________________ > From: Jagadish Venkatraman <jagadish1...@gmail.com> > Sent: Monday, March 25, 2019 10:49 AM > To: dev@samza.apache.org > Subject: Re: Empty Kafka topic partition Warning > > Hi Jeremiah, > > >> It looks like my configuration for changelog.replication.factor is not > being applied. Instead the default seems to be applied. I have messages on > 2/3 partitions. I am not seeing what I have incorrectly configured. > > The "replication.factor" config you mention determines how many brokers > each change-log partition is replicated to. This is un-related to the > behavior you are observing - where a single partition is empty. > > For context, each Samza task has its own store and writes to a single > change-log partition. An empty change-log partition means its corresponding > task did not perform writes to its store. > > Can you check if there are *input* topic-partitions that don't receive > messages? If so, tasks consuming from them wouldn't have data to process. > This results in their store and the corresponding change-log partition to > be empty. Also, can you check for any exceptions in your logs? > > For details on Samza's state management, I'd recommend the below resources: > Videos: Samza Architecture Part-1 > < > https://url.emailprotection.link/?bH1bmLuRP-9egfczNVLB6hd1gawa23HTbKoVFrWAVOlRfLffKffb7k6Xz4SIKeQI_-CACqwBAAIv3sh6ggDJS0pwrRpygaBi5RQfIazuEl6b3tQARLvOEhyiW0TJtkUGo8iFsUnmdOZ1JP5t8htxpLELL8Fc_bTyPPOf7HGCzqgafJ0dg6ZlqUd_hiohVUE_GRFkJ5akgiTrEVn6Atuz7ww~~ > > > Samza Architecture Part-2 > < > https://url.emailprotection.link/?bH1bmLuRP-9egfczNVLB6hd1gawa23HTbKoVFrWAVOlSf_VraXufE_XpIbZBkBEqImcqPh9L5FuLuDNcZ62q9Rxq9ovOrvm2AnsNc3Jar1dCQ3YPrRkbddUeagVD98AamxScT9ytgfFGR_xlLudhgIvywi062aHaNyy8ATooSdwA~ > > > Docs: Architecture overview > < > https://url.emailprotection.link/?b4GATHeTL1aGKLTs1Na3CfFbXFP0dMSRpyk1-M3oM2l0apvilQMvwdv4vkmcIDJo4yw6in8o7yvSedpOKyEvHxyiCQYkqUDP8o6FRVBFhQ7ehTfrVW4nBa8MBvfGU7FW8DkuH-VIG6Id5RgOWCS-Dryx2x3Pfc55FSH8rwNWDQ8Y~ > > > > > Best, > Jagadish > > On Mon, Mar 25, 2019 at 8:30 AM Jeremiah Adams <jad...@helixeducation.com> > wrote: > > > It looks like my configuration for changelog.replication.factor is not > > being applied. Instead the default seems to be applied. I have messages > on > > 2/3 partitions. I am not seeing what I have incorrectly configured. > > > > > > > stores.redelivery-store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory > > stores.redelivery-store.changelog=kafka.delivery-changelog > > stores.default.changelog.replication.factor=3 > > > > > > > > > > > > Jeremiah Adams > > Software Engineer > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > Blog | Twitter | Facebook | LinkedIn > > > > ________________________________________ > > From: Jagadish Venkatraman <jagadish1...@gmail.com> > > Sent: Friday, March 22, 2019 3:01 PM > > To: dev@samza.apache.org > > Subject: Re: Empty Kafka topic partition Warning > > > > Hi Jeremiah, > > > > >> why is the offset 0? > > > > This likely means that the change-log is empty and does not have any > > messages. > > > > Can you try consuming from partition-number: 0 using a KafkaConsumer? > > > > Best, > > Jagadish > > > > > > > > > > > > On Fri, Mar 22, 2019 at 11:45 AM Jeremiah Adams < > jad...@helixeducation.com > > > > > wrote: > > > > > I'm seeing these in our log periodically and havn't seen them before. > > > Does this imply that the topic associated with the change log is being > > > replayed from the beginning? > > > > > > > > > Also, why is the offset 0? It definitely should not be. We have > messages > > > across all partitions. > > > > > > > > > 2019-03-22 18:30:24 KafkaSystemAdmin [INFO] Fetching SSP metadata for: > > > [SystemStreamPartition [kafka, delivery-changelog, 0]] > > > 2019-03-22 18:30:24 KafkaSystemAdmin [WARN] Empty Kafka topic partition > > > delivery-changelog-0 with upcoming offset 0. Skipping newest offset and > > > setting oldest offset to 0 to consume from beginning > > > 2019-03-22 18:30:52 KafkaSystemAdmin [INFO] Fetching SSP metadata for: > > > [SystemStreamPartition [kafka, delivery-changelog, 0]] > > > 2019-03-22 18:30:52 KafkaSystemAdmin [WARN] Empty Kafka topic partition > > > delivery-changelog-0 with upcoming offset 0. Skipping newest offset and > > > setting oldest offset to 0 to consume from beginning > > > > > > > > > Jeremiah Adams > > > Software Engineer > > > > > > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~ > > < > > > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~ > > > > > > Blog< > > > https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~ > > > > | Twitter< > > > > > > https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~ > > > > | Facebook< > > > > > > https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~ > > > > | LinkedIn< > > > > > > https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0 > > > > > > > > > > > > -- > > Jagadish V, > > Graduate Student, > > Department of Computer Science, > > Stanford University > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
