[
https://issues.apache.org/jira/browse/SAMZA-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516057#comment-15516057
]
Branislav Cogic commented on SAMZA-952:
---------------------------------------
Patch attached. RB link: https://reviews.apache.org/r/52213/
> Add validation for broadcast inputs configured for partitions that don't exist
> ------------------------------------------------------------------------------
>
> Key: SAMZA-952
> URL: https://issues.apache.org/jira/browse/SAMZA-952
> Project: Samza
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Jake Maes
> Assignee: Branislav Cogic
> Attachments: SAMZA-952_0.patch
>
>
> We had an issue at LinkedIn where the user configured a broadcast input as:
> {noformat}
> task.broadcast.inputs=kafka-queueing.ads_glob#[0-9]
> {noformat}
> But the ads_glob topic only has 8 partitions.
> This caused the following exception in the container log:
> {noformat}
> 2016-05-24 19:59:20 SamzaContainer [ERROR] Caught exception in process loop.
> org.apache.samza.SamzaException: No metadata available for partition null.
> at
> org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1$$anonfun$apply$15.apply(OffsetManager.scala:412)
> at
> org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1$$anonfun$apply$15.apply(OffsetManager.scala:380)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at
> org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1.apply(OffsetManager.scala:380)
> at
> org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1.apply(OffsetManager.scala:378)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at
> org.apache.samza.checkpoint.OffsetManager.loadDefaults(OffsetManager.scala:378)
> at
> org.apache.samza.checkpoint.OffsetManager.start(OffsetManager.scala:175)
> at
> org.apache.samza.container.SamzaContainer.startOffsetManager(SamzaContainer.scala:604)
> at
> org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:550)
> at
> org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
> at
> org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
> at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
> {noformat}
> We should add some validation and a meaningful error message for cases when
> the broadcast inputs don't make sense compared to the metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
