[ https://issues.apache.org/jira/browse/SPARK-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu resolved SPARK-19853. ---------------------------------- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0 2.1.1 > Uppercase Kafka topics fail when startingOffsets are SpecificOffsets > -------------------------------------------------------------------- > > Key: SPARK-19853 > URL: https://issues.apache.org/jira/browse/SPARK-19853 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.1.0 > Reporter: Chris Bowden > Assignee: Genmao Yu > Priority: Trivial > Fix For: 2.1.1, 2.2.0 > > > When using the KafkaSource with Structured Streaming, consumer assignments > are not what the user expects if startingOffsets is set to an explicit set of > topics/partitions in JSON where the topic(s) happen to have uppercase > characters. When StartingOffsets is constructed, the original string value > from options is transformed toLowerCase to make matching on "earliest" and > "latest" case insensitive. However, the toLowerCase json is passed to > SpecificOffsets for the terminal condition, so topic names may not be what > the user intended by the time assignments are made with the underlying > KafkaConsumer. > From KafkaSourceProvider: > {code} > val startingOffsets = > caseInsensitiveParams.get(STARTING_OFFSETS_OPTION_KEY).map(_.trim.toLowerCase) > match { > case Some("latest") => LatestOffsets > case Some("earliest") => EarliestOffsets > case Some(json) => SpecificOffsets(JsonUtils.partitionOffsets(json)) > case None => LatestOffsets > } > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org