[
https://issues.apache.org/jira/browse/SPARK-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-8865:
---------------------------------
Description:
"zookeeper.connect" and "group.id" aren't necessary for anything in the kafka
direct stream.
But they're expected to be present in a kafka consumer config, and overriding
that behavior wasn't possible. So as a workaround, we set them to a blank
string. That way users don't have to define unnecessary settings in the kafka
param map passed to the KafkaUtils constructor. We talked through that during
the original development of the direct stream.
The code as it is released today is almost always going to set a blank string,
regardless of what users pass in, because contains on a java property object is
not the equivalent of containsKey, it is containsValue. The intention was that
if the user sets those properties (whatever personal reasons they have), the
values should not get overwritten with a blank string.
> Fix bug: init SimpleConsumerConfig with kafka params
> -----------------------------------------------------
>
> Key: SPARK-8865
> URL: https://issues.apache.org/jira/browse/SPARK-8865
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Reporter: guowei
> Assignee: guowei
> Priority: Minor
> Fix For: 1.3.2, 1.4.2, 1.5.0
>
>
> "zookeeper.connect" and "group.id" aren't necessary for anything in the kafka
> direct stream.
> But they're expected to be present in a kafka consumer config, and overriding
> that behavior wasn't possible. So as a workaround, we set them to a blank
> string. That way users don't have to define unnecessary settings in the kafka
> param map passed to the KafkaUtils constructor. We talked through that during
> the original development of the direct stream.
> The code as it is released today is almost always going to set a blank
> string, regardless of what users pass in, because contains on a java property
> object is not the equivalent of containsKey, it is containsValue. The
> intention was that if the user sets those properties (whatever personal
> reasons they have), the values should not get overwritten with a blank string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
