[
https://issues.apache.org/jira/browse/SAMZA-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015855#comment-14015855
]
Yan Fang commented on SAMZA-144:
--------------------------------
I am having a look at this issue and realizing that, when we support
stream-level consumer and porducer override, then, if the user specifies two
different Kafka brokers for two topics (streams), how do we define which broker
to use for the checkpoint? An easy solusion is, the user MUST specify
{code}
systems.my-kafka-system.producer.*
{code}
even thought they do stream-level consumer/producer override. Otherwise, we may
need to add some logic in checkpoint manager to define which Kafka broker to
use.
What do you think? Thank you.
> Add stream-level overrides for KafkaSystemFactory config
> --------------------------------------------------------
>
> Key: SAMZA-144
> URL: https://issues.apache.org/jira/browse/SAMZA-144
> Project: Samza
> Issue Type: Bug
> Components: kafka
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
>
> Currently, we allow Kafka systems to be defined with:
> {noformat}
> systems.my-kafka-system.consumer.*
> {noformat}
> And:
> {noformat}
> systems.my-kafka-system.producer.*
> {noformat}
> We don't support stream-level configurations, the way we do with StreamConfig
> configuration:
> {code}
> val STREAM_PREFIX = "systems.%s.streams.%s."
> val MSG_SERDE = STREAM_PREFIX + "samza.msg.serde"
> val KEY_SERDE = STREAM_PREFIX + "samza.key.serde"
> val CONSUMER_RESET_OFFSET = STREAM_PREFIX + "samza.reset.offset"
> {code}
> We should add stream-level configuration override support for Kafka consumer
> and producers with:
> {noformat}
> systems.my-kafka-system.streams.my-kafka-topic.consumer.*
> systems.my-kafka-system.streams.my-kafka-topic.producer.*
> {noformat}
> KafkaConfig and KafkaSystemFactory should be updated to handle these.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
