[
https://issues.apache.org/jira/browse/SAMZA-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018076#comment-14018076
]
Chris Riccomini commented on SAMZA-144:
---------------------------------------
On the second thought, I'm not sure we can allow overrides for ANYTHING that's
defined at the connection or fetch request level.
The problem with config passed in at the connection-level is that connections
are per-broker (for consumers), and shared between streams. It's non-sensical
to have one stream define Configuration that's passed in at the fetch-request
level is again per-broker, and shared among streams.
So, what does this leave? Referring to:
http://kafka.apache.org/08/configuration.html
On the consumer side, I can find just two:
* fetchSize is per-topic/per-fetch (used in DefaultFetchSimpleConsumer in
Samza).
* auto.offset.reset is per-topic
On the producer side, I can't find a single config that we can override if we
share connections among streams.
So, perhaps the better strategy is to JUST allow fetch size and
auto.offset.reset stream-level overrides in the consumer. What do you think?
> Add stream-level overrides for KafkaSystemFactory config
> --------------------------------------------------------
>
> Key: SAMZA-144
> URL: https://issues.apache.org/jira/browse/SAMZA-144
> Project: Samza
> Issue Type: Bug
> Components: kafka
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Yan Fang
>
> Currently, we allow Kafka systems to be defined with:
> {noformat}
> systems.my-kafka-system.consumer.*
> {noformat}
> And:
> {noformat}
> systems.my-kafka-system.producer.*
> {noformat}
> We don't support stream-level configurations, the way we do with StreamConfig
> configuration:
> {code}
> val STREAM_PREFIX = "systems.%s.streams.%s."
> val MSG_SERDE = STREAM_PREFIX + "samza.msg.serde"
> val KEY_SERDE = STREAM_PREFIX + "samza.key.serde"
> val CONSUMER_RESET_OFFSET = STREAM_PREFIX + "samza.reset.offset"
> {code}
> We should add stream-level configuration override support for Kafka consumer
> and producers with:
> {noformat}
> systems.my-kafka-system.streams.my-kafka-topic.consumer.*
> systems.my-kafka-system.streams.my-kafka-topic.producer.*
> {noformat}
> KafkaConfig and KafkaSystemFactory should be updated to handle these.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
