Hey Guozhang,
thanks for the reply. Actually I'm not trying to pass config into
StreamPartitionAssignor. My debugging was about some other configs that wasn't
transferred properly (completely irrelevant), but the effort was delayed
because the log would print "StreamsConfig values" for StreamPartitionAssignor,
where it will fallback all the config to defaults. So it would give me as an
end user the impression that my config passed in was "not successfully"
conveyed to each Stream Thread.
I'm thinking whether we could disable stream partitioner to print out its
config in the log, or have a more explicit way saying this is not Stream Thread
config? Hope I made myself clear to you.
Also yes, I could put some comment in the docs to let user use Consumer config
prefix if that could help.
Best,
Boyang
________________________________
From: Guozhang Wang <wangg...@gmail.com>
Sent: Tuesday, July 10, 2018 5:35 AM
To: dev
Subject: Re: Stream config in StreamPartitionAssignor
Hello Boyang,
Thanks for brining this up. Currently since the StreamsPartitionAssingor
can only be initiated within the Consumer instance, I think letting users
to pass in the config values with prefix is the preferrable way, i.e. we
can improve our docs to educate users about that. BTW I'm curious what
configs you want to pass into the StreamsPartitionAssignor? Currently since
there is no user configurable part inside that class, I do not know what
configs can take effects.
Guozhang
On Mon, Jul 9, 2018 at 2:06 PM, Boyang Chen <bche...@outlook.com> wrote:
> Hey there,
>
>
> over the weekend I was debugging the streams configuration not passed
> within threads. I noticed that one of the code path from KafkaConsumer
> (L743) was to initialize the StreamPartitionAssignor:
>
> this.assignors = config.getConfiguredInstances(
> ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> PartitionAssignor.class);
>
> However, it was using the ConsumerConfig instance (that config is passed
> in), so if I want to make some configuration change in the assignor, I need
> to put consumer prefix. To make the debugging even harder, there was an
> logAll() function in AbstractConfig which will print "StreamsConfig values"
> at the beginning, since it is indeed a stream config:
>
> @Override
> public void configure(final Map<String, ?> configs) {
> final StreamsConfig streamsConfig = new StreamsConfig(configs);
>
> (L190 in StreamPartitionAssignor)
>
>
> This would further confuse developer as they see two different sets of
> StreamsConfig: one from top level, one from this derived level per thread.
>
>
> My point is that we could either: 1. let developer be aware that they need
> to add consumer prefix to pass in configs to StreamPartitionAssignor 2. we
> found a way to pass in original StreamsConfig.
>
> I know this is a little bit lengthy description, let me know if you feel
> unclear about my proposal, or this is not a concern since most people
> already know the trick here, thank you!
>
>
--
-- Guozhang
