[
https://issues.apache.org/jira/browse/KAFKA-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297816#comment-17297816
]
A. Sophie Blee-Goldman commented on KAFKA-12313:
------------------------------------------------
Sure, feel free to kick off a discussion on this. This was just an idea that
came up during the discussion of KIP-659, so I'm not certain whether there's
consensus among the other devs. I guess it will be up to you to convince them
:)
You can look up the DISCUSS thread for KIP-659 in the kafka-dev mail archive
for the full picture.
By the way: by the end of KIP-659 we realized that we do still need some kind
of config for the two windowedSerde parameters (windowSize and innerClass).
This is because of the console consumer, which we want to enable to read
Windowed-type records but which cannot be passed a Deserializer object
directly. It relies on passing in the name of a Deserializer class and using
reflection to invoke the class's default constructor. So we do still need a
config. (In theory the same applies for the console producer & Serializer, but
it's more likely that Streams is the one writing Windowed records and a plain
consumer client is trying to read it rather than the other way around. Both are
valid use cases though)
That said, I think we can improve on the current situation. We still don't want
KafkaStreams users falling back on this config, for the reasons listed in the
ticket description. Here's what I've been thinking to expand on the original
proposal:
# Rename the "default.windowed.key.serde.inner" to "windowed.key.serde.inner"
– this brings the innerClass config name in line with the "window.size.ms"
config introduced in KIP-659, and emphasizes that it isn't a "default"
# Get rid of the "default.windowed.value.serde.inner" – there's really no
concept of a "WindowedValue" in Kafka Streams, unlike the "WindowedKey" which
appears in the DSL. The Windowed class in Streams itself assumes that the inner
class is a key. If a user really does want a WindowedValue in some custom
processor for some reason, they would have to write the WindowedValue class
themselves. In that case I think it's reasonable to assert that they should be
responsible for writing the WindowedValueSerializer class as well, and pass
this to the console consumer should they need to use it
# We could even take things a step further by requiring that users always pass
an actual WindowedSerde object to any operators which require it in the DSL,
and disallow falling back on the "windowed.key.serde.inner" at all. If we want
to go with this, we should make sure to do the same for the new
"window.size.ms" config as well.
WDYT?
> Consider deprecating the default.windowed.serde.inner.class configs
> -------------------------------------------------------------------
>
> Key: KAFKA-12313
> URL: https://issues.apache.org/jira/browse/KAFKA-12313
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
> Labels: needs-kip
> Fix For: 3.0.0
>
>
> During the discussion of KIP-659 we discussed whether it made sense to have a
> "default" class for the serdes of windowed inner classes across Streams.
> Using these configs instead of specifying an actual Serde object can lead to
> subtle bugs, since the WindowedDeserializer requires a windowSize in addition
> to the inner class. If the default constructor is invoked, as it will be when
> falling back on the config, this windowSize defaults to MAX_VALUE.
> If the downstream program doesn't care about the window end time in the
> output, then this can go unnoticed and technically there is no problem. But
> if anything does depend on the end time, or the user just wants to manually
> read the output for testing purposes, then the MAX_VALUE will result in a
> garbage timestamp.
> We should consider whether the convenience of specifying a config instead of
> instantiating a Serde in each operator is really worth the risk of a user
> accidentally failing to specify a windowSize
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
