[
https://issues.apache.org/jira/browse/KAFKA-9739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064031#comment-17064031
]
Bill Bejeck commented on KAFKA-9739:
------------------------------------
Hi Artur,
I've taken a look at the code sample you've sent. I can' fully reproduce the
issue at the moment as there is some business code I don't have access to like
the `{{mapRecordToLookupEntry}}` method, for example. But by looking at the
code sample and the graph visualization, I've been able to get enough
information. I can say that you don't need to run the optimizer as there aren't
redundant repartition topics in the topology.
While there certainly is an issue, and thanks for reporting it, in this case, I
think it's okay for you to run the stream application with the optimizations
set to 'none' for now.
The reason I say that the optimizer won't help is that from your topology, as
each repartition topic is unique to each sub-topology. There is no shared
key-changing operation that causes multiple redundant repartition topics. For
example, consider this topology snippet:
{code:java}
mystream = builder.stream(topic).selectKey((k,v) -> ....);
mystream.groupByKey().windowedBy(....).count().toStream().to(....)
mystream.groupByKey().count().toStream().to(....)
{code}
This topology results in two repartition topics, but since they are
key-changing operation, both will have virtually the same data. In this case,
the optimizer can replace the two repartition topics with one directly on the
initial stream.
However, from looking at the supplied code example, the repartitions occur on
separate streams and inside distinct sub topologies.
For example
1. In sub-topology 0, there is a `{{transform}}` call followed by an
aggregation, so the resulting repartition topic can't get optimized away.
2. In sub-topology one there is `{{flatMap}}` call (when building the
retryStreom) which triggers another repartition in the resulting join
3. In sub-topology 2, when creating the `{{recipientIdTable}}`, there is a
`flatMap` call followed by a `groupByKey` then an aggregation, which again is a
repartition that can't get optimized away.
Thanks again for reporting this issue, as I'll continue to look into it. But
for the meantime, for the reasons stated above, IMHO, you should be fine to run
with the optimization set to none.
Thanks,
Bill
> StreamsBuilder.build fails with StreamsException "Found a null
> keyChangingChild node for OptimizableRepartitionNode"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9739
> URL: https://issues.apache.org/jira/browse/KAFKA-9739
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.1
> Reporter: Artur
> Assignee: Bill Bejeck
> Priority: Major
> Attachments: streams-exception-log.txt,
> topology-definition-fragment.java, topology-description.txt
>
>
> We created a topology using Streams DSL (topology description is available in
> the attached topology-description.txt, no optimization).
> Topology works fine with {{topology.optimization=none}}, however it fails to
> build with StreamsException "Found a null keyChangingChild node for
> OptimizableRepartitionNode" if we set {{topology.optimization=all}}
> (exception stack trace is attached streams-exception-log.txt).
> We used [https://zz85.github.io/kafka-streams-viz/] to visualize topology and
> try to guess what might be upsetting optimizer, yet did not manage to figure
> it out ourselves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
