[
https://issues.apache.org/jira/browse/KAFKA-9739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072255#comment-17072255
]
ASF GitHub Bot commented on KAFKA-9739:
---------------------------------------
bbejeck commented on pull request #8400: KAFKA-9739: Fixes null key changing
child node
URL: https://github.com/apache/kafka/pull/8400
For some context, when building a streams application, the optimizer keeps
track of the key-changing operations and any repartition nodes that are
descendants of the key-changer. During the optimization phase (if enabled),
any repartition nodes are logically collapsed into one. The optimizer updates
the graph by inserting the single repartition node between the key-changing
node and its first child node. This graph update process is done by searching
for a node that has the key-changing node as one of its direct parents, and the
search starts from the repartition node, going up in the parent hierarchy.
The one exception to this rule is if there is a merge node that is a
descendant of the key-changing node, then during the optimization phase, the
map tracking key-changers to repartition nodes is updated to have the merge
node as the key. Then the optimization process updates the graph to place the
single repartition node between the merge node and its first child node.
The error in KAFKA-9739 occurred because there was an assumption that the
repartition nodes are children of the merge node. But in the topology from
KAFKA-9739, the repartition node was a parent of the merge node. So when
attempting to find the first child of the merge node, nothing was found
(obviously) resulting in `StreamException(Found a null keyChangingChild node
for..)`
This PR fixes this bug by first checking that all repartition nodes for
optimization are children of the merge node.
This PR includes a test with the topology from KAFKA-9739.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> StreamsBuilder.build fails with StreamsException "Found a null
> keyChangingChild node for OptimizableRepartitionNode"
> --------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9739
> URL: https://issues.apache.org/jira/browse/KAFKA-9739
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.1
> Reporter: Artur Poliachenko
> Assignee: Bill Bejeck
> Priority: Major
> Attachments: streams-exception-log.txt,
> topology-definition-fragment.java, topology-description.txt
>
>
> We created a topology using Streams DSL (topology description is available in
> the attached topology-description.txt, no optimization).
> Topology works fine with {{topology.optimization=none}}, however it fails to
> build with StreamsException "Found a null keyChangingChild node for
> OptimizableRepartitionNode" if we set {{topology.optimization=all}}
> (exception stack trace is attached streams-exception-log.txt).
> We used [https://zz85.github.io/kafka-streams-viz/] to visualize topology and
> try to guess what might be upsetting optimizer, yet did not manage to figure
> it out ourselves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
