[
https://issues.apache.org/jira/browse/KAFKA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick O'Keeffe updated KAFKA-12453:
-------------------------------------
Description:
Since the introduction of KStream.toTable() in Kafka 2.6.x, the decision about
whether a topology is eligible for optimisation is no longer a simple one, and
is related to whether toTable() operations are preceded by key changing
operators.
This decision requires expert level knowledge, and there are serious
implications associated with getting it wrong in terms of fault tolerance
Some idea spring to mind around how to guide developers to make the correct
decision:
# Topology.describe() could indicate whether this topology is eligible for
optimisation
# Topologies could be automatically optimised - note this may have an impact
at deployment time, in that an application reset may be required. The developer
would need to made aware of this and adjust the deployment instructions
accordingly
was:
Since the introduction of KStream.toTable() in Kafka 2.6.x, the decision about
whether a topology is eligible for optimisation is no longer a simple one, and
is related to whether toTable() operations are preceded by key changing
operators.
This decision requires expert level knowledge, and there are serious
implications associated with getting it wrong in terms of fault tolerance
Some idea spring to mind:
# Topology.describe() could indicate whether this topology is eligible for
optimisation
# Topologies could be automatically optimised - note this may have an impact
at deployment time, in that an application reset may be required. The developer
would need to made aware of this and adjust the deployment instructions
accordingly
> Guidance on whether a topology is eligible for optimisation
> -----------------------------------------------------------
>
> Key: KAFKA-12453
> URL: https://issues.apache.org/jira/browse/KAFKA-12453
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Patrick O'Keeffe
> Priority: Major
>
> Since the introduction of KStream.toTable() in Kafka 2.6.x, the decision
> about whether a topology is eligible for optimisation is no longer a simple
> one, and is related to whether toTable() operations are preceded by key
> changing operators.
> This decision requires expert level knowledge, and there are serious
> implications associated with getting it wrong in terms of fault tolerance
> Some idea spring to mind around how to guide developers to make the correct
> decision:
> # Topology.describe() could indicate whether this topology is eligible for
> optimisation
> # Topologies could be automatically optimised - note this may have an impact
> at deployment time, in that an application reset may be required. The
> developer would need to made aware of this and adjust the deployment
> instructions accordingly
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
