John Roesler created KAFKA-8307:
-----------------------------------
Summary: Kafka Streams should provide some mechanism to determine
topology equality and compatibility
Key: KAFKA-8307
URL: https://issues.apache.org/jira/browse/KAFKA-8307
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: John Roesler
Currently, Streams provides no mechanism to compare two topologies. This is a
common operation when users want to have tests verifying that they don't
accidentally alter their topology. They would save the known-good topology and
then add a unit test verifying the current code against that known-good state.
However, because there's no way to do this comparison properly, everyone is
reduced to using the string format of the topology (from
`Topology#describe().toString()`). The major drawback is that the string format
is meant for human consumption. It is neither machine-parseable nor stable. So,
these compatibility tests are doomed to fail when any minor, non-breaking,
change is made either to the application, or to the library. This trains
everyone to update the test whenever it fails, undermining its utility.
We should fix this problem, and provide both a mechanism to serialize the
topology and to compare two topologies for compatibility. All in all, I think
we need:
# a way to serialize/deserialize topology structure in a machine-parseable
format that is future-compatible. Offhand, I'd recommend serializing the
topology structure as JSON, and establishing a policy that attributes should
only be added to the object graph, never removed. Note, it's out of scope to be
able to actually run a deserialized topology; we only want to save and load the
structure (not the logic) to facilitate comparisons.
# a method to verify the *equality* of two topologies... This method tells you
that the two topologies are structurally identical. We can't know if the logic
of any operator has changed, only if the structure of the graph is changed. We
can consider whether other graph properties, like serdes, should be included.
# a method to verify the *compatibility* of two topologies... This method tells
you that moving from topology A to topology B does not require an application
reset. Note that this operation is not commutative: `A.compatibleWith(B)` does
not imply `B.compatibleWith(A)`. We can discuss whether `A.compatibleWith(B) &&
B.compatibleWith(A)` implies `A.equals(B)` (I think not necessarily, because we
may want "equality" to be stricter than "compatibility").
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
