Hi,
Thanks again for your feedbacks
MJS8 (follow up):
You are right, since we are not overriding any method we are free to make
it type-safe.
MJS8 (a):
I added "For output records, single-partition mode always produces
partition = 0. Since all records are emitted from the only partition
available, this preserves the behaviour of existing tests."
MJS11. The KIP says:
> Topics not declared via declareTopic() are automatically assigned 1
partition. For example, if declareTopic("a", 3) is declared but topic "b"
is not, topic "b" will have 1 partition. The driver will still operate in
multi-partition mode since topic "a" was declared with more than 1
partition.
You are right, all input topics should be declared before the build() call
since we need to initialize all tasks. Otherwise it can be confusing for
user.
For output topic, it is a little different we can afford to set them with 1
partition by default because the mock producer can handle them.
I added:
"An exception is thrown if a record is piped into an undeclared input topic
while the driver is operating in multi-partition mode (that is, at least
one topic has a partition count greater than 1).
Output topics do not need to be declared. Unlike input topics, they do not
participate in task initialization and are therefore automatically created
with a single partition."
Cheers !
Le ven. 12 juin 2026 à 02:13, Matthias J. Sax <[email protected]> a écrit :
> Thanks.
>
>
>
> MJS8 (follow up): For `equalsIgnorePartition()`, it seems we could use
> `TestRecord<K, V>, instead of undefined generic types? -- If the types
> don't match, we cannot really expect that both records are equal to each
> other to begin with, and the test code seems to be incorrect to begin
> with, and should not compile? -- Or is the test scenarios I miss, for
> which `TestRecord<?, ?>` would be required?
>
>
>
> MJS8 (a). Edit suggestion. The KIP says:
>
> > Existing code that does not set a partition explicitly will produce
> partition = -1 on both sides of an equality check, preserving backward
> compatibility.
>
> This applies to input TestRecords. It might be helpful to also add a
> sentence for output TestRecords for existing code, which runs in
> single-partition mode: all `TestRecords` would get partition=1, so it's
> also backward compatible.
>
>
>
> MJS11. The KIP says:
>
> > Topics not declared via declareTopic() are automatically assigned 1
> partition. For example, if declareTopic("a", 3) is declared but topic "b"
> is not, topic "b" will have 1 partition. The driver will still operate in
> multi-partition mode since topic "a" was declared with more than 1
> partition.
>
> Do I read this right, that we would allow to have not declared topics in
> multi-partition mode? Just wondering if we can actually implement it? My
> understanding was, but I could be wrong, that for multi-partition mode,
> it would only work correctly if all topics got declared? At least, I
> would believe all input topics need to be declared upfront? I can see
> that output topic don't need to be declared and can safely default to
> having one partition.
>
> I am totally ok with supporting not declared topis, just want to double
> check that the KIP does not say something we cannot build.
>
>
>
>
> -Matthias
>
>
> On 6/11/26 1:14 PM, Sébastien Viale wrote:
> > Hi all,
> >
> > Thank you for the follow-up.
> >
> > MJS6 (follow up)
> >
> > We decided to go with the builder as the single entry point, with
> implicit
> > mode detection based on declared partition counts (at least one topic
> > declared with more than 1 partition triggers multi-partition mode)
> >
> > Indeed, we think that a single entry point is simpler for users.
> >
> > We also propose to deprecate TopologyTestDriver constructors in order to
> > avoid having to ways to declare TDD
> >
> > I have updated the KIP accordingly.
> >
> >
> > MJS8
> >
> > You are right, I replaced Object by TestRecord
> >
> >
> > MS10: Nit
> >
> > Fixed also
> >
> >
> > cheers !
> >
> >
> >
> >
> > Le lun. 8 juin 2026 à 02:16, Matthias J. Sax <[email protected]> a écrit
> :
> >
> >> Thanks.
> >>
> >>
> >>
> >> MJS6 (follow up): The KIP says
> >>
> >>> Multi-partition mode is implicitly enabled when at least one topic is
> >> declared with a partition count greater than 1.
> >>
> >> and
> >>
> >>> If all topics are declared with a partition count of 1, or no topics
> are
> >> declared at all, the driver behaves identically to the legacy
> constructors
> >>
> >> I am wondering if this make it more complex for users to reason about
> >> the mode? Would it be simpler to use singe-partition mode only for `new
> >> TopologyTestDriver` and multi-partition mode always when the new builder
> >> is used?
> >>
> >> For this case, we would be in multi-partition mode even if all declared
> >> topics have 1 partition. -- And if the builder is used, we would always
> >> require all topics (even if all have 1 partition) to be declared.
> >>
> >> We would also disable the `getStateStore()` methods w/o a partition
> >> parameter when the builder is used, and only support them for `new
> >> TopologyTestDriver` (which is the only way to use single-partition
> mode).
> >>
> >> Given that we don't propose to deprecate `new TopologyTestDriver`, I
> >> believe this design would be cleaner and easier to understand for users?
> >> Two modes, and two different entry points:
> >>
> >> - new TopologyTestDriver == single-partition mode
> >> - Builder == multi-partition mode
> >>
> >> For this case, we would never deprecate `new TopologyTestDriver` I
> guess?
> >>
> >> If we want to support single-partition mode using the builder, I would
> >> propose to deprecate `new TopologyTestDriver` with this KIP. No need to
> >> keep it? Or is there a reason why we would defer this to a follow-up
> KIP?
> >>
> >>
> >>
> >> MJS8: Given that `equalsIgnorePartition()` is newly added method, it
> >> could take `TestRecord` as parameter? No need to go with `Object` (only
> >> required for `equals()` to use `Object` as we inherit it).
> >>
> >>
> >>
> >> MS10: Nit.
> >>
> >>> Only after build() can records be piped and outputs be read.
> >>
> >> This is obviously true, because only `build()` create the TTD object
> >> allowing to create `TestInput|OutputTopics`. Guess it's an relict from
> >> the previous version of the KIP w/o the builder class.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 6/5/26 12:09 AM, Sébastien Viale wrote:
> >>> Hi all,
> >>>
> >>> MJS6
> >>> Actually yes, the single-partition mode will behave the same as
> >>> multi-partition mode with one single partition for all topics
> >>> So yes, the TopologyTestDriver constructors could be deprecated /
> removed
> >>> in the future.
> >>>
> >>> MJS7:
> >>> Good catch indeed.
> >>>
> >>> We can use -1 as default the default value to clearly indicate that no
> >>> explicit partition was set, making the routing strategy unambiguous
> >>>
> >>> Concretely:
> >>>
> >>> - Constructors without explicit partition should default to -1.
> >>> - In pipeInput:
> >>> - if partition == -1: delegate to TopologyTestDriverPartitioner
> >>> - if partition >= 0: use it directly (with range validation)
> >>>
> >>> equals() / hashCode() methods include the partition field:
> >>>
> >>> - existing code will now also produce -1 when no partition is set
> >>> explicitly
> >>> - backward compatibility for equality checks is preserved
> >>>
> >>> Finally, -1 never appears in output records (readRecordsToList()),
> since
> >>> those always carry the resolved partition, it only exists as an input
> >>> sentinel value indicating “no explicit partition”.
> >>>
> >>> I'll update the KIP accordingly.
> >>> thanks
> >>>
> >>> Le jeu. 4 juin 2026 à 21:08, Matthias J. Sax <[email protected]> a
> écrit
> >> :
> >>>
> >>>> Thanks for the update.
> >>>>
> >>>>
> >>>> Few more questions:
> >>>>
> >>>> MJS6: Would it be simpler to (in the long run) drop single-partition
> >>>> mode all together? We would only keep it now for backward
> compatibility
> >>>> if `new TopologyTestDriver()` is used, but would always use
> >>>> multi-partition mode when then new builder is used?
> >>>>
> >>>> It seems to add complexity to keep single-partition mode for the
> builder
> >>>> case?
> >>>>
> >>>> If we say, the new builder always does multi-partition mode, and we
> >>>> deprecate/remove `new TopologyTestDriver()` later, single-partition
> mode
> >>>> would get removed, too.
> >>>>
> >>>> Or is there something specific about single-partition mode that makes
> it
> >>>> worth keeping for the new builder case? I would believe, with the only
> >>>> exception that multi-partition mode does create more tasks, if all
> >>>> topics are defined with a single partition, the record piping would be
> >>>> the same as in single-partition mode? Or is there a difference?
> >>>>
> >>>>
> >>>>
> >>>> MJS7:
> >>>>> Update equals() and hashCode() to include the partition field. For
> >>>> records created via existing constructors, partition defaults to 0,
> >>>> preserving backward compatibility for existing assertEquals calls.
> >>>>
> >>>> Should it default to `-1` (or `Optional.empty()`) instead of `0`?
> Making
> >>>> the default `0` seems to not work with "multi-partition mode" but
> break
> >>>> it -- as it would skip the hashing/partitioning, as `0` would be set
> >>>> explicitly?
> >>>>
> >>>> And for `-1` as default and topics with a single partition, `-1` will
> >>>> get hashed/mapped to partition-0 correctly.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 6/4/26 5:47 AM, Sébastien Viale wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Thanks for the feedback and suggestions.
> >>>>>
> >>>>>
> >>>>> MJS1:
> >>>>>
> >>>>> I fixed the typo `woraround` => workaround`
> >>>>>
> >>>>>
> >>>>> MJS2:
> >>>>>
> >>>>> I fixed the inconsistency:
> >>>>>
> >>>>> - removed duplicate entry for getKeyValueStore
> >>>>> - added missing *WithHeaders variants for state store APIs
> >>>>>
> >>>>>
> >>>>> MJS3:
> >>>>>
> >>>>> The existing usages of TopologyTestDriver remain fully unchanged. In
> >>>>> particular, current code relying on the default behavior (i.e., no
> >>>> explicit
> >>>>> topic declaration) will continue to work as before.
> >>>>>
> >>>>> Topic declaration is only required when a test explicitly opts into
> >>>>> multi-partition behavior. Otherwise, topics continue to be created
> >>>>> implicitly with the current semantics.
> >>>>>
> >>>>> We will introduce a TopologyTestDriverBuilder class:
> >>>>>
> >>>>> public class TopologyTestDriverBuilder {
> >>>>> public TopologyTestDriverBuilder(Topology topology);
> >>>>> public TopologyTestDriverBuilder withConfig(Properties
> config);
> >>>>> public TopologyTestDriverBuilder
> withInitialWallClockTime(Instant
> >>>>> initialWallClockTime);
> >>>>> public TopologyTestDriverBuilder declareTopic(String
> topicName,
> >>>>> int partitions);
> >>>>> public TopologyTestDriver build();
> >>>>> }
> >>>>>
> >>>>> In this model, multi-partition mode is implicitly enabled when at
> least
> >>>> one
> >>>>> topic is declared with a partition count greater than 1.
> >>>>>
> >>>>> As a result, the explicit withMultiPartitionMode() is no longer
> needed.
> >>>>>
> >>>>> The existing TopologyTestDriver constructors remain unchanged for
> >>>> backward
> >>>>> compatibility.
> >>>>>
> >>>>> If it is ok for you all, deprecation of those constructors is
> >>>> intentionally
> >>>>> deferred to a follow-up KIP to keep the scope of this change focused.
> >>>>>
> >>>>>
> >>>>> MJS4:
> >>>>>
> >>>>> For this KIP, we wanted initially to make the partitioning behavior
> >>>>> deterministic and test-controlled.
> >>>>>
> >>>>> Supporting partitioner.class would indeed be a natural extension, and
> >>>>> aligns well with production behavior. We propose deferring this to a
> >>>>> follow-up KIP to keep this one focused
> >>>>>
> >>>>> However, we will implement the internal routing logic with an
> internal
> >>>>> TopologyTestDriverPartitioner that implements Partitioner. Which
> would
> >>>>> provide a natural extension point for future support of custom
> >>>> partitioners.
> >>>>>
> >>>>>
> >>>>> MJS5:
> >>>>> -
> >>>>>
> >>>>> Your proposal seems like the best compromise:
> >>>>> - - TestRecord.equals() and TestRecord.hashCode() will be updated to
> >>>>> include the partition field.
> >>>>> - For existing code using the current constructors (without a
> partition
> >>>>> argument), the partition defaults to 0, preserving backward
> >> compatibility
> >>>>> for existing assertEquals() assertions.
> >>>>>
> >>>>> - - A new utility method TestRecord.equalsIgnorePartition(TestRecord
> >>>>> other) will
> >>>>> be added for cases where the test author does not want to assert on
> >>>>> partition placement.
> >>>>> - This allows use with
> >>>> assertTrue(expected.equalsIgnorePartition(actual)) when
> >>>>> partition is irrelevant to the test.
> >>>>> -
> >>>>> - The KIP has been updated.
> >>>>> -
> >>>>> - Cheers
> >>>>>
> >>>>>
> >>>>> Le mer. 3 juin 2026 à 22:56, Matthias J. Sax <[email protected]> a
> >> écrit
> >>>> :
> >>>>>
> >>>>>> Thanks for the great discussion and KIP refinements.
> >>>>>>
> >>>>>> Couple of comments:
> >>>>>>
> >>>>>>
> >>>>>> MJS1: (nit) there is a typo `woraround`
> >>>>>>
> >>>>>>
> >>>>>> MJS2: the KIP lists `getKeyValueStore` twice, and it's missing the
> >> newly
> >>>>>> added `*WithHeader` variants.
> >>>>>>
> >>>>>>
> >>>>>> MJS3: About the builder pattern. In general I like it, but it seems
> >> the
> >>>>>> applied patter is a little bit odd, and also not backward compatible
> >> (or
> >>>>>> maybe I misunderstand the proposal)?
> >>>>>>
> >>>>>> Existing code, would only call `new TopologyTestDriver()`, and it
> >> seems
> >>>>>> this code would break, and would need to get updated to add the
> >>>>>> dedicated setup phase? W/o declaring the topics explicitly, it seems
> >>>>>> later calls to `createInputTopic` and `createOutputTopic` would fail
> >> (at
> >>>>>> least on what the KIP says I believe; could also be a phrasing
> problem
> >>>>>> only)?
> >>>>>>
> >>>>>> I also think, that we would need to have some "builder helper
> class".
> >>>>>> Adding the methods to `TopologyTestDriver` can leave the object in
> an
> >>>>>> weird state. A classic builder pattern would suggest, that we add
> >>>>>>
> >>>>>> public class TTDBuilder { // or similar/better name
> >>>>>>
> >>>>>> public TTDBuilder(Topology topology);
> >>>>>>
> >>>>>> TTDBuilder withConfig(Properties config);
> >>>>>> TTDBuilder withInitialWallClockTime(Instant
> >> initialWallClockTime);
> >>>>>>
> >>>>>> TTDBuilder declareTopic(String topicName, int partitions);
> >>>>>>
> >>>>>> TopologyTestDriver build();
> >>>>>> }
> >>>>>>
> >>>>>> For this case, I don't think that we even need
> >>>> `withMultiPartitionMode()`?
> >>>>>>
> >>>>>> We would also deprecate the existing constructors `new
> >>>>>> TopologyTestDriver`, which we still need to support thought. For
> `new
> >>>>>> TopologyTestDriver()` we would just accept all topic names for
> >>>>>> `createInputTopic` and `createOutputTopic` and create these topic
> >> with a
> >>>>>> single partition. For this case, we would also still allow to use
> the
> >>>>>> existing `getStateStore()` methods, which we might also want to
> >>>> deprecate?
> >>>>>>
> >>>>>> (Maybe this is already proposed, but the KIP itself is not 100%
> clear
> >>>>>> about it -- at least to me.)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> MJS4: about partitioning; if the passed `Properties` configure
> >>>>>> `partitioner.class`, I think we could just use it allowing to
> plugin a
> >>>>>> global customer partitioner? Could also be a follow up KIP (maybe
> >>>>>> including the ability to define a customer partitioner per input
> >> topic).
> >>>>>>
> >>>>>> In general, I am wondering if we should implement the discussed
> >>>>>> partition-routing strategy just as an internal `TDDPartitioner
> extends
> >>>>>> Partitioner` (of course, this is an implementation detail). This
> would
> >>>>>> setup the code nicely to support `partitioner.class` config.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> MJS5: About testing and the new `partition` field on `TestRecord`. I
> >>>>>> think that both `equals()` and `hashCode()` should be considered.
> For
> >>>>>> existing code, all `TestRecord`s get set partition=0 and thus it's
> >>>>>> backward compatible. For new code, checking the expected partition
> >> sound
> >>>>>> desirable?
> >>>>>>
> >>>>>> I still believe that for some tests, the partition is not necessary.
> >> For
> >>>>>> this case, we could add
> >> `TestRecord.equalsIgnorePartition(TestRecord)`,
> >>>>>> and users can use `assertTrue()` for this case?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 6/1/26 10:46 AM, Sébastien Viale wrote:
> >>>>>>> Thanks for your response.
> >>>>>>>
> >>>>>>> BB1:
> >>>>>>> I have removed the contradictory statement.
> >>>>>>>
> >>>>>>> LB5:
> >>>>>>> OK, I will keep the KIP as it is.
> >>>>>>>
> >>>>>>> I will call for the vote
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> Le lun. 1 juin 2026 à 16:33, Bill Bejeck <[email protected]> a
> >> écrit
> >>>> :
> >>>>>>>
> >>>>>>>> Hi Sebastian,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP! This will be a great, much needed addition.
> >>>>>>>> I realize I'm a bit late to the discussion. Overall things look
> >> good
> >>>>>> to me
> >>>>>>>> and I have one nit comment. I also think the KIP is ready for a
> >> vote.
> >>>>>>>>
> >>>>>>>> Regarding the treatment of null keys I agree with Lucas's comment
> >> for
> >>>>>> the
> >>>>>>>> treatment of null keys (round-robin) distribution.
> >>>>>>>> In "Partition Routing" section point 2 seems to reflection this
> >>>> change
> >>>>>> but
> >>>>>>>> then then it concludes with "a null key goes to a default
> partition
> >> (0
> >>>>>> in
> >>>>>>>> the test driver)" . I find this a little contradictory, can we
> >> either
> >>>>>>>> clarify or remove the statement?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Bill
> >>>>>>>>
> >>>>>>>> On Mon, Jun 1, 2026 at 8:33 AM Lucas Brutschy via dev <
> >>>>>>>> [email protected]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Sebastian,
> >>>>>>>>>
> >>>>>>>>> Thanks for the response!
> >>>>>>>>>
> >>>>>>>>> LB5: I was thinking of this solution but I think it introduces
> more
> >>>>>>>>> problems than it solves. Long-term, I feel it would be a footgun
> if
> >>>>>>>>> people try to compare test records _with_ partitions. And the
> >> current
> >>>>>>>>> implementation with a reasonable default partition should already
> >> be
> >>>>>>>>> backward compatible. There are other possible solutions, like a
> >>>> custom
> >>>>>>>>> assertion or an operation to "wipe" the partition, but they are
> >>>>>>>>> probably not worth the effort. As I mentioned before, I don't
> see a
> >>>>>>>>> better solution to this -- I would just keep the KIP as it is. I
> >>>>>>>>> mostly included it in the discussion to make people aware and
> maybe
> >>>>>>>>> spark an idea in someone.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Lucas
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
