tomstepp commented on code in PR #33596:
URL: https://github.com/apache/beam/pull/33596#discussion_r1938689204
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java:
##########
@@ -1573,6 +1586,27 @@ &&
runnerPrefersLegacyRead(input.getPipeline().getOptions()))) {
return input.apply(new ReadFromKafkaViaSDF<>(this, keyCoder,
valueCoder));
}
+ private void checkRedistributeConfiguration() {
+ if (getRedistributeNumKeys() == 0 && isRedistributed()) {
+ LOG.warn(
+ "withRedistribute without withRedistributeNumKeys will create a
key per record, which is sub-optimal for most use cases.");
+ }
+ if (isAllowDuplicates()) {
+ checkState(
Review Comment:
I was trying to make the Kafka read config checking more consistent, to
align with the direction we were going in with the offset deduplication config.
I've changed it back to a warning in case that is less friction for customer
upgrades and consistency with prior impl. If we want to change it back to a
check, perhaps we can handle it on a separate change than this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
