Build failed in Jenkins: kafka-trunk-jdk8 #4250

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9544; Fix flaky test

[github] MINOR: fix omitted 'next.' in interactive queries documentation (#7883)

[github] KAFKA-9533: ValueTransform forwards `null` values (#8108)


--
[...truncated 2.88 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task 

Build failed in Jenkins: kafka-trunk-jdk8 #4249

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9558; Fix retry logic in KafkaAdminClient listOffsets (#8119)


--
[...truncated 1.64 MB...]
kafka.coordinator.group.GroupMetadataManagerTest > 
testLoadGroupAndOffsetsFromDifferentSegments STARTED

kafka.coordinator.group.GroupMetadataManagerTest > 
testLoadGroupAndOffsetsFromDifferentSegments PASSED

kafka.coordinator.group.GroupMetadataManagerTest > 
testOffsetExpirationSemantics STARTED

kafka.coordinator.group.GroupMetadataManagerTest > 
testOffsetExpirationSemantics PASSED

kafka.coordinator.group.GroupMetadataManagerTest > testExpireOffset STARTED

kafka.coordinator.group.GroupMetadataManagerTest > testExpireOffset PASSED

kafka.coordinator.group.GroupMetadataManagerTest > 
testExpireGroupWithOffsetsOnly STARTED

kafka.coordinator.group.GroupMetadataManagerTest > 
testExpireGroupWithOffsetsOnly PASSED

kafka.coordinator.group.GroupMetadataManagerTest > 
testDoNotLoadAbortedTransactionalOffsetCommits STARTED

kafka.coordinator.group.GroupMetadataManagerTest > 
testDoNotLoadAbortedTransactionalOffsetCommits PASSED

kafka.coordinator.group.GroupMetadataManagerTest > testStoreEmptyGroup STARTED

kafka.coordinator.group.GroupMetadataManagerTest > testStoreEmptyGroup PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody PASSED

kafka.tools.ConsumerPerformanceTest > testConfigWithUnrecognizedOption STARTED

kafka.tools.ConsumerPerformanceTest > testConfigWithUnrecognizedOption PASSED

kafka.tools.ConsumerPerformanceTest > testBrokerListOverride STARTED

kafka.tools.ConsumerPerformanceTest > testBrokerListOverride PASSED

kafka.tools.ConsumerPerformanceTest > testConfigBootStrapServer STARTED

kafka.tools.ConsumerPerformanceTest > testConfigBootStrapServer PASSED

kafka.tools.ConsumerPerformanceTest > testConfigBrokerList STARTED

kafka.tools.ConsumerPerformanceTest > testConfigBrokerList PASSED

kafka.tools.ConsumerPerformanceTest > testNonDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testNonDetailedHeaderMatchBody PASSED

kafka.tools.ReplicaVerificationToolTest > testReplicaBufferVerifyChecksum 
STARTED

kafka.tools.ReplicaVerificationToolTest > testReplicaBufferVerifyChecksum PASSED

kafka.tools.MirrorMakerIntegrationTest > testCommaSeparatedRegex STARTED

kafka.tools.MirrorMakerIntegrationTest > testCommaSeparatedRegex PASSED

kafka.tools.MirrorMakerIntegrationTest > 
testCommitOffsetsRemoveNonExistentTopics STARTED

kafka.tools.MirrorMakerIntegrationTest > 
testCommitOffsetsRemoveNonExistentTopics PASSED

kafka.tools.MirrorMakerIntegrationTest > testCommitOffsetsThrowTimeoutException 
STARTED

kafka.tools.MirrorMakerIntegrationTest > testCommitOffsetsThrowTimeoutException 
PASSED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage STARTED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandlerWithHeaders 
STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandlerWithHeaders 
PASSED

unit.kafka.utils.ThrottlerTest > testThrottleDesiredRate STARTED

unit.kafka.utils.ThrottlerTest > testThrottleDesiredRate PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[0] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[0] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[1] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[1] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[2] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[2] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[3] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[3] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[4] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[4] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[5] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[5] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[6] 
STARTED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[6] PASSED

unit.kafka.cluster.AssignmentStateTest > testPartitionAssignmentStatus[7] 
STARTED

unit.kafka.cluster.AssignmentStateTest > 

Jenkins build is back to normal : kafka-2.5-jdk8 #33

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-9577) Client encountering SASL_HANDSHAKE protocol version errors on 2.5 / trunk

[Lucas Bradstreet (Jira)]

Lucas Bradstreet created KAFKA-9577:
---

 Summary: Client encountering SASL_HANDSHAKE protocol version 
errors on 2.5 / trunk
 Key: KAFKA-9577
 URL: https://issues.apache.org/jira/browse/KAFKA-9577
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.5.0
Reporter: Lucas Bradstreet


I am trying 2.5.0 with sasl turned on and my consumer clients receive:
{noformat}
org.apache.kafka.common.errors.UnsupportedVersionException: The SASL_HANDSHAKE 
protocol does not support version 2
{noformat}
I believe this is due to 
[https://github.com/apache/kafka/commit/0a2569e2b9907a1217dd50ccbc320f8ad0b42fd0]
 which added flexible version support and bumped the protocol version.

It appears that the SaslClientAuthenticator uses the max version for 
SASL_HANDSHAKE returned by the broker's api versions request, and then uses 
that version even though it may not support it. See 
[https://github.com/apache/kafka/blob/eb09efa9ac79efa484307bdcf03ac8eb8a3a94e2/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java#L290].
 

This may make it hard to ever evolve this schema. In the short term I suggest 
we roll back the version bump and flexible schema until we figure out a path 
forward.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk11 #1172

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-9533) ValueTransform forwards `null` values

[Bill Bejeck (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck resolved KAFKA-9533.

Resolution: Fixed

> ValueTransform forwards `null` values
> -
>
> Key: KAFKA-9533
> URL: https://issues.apache.org/jira/browse/KAFKA-9533
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.4.0
>Reporter: Michael Viamari
>Assignee: Michael Viamari
>Priority: Minor
>
> According to the documentation for `KStream#transformValues`, nulls returned 
> from `ValueTransformer#transform` are not forwarded. (see 
> [KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-]
> However, this does not appear to be the case. In 
> `KStreamTransformValuesProcessor#process` the result of the transform is 
> forwarded directly.
> {code:java}
>  @Override
>  public void process(final K key, final V value) {
>  context.forward(key, valueTransformer.transform(key, value));
>  }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

ConsumerRebalanceListener not being invoked

[Sohil Shah]

Hello Kafka Team,

I just cannot get my registered "ConsumerRebalanceListener" to be invoked.

GitHub Commit for a full picture if that helps:
https://github.com/slydogshah/appGalCloud/commit/323a447929a17d8449b931a2972a746251dce495

Here is my KafkaConsumer bootstrap code


Properties config = new Properties();
config.put("client.id", InetAddress.getLocalHost().getHostName());
config.put("group.id", "foodRunnerSyncProtocol_notifications");
config.put("bootstrap.servers", "localhost:9092");
config.put("key.deserializer",
org.apache.kafka.common.serialization.StringDeserializer.class);
config.put("value.deserializer",
org.springframework.kafka.support.serializer.JsonDeserializer.class);
config.put("key.serializer",
org.apache.kafka.common.serialization.StringSerializer.class);
config.put("value.serializer",
org.springframework.kafka.support.serializer.JsonSerializer.class);
//config.put("session.timeout.ms", 3);

this.kafkaConsumer = new KafkaConsumer(config);

KafkaRebalanceListener rebalanceListener = new
KafkaRebalanceListener(this.topicPartitions);
this.kafkaConsumer.subscribe(topics, rebalanceListener);


public class KafkaRebalanceListener implements ConsumerRebalanceListener {
private static Logger logger =
LoggerFactory.getLogger(KafkaRebalanceListener.class);

private Map> topicPartitions;

public KafkaRebalanceListener(Map>
topicPartitions) {
this.topicPartitions = topicPartitions;
}

@Override
public void onPartitionsRevoked(Collection collection) {

logger.info("PARTITIONS_REVOKED**");
logger.info("");
}

@Override
public void onPartitionsAssigned(Collection partitions) {
List partitionList =
Arrays.asList(partitions.toArray(new TopicPartition[0]));

for (TopicPartition topicPartition : partitionList) {
String registeredTopic = topicPartition.topic();

List local = topicPartitions.get(registeredTopic);
if (local != null) {
local.add(topicPartition);
logger.info("**");
logger.info("NUMBER_OF_PARTITIONS registered for :(" +
registeredTopic + ") " + topicPartitions.size());
logger.info("**");
} else {
topicPartitions.put(registeredTopic,
Arrays.asList(topicPartition));
}
}
}

@Override
public void onPartitionsLost(Collection partitions) {
logger.info("PARTITIONS_LOST**");
logger.info("");

}
}


and Logs

2020-02-19 11:26:48,387 INFO  [io.app.clo.mes.KafkaDaemon] (main) **
2020-02-19 11:26:48,387 INFO  [io.app.clo.mes.KafkaDaemon] (main)
STARTING_KAFKA_DAEMON
2020-02-19 11:26:48,387 INFO  [io.app.clo.mes.KafkaDaemon] (main) **
2020-02-19 11:26:53,408 INFO  [org.apa.kaf.cli.con.ConsumerConfig]
(ForkJoinPool.commonPool-worker-9) ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.dns.lookup = default
client.id = Babys-MacBook-Pro.local
client.rack =
connections.max.idle.ms = 54
default.api.timeout.ms = 6
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = foodRunnerSyncProtocol_notifications
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 30
max.poll.records = 500
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 3
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 6
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300

Build failed in Jenkins: kafka-2.5-jdk8 #32

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-9558; Fix retry logic in KafkaAdminClient listOffsets (#8119)


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

[Colin McCabe]

On Thu, Feb 13, 2020, at 08:24, Mickael Maison wrote:
> Hi Colin,
> 
> > Can you please specify what the result is when a newer client tries to use
> > this on an older broker?  Does that result in an
> > UnsupportedVersionException?
> Yes, in case this feature is not supported by the broker,
> UnsupportedVersionException should be raised. Your question made me
> look at it again and to properly handle all cases, I decided to bump
> the ListGroups API version. That way, we can't end up in cases where
> brokers know about the protocol version but not about the new optional
> field. I've updated the KIP accordingly

Yes, a protocol version bump seems fine here.

> 
> > I would prefer an Optional in the Java API so that “show all groups” can
> > be EMPTY.
> I think it's nicer to provide a separate method in
> ListConsumerGroupsOptions to request all states without listing them
> rather than relying on empty Optional. I added a new method to the
> proposal: "inAnyState()".

I guess I don't feel that strongly about it.  However, if we are going to rely 
on inAnyState() / inStates(...) then let's make sure that the latter throws an 
exception when getting an empty list as an argument (since it will do something 
other than the obvious behavior of listing nothing).

> 
> Hi David,
> 
> > 1. We already have the "--state" option in the command line tool which can
> > be used
> > with "--describe" and we will have "--states" which can be used with
> > "--list". I feel this
> > is going to be confusing for users. I wonder if we could combine both cases
> > to reduce
> > the confusion but I am not sure it would be better. What do you think?
> Yes absolutely, it makes sense to reuse this existing flag. Good catch!
> 
> > 2. Regarding the output of the command line when "--states" is used, I
> > wonder if it
> > wouldn't be better to use a proper table with a header. We could use only
> > when
> > filters such as "--states" are used.
> Yes that's a good idea.
> 
> I've updated the KIP to take all these suggestions into account.
> Thanks again for the feedback

Thanks, Mickael.  It looks good.

best,
Colin


> 
> On Mon, Feb 10, 2020 at 1:13 PM David Jacot  wrote:
> >
> > Hi Michael,
> >
> > Thank you for the updated KIP. I have few comments/questions:
> >
> > 1. We already have the "--state" option in the command line tool which can
> > be used
> > with "--describe" and we will have "--states" which can be used with
> > "--list". I feel this
> > is going to be confusing for users. I wonder if we could combine both cases
> > to reduce
> > the confusion but I am not sure it would be better. What do you think?
> >
> > 2. Regarding the output of the command line when "--states" is used, I
> > wonder if it
> > wouldn't be better to use a proper table with a header. We could use only
> > when
> > filters such as "--states" are used.
> >
> > Best,
> > David
> >
> > On Thu, Feb 6, 2020 at 10:44 PM Colin McCabe  wrote:
> >
> > > Hi Mickael,
> > >
> > > Can you please specify what the result is when a newer client tries to use
> > > this on an older broker?  Does that result in an
> > > UnsupportedVersionException?
> > >
> > > I would prefer an Optional in the Java API so that “show all groups” can
> > > be EMPTY.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Mon, Jan 27, 2020, at 07:53, Mickael Maison wrote:
> > > > Hi David,
> > > >
> > > > Did that answer your questions? or do you have any further feedback?
> > > >
> > > > Thanks
> > > >
> > > > On Thu, Jan 16, 2020 at 4:11 PM Mickael Maison 
> > > > 
> > > wrote:
> > > > >
> > > > > Hi David,
> > > > >
> > > > > Thanks for taking a look.
> > > > > 1) Yes, updated
> > > > >
> > > > > 2) I had not considered that but indeed this would be useful if the
> > > > > request contained multiple states and would avoid doing another call.
> > > > > The ListGroups response already includes the group ProtocolType, so I
> > > > > guess we could add the State as well. The response will still be
> > > > > significantly smaller than DescribeGroups. With this change, one thing
> > > > > to note is that having Describe on the Cluster resource will allow
> > > > > retrieving the state of all groups. Currently retrieving the state of
> > > > > a group requires Describe on the Group.
> > > > >
> > > > > 3) Yes if ListGroups response includes the state, it makes sense to
> > > > > expose it via the command line tool and the AdminClient. With
> > > > > ConsumerGroupCommand, to avoid compatibility issues we can only print
> > > > > states when the --states flag is specified.
> > > > >
> > > > > I've updated the KIP accordingly.
> > > > >
> > > > > On Mon, Jan 13, 2020 at 12:20 PM David Jacot 
> > > wrote:
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > Please, excuse me for my late feedback. I've got a few
> > > questions/comments
> > > > > > while reviewing the KIP.
> > > > > >
> > > > > > 1. I would suggest to clearly state in the documentation of the
> > > state field
> > > > > > 

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[John Roesler]

Thanks for your remarks, Bruno!

I'm in favor of standardizing on terminology like "not forwarding
idempotent updates" or "dropping idempotent updates". Maybe we
should make a pass on the KIP and just convert everything to this
phrasing. In retrospect, even the term "emit-on-change" has too much
semantic baggage, since it implies the semantics from the SECRET
paper, which we don't really want to imply here.

I'm also in favor of the metric as you propose.

Likewise with stream aggregations, I was also under the impression
that we agreed on dropping idempotent updates to the aggregation
result, any time we find that our "new" (key, value, timestamp) result
is identical to the prior one.

Also, I'm +1 on all your recommendations for updating the KIP document
for clarity.

Regarding the opt-out config. Perhaps I'm suffering from a failure of
imagination, but I don't see how the current proposal could really have
a measurable impact on latency. If all we do is make a single extra pass
to compare two byte arrays for equality, only in the cases where we already
have the byte arrays available, it seems unlikely to measurably affect the
processing of non-idempotent updates. It seems guaranteed to _decrease_
the latency of processing idempotent updates, since we get to skip a
store#put, at least one producer#send, and also all downstream processing,
including all the disk and network operations associated with downstream
operations.

It seems like if we're pretty sure this change would only _help_, we shouldn't
introduce the operational burden of an extra configuration. If we want to
be more aggressive about dropping idempotent operations in the future,
such as depending on equals() or adding a ChangeDetector interface, then
we should consider adding a configuration as part of that future work. In
fact, if we add a simple "opt-in/opt-out" switch right now, we might find
that it's actually insufficient for whatever future feature we might propose,
then we have a mess of deprecating the opt-out config and replacing it.

What do you think?
-John

On Wed, Feb 19, 2020, at 09:50, Bruno Cadonna wrote:
> Hi all,
> 
> Sorry for the late reply!
> 
> I am also in favour of baby steps.
> 
> I am undecided whether the KIP should contain a opt-out config or not.
> The overhead of emit-on-change might affect latency. For applications
> where low latency is crucial and there are not too many idempotent
> updates, it would be better to fall back to emit-on-update. However,
> we do not know how much emit-on-change impacts latency. We would first
> need to benchmark that before we can decide about the opt-out-config.
> 
> A metric of dropped idempotent updates seems useful to me to be
> informed about potential upstream applications or upstream operators
> that produce too many idempotent updates. The KIP should state the
> name of the metric, its group, its tags, and its recording level (see
> KIP-444 or KIP-471 for examples). I propose DEBUG as reporting level.
> 
> Richard, what competing proposals for emit-on-change for aggregations
> do you mean? I have the feeling that we agreed to get rid of
> idempotent updates if the aggregate is updated with the same key,
> value, AND timestamp. I am also fine if we do not include this into
> this KIP (remember: baby steps).
> 
> You write that "emit-on-change is more correct". Since we agreed that
> this is an optimization, IMO you cannot argue this way.
> 
> Please put "Alternative Approaches" under "Rejected Alternatives", so
> that it becomes clear that we are not going to implement them. In
> general, I think the KIP needs a bit of clean-up (probably, you
> already planned for it). "Design Reasoning" is a bit of behavior
> changes, rejected alternatives and duplicates a bit the content in
> those sections.
> 
> I do not like the name "no-op operations" or "no-ops", because they
> are rather generic. I like more "idempotent updates".
> 
> Best,
> Bruno
> 
> 
> On Tue, Feb 18, 2020 at 7:25 PM Richard Yu  wrote:
> >
> > Hi all,
> >
> > We are definitely making progress!
> >
> > @John should I emphasize in the proposed behavior changes that we are only
> > doing binary equality checks for stateful operators?
> > It looks like we have come close to finalizing this part of the KIP. (I
> > will note in the KIP that this proposal is intended for optimization, not
> > semantics correctness)
> >
> > I do think maybe we still have one other detail we need to discuss. So far,
> > there has been quite a bit of back and forth about what the behavior of
> > aggregations should look like in emit on change. I have seen
> > multiple competing proposals, so I am not completely certain which one we
> > should go with, or how we will be able to compromise in between them.
> >
> > Let me know what your thoughts are on this matter, since we are probably
> > close to wrapping up most other stuff.
> > @Matthias J. Sax   and @Bruno, see what you think
> > about this.
> >
> > Best,
> > Richard
> >
> >
> >
> > 

[jira] [Created] (KAFKA-9576) Topic creation failure causing Stream thread death

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9576:
--

 Summary: Topic creation failure causing Stream thread death
 Key: KAFKA-9576
 URL: https://issues.apache.org/jira/browse/KAFKA-9576
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Boyang Chen


The failure to create an internal topic could lead to the stream thread death 
due to timeout:
{code:java}
[2020-02-14T03:03:00-08:00] 
(streams-soak-2-4-eos_soak_i-01c4a64bbd04974db_streamslog) [2020-02-14 
11:03:00,083] ERROR 
[stream-soak-test-c818a925-a8fd-4a81-9a26-1c744d52ff2f-StreamThread-1] 
stream-thread 
[stream-soak-test-c818a925-a8fd-4a81-9a26-1c744d52ff2f-StreamThread-1] 
Encountered the following unexpected Kafka exception during processing, this 
usually indicate Streams internal errors: 
(org.apache.kafka.streams.processor.internals.StreamThread)
[2020-02-14T03:03:00-08:00] 
(streams-soak-2-4-eos_soak_i-01c4a64bbd04974db_streamslog) 
org.apache.kafka.streams.errors.StreamsException: Could not create topic 
stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-19-changelog.
        at 
org.apache.kafka.streams.processor.internals.InternalTopicManager.getNumPartitions(InternalTopicManager.java:209)
        at 
org.apache.kafka.streams.processor.internals.InternalTopicManager.validateTopics(InternalTopicManager.java:223)
        at 
org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:106)
        at 
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.prepareTopic(StreamsPartitionAssignor.java:1229)
        at 
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:588)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:548)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:650)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1300(AbstractCoordinator.java:111)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:572)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:555)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1026)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1006)
        at 
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
        at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
        at 
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:599)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:409)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:294)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:400)
        at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:340)
        at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:471)
        at 
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1267)
        at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
        at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:843)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:743)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:698)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:671)
[2020-02-14T03:03:00-08:00] 
(streams-soak-2-4-eos_soak_i-01c4a64bbd04974db_streamslog) Caused by: 
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the 
call.
{code}
Potentially we could consider handling this error more gracefully if the 

[jira] [Created] (KAFKA-9575) "Notable changes in 2.5.0" doesn't mention ZooKeeper 3.5.7

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-9575:


 Summary: "Notable changes in 2.5.0" doesn't mention ZooKeeper 3.5.7
 Key: KAFKA-9575
 URL: https://issues.apache.org/jira/browse/KAFKA-9575
 Project: Kafka
  Issue Type: Improvement
  Components: docs, documentation
Affects Versions: 2.5.0
Reporter: Ron Dagostino
Assignee: Ron Dagostino
 Fix For: 2.5.0


There is a paragraph in the 2.4.0 upgrade notes talking about ZooKeeper bugs 
that make manual intervention recommended while upgrading from ZoKeeper 3.4.  
Both of the ZooKeeper bugs that are mentioned in the paragraph are fixed in 
3.5.7, so at a minimum we should mention the ZooKeeper 3.5.7 upgrade in the AK 
2.5.0 upgrade notes section.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9473) A large number of core system tests failing due to Kafka server failed to start on trunk

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-9473.

Resolution: Fixed

> A large number of core system tests failing due to Kafka server failed to 
> start on trunk
> 
>
> Key: KAFKA-9473
> URL: https://issues.apache.org/jira/browse/KAFKA-9473
> Project: Kafka
>  Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Critical
>
> By running a full set of core system tests, we detected 38/166 test failures 
> which are due to 
> `FAIL: Kafka server didn't finish startup in 60 seconds`
> need further investigation on this.
> [https://jenkins.confluent.io/job/system-test-kafka-branch-builder/3701/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9544) Flaky Test `KafkaAdminClientTest.testDefaultApiTimeoutOverride`

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9544.

Resolution: Fixed

> Flaky Test `KafkaAdminClientTest.testDefaultApiTimeoutOverride`
> ---
>
> Key: KAFKA-9544
> URL: https://issues.apache.org/jira/browse/KAFKA-9544
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> {code}
> org.junit.runners.model.TestTimedOutException: test timed out after 12 
> milliseconds
>   at java.lang.Object.wait(Native Method)
>   at java.lang.Thread.join(Thread.java:1260)
>   at 
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:594)
>   at org.apache.kafka.clients.admin.Admin.close(Admin.java:98)
>   at org.apache.kafka.clients.admin.Admin.close(Admin.java:81)
>   at 
> org.apache.kafka.clients.admin.AdminClientUnitTestEnv.close(AdminClientUnitTestEnv.java:116)
>   at 
> org.apache.kafka.clients.admin.KafkaAdminClientTest.testApiTimeout(KafkaAdminClientTest.java:2642)
>   at 
> org.apache.kafka.clients.admin.KafkaAdminClientTest.testDefaultApiTimeoutOverride(KafkaAdminClientTest.java:2595)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-569: DescribeConfigsResponse - Update the schema to include datatype of the field

[Shailesh Panwar]

Bump.

Thanks
Shailesh

On Tue, Feb 11, 2020 at 1:00 PM Shailesh Panwar 
wrote:

> Hi all,
> We would like to extend the DescribeConfigsResponse schema to include the
> data-type of the fields.
>
> The KIP can be found here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-569%3A+DescribeConfigsResponse+-+Update+the+schema+to+include+datatype+of+the+field
>
> Thanks
> Shailesh
>

Re: [DISCUSS] KIP-570: Add leader epoch in StopReplicaRequest

[David Jacot]

Hey Jason,

You're right. The leader epoch is not bumped when a topic is deleted. Using
a sentinel (e.g. -1) which would override any existing epoch sounds like a
good approach in this case. Let me update the KIP to reflect that.

Thanks,
David

On Fri, Feb 14, 2020 at 8:11 PM Jason Gustafson  wrote:

> Hey David,
>
> Thanks, it makes sense to prevent reordering, especially for the case of
> reassignment. When a topic is deleted, however, I am not sure we will have
> a bumped epoch to send. I guess for that case, we could send a sentinel
> which would take the existing semantics of overriding any existing epoch?
>
> -Jason
>
> On Tue, Feb 11, 2020 at 12:48 PM David Jacot  wrote:
>
> > Hi all,
> >
> > I've put together a very small KIP which proposes to add the leader epoch
> > in the
> > StopReplicaRequest in order to make it robust to reordering:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-570%3A+Add+leader+epoch+in+StopReplicaRequest
> >
> > Please take a look at the KIP and let me know what you think.
> >
> > Best,
> > David
> >
>

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[John Roesler]

Thanks for the response, Dongjin,

I'm sorry, but I'm still not following. It seems like the view you would
get on the "current state of the buffer" would always be equivalent to
the view of the upstream table.

Let me try an example, and maybe you can point out the flaw in my 
reasoning.

Let's say we're doing 10 ms windows with a grace period of zero.
Let's also say we're computing a windowed count, and that we have
a "final results" suppression after the count. Let's  materialize the
count as "Count" and the suppressed result as "Final Count".

Suppose we get an input event:
(time=10, key=A, value=...)

Then, Count will look like:

| window | key | value |
| 10 | A   | 1 |

The (internal) suppression buffer will contain:

| window | key | value |
| 10 | A   | 1 |

The record is still buffered because the window isn't closed yet.
Final Count is an empty table:

| window | key | value |

---

Now, we get a second event:
(time=15, key=A, value=...)

Then, Count will look like:

| window | key | value |
| 10 | A   | 2 |

The (internal) suppression buffer will contain:

| window | key | value |
| 10 | A   | 2 |

The record is still buffered because the window isn't closed yet.
Final Count is an empty table:

| window | key | value |


---

Finally, we get a third event:
(time=20, key=A, value=...)

Then, Count will look like:

| window | key | value |
| 10 | A   | 2 |
| 20 | A   | 1 |

The (internal) suppression buffer will contain:

| window | key | value |
| 20 | A   | 1 |

Note that window 10 has been flushed out, because it's now closed.
And window 20 is buffered because it isn't closed yet.
Final Count is now:

| window | key | value |
| 10 | A   | 2 |


---

Reading your email, I can't figure out what value there is in querying the
internal suppression buffer, since it only contains exactly the same value as
the upstream table, for each key that is still buffered. But it feels like
you're saying you're trying to do something different than just query the
windowed key and get back the current count?

Thanks,
-John


On Wed, Feb 19, 2020, at 09:49, Dongjin Lee wrote:
> Hi John,
> 
> 'The intermediate state of the suppression' in KIP does not mean the state
> of upstream KTable - sure, the state of the upstream KTable can be queried
> by materializing the operator immediately before the suppress as you shown.
> What I meant in KIP was the final state of the buffer, which is not emitted
> yet. (I agree, the current description may be confusing; it would be better
> to change it with 'the current state of the suppression' or 'the results of
> the suppression', like the Jira issue
>  states.)
> 
> For a little bit more about the motivation, here is one of my experience: I
> had to build a monitoring application which collects signals from IoT
> devices (say, a semiconductor production line.) If the number of collected
> signals within the time window is much less than the expected, there may be
> some problems like network hiccup in the systems. We wanted to build the
> system in the form of a dashboard, but could not by lack of materializing
> feature. It was precisely the case of querying only the final results of a
> windowed aggregation, as the Jira issue
>  states. We finally ended
> in implementing the system in an email alerting system like this
> 
> and had to collect the keys and windows of trouble by hand.
> 
> I think these kinds of use cases would be much common. Should it be
> described in the KIP much more in detail?
> 
> Thanks,
> Dongjin
> 
> On Sat, Feb 15, 2020 at 4:43 AM John Roesler  wrote:
> 
> > Hi Dongjin,
> >
> > Thanks for the KIP!
> >
> > Can you explain more about why the internal data structures of suppression
> > should be queriable? The motivation just says that users might want to do
> > it, which seems like it could justify literally anything :)
> >
> > One design point of Suppression is that if you wanted to query the “final
> > state”, you can Materialize the suppress itself (which is why it needs the
> > variant); if you wanted to query the “intermediate state”, you can
> > materialize the operator immediately before the suppress.
> >
> > Example:
> >
> > ...count(Materialized.as(“intermediate”))
> >   .supress(untilWindowClosed(), Materialized.as(“final”))
> >
> > I’m not sure what use case would require actually fetching from the
> > internal buffers.
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Feb 14, 2020, at 07:55, Dongjin Lee wrote:
> > > Hi devs,
> > >
> > > I'd like to reboot the discussion on KIP-508, which aims to support a
> > > Materialized variant of KTable#suppress. It was initially submitted
> > several
> > > months ago but closed by the inactivity.
> > >
> > > - KIP:
> > >
> > 

JIRA access request

[张祥]

Hi,

I am a newbie, could somebody please open JIRA access for me ? Thanks.

Re: [KAFKA-557] Add emit on change support for Kafka Streams

[Bruno Cadonna]

Hi all,

Sorry for the late reply!

I am also in favour of baby steps.

I am undecided whether the KIP should contain a opt-out config or not.
The overhead of emit-on-change might affect latency. For applications
where low latency is crucial and there are not too many idempotent
updates, it would be better to fall back to emit-on-update. However,
we do not know how much emit-on-change impacts latency. We would first
need to benchmark that before we can decide about the opt-out-config.

A metric of dropped idempotent updates seems useful to me to be
informed about potential upstream applications or upstream operators
that produce too many idempotent updates. The KIP should state the
name of the metric, its group, its tags, and its recording level (see
KIP-444 or KIP-471 for examples). I propose DEBUG as reporting level.

Richard, what competing proposals for emit-on-change for aggregations
do you mean? I have the feeling that we agreed to get rid of
idempotent updates if the aggregate is updated with the same key,
value, AND timestamp. I am also fine if we do not include this into
this KIP (remember: baby steps).

You write that "emit-on-change is more correct". Since we agreed that
this is an optimization, IMO you cannot argue this way.

Please put "Alternative Approaches" under "Rejected Alternatives", so
that it becomes clear that we are not going to implement them. In
general, I think the KIP needs a bit of clean-up (probably, you
already planned for it). "Design Reasoning" is a bit of behavior
changes, rejected alternatives and duplicates a bit the content in
those sections.

I do not like the name "no-op operations" or "no-ops", because they
are rather generic. I like more "idempotent updates".

Best,
Bruno


On Tue, Feb 18, 2020 at 7:25 PM Richard Yu  wrote:
>
> Hi all,
>
> We are definitely making progress!
>
> @John should I emphasize in the proposed behavior changes that we are only
> doing binary equality checks for stateful operators?
> It looks like we have come close to finalizing this part of the KIP. (I
> will note in the KIP that this proposal is intended for optimization, not
> semantics correctness)
>
> I do think maybe we still have one other detail we need to discuss. So far,
> there has been quite a bit of back and forth about what the behavior of
> aggregations should look like in emit on change. I have seen
> multiple competing proposals, so I am not completely certain which one we
> should go with, or how we will be able to compromise in between them.
>
> Let me know what your thoughts are on this matter, since we are probably
> close to wrapping up most other stuff.
> @Matthias J. Sax   and @Bruno, see what you think
> about this.
>
> Best,
> Richard
>
>
>
> On Tue, Feb 18, 2020 at 9:06 AM John Roesler  wrote:
>
> > Thanks, Matthias!
> >
> > Regarding numbers, it would be hard to know how many applications
> > would benefit, since we don't know how many applications there are,
> > or anything about their data sets or topologies. We could do a survey,
> > but it seems overkill if we take the conservative approach.
> >
> > I have my own practical stream processing experience that tells me this
> > is absolutely critical for any moderate-to-large relational stream
> > processing use cases. I'll leave it to you to decide if you find that
> > convincing, but it's definitely not an _assumption_. I've also heard from
> > a few Streams users who have already had to implement their own
> > noop-suppression transformers in order to get to production scale.
> >
> > Regardless, it sounds like we can agree on taking an opportunistic approach
> > and targeting the optimization just to use a binary-equality check at
> > stateful operators. (I'd also suggest in sink nodes, when we are about to
> > send old and new values, since they are also already present and serialized
> > at that point.) We could make the KIP even more vague, and just say that
> > we'll drop no-op updates "when possible".
> >
> > I'm curious what Bruno and the others think about this. If it seems like
> > a good starting point, perhaps we could move to a vote soon and get to
> > work on the implementation!
> >
> > Thanks,
> > -John
> >
> > On Mon, Feb 17, 2020, at 20:54, Matthias J. Sax wrote:
> > > Talking about optimizations and reducing downstream load:
> > >
> > > Do we actually have any numbers? I have the impression that this KIP is
> > > more or less build on the _assumption_ that there is a problem. Yes,
> > > there are some use cases that would benefit from this; But how many
> > > applications would actually benefit? And how much load reduction would
> > > they get?
> > >
> > > The simplest approach (following John idea to make baby steps) would be
> > > to apply the emit-on-change pattern only if there is a store. For this
> > > case we need to serialize old and new result anyway and thus a simple
> > > byte-array comparison is no overhead.
> > >
> > > Sending `oldValues` by default would become expensive because 

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[Dongjin Lee]

Hi John,

'The intermediate state of the suppression' in KIP does not mean the state
of upstream KTable - sure, the state of the upstream KTable can be queried
by materializing the operator immediately before the suppress as you shown.
What I meant in KIP was the final state of the buffer, which is not emitted
yet. (I agree, the current description may be confusing; it would be better
to change it with 'the current state of the suppression' or 'the results of
the suppression', like the Jira issue
 states.)

For a little bit more about the motivation, here is one of my experience: I
had to build a monitoring application which collects signals from IoT
devices (say, a semiconductor production line.) If the number of collected
signals within the time window is much less than the expected, there may be
some problems like network hiccup in the systems. We wanted to build the
system in the form of a dashboard, but could not by lack of materializing
feature. It was precisely the case of querying only the final results of a
windowed aggregation, as the Jira issue
 states. We finally ended
in implementing the system in an email alerting system like this

and had to collect the keys and windows of trouble by hand.

I think these kinds of use cases would be much common. Should it be
described in the KIP much more in detail?

Thanks,
Dongjin

On Sat, Feb 15, 2020 at 4:43 AM John Roesler  wrote:

> Hi Dongjin,
>
> Thanks for the KIP!
>
> Can you explain more about why the internal data structures of suppression
> should be queriable? The motivation just says that users might want to do
> it, which seems like it could justify literally anything :)
>
> One design point of Suppression is that if you wanted to query the “final
> state”, you can Materialize the suppress itself (which is why it needs the
> variant); if you wanted to query the “intermediate state”, you can
> materialize the operator immediately before the suppress.
>
> Example:
>
> ...count(Materialized.as(“intermediate”))
>   .supress(untilWindowClosed(), Materialized.as(“final”))
>
> I’m not sure what use case would require actually fetching from the
> internal buffers.
>
> Thanks,
> John
>
>
> On Fri, Feb 14, 2020, at 07:55, Dongjin Lee wrote:
> > Hi devs,
> >
> > I'd like to reboot the discussion on KIP-508, which aims to support a
> > Materialized variant of KTable#suppress. It was initially submitted
> several
> > months ago but closed by the inactivity.
> >
> > - KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make+Suppression+State+Queriable
> > - Jira: https://issues.apache.org/jira/browse/KAFKA-8403
> >
> > All kinds of feedback will be greatly appreciated.
> >
> > Best,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> > *github:  github.com/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> speakerdeck.com/dongjin
> > *
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*
*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[jira] [Created] (KAFKA-9574) Contact page links to spam/ads site on search-hadoop[.]com domain

[Gert van Dijk (Jira)]

Gert van Dijk created KAFKA-9574:


 Summary: Contact page links to spam/ads site on 
search-hadoop[.]com domain
 Key: KAFKA-9574
 URL: https://issues.apache.org/jira/browse/KAFKA-9574
 Project: Kafka
  Issue Type: Bug
  Components: website
Reporter: Gert van Dijk


The current live page at [https://kafka.apache.org/contact] displays:
{quote}A searchable archive of the mailing lists is available at 
search-hadoop[.]com
{quote}
But this is shows as a scam/ads serving site in my browser.

Git history shows me that this link has been present for many years. WHOIS 
domain info suggests that the domain is transferred 2020-01-22, so I assume 
this site is now a different site than was intended to link to.

Is there another site that allows to search through all Kafka archives? If not, 
I suggest to remove the whole paragraph.

I did not find any other links to the domain on the kafka-site repo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9573) TestUpgrade system test failed on Java11.

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created KAFKA-9573:
--

 Summary: TestUpgrade system test failed on Java11.
 Key: KAFKA-9573
 URL: https://issues.apache.org/jira/browse/KAFKA-9573
 Project: Kafka
  Issue Type: Improvement
Reporter: Nikolay Izhikov


Test was runed on Jdk11
Test result:

{noformat}

test_id:
kafkatest.tests.core.upgrade_test.TestUpgrade.test_upgrade.from_kafka_version=0.9.0.1.to_message_format_version=None.security_protocol=SASL_SSL.compression_types=.none
status: FAIL
run time:   1 minute 28.387 seconds


Kafka server didn't finish startup in 60 seconds
Traceback (most recent call last):
  File 
"/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 
132, in run
data = self.run_test()
  File 
"/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 
189, in run_test
return self.test_context.function(self.test)
  File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", line 
428, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/opt/kafka-dev/tests/kafkatest/tests/core/upgrade_test.py", line 133, 
in test_upgrade
self.kafka.start()
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 242, in 
start
Service.start(self)
  File "/usr/local/lib/python2.7/dist-packages/ducktape/services/service.py", 
line 234, in start
self.start_node(node)
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 357, in 
start_node
err_msg="Kafka server didn't finish startup in %d seconds" % timeout_sec)
  File 
"/usr/local/lib/python2.7/dist-packages/ducktape/cluster/remoteaccount.py", 
line 705, in wait_until
allow_fail=True) == 0, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ducktape/utils/util.py", line 
41, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg)
TimeoutError: Kafka server didn't finish startup in 60 seconds
{noformat}

Detailed output:

{noformat}
[0.001s][warning][gc] -Xloggc is deprecated. Will use 
-Xlog:gc:/opt/kafka-0.9.0.1/bin/../logs/kafkaServer-gc.log instead.
Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
~ 
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9572) Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some Records

[Bruno Cadonna (Jira)]

Bruno Cadonna created KAFKA-9572:


 Summary: Sum Computation with Exactly-Once Enabled and Injected 
Failures Misses Some Records
 Key: KAFKA-9572
 URL: https://issues.apache.org/jira/browse/KAFKA-9572
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.4.0
Reporter: Bruno Cadonna
 Attachments: 7-changelog-1.txt, data-1.txt, streams22.log, 
streams23.log, streams30.log, sum-1.txt

System test {{StreamsEosTest.test_failure_and_recovery}} failed due to a 
wrongly computed aggregation under exactly-once (EOS). The specific error is:
{code:java}
Exception in thread "main" java.lang.RuntimeException: Result verification 
failed for ConsumerRecord(topic = sum, partition = 1, leaderEpoch = 0, offset = 
2805, CreateTime = 1580719595164, serialized key size = 4, serialized value 
size = 8, headers = RecordHeaders(headers = [], isReadOnly = false), key = 
[B@6c779568, value = [B@f381794) expected <6069,17269> but was <6069,10698>
at 
org.apache.kafka.streams.tests.EosTestDriver.verifySum(EosTestDriver.java:444)
at 
org.apache.kafka.streams.tests.EosTestDriver.verify(EosTestDriver.java:196)
at 
org.apache.kafka.streams.tests.StreamsEosTest.main(StreamsEosTest.java:69)
{code} 
That means, the sum computed by the Streams app seems to be wrong for key 6069. 
I checked the dumps of the log segments of the input topic partition (attached: 
data-1.txt) and indeed two input records are not considered in the sum. With 
those two missed records the sum would be correct. More concretely, the input 
values for key 6069 are:
# 147
# 9250
# 5340 
# 1231
# 1301

The sum of this values is 17269 as stated in the exception above as expected 
sum. If you subtract values 3 and 4, i.e., 5340 and 1231 from 17269, you get 
10698 , which is the actual sum in the exception above. Somehow those two 
values are missing.

In the log dump of the output topic partition (attached: sum-1.txt), the sum is 
correct until the 4th value 1231 , i.e. 15968, then it is overwritten with 
10698.

In the log dump of the changelog topic of the state store that stores the sum 
(attached: 7-changelog-1.txt), the sum is also overwritten as in the output 
topic.

I attached the logs of the three Streams instances involved.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)