date:20210723

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #62

2021-07-23 Thread Apache Jenkins Server

See

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #361

2021-07-23 Thread Apache Jenkins Server

See

[jira] [Resolved] (KAFKA-13113) Add unregister support to the RaftClient.

2021-07-23 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13113.
-
Resolution: Fixed

> Add unregister support to the RaftClient.
> -
>
> Key: KAFKA-13113
> URL: https://issues.apache.org/jira/browse/KAFKA-13113
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kraft
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Blocker
>  Labels: kip-500
> Fix For: 3.0.0
>
>
> Implement the following API:
> {code:java}
> interface RaftClient {
>   ListenerContext register(Listener);
>   void unregister(ListenerContext);
> }
> interface ListenerContext {
> }
> interface Listener {
>   void handleCommit(ListenerContext, BatchReader);
>   void handleSnapshot(ListenerContext, SnapshotReader);
>   void handleLeaderChange(ListenerContext, LeaderAndEpoch);
> } {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #360

2021-07-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 485901 lines...]
[2021-07-24T03:15:23.466Z] 
[2021-07-24T03:15:23.466Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicPartition() PASSED
[2021-07-24T03:15:23.466Z] 
[2021-07-24T03:15:23.466Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicOnly() STARTED
[2021-07-24T03:15:27.038Z] 
[2021-07-24T03:15:27.038Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicOnly() PASSED
[2021-07-24T03:15:27.038Z] 
[2021-07-24T03:15:27.038Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithTopicPartition() STARTED
[2021-07-24T03:15:28.789Z] 
[2021-07-24T03:15:28.789Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithTopicPartition() PASSED
[2021-07-24T03:15:28.789Z] 
[2021-07-24T03:15:28.789Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsNonExistingGroup() STARTED
[2021-07-24T03:15:31.582Z] 
[2021-07-24T03:15:31.582Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsNonExistingGroup() PASSED
[2021-07-24T03:15:31.582Z] 
[2021-07-24T03:15:31.582Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithUnknownTopicOnly() STARTED
[2021-07-24T03:15:34.228Z] 
[2021-07-24T03:15:34.228Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithUnknownTopicOnly() PASSED
[2021-07-24T03:15:34.228Z] 
[2021-07-24T03:15:34.228Z] TopicCommandIntegrationTest > 
testAlterPartitionCount() STARTED
[2021-07-24T03:15:38.837Z] 
[2021-07-24T03:15:38.837Z] TopicCommandIntegrationTest > 
testAlterPartitionCount() PASSED
[2021-07-24T03:15:38.837Z] 
[2021-07-24T03:15:38.837Z] TopicCommandIntegrationTest > 
testCreatePartitionsDoesNotRetryThrottlingQuotaExceededException() STARTED
[2021-07-24T03:15:42.752Z] 
[2021-07-24T03:15:42.752Z] TopicCommandIntegrationTest > 
testCreatePartitionsDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-07-24T03:15:42.752Z] 
[2021-07-24T03:15:42.752Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExistWithIfExists() STARTED
[2021-07-24T03:15:47.361Z] 
[2021-07-24T03:15:47.361Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExistWithIfExists() PASSED
[2021-07-24T03:15:47.361Z] 
[2021-07-24T03:15:47.361Z] TopicCommandIntegrationTest > 
testCreateWithDefaultReplication() STARTED
[2021-07-24T03:15:51.102Z] 
[2021-07-24T03:15:51.102Z] TopicCommandIntegrationTest > 
testCreateWithDefaultReplication() PASSED
[2021-07-24T03:15:51.102Z] 
[2021-07-24T03:15:51.102Z] TopicCommandIntegrationTest > 
testDescribeAtMinIsrPartitions() STARTED
[2021-07-24T03:15:59.516Z] 
[2021-07-24T03:15:59.516Z] TopicCommandIntegrationTest > 
testDescribeAtMinIsrPartitions() PASSED
[2021-07-24T03:15:59.516Z] 
[2021-07-24T03:15:59.516Z] TopicCommandIntegrationTest > 
testCreateWithNegativeReplicationFactor() STARTED
[2021-07-24T03:16:03.090Z] 
[2021-07-24T03:16:03.090Z] TopicCommandIntegrationTest > 
testCreateWithNegativeReplicationFactor() PASSED
[2021-07-24T03:16:03.090Z] 
[2021-07-24T03:16:03.090Z] TopicCommandIntegrationTest > 
testCreateWithInvalidReplicationFactor() STARTED
[2021-07-24T03:16:06.666Z] 
[2021-07-24T03:16:06.666Z] TopicCommandIntegrationTest > 
testCreateWithInvalidReplicationFactor() PASSED
[2021-07-24T03:16:06.666Z] 
[2021-07-24T03:16:06.666Z] TopicCommandIntegrationTest > 
testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() STARTED
[2021-07-24T03:16:10.240Z] 
[2021-07-24T03:16:10.240Z] TopicCommandIntegrationTest > 
testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-07-24T03:16:10.240Z] 
[2021-07-24T03:16:10.241Z] TopicCommandIntegrationTest > 
testListTopicsWithExcludeInternal() STARTED
[2021-07-24T03:16:14.850Z] 
[2021-07-24T03:16:14.850Z] TopicCommandIntegrationTest > 
testListTopicsWithExcludeInternal() PASSED
[2021-07-24T03:16:14.850Z] 
[2021-07-24T03:16:14.850Z] TopicCommandIntegrationTest > 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress() STARTED
[2021-07-24T03:16:18.424Z] 
[2021-07-24T03:16:18.424Z] TopicCommandIntegrationTest > 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress() PASSED
[2021-07-24T03:16:18.424Z] 
[2021-07-24T03:16:18.424Z] TopicCommandIntegrationTest > 
testCreateWithNegativePartitionCount() STARTED
[2021-07-24T03:16:23.240Z] 
[2021-07-24T03:16:23.240Z] TopicCommandIntegrationTest > 
testCreateWithNegativePartitionCount() PASSED
[2021-07-24T03:16:23.240Z] 
[2021-07-24T03:16:23.240Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExist() STARTED
[2021-07-24T03:16:26.872Z] 
[2021-07-24T03:16:26.872Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExist() PASSED

Jenkins build became unstable: Kafka » Kafka Branch Builder » 3.0 #61

2021-07-23 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

2021-07-23 Thread Luke Chen

Maybe best to do a POC PR to see if we can do the fix without a KIP?
--> Let me give it a try, and let you know! :)

Thank you.
Luke


On Sat, Jul 24, 2021 at 6:46 AM Matthias J. Sax  wrote:

> Oh. I though `TimeWindow` (singular) is part of the public API... For
> this case, I agree that we might not need a KIP, if there are no
> compatibility concerns to internally switch from `TimeWindow` to (newly
> added) `SlidingWindow` (that will also be internal).
>
> Maybe best to do a POC PR to see if we can do the fix without a KIP of
> if we need one?
>
> @Luke: would this work for you?
>
>
> -Matthias
>
>
> On 7/23/21 1:00 PM, Sophie Blee-Goldman wrote:
> > @Matthias that operator doesn't need to be deprecated/updated as the
> > argument to it is a SlidingWindow*s*, not
> > a SlidingWindow (which is what this KIP is proposing to add).
> > The SlidingWindows class which is part of the public
> > API is really just a config container class, it doesn't hold the actual
> > data like the TimeWindow/SlidingWindow does.
> >
> > In fact, I'm not sure we even need a KIP at all for this change. The
> > Window/TimeWindow/SlidingWindow classes
> > are/would be internal, as they are only used within the sliding windowed
> > aggregation itself.
> >
> > On Fri, Jul 23, 2021 at 11:04 AM Matthias J. Sax 
> wrote:
> >
> >> Thanks for the KIP. Would be nice to fix this bug...
> >>
> >> Couple of comments:
> >>
> >> (1) nit: constructor should be `SlidingWindow` (not `TimeWindow` --
> >> guess just a c error)
> >>
> >> (2) Adding a new window type it not sufficient. We also need to update
> >> `KGroupedStream#windowedBy()` to allow uses to use the newly added
> >> window. I don't think we can change `SlidingWindows` in a backward
> >> compatible way, thus, we need to add a new class and new overloads for
> >> `windowedBy()` to make the transition. (Also for `CogroupedKStream`.)
> >>
> >> (3) #2 implies we should deprecate the exiting
> >> `windowBy(SlidingWindows)` methods.
> >>
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 7/23/21 6:46 AM, Luke Chen wrote:
> >>> Hi, Kafka.
> >>>
> >>> I'd like to discuss the KIP-765: Introduce new SlidingWindow type for
> >>> [start,end] time. This is an existing bug in slidingWindows, that we
> used
> >>> the wrong window type so the window time interval will have 1 ms less
> at
> >>> the end time. This KIP will fix this issue.
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-765%3A+Introduce+new+SlidingWindow+type+for+%5Bstart%2Cend%5D+time
> >>>
> >>> Thank you.
> >>>
> >>> Luke
> >>>
> >>
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #359

2021-07-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 486258 lines...]
[2021-07-24T01:07:14.682Z] 
[2021-07-24T01:07:14.682Z] AclAuthorizerTest > testWildCardAcls() PASSED
[2021-07-24T01:07:14.682Z] 
[2021-07-24T01:07:14.682Z] AclAuthorizerTest > testCreateDeleteTiming() STARTED
[2021-07-24T01:07:16.043Z] 
[2021-07-24T01:07:16.043Z] AclAuthorizerTest > testCreateDeleteTiming() PASSED
[2021-07-24T01:07:16.043Z] 
[2021-07-24T01:07:16.043Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWithAllHostAce() STARTED
[2021-07-24T01:07:16.043Z] 
[2021-07-24T01:07:16.043Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWithAllHostAce() PASSED
[2021-07-24T01:07:16.043Z] 
[2021-07-24T01:07:16.043Z] AclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2() STARTED
[2021-07-24T01:07:17.238Z] 
[2021-07-24T01:07:17.238Z] AclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2() PASSED
[2021-07-24T01:07:17.238Z] 
[2021-07-24T01:07:17.238Z] AclAuthorizerTest > testTopicAcl() STARTED
[2021-07-24T01:07:17.238Z] 
[2021-07-24T01:07:17.238Z] AclAuthorizerTest > testTopicAcl() PASSED
[2021-07-24T01:07:17.238Z] 
[2021-07-24T01:07:17.238Z] AclAuthorizerTest > testSuperUserHasAccess() STARTED
[2021-07-24T01:07:18.175Z] 
[2021-07-24T01:07:18.175Z] AclAuthorizerTest > testSuperUserHasAccess() PASSED
[2021-07-24T01:07:18.175Z] 
[2021-07-24T01:07:18.175Z] AclAuthorizerTest > 
testDeleteAclOnPrefixedResource() STARTED
[2021-07-24T01:07:18.175Z] 
[2021-07-24T01:07:18.175Z] AclAuthorizerTest > 
testDeleteAclOnPrefixedResource() PASSED
[2021-07-24T01:07:18.175Z] 
[2021-07-24T01:07:18.175Z] AclAuthorizerTest > testDenyTakesPrecedence() STARTED
[2021-07-24T01:07:19.411Z] 
[2021-07-24T01:07:19.411Z] AclAuthorizerTest > testDenyTakesPrecedence() PASSED
[2021-07-24T01:07:19.411Z] 
[2021-07-24T01:07:19.411Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeIsolationUnrelatedDenyWontDominateAllow() STARTED
[2021-07-24T01:07:19.411Z] 
[2021-07-24T01:07:19.411Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeIsolationUnrelatedDenyWontDominateAllow() PASSED
[2021-07-24T01:07:19.411Z] 
[2021-07-24T01:07:19.411Z] AclAuthorizerTest > 
testSingleCharacterResourceAcls() STARTED
[2021-07-24T01:07:20.522Z] 
[2021-07-24T01:07:20.522Z] AclAuthorizerTest > 
testSingleCharacterResourceAcls() PASSED
[2021-07-24T01:07:20.522Z] 
[2021-07-24T01:07:20.522Z] AclAuthorizerTest > testNoAclFoundOverride() STARTED
[2021-07-24T01:07:20.522Z] 
[2021-07-24T01:07:20.522Z] AclAuthorizerTest > testNoAclFoundOverride() PASSED
[2021-07-24T01:07:20.522Z] 
[2021-07-24T01:07:20.522Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWildcardResourceDenyDominate() STARTED
[2021-07-24T01:07:21.986Z] 
[2021-07-24T01:07:21.986Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWildcardResourceDenyDominate() PASSED
[2021-07-24T01:07:21.986Z] 
[2021-07-24T01:07:21.986Z] AclAuthorizerTest > testEmptyAclThrowsException() 
STARTED
[2021-07-24T01:07:21.986Z] 
[2021-07-24T01:07:21.986Z] AclAuthorizerTest > testEmptyAclThrowsException() 
PASSED
[2021-07-24T01:07:21.986Z] 
[2021-07-24T01:07:21.986Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeNoAclFoundOverride() STARTED
[2021-07-24T01:07:22.904Z] 
[2021-07-24T01:07:22.904Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeNoAclFoundOverride() PASSED
[2021-07-24T01:07:22.904Z] 
[2021-07-24T01:07:22.904Z] AclAuthorizerTest > 
testSuperUserWithCustomPrincipalHasAccess() STARTED
[2021-07-24T01:07:22.904Z] 
[2021-07-24T01:07:22.904Z] AclAuthorizerTest > 
testSuperUserWithCustomPrincipalHasAccess() PASSED
[2021-07-24T01:07:22.904Z] 
[2021-07-24T01:07:22.904Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWithAllOperationAce() STARTED
[2021-07-24T01:07:24.307Z] 
[2021-07-24T01:07:24.307Z] AclAuthorizerTest > 
testAuthorizeByResourceTypeWithAllOperationAce() PASSED
[2021-07-24T01:07:24.307Z] 
[2021-07-24T01:07:24.307Z] AclAuthorizerTest > 
testAllowAccessWithCustomPrincipal() STARTED
[2021-07-24T01:07:24.307Z] 
[2021-07-24T01:07:24.307Z] AclAuthorizerTest > 
testAllowAccessWithCustomPrincipal() PASSED
[2021-07-24T01:07:24.307Z] 
[2021-07-24T01:07:24.307Z] AclAuthorizerTest > 
testDeleteAclOnWildcardResource() STARTED
[2021-07-24T01:07:24.307Z] 
[2021-07-24T01:07:24.307Z] AclAuthorizerTest > 
testDeleteAclOnWildcardResource() PASSED
[2021-07-24T01:07:24.307Z] 
[2021-07-24T01:07:24.307Z] AclAuthorizerTest > 
testAuthorizerZkConfigFromKafkaConfig() STARTED
[2021-07-24T01:07:25.236Z] 
[2021-07-24T01:07:25.236Z] AclAuthorizerTest > 
testAuthorizerZkConfigFromKafkaConfig() PASSED
[2021-07-24T01:07:25.236Z] 
[2021-07-24T01:07:25.236Z] AclAuthorizerTest > testChangeListenerTiming() 
STARTED
[2021-07-24T01:07:25.236Z] 
[2021-07-24T01:07:25.236Z] AclAuthorizerTest > testChangeListenerTiming() PASSED
[2021-07-24T01:07:25.236Z] 
[2021-07-24T01:07:25.236Z] AclAuthorizerTest >

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #56

2021-07-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 465925 lines...]
[2021-07-24T00:40:48.699Z] 
[2021-07-24T00:40:48.699Z] KafkaZkClientTest > testTopicAssignments() STARTED
[2021-07-24T00:40:48.699Z] 
[2021-07-24T00:40:48.699Z] KafkaZkClientTest > testTopicAssignments() PASSED
[2021-07-24T00:40:48.699Z] 
[2021-07-24T00:40:48.699Z] KafkaZkClientTest > 
testControllerManagementMethods() STARTED
[2021-07-24T00:40:48.699Z] 
[2021-07-24T00:40:48.699Z] KafkaZkClientTest > 
testControllerManagementMethods() PASSED
[2021-07-24T00:40:48.699Z] 
[2021-07-24T00:40:48.699Z] KafkaZkClientTest > testTopicAssignmentMethods() 
STARTED
[2021-07-24T00:40:49.783Z] 
[2021-07-24T00:40:49.783Z] KafkaZkClientTest > testTopicAssignmentMethods() 
PASSED
[2021-07-24T00:40:49.783Z] 
[2021-07-24T00:40:49.783Z] KafkaZkClientTest > testConnectionViaNettyClient() 
STARTED
[2021-07-24T00:40:49.783Z] 
[2021-07-24T00:40:49.783Z] KafkaZkClientTest > testConnectionViaNettyClient() 
PASSED
[2021-07-24T00:40:49.783Z] 
[2021-07-24T00:40:49.783Z] KafkaZkClientTest > testPropagateIsrChanges() STARTED
[2021-07-24T00:40:49.783Z] 
[2021-07-24T00:40:49.783Z] KafkaZkClientTest > testPropagateIsrChanges() PASSED
[2021-07-24T00:40:49.783Z] 
[2021-07-24T00:40:49.783Z] KafkaZkClientTest > testControllerEpochMethods() 
STARTED
[2021-07-24T00:40:50.917Z] 
[2021-07-24T00:40:50.917Z] KafkaZkClientTest > testControllerEpochMethods() 
PASSED
[2021-07-24T00:40:50.917Z] 
[2021-07-24T00:40:50.917Z] KafkaZkClientTest > testDeleteRecursive() STARTED
[2021-07-24T00:40:50.917Z] 
[2021-07-24T00:40:50.917Z] KafkaZkClientTest > testDeleteRecursive() PASSED
[2021-07-24T00:40:50.917Z] 
[2021-07-24T00:40:50.917Z] KafkaZkClientTest > testGetTopicPartitionStates() 
STARTED
[2021-07-24T00:40:50.917Z] 
[2021-07-24T00:40:50.917Z] KafkaZkClientTest > testGetTopicPartitionStates() 
PASSED
[2021-07-24T00:40:50.917Z] 
[2021-07-24T00:40:50.917Z] KafkaZkClientTest > 
testCreateConfigChangeNotification() STARTED
[2021-07-24T00:40:52.001Z] 
[2021-07-24T00:40:52.001Z] KafkaZkClientTest > 
testCreateConfigChangeNotification() PASSED
[2021-07-24T00:40:52.001Z] 
[2021-07-24T00:40:52.001Z] KafkaZkClientTest > testDelegationTokenMethods() 
STARTED
[2021-07-24T00:40:52.001Z] 
[2021-07-24T00:40:52.001Z] KafkaZkClientTest > testDelegationTokenMethods() 
PASSED
[2021-07-24T00:40:52.001Z] 
[2021-07-24T00:40:52.001Z] LogCleanerIntegrationTest > 
testMarksPartitionsAsOfflineAndPopulatesUncleanableMetrics() STARTED
[2021-07-24T00:40:53.134Z] 
[2021-07-24T00:40:53.135Z] LogCleanerIntegrationTest > 
testMarksPartitionsAsOfflineAndPopulatesUncleanableMetrics() PASSED
[2021-07-24T00:40:53.135Z] 
[2021-07-24T00:40:53.135Z] LogCleanerIntegrationTest > testIsThreadFailed() 
STARTED
[2021-07-24T00:40:53.135Z] 
[2021-07-24T00:40:53.135Z] LogCleanerIntegrationTest > testIsThreadFailed() 
PASSED
[2021-07-24T00:40:53.135Z] 
[2021-07-24T00:40:53.135Z] LogCleanerIntegrationTest > 
testMaxLogCompactionLag() STARTED
[2021-07-24T00:40:55.414Z] 
[2021-07-24T00:40:55.414Z] SslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl() PASSED
[2021-07-24T00:40:55.414Z] 
[2021-07-24T00:40:55.414Z] SslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls() STARTED
[2021-07-24T00:41:01.307Z] 
[2021-07-24T00:41:01.307Z] GssapiAuthenticationTest > testReLogin() PASSED
[2021-07-24T00:41:01.307Z] 
[2021-07-24T00:41:01.307Z] ControllerFailoverTest > 
testHandleIllegalStateException() STARTED
[2021-07-24T00:41:02.504Z] 
[2021-07-24T00:41:02.504Z] ControllerFailoverTest > 
testHandleIllegalStateException() PASSED
[2021-07-24T00:41:02.504Z] 
[2021-07-24T00:41:02.504Z] ZkNodeChangeNotificationListenerTest > 
testProcessNotification() STARTED
[2021-07-24T00:41:03.531Z] 
[2021-07-24T00:41:03.531Z] ZkNodeChangeNotificationListenerTest > 
testProcessNotification() PASSED
[2021-07-24T00:41:03.531Z] 
[2021-07-24T00:41:03.531Z] ZkNodeChangeNotificationListenerTest > 
testSwallowsProcessorException() STARTED
[2021-07-24T00:41:03.531Z] 
[2021-07-24T00:41:03.531Z] ZkNodeChangeNotificationListenerTest > 
testSwallowsProcessorException() PASSED
[2021-07-24T00:41:03.733Z] 
[2021-07-24T00:41:03.733Z] SslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls() PASSED
[2021-07-24T00:41:03.733Z] 
[2021-07-24T00:41:03.733Z] SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe() STARTED
[2021-07-24T00:41:04.556Z] 
[2021-07-24T00:41:04.556Z] 1328 tests completed, 1 failed, 7 skipped
[2021-07-24T00:41:04.556Z] There were failing tests. See the report at: 
file:///home/jenkins/workspace/Kafka_kafka_2.8/core/build/reports/tests/integrationTest/index.html
[2021-07-24T00:41:05.579Z] 
[2021-07-24T00:41:05.579Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 7.0.
[2021-07-24T00:41:05.579Z] Use '--warning-mode all' to show the individual 
deprecation warnings.

[jira] [Resolved] (KAFKA-13008) Stream will stop processing data for a long time while waiting for the partition lag

2021-07-23 Thread Guozhang Wang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-13008.
---
  Assignee: Guozhang Wang
Resolution: Fixed

> Stream will stop processing data for a long time while waiting for the 
> partition lag
> 
>
> Key: KAFKA-13008
> URL: https://issues.apache.org/jira/browse/KAFKA-13008
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.0.0
>Reporter: Luke Chen
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: image-2021-07-07-11-19-55-630.png
>
>
> In KIP-695, we improved the task idling mechanism by checking partition lag. 
> It's a good improvement for timestamp sync. But I found it will cause the 
> stream stop processing the data for a long time while waiting for the 
> partition metadata.
>  
> I've been investigating this case for a while, and figuring out the issue 
> will happen in below situation (or similar situation):
>  # start 2 streams (each with 1 thread) to consume from a topicA (with 3 
> partitions: A-0, A-1, A-2)
>  # After 2 streams started, the partitions assignment are: (I skipped some 
> other processing related partitions for simplicity)
>  stream1-thread1: A-0, A-1 
>  stream2-thread1: A-2
>  # start processing some data, assume now, the position and high watermark is:
>  A-0: offset: 2, highWM: 2
>  A-1: offset: 2, highWM: 2
>  A-2: offset: 2, highWM: 2
>  # Now, stream3 joined, so trigger rebalance with this assignment:
>  stream1-thread1: A-0 
>  stream2-thread1: A-2
>  stream3-thread1: A-1
>  # Suddenly, stream3 left, so now, rebalance again, with the step 2 
> assignment:
>  stream1-thread1: A-0, *A-1* 
>  stream2-thread1: A-2
>  (note: after initialization, the  position of A-1 will be: position: null, 
> highWM: null)
>  # Now, note that, the partition A-1 used to get assigned to stream1-thread1, 
> and now, it's back. And also, assume the partition A-1 has slow input (ex: 1 
> record per 30 mins), and partition A-0 has fast input (ex: 10K records / 
> sec). So, now, the stream1-thread1 won't process any data until we got input 
> from partition A-1 (even if partition A-0 is buffered a lot, and we have 
> `{{max.task.idle.ms}}` set to 0).
>  
> The reason why the stream1-thread1 won't process any data is because we can't 
> get the lag of partition A-1. And why we can't get the lag? It's because
>  # In KIP-695, we use consumer's cache to get the partition lag, to avoid 
> remote call
>  # The lag for a partition will be cleared if the assignment in this round 
> doesn't have this partition. check 
> [here|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java#L272].
>  So, in the above example, the metadata cache for partition A-1 will be 
> cleared in step 4, and re-initialized (to null) in step 5
>  # In KIP-227, we introduced a fetch session to have incremental fetch 
> request/response. That is, if the session existed, the client(consumer) will 
> get the update only when the fetched partition have update (ex: new data). 
> So, in the above case, the partition A-1 has slow input (ex: 1 record per 30 
> mins), it won't have update until next 30 mins, or wait for the fetch session 
> become inactive for (default) 2 mins to be evicted. Either case, the metadata 
> won't be updated for a while.
>  
> In KIP-695, if we don't get the partition lag, we can't determine the 
> partition data status to do timestamp sync, so we'll keep waiting and not 
> processing any data. That's why this issue will happen.
>  
> *Proposed solution:*
>  # If we don't get the current lag for a partition, or the current lag > 0, 
> we start to wait for max.task.idle.ms, and reset the deadline when we get the 
> partition lag, like what we did in previous KIP-353
>  # Introduce a waiting time config when no partition lag, or partition lag 
> keeps > 0 (need KIP)
> [~vvcephei] [~guozhang] , any suggestions?
>  
> cc [~ableegoldman]  [~mjsax] , this is the root cause that in 
> [https://github.com/apache/kafka/pull/10736,] we discussed and thought 
> there's a data lose situation. FYI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-13021) Improve Javadocs for API Changes and address followup from KIP-633

2021-07-23 Thread A. Sophie Blee-Goldman (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-13021.

Resolution: Fixed

> Improve Javadocs for API Changes and address followup from KIP-633
> --
>
> Key: KAFKA-13021
> URL: https://issues.apache.org/jira/browse/KAFKA-13021
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation, streams
>Affects Versions: 3.0.0
>Reporter: Israel Ekpo
>Assignee: Israel Ekpo
>Priority: Major
> Fix For: 3.0.0
>
>
> There are Javadoc changes from the following PR that needs to be completed 
> prior to the 3.0 release. This Jira item is to track that work
> [https://github.com/apache/kafka/pull/10926]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build became unstable: Kafka » Kafka Branch Builder » trunk #358

2021-07-23 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

2021-07-23 Thread Matthias J. Sax

Oh. I though `TimeWindow` (singular) is part of the public API... For
this case, I agree that we might not need a KIP, if there are no
compatibility concerns to internally switch from `TimeWindow` to (newly
added) `SlidingWindow` (that will also be internal).

Maybe best to do a POC PR to see if we can do the fix without a KIP of
if we need one?

@Luke: would this work for you?


-Matthias


On 7/23/21 1:00 PM, Sophie Blee-Goldman wrote:
> @Matthias that operator doesn't need to be deprecated/updated as the
> argument to it is a SlidingWindow*s*, not
> a SlidingWindow (which is what this KIP is proposing to add).
> The SlidingWindows class which is part of the public
> API is really just a config container class, it doesn't hold the actual
> data like the TimeWindow/SlidingWindow does.
> 
> In fact, I'm not sure we even need a KIP at all for this change. The
> Window/TimeWindow/SlidingWindow classes
> are/would be internal, as they are only used within the sliding windowed
> aggregation itself.
> 
> On Fri, Jul 23, 2021 at 11:04 AM Matthias J. Sax  wrote:
> 
>> Thanks for the KIP. Would be nice to fix this bug...
>>
>> Couple of comments:
>>
>> (1) nit: constructor should be `SlidingWindow` (not `TimeWindow` --
>> guess just a c error)
>>
>> (2) Adding a new window type it not sufficient. We also need to update
>> `KGroupedStream#windowedBy()` to allow uses to use the newly added
>> window. I don't think we can change `SlidingWindows` in a backward
>> compatible way, thus, we need to add a new class and new overloads for
>> `windowedBy()` to make the transition. (Also for `CogroupedKStream`.)
>>
>> (3) #2 implies we should deprecate the exiting
>> `windowBy(SlidingWindows)` methods.
>>
>>
>>
>> -Matthias
>>
>>
>>
>> On 7/23/21 6:46 AM, Luke Chen wrote:
>>> Hi, Kafka.
>>>
>>> I'd like to discuss the KIP-765: Introduce new SlidingWindow type for
>>> [start,end] time. This is an existing bug in slidingWindows, that we used
>>> the wrong window type so the window time interval will have 1 ms less at
>>> the end time. This KIP will fix this issue.
>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-765%3A+Introduce+new+SlidingWindow+type+for+%5Bstart%2Cend%5D+time
>>>
>>> Thank you.
>>>
>>> Luke
>>>
>>
>

Jenkins build is back to stable : Kafka » Kafka Branch Builder » 3.0 #59

2021-07-23 Thread Apache Jenkins Server

See

[jira] [Resolved] (KAFKA-13127) Fix stray partition lookup logic

2021-07-23 Thread Colin McCabe (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-13127.
--
Resolution: Fixed

> Fix stray partition lookup logic
> 
>
> Key: KAFKA-13127
> URL: https://issues.apache.org/jira/browse/KAFKA-13127
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 3.0.0
>
>
> The result of `BrokerMetadataPublisher.findGhostReplicas` is inverted. It 
> returns all of the non-stray replicas. This causes all of these partitions to 
> get deleted on startup by mistake.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-13116) KIP-724: Adjust system tests due to KAFKA-12944

2021-07-23 Thread Ismael Juma (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-13116.
-
Resolution: Fixed

> KIP-724: Adjust system tests due to KAFKA-12944
> ---
>
> Key: KAFKA-13116
> URL: https://issues.apache.org/jira/browse/KAFKA-13116
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Several system tests involving legacy message formats are failing due to 
> KAFKA-12944:
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2021-07-21--001.system-test-kafka-trunk--1626872410--confluentinc--master–038bdaa4df/report.html
> All system tests that write data with legacy message formats need to use IBP 
> 2.8 or lower.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

2021-07-23 Thread Sophie Blee-Goldman

@Matthias that operator doesn't need to be deprecated/updated as the
argument to it is a SlidingWindow*s*, not
a SlidingWindow (which is what this KIP is proposing to add).
The SlidingWindows class which is part of the public
API is really just a config container class, it doesn't hold the actual
data like the TimeWindow/SlidingWindow does.

In fact, I'm not sure we even need a KIP at all for this change. The
Window/TimeWindow/SlidingWindow classes
are/would be internal, as they are only used within the sliding windowed
aggregation itself.

On Fri, Jul 23, 2021 at 11:04 AM Matthias J. Sax  wrote:

> Thanks for the KIP. Would be nice to fix this bug...
>
> Couple of comments:
>
> (1) nit: constructor should be `SlidingWindow` (not `TimeWindow` --
> guess just a c error)
>
> (2) Adding a new window type it not sufficient. We also need to update
> `KGroupedStream#windowedBy()` to allow uses to use the newly added
> window. I don't think we can change `SlidingWindows` in a backward
> compatible way, thus, we need to add a new class and new overloads for
> `windowedBy()` to make the transition. (Also for `CogroupedKStream`.)
>
> (3) #2 implies we should deprecate the exiting
> `windowBy(SlidingWindows)` methods.
>
>
>
> -Matthias
>
>
>
> On 7/23/21 6:46 AM, Luke Chen wrote:
> > Hi, Kafka.
> >
> > I'd like to discuss the KIP-765: Introduce new SlidingWindow type for
> > [start,end] time. This is an existing bug in slidingWindows, that we used
> > the wrong window type so the window time interval will have 1 ms less at
> > the end time. This KIP will fix this issue.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-765%3A+Introduce+new+SlidingWindow+type+for+%5Bstart%2Cend%5D+time
> >
> > Thank you.
> >
> > Luke
> >
>

Jenkins build is back to stable : Kafka » Kafka Branch Builder » trunk #357

2021-07-23 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

2021-07-23 Thread Matthias J. Sax

Thanks for the KIP. Would be nice to fix this bug...

Couple of comments:

(1) nit: constructor should be `SlidingWindow` (not `TimeWindow` --
guess just a c error)

(2) Adding a new window type it not sufficient. We also need to update
`KGroupedStream#windowedBy()` to allow uses to use the newly added
window. I don't think we can change `SlidingWindows` in a backward
compatible way, thus, we need to add a new class and new overloads for
`windowedBy()` to make the transition. (Also for `CogroupedKStream`.)

(3) #2 implies we should deprecate the exiting
`windowBy(SlidingWindows)` methods.

-Matthias

On 7/23/21 6:46 AM, Luke Chen wrote:
> Hi, Kafka.
> 
> I'd like to discuss the KIP-765: Introduce new SlidingWindow type for
> [start,end] time. This is an existing bug in slidingWindows, that we used
> the wrong window type so the window time interval will have 1 ms less at
> the end time. This KIP will fix this issue.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-765%3A+Introduce+new+SlidingWindow+type+for+%5Bstart%2Cend%5D+time
> 
> Thank you.
> 
> Luke
>

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

2021-07-23 Thread Konstantine Karantasis

Thanks for the PR and the follow up Sophie.

We can still get this in and there's no risk to do so, given the proposed
changes.
Therefore, I agree to cherry-pick to 3.0 since the PR is about to get
merged.

Konstantine

On Thu, Jul 22, 2021 at 9:12 PM Sophie Blee-Goldman
 wrote:

> Hey Konstantine,
>
> A javadocs ticket of ours was demoted to a non-blocker earlier this week
> due to lack of action,
> but I now have a PR ready and under review. It's picking up some essential
> followup that was
> missed during the implementation of KIP-633 and is pretty essential. I
> tagged you on the PR,
> it's technically touching on a few things that aren't just docs, but only
> to add a handful of checks
> that already existed on the old APIs and just got missed on the new APIs.
> Anything beyond that
> I left as a TODO to follow up on after 3.0.
>
> KAFKA-13021  ---
> https://github.com/apache/kafka/pull/4
>
> I think we should be able to get it merged by tomorrow. Assuming we do, can
> I promote it back
> to blocker status and pick the fix to the 3.0 branch?
>
> Thanks!
> Sophie
>
> On Thu, Jul 22, 2021 at 4:29 PM Konstantine Karantasis
>  wrote:
>
> > Thanks for raising this John.
> >
> > While we are working to eliminate the existing blockers I think it would
> be
> > great to use this time in order to test the upgrade path that you
> mention.
> >
> > Before we approve a release candidate (once such a RC is generated) we
> > should confirm that the upgrade works as expected.
> > So, I agree with you that this is not an RC generation blocker per se but
> > it's a release blocker overall.
> >
> > Konstantine
> >
> >
> > On Thu, Jul 22, 2021 at 4:21 PM John Roesler 
> wrote:
> >
> > > Hello Konstantine,
> > >
> > > Someone just called to my attention that KAFKA-12724 had not
> > > been marked as a 3.0 blocker. We never added 2.8 to the
> > > Streams upgrade system test suite. This isn't a blocker in
> > > that it is a problem, but we should make sure that Streams
> > > is actually upgradable before releasing 3.0.
> > >
> > > I'm sorry for the oversight. For what it's worth, I think we
> > > could proceed with a release candidate while we continue to
> > > address the missing system test.
> > >
> > > Thanks,
> > > -John
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-12724
> > >
> > > On Wed, 2021-07-21 at 14:00 -0700, Konstantine Karantasis
> > > wrote:
> > > > Thanks for the heads up Colin.
> > > >
> > > > KAFKA-13112 seems important and of course relevant to what we ship
> with
> > > > 3.0.
> > > > Same for the test failures captured by KAFKA-13095 and KAFKA-12851.
> > > Fixing
> > > > those will increase the stability of our builds.
> > > >
> > > > Therefore, considering these tickets as blockers currently makes
> sense
> > to
> > > > me.
> > > >
> > > > Konstantine
> > > >
> > > >
> > > > On Wed, Jul 21, 2021 at 11:46 AM Colin McCabe 
> > > wrote:
> > > >
> > > > > Hi Konstantine,
> > > > >
> > > > > Thanks for your work on this release! We discovered three blocker
> > bugs
> > > > > which are worth bringing up here:
> > > > >
> > > > > 1. KAFKA-13112: Controller's committed offset get out of sync with
> > raft
> > > > > client listener context
> > > > > 2. KAFKA-13095: TransactionsTest is failing in kraft mode
> > > > > 3. KAFKA-12851: Flaky Test
> > > > > RaftEventSimulationTest.canMakeProgressIfMajorityIsReachable
> > > > >
> > > > > There are two subtasks for #1 which we are working on. We suspect
> > that
> > > #3
> > > > > has been fixed by a previous fix we made... we're looking into it.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Mon, Jul 19, 2021, at 20:23, Konstantine Karantasis wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Since last week, we have reached the stage of Code Freeze for the
> > > 3.0.0
> > > > > > Apache Kafka release.
> > > > > >
> > > > > > From this point forward and until the official release of 3.0.0,
> > only
> > > > > > critical fixes for blocker issues should be merged to the 3.0
> > release
> > > > > > branch.
> > > > > >
> > > > > > The release plan currently includes ten (10) such known blockers.
> > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> > > > > >
> > > > > > Besides these issues, any new issue that potentially gets
> > discovered
> > > will
> > > > > > need to be reported on dev@kafka.apache.org (under this thread)
> > and
> > > be
> > > > > > evaluated as a release blocker. At this point, the bar for such
> > > issues is
> > > > > > high; they need to be regressions or critical issues without an
> > > > > acceptable
> > > > > > workaround to be considered as release blockers.
> > > > > >
> > > > > > Exceptions of changes that may be merged to the 3.0 branch
> without
> > a
> > > > > > mention on the dev mailing list are fixes for test failures that
> > will
> > > > > help
> > > > > > stabilize the build and small documentation

[jira] [Resolved] (KAFKA-8529) Flakey test ConsumerBounceTest#testCloseDuringRebalance

2021-07-23 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8529.

Resolution: Fixed

> Flakey test ConsumerBounceTest#testCloseDuringRebalance
> ---
>
> Key: KAFKA-8529
> URL: https://issues.apache.org/jira/browse/KAFKA-8529
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Boyang Chen
>Priority: Major
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/5450/consoleFull]
>  
> *16:16:10* kafka.api.ConsumerBounceTest > testCloseDuringRebalance 
> STARTED*16:16:22* kafka.api.ConsumerBounceTest.testCloseDuringRebalance 
> failed, log available in 
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.ConsumerBounceTest.testCloseDuringRebalance.test.stdout*16:16:22*
>  *16:16:22* kafka.api.ConsumerBounceTest > testCloseDuringRebalance 
> FAILED*16:16:22* java.lang.AssertionError: Rebalance did not complete in 
> time*16:16:22* at org.junit.Assert.fail(Assert.java:89)*16:16:22* 
> at org.junit.Assert.assertTrue(Assert.java:42)*16:16:22* at 
> kafka.api.ConsumerBounceTest.waitForRebalance$1(ConsumerBounceTest.scala:402)*16:16:22*
>  at 
> kafka.api.ConsumerBounceTest.checkCloseDuringRebalance(ConsumerBounceTest.scala:416)*16:16:22*
>  at 
> kafka.api.ConsumerBounceTest.testCloseDuringRebalance(ConsumerBounceTest.scala:379)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #356

2021-07-23 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-13131) Consumer offsets lost during partition reassignment

2021-07-23 Thread Sergejs Andrejevs (Jira)

Sergejs Andrejevs created KAFKA-13131:
-

 Summary: Consumer offsets lost during partition reassignment
 Key: KAFKA-13131
 URL: https://issues.apache.org/jira/browse/KAFKA-13131
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Sergejs Andrejevs


While doing replicas reassignment of a *___consumer_offsets_* partition from 
one set of brokers to another, the consumer group offset got lost (seems to be 
reset to earliest).

Initial setup:
 __consumer_offsets-18 
 Replicas: 9,7,6

Desired setup:
 __consumer_offsets-18
 Replicas: 11,10,5

File_with_desired_state:
{code:java}
{
  "version": 1,
  "partitions": [
{
  "topic": "__consumer_offsets",
  "partition": 18,
  "replicas": [
11,
10,
5
  ],
  "log_dirs": [
"/path_replica_1",
"/path_replica_2",
"/path_replica_3"
  ]
}
  ]
}
{code}
Reassignment command:
{code:java}
/opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 
--execute --reassignment-json-file File_with_desired_state --throttle 104857600 
--replica-alter-log-dirs-throttle 104857600
{code}
The error in logs at the broker:
{code:java}
[2021-07-22 05:28:11,777] ERROR [GroupMetadataManager brokerId=11] Error 
loading offsets from __consumer_offsets-18 
(kafka.coordinator.group.GroupMetadataManager)
java.lang.NullPointerException
at kafka.log.OffsetIndex.$anonfun$lookup$1(OffsetIndex.scala:90)
at kafka.log.AbstractIndex.maybeLock(AbstractIndex.scala:338)
at kafka.log.OffsetIndex.lookup(OffsetIndex.scala:89)
at kafka.log.LogSegment.translateOffset(LogSegment.scala:274)
at kafka.log.LogSegment.read(LogSegment.scala:298)
at kafka.log.Log.$anonfun$read$2(Log.scala:1522)
at kafka.log.Log.read(Log.scala:2340)
at 
kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:589)
at 
kafka.coordinator.group.GroupMetadataManager.$anonfun$scheduleLoadGroupAndOffsets$2(GroupMetadataManager.scala:537)
at 
kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
It was tried to reproduce at test environments, but so far unsuccessfully.


 Let me know if any other configuration/parameters/details shall be added.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

2021-07-23 Thread Luke Chen

Hi, Kafka.

I'd like to discuss the KIP-765: Introduce new SlidingWindow type for
[start,end] time. This is an existing bug in slidingWindows, that we used
the wrong window type so the window time interval will have 1 ms less at
the end time. This KIP will fix this issue.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-765%3A+Introduce+new+SlidingWindow+type+for+%5Bstart%2Cend%5D+time

Thank you.

Luke

[jira] [Created] (KAFKA-13130) Deprecate long based range queries in SessionStore

2021-07-23 Thread Patrick Stuedi (Jira)

Patrick Stuedi created KAFKA-13130:
--

 Summary: Deprecate long based range queries in SessionStore
 Key: KAFKA-13130
 URL: https://issues.apache.org/jira/browse/KAFKA-13130
 Project: Kafka
  Issue Type: Improvement
  Components: streams, streams-test-utils
Reporter: Patrick Stuedi
Assignee: Patrick Stuedi
 Fix For: 3.1.0


Migrate long based queries in ReadOnlySessionStore (fetchSession*, 
findSession*, etc.) to object based interfaces. Deprecate old long based 
interface, similar to how it was done for the ReadOnlyWindowStore.

Related KIPs:

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-358%3A+Migrate+Streams+API+to+Duration+instead+of+long+ms+times]

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards]

 

Related Jiras:

https://issues.apache.org/jira/browse/KAFKA-12419

https://issues.apache.org/jira/browse/KAFKA-12526

https://issues.apache.org/jira/browse/KAFKA-12451

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-655: Windowed "Distinct" Operation for KStream

2021-07-23 Thread Bruno Cadonna


Hi Ivan and John,

1. John, could you clarify why comparing serialized values seems the way 
to go, now?


2. Ivan, Could you please answer my questions that I posted earlier? I 
will repost it here:
Ivan, could you please make this matter a bit clearer in the KIP? 
Actually, thinking about it again, I do currently not see why it should 
not make sense in hopping windows. Regarding this, I do not understand 
the following sentence:


"hopping and sliding windows do not make much sense for distinct() 
because they produce multiple intersected windows, so that one record 
can be multiplied instead of deduplication."


Ivan, what do you mean with "multiplied"?

3. As I said earlier, I do not think that SQL and the Java Stream API 
are good arguments to not use a verb. However, if everybody else is fine 
with it, I can get behind it.


John, good catch about the missing overloads!
BTW, the overload with Named should be there regardless of stateful or 
stateless.


Best,
Bruno

On 22.07.21 20:58, John Roesler wrote:

Hi Ivan,

Thanks for the reply.

1. I think I might have gotten myself confused. I was
thinking of this operation as stateless, but now I'm not
sure what I was thinking... This operator has to be
stateful, right? In that case, I agree that comparing
serialized values seems to be way to do it.

2. Thanks for the confirmation

3. I continue to be satisfied to let you all hash it out.

Thanks,
-John

On Tue, 2021-07-20 at 11:42 +0300, Ivan Ponomarev wrote:

Hi all,

1. Actually I always thought about the serialized byte array only -- at
least this is what local stores depend upon, and what Kafka itself
depends upon when doing log compaction.

I can imagine a case where two different byte arrays deserialize to
objects which are `equals` to each other. But I think we can ignore this
for now because IMO the risks related to buggy `equals` implementations
outweigh the potential benefits.

I will mention the duplicate definition in the KIP.

2. I agree with John, he got my point.

3. Let me gently insist on `distinct`. I believe that an exception to
the rule is appropriate here, because the name `distinct()` is ubiquitous.

It's not only about Java Streams API (or .NET LINQ, which appeared
earlier and also has `Distinct`): Spark's DataFrame has `distinct()`
method, Hazelcast Jet has `distinct()` method, and I bet I can find more
examples if I search. When we teach KStreams, we always say that
KStreams are just like other streaming APIs, and they have roots in SQL
queries. Users know what `distinct` is and they expect it to be in the API.


Regards,

Ivan


13.07.2021 0:10, John Roesler пишет:

Hi all,

Bruno raised some very good points. I’d like to chime in with additional 
context.

1. Great point. We faced a similar problem defining KIP-557. For 557, we chose 
to use the serialized byte array instead of the equals() method, but I think 
the situation in KIP-655 is a bit different. I think it might make sense to use 
the equals() method here, but am curious what Ivan thinks.

2. I figured we'd do nothing. I thought Ivan was just saying that it doesn't 
make a ton of sense to use it, which I agree with, but it doesn't seem like 
that means we should prohibit it.

3. FWIW, I don't have a strong feeling either way.

Thanks,
-John

On Mon, Jul 12, 2021, at 09:14, Bruno Cadonna wrote:

Hi Ivan,

Thank you for the KIP!

Some aspects are not clear to me from the KIP and I have a proposal.

1. The KIP does not describe the criteria that define a duplicate. Could
you add a definition of duplicate to the KIP?

2. The KIP does not describe what happens if distinct() is applied on a
hopping window. On the DSL level, I do not see how you can avoid that
users apply distinct() on a hopping window, i.e., you cannot avoid it at
compile time, you need to check it at runtime and throw an exception. Is
this correct or am I missing something?

3. I would also like to back a proposal by Sophie. She proposed to use
deduplicate() instead of distinct(), since the other DSL operations are
also verbs. I do not think that SQL and the Java Stream API are good
arguments to not use a verb.

Best,
Bruno


On 10.07.21 19:11, John Roesler wrote:

Hi Ivan,

Sorry for the silence!

I have just re-read the proposal.

To summarize, you are now only proposing the zero-arg distict() method to be 
added to TimeWindowedKStream and SessionWindowedKStream, right?

I’m in favor of this proposal.

Thanks,
John

On Sat, Jul 10, 2021, at 10:18, Ivan Ponomarev wrote:

Hello everyone,

I would like to remind you about KIP-655 and KIP-759 just in case they
got lost in your inbox.

Now the initial proposal is split into two independent and smaller ones,
so it must be easier to review them. Of course, if you have time.

Regards,

Ivan


24.06.2021 18:11, Ivan Ponomarev пишет:

Hello all,

I have rewritten the KIP-655 summarizing what was agreed upon during
this discussion (now the proposal is much simpler and less invasive).

I have also created KIP-759

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #62

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #361

[jira] [Resolved] (KAFKA-13113) Add unregister support to the RaftClient.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #360

Jenkins build became unstable: Kafka » Kafka Branch Builder » 3.0 #61

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #359

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #56

[jira] [Resolved] (KAFKA-13008) Stream will stop processing data for a long time while waiting for the partition lag

[jira] [Resolved] (KAFKA-13021) Improve Javadocs for API Changes and address followup from KIP-633

Jenkins build became unstable: Kafka » Kafka Branch Builder » trunk #358

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

Jenkins build is back to stable : Kafka » Kafka Branch Builder » 3.0 #59

[jira] [Resolved] (KAFKA-13127) Fix stray partition lookup logic

[jira] [Resolved] (KAFKA-13116) KIP-724: Adjust system tests due to KAFKA-12944

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

Jenkins build is back to stable : Kafka » Kafka Branch Builder » trunk #357

Re: [DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[jira] [Resolved] (KAFKA-8529) Flakey test ConsumerBounceTest#testCloseDuringRebalance

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #356

[jira] [Created] (KAFKA-13131) Consumer offsets lost during partition reassignment

[DISCUSS] KIP-765: Introduce new SlidingWindow type for [start,end] time

[jira] [Created] (KAFKA-13130) Deprecate long based range queries in SessionStore

Re: [DISCUSS] KIP-655: Windowed "Distinct" Operation for KStream

25 matches

Site Navigation

Mail list logo

Footer information