Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #509

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 493244 lines...]
[2021-10-06T04:27:39.870Z] PlaintextConsumerTest > 
testAutoCommitOnCloseAfterWakeup() STARTED
[2021-10-06T04:27:41.738Z] 
[2021-10-06T04:27:41.738Z] PlaintextConsumerTest > testAutoCommitOnRebalance() 
PASSED
[2021-10-06T04:27:41.738Z] 
[2021-10-06T04:27:41.738Z] PlaintextConsumerTest > 
testInterceptorsWithWrongKeyValue() STARTED
[2021-10-06T04:27:43.300Z] 
[2021-10-06T04:27:43.300Z] PlaintextConsumerTest > 
testAutoCommitOnCloseAfterWakeup() PASSED
[2021-10-06T04:27:43.300Z] 
[2021-10-06T04:27:43.300Z] PlaintextConsumerTest > testMaxPollRecords() STARTED
[2021-10-06T04:27:46.077Z] 
[2021-10-06T04:27:46.077Z] PlaintextConsumerTest > 
testInterceptorsWithWrongKeyValue() PASSED
[2021-10-06T04:27:46.077Z] 
[2021-10-06T04:27:46.077Z] PlaintextConsumerTest > 
testPerPartitionLeadWithMaxPollRecords() STARTED
[2021-10-06T04:27:47.116Z] 
[2021-10-06T04:27:47.116Z] ConsumerBounceTest > 
testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize() PASSED
[2021-10-06T04:27:47.116Z] 
[2021-10-06T04:27:47.116Z] ConsumerBounceTest > 
testSubscribeWhenTopicUnavailable() STARTED
[2021-10-06T04:27:48.681Z] 
[2021-10-06T04:27:48.681Z] PlaintextConsumerTest > testMaxPollRecords() PASSED
[2021-10-06T04:27:48.681Z] 
[2021-10-06T04:27:48.681Z] PlaintextConsumerTest > testAutoOffsetReset() STARTED
[2021-10-06T04:27:50.244Z] 
[2021-10-06T04:27:50.244Z] PlaintextConsumerTest > 
testPerPartitionLeadWithMaxPollRecords() PASSED
[2021-10-06T04:27:50.244Z] 
[2021-10-06T04:27:50.244Z] PlaintextConsumerTest > testHeaders() STARTED
[2021-10-06T04:27:53.063Z] 
[2021-10-06T04:27:53.063Z] PlaintextConsumerTest > testAutoOffsetReset() PASSED
[2021-10-06T04:27:53.063Z] 
[2021-10-06T04:27:53.063Z] PlaintextConsumerTest > 
testPerPartitionLagWithMaxPollRecords() STARTED
[2021-10-06T04:27:54.630Z] 
[2021-10-06T04:27:54.630Z] PlaintextConsumerTest > testHeaders() PASSED
[2021-10-06T04:27:54.630Z] 
[2021-10-06T04:27:54.630Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInAssignment() STARTED
[2021-10-06T04:27:58.869Z] 
[2021-10-06T04:27:58.869Z] PlaintextConsumerTest > 
testPerPartitionLagWithMaxPollRecords() PASSED
[2021-10-06T04:27:58.869Z] 
[2021-10-06T04:27:58.869Z] PlaintextConsumerTest > testFetchInvalidOffset() 
STARTED
[2021-10-06T04:28:00.341Z] 
[2021-10-06T04:28:00.341Z] PlaintextConsumerTest > 
testMaxPollIntervalMsDelayInAssignment() PASSED
[2021-10-06T04:28:00.341Z] 
[2021-10-06T04:28:00.341Z] PlaintextConsumerTest > 
testHeadersSerializerDeserializer() STARTED
[2021-10-06T04:28:02.993Z] 
[2021-10-06T04:28:02.993Z] PlaintextConsumerTest > testFetchInvalidOffset() 
PASSED
[2021-10-06T04:28:02.993Z] 
[2021-10-06T04:28:02.993Z] PlaintextConsumerTest > testAutoCommitIntercept() 
STARTED
[2021-10-06T04:28:05.782Z] 
[2021-10-06T04:28:05.782Z] ConsumerBounceTest > 
testSubscribeWhenTopicUnavailable() PASSED
[2021-10-06T04:28:05.782Z] 
[2021-10-06T04:28:05.782Z] ConsumerBounceTest > 
testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup() STARTED
[2021-10-06T04:28:05.782Z] 
[2021-10-06T04:28:05.782Z] PlaintextConsumerTest > 
testHeadersSerializerDeserializer() PASSED
[2021-10-06T04:28:05.782Z] 
[2021-10-06T04:28:05.782Z] PlaintextConsumerTest > 
testDeprecatedPollBlocksForAssignment() STARTED
[2021-10-06T04:28:08.546Z] 
[2021-10-06T04:28:08.546Z] PlaintextConsumerTest > testAutoCommitIntercept() 
PASSED
[2021-10-06T04:28:08.546Z] 
[2021-10-06T04:28:08.546Z] PlaintextConsumerTest > 
testFetchHonoursMaxPartitionFetchBytesIfLargeRecordNotFirst() STARTED
[2021-10-06T04:28:11.321Z] 
[2021-10-06T04:28:11.321Z] PlaintextConsumerTest > 
testDeprecatedPollBlocksForAssignment() PASSED
[2021-10-06T04:28:11.321Z] 
[2021-10-06T04:28:11.321Z] PlaintextConsumerTest > 
testPartitionPauseAndResume() STARTED
[2021-10-06T04:28:12.968Z] 
[2021-10-06T04:28:12.968Z] PlaintextConsumerTest > 
testFetchHonoursMaxPartitionFetchBytesIfLargeRecordNotFirst() PASSED
[2021-10-06T04:28:12.968Z] 
[2021-10-06T04:28:12.968Z] PlaintextConsumerTest > testCommitSpecifiedOffsets() 
STARTED
[2021-10-06T04:28:15.659Z] 
[2021-10-06T04:28:15.659Z] PlaintextConsumerTest > 
testPartitionPauseAndResume() PASSED
[2021-10-06T04:28:15.659Z] 
[2021-10-06T04:28:15.659Z] PlaintextConsumerTest > 
testQuotaMetricsNotCreatedIfNoQuotasConfigured() STARTED
[2021-10-06T04:28:18.436Z] 
[2021-10-06T04:28:18.436Z] PlaintextConsumerTest > testCommitSpecifiedOffsets() 
PASSED
[2021-10-06T04:28:18.436Z] 
[2021-10-06T04:28:18.436Z] PlaintextConsumerTest > 
testPerPartitionLeadMetricsCleanUpWithSubscribe() STARTED
[2021-10-06T04:28:18.955Z] 
[2021-10-06T04:28:18.955Z] PlaintextConsumerTest > 
testQuotaMetricsNotCreatedIfNoQuotasConfigured() PASSED
[2021-10-06T04:28:18.955Z] 
[2021-10-06T04:28:18.955Z] PlaintextConsumerTest > 
testPerPartitionLagMetricsCleanUpWithSubscribe() STARTED
[2021-10-06T04:28:23.124Z] 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #508

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 490623 lines...]
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > testGetAclsPrincipal() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > testGetAclsPrincipal() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > 
testWritesExtendedAclChangeEventIfInterBrokerProtocolNotSet() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > 
testWritesExtendedAclChangeEventIfInterBrokerProtocolNotSet() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > 
testAccessAllowedIfAllowAclExistsOnWildcardResource() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > 
testAccessAllowedIfAllowAclExistsOnWildcardResource() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > testLoadCache() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] AclAuthorizerTest > testLoadCache() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > 
testPeriodicTokenExpiry() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > 
testPeriodicTokenExpiry() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testDescribeToken() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testDescribeToken() 
PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testCreateToken() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testCreateToken() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testExpireToken() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testExpireToken() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testRenewToken() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testRenewToken() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testRemoveTokenHmac() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] DelegationTokenManagerTest > testRemoveTokenHmac() 
PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testIsZkSecurityEnabled() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testIsZkSecurityEnabled() 
PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testKafkaZkClient() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testKafkaZkClient() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testZkAntiMigration() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testZkAntiMigration() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testConsumerOffsetPathAcls() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testConsumerOffsetPathAcls() 
PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testZkMigration() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testZkMigration() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testChroot() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testChroot() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testDelete() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testDelete() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testDeleteRecursive() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] ZkAuthorizationTest > testDeleteRecursive() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] LogCleanerIntegrationTest > 
testMarksPartitionsAsOfflineAndPopulatesUncleanableMetrics() STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] LogCleanerIntegrationTest > 
testMarksPartitionsAsOfflineAndPopulatesUncleanableMetrics() PASSED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] LogCleanerIntegrationTest > testIsThreadFailed() 
STARTED
[2021-10-05T20:32:39.065Z] 
[2021-10-05T20:32:39.065Z] LogCleanerIntegrationTest > testIsThreadFailed() 
PASSED

[jira] [Created] (KAFKA-13350) Handle task corrupted exception on a per state store basis

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-13350:
---

 Summary: Handle task corrupted exception on a per state store basis
 Key: KAFKA-13350
 URL: https://issues.apache.org/jira/browse/KAFKA-13350
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


When we hit an `OffsetOutOfRangeException` during restore, we close a tasks as 
dirty and retry the restore process from scratch. For this case, we wipe out 
the task's state stores.

If a task has multiple state stores, we also wipe out state that is actually 
clean and thus need to redo work for no reason. Instead of wiping out all state 
store, we should only wipe out the single state store that corresponds to the 
changelog topic partition that hit the `OffsetOutOfRangeException`, but 
preserve the restore progress for all other state stores.

We need to consider persistent and in-memory stores: for persistent stores, it 
would be fine to close the not affected stores cleanly and also write the 
checkpoint file. For in-memory stores however, we should not close the store to 
avoid dropping the in-memory data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-779: Allow Source Tasks to Handle Producer Exceptions

[Knowles Atchison Jr]

Hello all,

I would like to discuss the following KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-779%3A+Allow+Source+Tasks+to+Handle+Producer+Exceptions

The main purpose is to allow Source Tasks the ability to see underlying
Producer Exceptions and decide what to do rather than being killed. In our
use cases we would want to log/write off some information and continue
processing.

PR is here:

https://github.com/apache/kafka/pull/11382

Any comments and feedback are welcome.


Knowles

Re: Wiki Permissions

[Bill Bejeck]

Knowles,

My apologies, I forgot to hit save from before.  Try again you should be
all set now.

-Bill

On Tue, Oct 5, 2021 at 2:43 PM Knowles Atchison Jr 
wrote:

> Bill,
>
> Thank you. I'm still seeing the "Sorry, you don't have permission to create
> content. Contact your space administrator to request access." on the Create
> KIP button. Am I not looking in the right place?
>
> Knowles
>
> On Tue, Oct 5, 2021 at 2:35 PM Bill Bejeck  wrote:
>
> > Done.  Thanks for your interest in Apache Kafka.
> >
> > -Bill
> >
> > On Tue, Oct 5, 2021 at 10:54 AM Knowles Atchison Jr <
> katchiso...@gmail.com
> > >
> > wrote:
> >
> > > Good morning,
> > >
> > > I would like to author a KIP.
> > >
> > > May I please have permissions granted for the wiki?
> > >
> > > username: katchison
> > >
> > > Thank you.
> > >
> > > Knowles
> > >
> >
>

CVE Back Port?

[Gary Russell]

Is there any chance that the fix for this CVE [1] can be back ported (and 
released) on the 2.5, 2.6 and 2.7 branches?

We have 3 (soon to be 4) supported branches, based on the 2.5.x, 2.6.x, 2.7.x, 
(and soon 3.0.0) clients.

Our versioning rules forbid moving to a new minor release for a dependency (e.g 
2.7.x to 2.8.x) in a patch release.

Yes, the user can override the version to 2.8.1 (works on all of our supported 
branches), but the problem is (s)he gets a vulnerable version transitively and 
has to know to do so.

Or, is this CVE on the broker side only (and not on the clients)? (I have been 
unable to find the actual fix in the commit log).

Thanks for your consideration.

The Spring team.

[1]: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38153

Re: Wiki Permissions

[Knowles Atchison Jr]

Bill,

Thank you. I'm still seeing the "Sorry, you don't have permission to create
content. Contact your space administrator to request access." on the Create
KIP button. Am I not looking in the right place?

Knowles

On Tue, Oct 5, 2021 at 2:35 PM Bill Bejeck  wrote:

> Done.  Thanks for your interest in Apache Kafka.
>
> -Bill
>
> On Tue, Oct 5, 2021 at 10:54 AM Knowles Atchison Jr  >
> wrote:
>
> > Good morning,
> >
> > I would like to author a KIP.
> >
> > May I please have permissions granted for the wiki?
> >
> > username: katchison
> >
> > Thank you.
> >
> > Knowles
> >
>

[jira] [Created] (KAFKA-13349) Allow Iterator.remove on KeyValueIterator

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-13349:
-

 Summary: Allow Iterator.remove on KeyValueIterator
 Key: KAFKA-13349
 URL: https://issues.apache.org/jira/browse/KAFKA-13349
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


Today Stream's state store's range iterator does not support `remove`. We could 
consider adding such support for all the built-in state stores:

* RocksDB's native iterator does not support removal, but we can always do a 
delete(key) concurrently while the iterator is open on the snapshot.
* In-Memory: straight forward implementation.

The benefit of that is then for range-and-delete truncation operation we do not 
necessarily have to be cautious about concurrent modification exceptions. This 
could also help GC with in-memory stores.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Wiki Permissions

[Bill Bejeck]

Done.  Thanks for your interest in Apache Kafka.

-Bill

On Tue, Oct 5, 2021 at 10:54 AM Knowles Atchison Jr 
wrote:

> Good morning,
>
> I would like to author a KIP.
>
> May I please have permissions granted for the wiki?
>
> username: katchison
>
> Thank you.
>
> Knowles
>

[jira] [Created] (KAFKA-13348) Allow Source Tasks to Handle Producer Exceptions

[Knowles Atchison Jr (Jira)]

Knowles Atchison Jr created KAFKA-13348:
---

 Summary: Allow Source Tasks to Handle Producer Exceptions
 Key: KAFKA-13348
 URL: https://issues.apache.org/jira/browse/KAFKA-13348
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 3.0.0, 2.8.1, 3.1.0
Reporter: Knowles Atchison Jr


KAFKA-8586 added capture of Producer Exceptions which will kill the connector.

There is a need to allow the connector itself to be aware of these errors, 
handle it in some manner, and continuing processing records.

The proposed change is to add a function to SourceTask that allows handling of 
the SourceRecord and Exception as thrown from the Producer. The SourceTask can 
examine these items and determine if it is appropriate to die (current 
behavior) or let the record be thrown away and continue processing.

The current behavior will be maintained by defaulting to returning false from 
this function. If the implementing SourceTask override of this function returns 
true, Kafka Connect will ignore this error record and continue processing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13347) kafka-topics.sh required arguments and defaults

[Scott M Messner (Jira)]

Scott M Messner created KAFKA-13347:
---

 Summary: kafka-topics.sh required arguments and defaults
 Key: KAFKA-13347
 URL: https://issues.apache.org/jira/browse/KAFKA-13347
 Project: Kafka
  Issue Type: Bug
  Components: docs
Affects Versions: 3.0.0
 Environment: WSL Debian GNU/Linux 10 (buster)
Reporter: Scott M Messner


Step 3 of the quickstart suggests that partitions and replication-factor are 
optional [kafka.apache.org/quickstart.|https://kafka.apache.org/quickstart] 
Also, the options list provided by bin/kafka-topics.sh explicitly mentions that 
" If not supplied for create, defaults to the cluster default." 
(kafka_2.13-3.0.0)

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 
localhost:9092
Missing required argument "[partitions]"

...
--partitions  The number of partitions for the topic
 being created or altered (WARNING:
 If partitions are increased for a
 topic that has a key, the partition
 logic or ordering of the messages
 will be affected). *{color:#172b4d}If not supplied{color}*
 *{color:#172b4d}for create, defaults to the cluster{color}*
 *{color:#172b4d}default.{color}*

...
--replication-factor  partition in the topic being
 created. *If not supplied, defaults*
 *to the cluster default.*



 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-778 KRaft Upgrades

[David Arthur]

Jose, thanks for the thorough review and comments!

I am out of the office until next week, so I probably won't be able to
update the KIP until then. Here are some replies to your questions:

1. Generate snapshot on upgrade
> > Metadata snapshot is generated and sent to the other nodes
> Why does the Active Controller need to generate a new snapshot and
> force a snapshot fetch from the replicas (inactive controller and
> brokers) on an upgrade? Isn't writing the FeatureLevelRecord good
> enough to communicate the upgrade to the replicas?


You're right, we don't necessarily need to _transmit_ a snapshot, since
each node can generate its own equivalent snapshot

2. Generate snapshot on downgrade
> > Metadata snapshot is generated and sent to the other inactive
> controllers and to brokers (this snapshot may be lossy!)
> Why do we need to send this downgraded snapshot to the brokers? The
> replicas have seen the FeatureLevelRecord and noticed the downgrade.
> Can we have the replicas each independently generate a downgraded
> snapshot at the offset for the downgraded FeatureLevelRecord? I assume
> that the active controller will guarantee that all records after the
> FatureLevelRecord use the downgraded version. If so, it would be good
> to mention that explicitly.


Similar to above, yes a broker that detects a downgrade via
FeatureLevelRecord could generate its own downgrade snapshot and reload its
state from that. This does get a little fuzzy when we consider cases where
brokers are on different software versions and could be generating a
downgrade snapshot for version X, but using different versions of the code.
It might be safer to let the controller generate the snapshot so each
broker (regardless of software version) gets the same records. However, for
upgrades (or downgrades) we expect the whole cluster to be running the same
software version before triggering the metadata.version change, so perhaps
this isn't a likely scenario. Thoughts?


3. Max metadata version
> >For the first release that supports metadata.version, we can simply
> initialize metadata.version with the current (and only) version. For future
> releases, we will need a mechanism to bootstrap a particular version. This
> could be done using the meta.properties file or some similar mechanism. The
> reason we need the allow for a specific initial version is to support the
> use case of starting a Kafka cluster at version X with an older
> metadata.version.


I assume that the Active Controller will learn the metadata version of
> the broker through the BrokerRegistrationRequest. How will the Active
> Controller learn about the max metadata version of the inactive
> controller nodes? We currently don't send a registration request from
> the inactive controller to the active controller.


This came up during the design, but I neglected to add it to the KIP. We
will need a mechanism for determining the supported features of each
controller similar to how brokers use BrokerRegistrationRequest. Perhaps
controllers could write a FeatureLevelRecord (or similar) to the metadata
log indicating their supported version. WDYT?

Why do you need to bootstrap a particular version? Isn't the intent
> that the broker will learn the active metadata version by reading the
> metadata before unfencing?


This bootstrapping is needed for when a KRaft cluster is first started. If
we don't have this mechanism, the cluster can't really do anything until
the operator finalizes the metadata.version with the tool. The
bootstrapping will be done by the controller and the brokers will see this
version as a record (like you say). I'll add some text to clarify this.


4. Reject Registration - This is related to the bullet point above.
> What will be the behavior of the active controller if the broker sends
> a metadata version that is not compatible with the cluster wide
> metadata version?


If a broker starts up with a lower supported version range than the current
cluster metadata.version, it should log an error and shutdown. This is in
line with KIP-584.

5. Discover upgrade
> > This snapshot will be a convenient way to let broker and controller
> components rebuild their entire in-memory state following an upgrade.
> Can we rely on the presence of the FeatureLevelRecord for the metadata
> version for this functionality? If so, it avoids having to reload the
> snapshot.


For upgrades, yes probably since we won't need to "rewrite" any records in
this case. For downgrades, we will need to generate the snapshot and reload
everything.

6. Metadata version specification
> >  V4(version=4, isBackwardsCompatible=false, description="New metadata
> record type Bar"),


Very cool. Do you have plans to generate Apache Kafka HTML
> documentation for this information? Would be helpful to display this
> information to the user using the kafka-features.sh and feature RPC?


Hm good idea :) I'll add a brief section on documentation. This would
certainly be very useful

Wiki Permissions

[Knowles Atchison Jr]

Good morning,

I would like to author a KIP.

May I please have permissions granted for the wiki?

username: katchison

Thank you.

Knowles

Re: [DISCUSS] KIP-776: Add Consumer#peek for debugging/tuning

[Luke Chen]

Hi Mickael,
Thanks for your comments.
I've added a use case into the KIP, and added Boyang's comment into
rejected alternatives section.

Hope that makes sense and strengthens the motivation.
If you have other suggestions, please let me know.

Thank you.
Luke

On Mon, Oct 4, 2021 at 10:23 PM Mickael Maison 
wrote:

> Hi Luke,
>
> Thanks for the KIP.
>
> Can you clarify the use cases you have in mind exactly? This would
> strengthen the motivation which I find a bit weak at the moment.
>
> As mentioned by Boyang, it's possible to achieve something similar
> with the existing APIs, for example with poll/seek or with
> listOffsets. Can you list them in the rejected alternatives section
> and explain why they were rejected.
>
> Thanks
>
>
> On Tue, Sep 21, 2021 at 9:47 AM Luke Chen  wrote:
> >
> > Thanks for your feedback, Sagar, Boyang.
> >
> > I've added an additional API to take the Set as the
> > partitions to fetch from. Good suggestion!
> > I also updated the java doc in the KIP.
> >
> > And for the question that the behavior can also be achieved by using
> manual
> > offset commit + offset position rewind. That's true.
> > But I have the same thoughts as Sagar, which is that, it's for advanced
> > users.
> > Another reason is for simplicity. If you've ever used the peek API from
> > java collection (ex: Queue#peek), you should know what I'm talking about.
> > When you have data in a queue, if you want to know what the first data is
> > in the queue, you'd use peek(). You can also achieve it by remove() the
> 1st
> > element from queue, and then added it back to the right position, but I
> > believe that's not what you'd do.
> >
> > Thank you.
> > Luke
> >
> >
> > On Tue, Sep 21, 2021 at 1:02 AM Sagar  wrote:
> >
> > > Thanks Luke for the KIP. I think it makes sense.
> > >
> > > @Boyang,
> > >
> > > While it is possible to get the functionality using manual offset
> commit +
> > > offset position rewind as you stated, IMHO it could still be a very
> handy
> > > addition to the APIs.  The way I see it, manual offset commit + offset
> > > position rewind is for slightly more advanced users and the addition of
> > > peek() API would make it trivial to get the mentioned functionality.
> > >
> > > I agree to the point of adding a mechanism to fetch a more fine
> grained set
> > > of records. Maybe, add another API which takes a Set?
> In
> > > this case, we would probably need to add a behaviour to throw some
> > > exception when a user tries to peek from a TopicPartition that he/she
> isn't
> > > subscribed to.
> > >
> > > nit: In the javadoc, this line =>
> > >
> > > This method returns immediately if there are records available or
> > > exception thrown.
> > >
> > >
> > > should probably be =>
> > >
> > >
> > > This method returns immediately if there are no records available or
> > > exception thrown.
> > >
> > >
> > > Thanks!
> > >
> > > Sagar.
> > >
> > >
> > >
> > >
> > > On Mon, Sep 20, 2021 at 4:22 AM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Luke for the KIP.
> > > >
> > > > I think I understand the motivation is to avoid affecting offset
> > > positions
> > > > of the records, but the feature could be easily realized on the user
> side
> > > > by using manual offset commit + offset position rewind. So the new
> peek()
> > > > function doesn't provide any new functionality IMHO, weakening the
> > > > motivation a bit.
> > > >
> > > > Additionally, for the peek() case, I believe that users may want to
> have
> > > > more fine-grained exposure of records, such as from specific
> partitions
> > > > instead of getting random records. It's probably useful to define an
> > > option
> > > > handle class in the parameters to help clarify what specific records
> to
> > > be
> > > > returned.
> > > >
> > > > Boyang
> > > >
> > > > On Sun, Sep 19, 2021 at 1:51 AM Luke Chen  wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to discuss the following proposal to add Consumer#peek for
> > > > > debugging/tuning.
> > > > >
> > > > > The main purpose for Consumer#peek is to allow users:
> > > > >
> > > > >1. peek what records existed at broker side and not increasing
> the
> > > > >position offsets.
> > > > >2. throw exceptions when there is connection error existed
> between
> > > > >consumer and broker (or other exceptions will be thrown by
> "poll")
> > > > >
> > > > >
> > > > > detailed description can be found her:
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188746244
> > > > >
> > > > >
> > > > > Any comments and feedback are welcomed.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > >
> > >
>

[jira] [Created] (KAFKA-13346) Kafka Streams fails due to unexpected exception

[Amit Gupta (Jira)]

Amit Gupta created KAFKA-13346:
--

 Summary: Kafka Streams fails due to unexpected exception
 Key: KAFKA-13346
 URL: https://issues.apache.org/jira/browse/KAFKA-13346
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Amit Gupta


Hello,

We are using Kafka Streams and we observe that some times on some of the hosts 
running streams application, Kafka streams instance fails with unexpected 
exception.

 
|org.apache.kafka.streams.errors.ProcessorStateException: Error opening store 
state-store at location .
at 
org.apache.kafka.streams.state.internals.RocksDBStore.openRocksDB(RocksDBStore.java:214)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:188)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:224)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:42)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$0(MeteredKeyValueStore.java:101)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:836)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:101)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.ProcessorStateManager.registerStateStores(ProcessorStateManager.java:199)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:76)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StandbyTask.initializeIfNeeded(StandbyTask.java:95)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:426)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:660)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
 ~[kafka-streams-2.6.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
 ~[kafka-streams-2.6.0.jar:?]
Caused by: org.rocksdb.RocksDBException: lock : 
./0_468/rocksdb/state-store/LOCK: No locks available
at org.rocksdb.RocksDB.open(Native Method) ~[rocksdbjni-5.18.3.jar:?]
at org.rocksdb.RocksDB.open(RocksDB.java:286) ~[rocksdbjni-5.18.3.jar:?]
at 
org.apache.kafka.streams.state.internals.RocksDBStore.openRocksDB(RocksDBStore.java:211)
 ~[kafka-streams-2.6.0.jar:?]
... 15 more|



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Jira contributor request

[Matthias J. Sax]

Done.

On 10/4/21 3:04 PM, Bartłomiej S wrote:

Hi,

I would like to become a contributor in JIRA, would you please grant
me required permission?

Jira ID: voookie

Best Regards,
Bartlomiej