[jira] [Created] (KAFKA-15343) Fix MirrorConnectIntegrationTests causing ci build failures.

[Prasanth Kumar (Jira)]

Prasanth Kumar created KAFKA-15343:
--

 Summary: Fix MirrorConnectIntegrationTests causing ci build 
failures.
 Key: KAFKA-15343
 URL: https://issues.apache.org/jira/browse/KAFKA-15343
 Project: Kafka
  Issue Type: Bug
  Components: build
Affects Versions: 3.6.0
Reporter: Prasanth Kumar


There are several instances of tests interacting badly with gradle daemon(s) 
running on ports that the kafka broker previously used. After going through the 
debug logs we observed a few retrying kafka clients trying to connect to broker 
which got shutdown and the gradle worker chose the same port on which broker 
was running. Later in the build, the gradle daemon attempted to connect to the 
worker and could not, triggering a failure. Ideally gradle would not exit when 
connected to from an invalid client - in testing with netcat, it would often 
handle these without dying. However there appear to be some cases where the 
daemon dies completely. Both the broker code and the gradle workers bind to 
port 0, resulting in the OS assigning it an unused port. This does avoid 
conflicts, but does not ensure that long lived clients do not attempt to 
connect to these ports afterwards. It's possible that closing the client in 
between may be enough to work around this issue. Till then we will disable the 
test to avoid the ci blocker from testing the code changes.



*MirrorConnectorsIntegrationBaseTest and extending Tests*


{code:java}
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+ [DEBUG] 
[TestEventLogger] 
MirrorConnectorsWithCustomForwardingAdminIntegrationTest > 
testReplicateSourceDefault() STANDARD_OUT
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+ [DEBUG] 
[TestEventLogger] [2023-07-04 11:47:46,799]
 INFO primary REST service: http://localhost:43809/connectors 
(org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest:224)
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+ [DEBUG] 
[TestEventLogger] [2023-07-04 11:47:46,799] 
INFO backup REST service: http://localhost:43323/connectors 
(org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest:225)
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+ [DEBUG] 
[TestEventLogger] [2023-07-04 11:47:46,799] 
INFO primary brokers: localhost:37557 
(org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest:226)
[2023-07-04T11:59:12.968Z] 2023-07-04T11:59:12.900+ [DEBUG] 
[org.gradle.internal.remote.internal.inet.TcpIncomingConnector] 
Accepted connection from /127.0.0.1:47660 to /127.0.0.1:37557.
[2023-07-04T11:59:13.233Z] 
org.gradle.internal.remote.internal.MessageIOException: Could not read message 
from '/127.0.0.1:47660'.
[2023-07-04T11:59:12.970Z] 2023-07-04T11:59:12.579+ [DEBUG] 
[org.gradle.internal.remote.internal.inet.TcpIncomingConnector] Listening on 
[d6bf30cb-bca2-46d9-8aeb-b9fd0497f54d port:37557, 
addresses:[localhost/127.0.0.1]].
[2023-07-04T11:59:46.519Z] 2023-07-04T11:59:13.014+ [ERROR] 
[system.err] org.gradle.internal.remote.internal.ConnectException: Could not 
connect to server [d6bf30cb-bca2-46d9-8aeb-b9fd0497f54d port:37557, 
addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1]. {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2102

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2101

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Bruno Cadonna]

Hi Nick!

Thanks for the updates!

1.
Why does StateStore#flush() default to 
StateStore#commit(Collections.emptyMap())?
Since calls to flush() will not exist anymore after this KIP is 
released, I would rather throw an unsupported operation exception by 
default.



2.
When would a state store return -1 from 
StateStore#approximateNumUncommittedBytes() while being transactional?


Wouldn't StateStore#approximateNumUncommittedBytes() also return 0 if 
the state store is transactional but nothing has been written to the 
state store yet?



3.
Sorry for bringing this up again. Does this KIP really need to introduce 
StateStoreContext#isolationLevel()? StateStoreContext has already 
appConfigs() which basically exposes the same information, i.e., if EOS 
is enabled or not.

In one of your previous e-mails you wrote:

"My idea was to try to keep the StateStore interface as loosely coupled
from the Streams engine as possible, to give implementers more freedom, 
and reduce the amount of internal knowledge required."


While I understand the intent, I doubt that it decreases the coupling of 
a StateStore interface and the Streams engine. READ_COMMITTED only 
applies to IQ but not to reads by processors. Thus, implementers need to 
understand how Streams accesses the state stores.


I would like to hear what others think about this.


4.
Great exposing new metrics for transactional state stores! However, I 
would prefer to add new metrics and deprecate (in the docs) the old 
ones. You can find examples of deprecated metrics here: 
https://kafka.apache.org/documentation/#selector_monitoring



5.
Why does the KIP mention position files? I do not think they are related 
to transactions or flushes.



6.
I think we will also need to adapt/add integration tests besides unit 
tests. Additionally, we probably need integration or system tests to 
verify that upgrades and downgrades between transactional and 
non-transactional state stores work as expected.



Best,
Bruno





On 7/21/23 10:34 PM, Nick Telford wrote:

One more thing: I noted John's suggestion in the KIP, under "Rejected
Alternatives". I still think it's an idea worth pursuing, but I believe
that it's out of the scope of this KIP, because it solves a different set
of problems to this KIP, and the scope of this one has already grown quite
large!

On Fri, 21 Jul 2023 at 21:33, Nick Telford  wrote:


Hi everyone,

I've updated the KIP (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-892%3A+Transactional+Semantics+for+StateStores)
with the latest changes; mostly bringing back "Atomic Checkpointing" (for
what feels like the 10th time!). I think the one thing missing is some
changes to metrics (notably the store "flush" metrics will need to be
renamed to "commit").

The reason I brought back Atomic Checkpointing was to decouple store flush
from store commit. This is important, because with Transactional
StateStores, we now need to call "flush" on *every* Task commit, and not
just when the StateStore is closing, otherwise our transaction buffer will
never be written and persisted, instead growing unbounded! I experimented
with some simple solutions, like forcing a store flush whenever the
transaction buffer was likely to exceed its configured size, but this was
brittle: it prevented the transaction buffer from being configured to be
unbounded, and it still would have required explicit flushes of RocksDB,
yielding sub-optimal performance and memory utilization.

I deemed Atomic Checkpointing to be the "right" way to resolve this
problem. By ensuring that the changelog offsets that correspond to the most
recently written records are always atomically written to the StateStore
(by writing them to the same transaction buffer), we can avoid forcibly
flushing the RocksDB memtables to disk, letting RocksDB flush them only
when necessary, without losing any of our consistency guarantees. See the
updated KIP for more info.

I have fully implemented these changes, although I'm still not entirely
happy with the implementation for segmented StateStores, so I plan to
refactor that. Despite that, all tests pass. If you'd like to try out or
review this highly experimental and incomplete branch, it's available here:
https://github.com/nicktelford/kafka/tree/KIP-892-3.5.0. Note: it's built
against Kafka 3.5.0 so that I had a stable base to build and test it on,
and to enable easy apples-to-apples comparisons in a live environment. I
plan to rebase it against trunk once it's nearer completion and has been
proven on our main application.

I would really appreciate help in reviewing and testing:
- Segmented (Versioned, Session and Window) stores
- Global stores

As I do not currently use either of these, so my primary test environment
doesn't test these areas.

I'm going on Parental Leave starting next week for a few weeks, so will
not have time to move this forward until late August. That said, your
feedback is welcome and appreciated, I just won't be able to respond as

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2100

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2099

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 409232 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldThrowUnsupportedOperationExceptionForForbiddenMethods() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldThrowUnsupportedOperationExceptionForForbiddenMethods() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateChangelogPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateChangelogPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateState() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateState() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateIsActive() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateIsActive() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateId() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateId() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateInputPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateInputPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateCommitRequested() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateCommitRequested() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateNeedsInitializationOrRestoration() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateNeedsInitializationOrRestoration() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldThrowIfAddTaskActionIsCreatedWithNullTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldThrowIfAddTaskActionIsCreatedWithNullTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldCreateAddTaskAction() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldCreateAddTaskAction() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldCreateRemoveTaskAction() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldCreateRemoveTaskAction() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldThrowIfRemoveTaskActionIsCreatedWithNullTaskId() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldThrowIfRemoveTaskActionIsCreatedWithNullTaskId() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToSuspendShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToSuspendShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldOnlyKeepLastUpdateAction() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldOnlyKeepLastUpdateAction() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToRecycle() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToRecycle() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() STARTED

Gradle Test Run :streams:test > 

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Calvin Liu]

   1. Yes, the new protocol requires 2 things to advance the HWM. a) The
   messages have been replicated to the controller-committed ISR members. b)
   The number of ISR members should be at least the min ISR.
   2. With the current protocol, we are not able to select broker 1 as the
   leader. If we first imply we have the new HWM requirement in place, then
   broker 1 is a good candidate to choose. The following part of the KIP (ELR)
   part will explain a new mechanism to enable us to choose broker 1. Note, if
   both HWM and ELR are in place, broker 1 will be actually elected in T3.


On Fri, Aug 11, 2023 at 10:05 AM Jeff Kim 
wrote:

> Hi Calvin,
>
> Thanks for the KIP! I'm still digesting it but I have two questions:
>
> > In the scenario raised in the motivation section, the server may receive
> ack=1 messages during T1 and advance High Watermark when the leader
> is the only one in ISR.
>
> To confirm, the current protocol allows advancing the HWM if all brokers in
> the ISR append to their logs (in this case only the leader). And we're
> proposing
> to advance the HWM only when  brokers
> replicate. Is this correct?
>
> > Then, if we elect broker 1 as the leader at T4, though we can guarantee
> the safety of ack=all messages, the High Watermark may move backward
> which causes further impacts on the consumers.
>
> How can broker 1 become the leader if it was ineligible in T3? Or are
> you referring to broker 2?
>
> Thanks,
> Jeff
>
> On Thu, Aug 10, 2023 at 6:48 PM Calvin Liu 
> wrote:
>
> > Hi everyone,
> > I'd like to discuss a series of enhancement to the replication protocol.
> >
> > A partition replica can experience local data loss in unclean shutdown
> > scenarios where unflushed data in the OS page cache is lost - such as an
> > availability zone power outage or a server error. The Kafka replication
> > protocol is designed to handle these situations by removing such replicas
> > from the ISR and only re-adding them once they have caught up and
> therefore
> > recovered any lost data. This prevents replicas that lost an arbitrary
> log
> > suffix, which included committed data, from being elected leader.
> > However, there is a "last replica standing" state which when combined
> with
> > a data loss unclean shutdown event can turn a local data loss scenario
> into
> > a global data loss scenario, i.e., committed data can be removed from all
> > replicas. When the last replica in the ISR experiences an unclean
> shutdown
> > and loses committed data, it will be reelected leader after starting up
> > again, causing rejoining followers to truncate their logs and thereby
> > removing the last copies of the committed records which the leader lost
> > initially.
> >
> > The new KIP will maximize the protection and provides MinISR-1 tolerance
> to
> > data loss unclean shutdown events.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> >
>

Re: [DISCUSSION] KIP-965: Support disaster recovery between clusters by MirrorMaker

[Igor Fomenko]

I agree that synchronizing ACL between clusters is a very useful feature
for DR scenarios.
In fact, I would prefer this to be a default setting since almost every
prod implementation requires a DR cluster.

There are some scenarios when replication between clusters is done for some
other reasons (like cluster migration for example) but this in my opinion
is less common than DR scenario.

Igor


On Tue, Aug 8, 2023 at 9:33 AM Ryanne Dolan  wrote:

> hudeqi, I'd call the configuration property something that describes what
> it does rather than it's intended use-case.
>
> Ryanne
>
> On Tue, Aug 8, 2023, 4:46 AM hudeqi <16120...@bjtu.edu.cn> wrote:
>
> > Hi, all. I want to submit a kip, and hope get some review and good
> > suggestions. the kip is here:
> https://cwiki.apache.org/confluence/x/k5KzDw
> >
> > Motivation:
> >
> >
> > When mirroring ACLs, MirrorMaker downgrades allow ALL ACLs to allow READ.
> > The rationale to is prevent other clients to produce to remote topics,
> > which is mentioned in KIP-382: MirrorMaker 2.0.
> >
> > However in disaster recovery scenarios, where the target cluster is not
> > used and just a "hot standby", it would be preferable to have exactly the
> > same ACLs on both clusters to speed up failover. Therefore, in this
> > scenario, we need to synchronize the topic write ACL, group ACL, and
> > absolute user scram credential of the source cluster topic to the target
> > cluster, so that when the user directly switches the read and write
> service
> > to the target cluster, it can be ran directly.
> >
> > Proposed changes:
> >
> > Add a config parameter: disaster.recovery.enabled in MirrorMakerConfig,
> > the default is false, it will leave the current sync behavior unchanged,
> if
> > set true, it will synchronize the topic write ACL, group ACL, and
> > absolute user scram credential of the source cluster replicated topics to
> > the target cluster.
> >
> > topic write ACL: Filter all topic read Acl informations
> related
> > to the topics replicated with the source cluster.
> > user scram credential: Filter the user scram credential to be
> synchronized
> > according to the topic acl information to be synchronized and create user
> > in target cluster.
> > group ACL: The group Acl information is obtained by filtering the user
> > obtained above.
> >
> > Looking forward to your reply.
> >
> > Best, hudeqi
>

[jira] [Created] (KAFKA-15342) Considering upgrading to Mockito 5.4.1 or later

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-15342:
-

 Summary: Considering upgrading to Mockito 5.4.1 or later
 Key: KAFKA-15342
 URL: https://issues.apache.org/jira/browse/KAFKA-15342
 Project: Kafka
  Issue Type: Task
  Components: unit tests
Reporter: Chris Egerton
Assignee: Chris Egerton
 Fix For: 4.0.0


We're currently stuck on Mockito 4.x.y because the 5.x.y line requires Java 11 
and, until we begin to work on Kafka 4.0.0, we continue to support Java 8.

Either directly before, or after releasing Kafka 4.0.0, we should try to 
upgrade to a version of Mockito on the 5.x.y line.

If we're able to use a version that includes 
[https://github.com/mockito/mockito/pull/3078|https://github.com/mockito/mockito/pull/3078,]
 (which should be included in either a 5.4.1 or 5.5.0 release), we should also 
revert the change made for https://issues.apache.org/jira/browse/KAFKA-14682, 
which is just a temporary workaround.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [DISCUSS] KIP-965: Support disaster recovery between clusters by MirrorMaker

[hudeqi]

bump this discuss thread.

best,
hudeqi

hudeqi 16120...@bjtu.edu.cn写道：
> Thanks for your suggestion, Ryanne. I have updated the configuration name in 
> cwiki.
> 
> best,
> hudeqi

Re: Re: [DISCUSS] KIP-943: Add independent "offset.storage.segment.bytes" for connect-distributed.properties

[hudeqi]

bump this discuss thread.

best,
hudeqi

hudeqi 16120...@bjtu.edu.cn写道：
> Sorry for not getting email reminders and ignoring your reply for getting 
> back so late, Yash Mayya, Greg Harris, Sagar.
> 
> Thank you for your thoughts and suggestions, I learned a lot, I will give my 
> thoughts and answers in a comprehensive way:
> 1. The default configuration of 50MB is the online configuration I actually 
> used to solve this problem, and the effect is better (see the description of 
> jira:https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-15086?filter=allopenissues.
>  In fact, I think it may be better to set this value smaller, so I abandoned 
> the default value like __consumer_offsets, but I don't know how much the 
> default value is the best.). Secondly, I also set the default value of 50MB 
> online through ConfigDef#defineInternal, and if the value configured by the 
> user is greater than the default value, the warning log will be displayed, 
> but the only difference from your said is that I will overwrite the value 
> configured by the user with the default value (emmm, this point was denied by 
> Chris Egerton: https://github.com/apache/kafka/pull/13852, in fact, you all 
> agree that should not directly override the user-configured value, and now I 
> agree with this). 
> 2. I think the potential bug that Greg mentioned may lead to inconsistent 
> state between workers is a great point. It is true that we cannot directly 
> change the configuration for an existing internal topics. Perhaps a more 
> tricky and disgusting approach is that we manually find that the active 
> segment sizes of all current partitions are relatively small, first stop all 
> connect instances, then change the topic configuration, and finally start the 
> instances.
> 
> To sum up, I think whether the scope of the KIP could be reduced to: only set 
> the default value of the 'segment.bytes' of the internal topics and make a 
> warning for the bigger value configured by the user. What do you think? If 
> there's a better way I'm all ears.
> 
> best,
> hudeqi

[jira] [Resolved] (KAFKA-14682) Unused stubbings are not reported by Mockito during CI builds

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14682.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> Unused stubbings are not reported by Mockito during CI builds
> -
>
> Key: KAFKA-14682
> URL: https://issues.apache.org/jira/browse/KAFKA-14682
> Project: Kafka
>  Issue Type: Test
>  Components: unit tests
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.6.0
>
>
> We've started using [strict 
> stubbing|https://javadoc.io/static/org.mockito/mockito-core/4.6.1/org/mockito/junit/MockitoJUnitRunner.StrictStubs.html]
>  for unit tests written with Mockito, which is supposed to automatically fail 
> tests when they set up mock expectations that go unused.
> However, these failures are not reported during Jenkins builds, even if they 
> are reported when building/testing locally.
> In at least one case, this difference appears to be because our [Jenkins 
> build|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/Jenkinsfile#L32-L35]
>  uses the custom {{unitTest}} and {{integrationTest}} tasks defined in the 
> project's [Gradle build 
> file|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/build.gradle#L452-L543],
>  instead of the {{test}} task. Some IDEs (such as IntelliJ) may use the 
> latter instead of the former, which can cause tests to fail due to 
> unnecessary stubbings when being run in that IDE but not when being built on 
> Jenkins.
> It's possible that, because the custom test tasks filter out some tests from 
> running, Mockito does not check for unnecessary stubbings in order to avoid 
> incorrectly failing tests that set up mocks in, e.g., a {{@BeforeEach}} 
> method.
>  
> This exact behavior has been reported elsewhere as a [Gradle 
> issue|https://github.com/gradle/gradle/issues/10694]; based on [discussion on 
> that 
> thread|https://github.com/gradle/gradle/issues/10694#issuecomment-1374911274],
>  it appears this is a known and somewhat-intentional limitation of Mockito:
> {quote}I spent some time trying to solve this and eventually I stumbled upon 
> this piece in Mockito's JUnit runner:
> [https://github.com/mockito/mockito/blob/main/src/main/java/org/mockito/internal/runners/StrictRunner.java#L47-L53]
> // only report when:
> // 1. if all tests from given test have ran (filter requested is false)
> // Otherwise we would report unnecessary stubs even if the user runs just 
> single test
> // from the class
> // 2. tests are successful (we don't want to add an extra failure on top of 
> any existing
> // failure, to avoid confusion)
>  
> (1) suggests that skipping unused stub validation is the intended behavior 
> when the user filters a single test from the class. However, this behavior 
> applies to any type of filter.
> And Gradle indeed applies a {{CategoryFilter}} if categories are configured: 
> [https://github.com/rieske/gradle/blob/e82029abb559d620fdfecb4708a95c6313af625c/subprojects/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junit/JUnitTestClassExecutor.java#L70-L96]
> Which then causes Mockito to not validate unused stubs.
> {quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2098

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 408705 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldThrowUnsupportedOperationExceptionForForbiddenMethods() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldThrowUnsupportedOperationExceptionForForbiddenMethods() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateChangelogPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateChangelogPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateState() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateState() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateIsActive() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateIsActive() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateId() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateId() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateInputPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateInputPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateCommitRequested() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateCommitRequested() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateNeedsInitializationOrRestoration() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > ReadOnlyTaskTest > 
shouldDelegateNeedsInitializationOrRestoration() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldThrowIfAddTaskActionIsCreatedWithNullTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldThrowIfAddTaskActionIsCreatedWithNullTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 85 > TaskAndActionTest > 
shouldCreateAddTaskAction() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 85 > 

Re: [VOTE] KIP-942: Add Power(ppc64le) support

[Divij Vaidya]

+1 (binding)

--
Divij Vaidya


On Wed, Jul 26, 2023 at 9:04 AM Vaibhav Nazare
 wrote:
>
> I'd like to call a vote on KIP-942

[jira] [Resolved] (KAFKA-13836) Improve KRaft broker heartbeat logic

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-13836.
-
Resolution: Won't Fix

> Improve KRaft broker heartbeat logic
> 
>
> Key: KAFKA-13836
> URL: https://issues.apache.org/jira/browse/KAFKA-13836
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Deng Ziming
>Assignee: Deng Ziming
>Priority: Major
>
> # Don't advertise an offset to the controller until it has been published
>  # only unfence a broker when it has seen it's own registration



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Taras Ledkov]

Hi Kafka Team.

I would like to start a discussion for KIP-967: Support custom SSL 
configuration for Kafka Connect RestServer [1].
The purpose of this KIP is add ability to use custom SSL factory to configure 
Kafka Connect RestServer.
It looks like the interface 'SslEngineFactory' may be used with simple 
adapters. 

The prototype of the patch is available on PR#14203 [2].
It is not a final/clean patch yet. Just for demo & discuss. 

Thanks in advance for leaving a review!

[1]. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
[2]. https://github.com/apache/kafka/pull/14203

--
With best regards,
Taras Ledkov

[jira] [Resolved] (KAFKA-15289) Support KRaft mode in RequestQuotaTest

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15289.
-
Resolution: Fixed

> Support KRaft mode in RequestQuotaTest
> --
>
> Key: KAFKA-15289
> URL: https://issues.apache.org/jira/browse/KAFKA-15289
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Deng Ziming
>Priority: Major
>  Labels: newbee
>
> we are calling `zkBrokerApis` in RequestQuotaTest, we should ensure kraft 
> broker apis are also supported, so use clientApis as far as possible.use 
> zkBrokerApis.clientApis instead of ApiKeys.zkBrokerApis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15341) Enabling TS for a topic during rolling restart causes problems

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15341:


 Summary: Enabling TS for a topic during rolling restart causes 
problems
 Key: KAFKA-15341
 URL: https://issues.apache.org/jira/browse/KAFKA-15341
 Project: Kafka
  Issue Type: Bug
Reporter: Divij Vaidya
 Fix For: 3.6.0


When we are in a rolling restart to enable TS at system level, some brokers 
have TS enabled on them and some don't. We send an alter config call to enable 
TS for a topic, it hits a broker which has TS enabled, this broker forwards it 
to the controller and controller will send the config update to all brokers. 
When another broker which doesn't have TS enabled (because it hasn't undergone 
the restart yet) gets this config change, it "should" fail to apply it. But 
failing now is too late since alterConfig has already succeeded since 
controller->broker config propagation is done async.

With this JIRA, we want to have controller check if TS is enabled on all 
brokers before applying alter config to turn on TS for a topic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-962 Relax non-null key requirement in Kafka Streams

[Florin Akermann]

Thanks all.

The 72h window is through.

The vote on KIP-962 passes with:
2 binding +1
1 non-binding +1
no vetoes

Florin

On Fri, 11 Aug 2023 at 15:43, Bill Bejeck  wrote:

> +1(binding)
>
> On Fri, Aug 11, 2023 at 7:33 AM Lucas Brutschy
>  wrote:
>
> > +1 (non-binding)
> >
> > On Fri, Aug 11, 2023 at 1:08 AM Matthias J. Sax 
> wrote:
> > >
> > > +1 (binding)
> > >
> > > On 8/10/23 12:31 PM, Florin Akermann wrote:
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams
> > > >
> >
>