Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1676

2023-03-13 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 447747 lines...]
[2023-03-13T23:14:13.157Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateTopLevelPaths() PASSED
[2023-03-13T23:14:13.157Z] 
[2023-03-13T23:14:13.157Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED
[2023-03-13T23:14:13.157Z] 
[2023-03-13T23:14:13.157Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED
[2023-03-13T23:14:13.157Z] 
[2023-03-13T23:14:13.157Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED
[2023-03-13T23:14:14.094Z] 
[2023-03-13T23:14:14.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2023-03-13T23:14:14.094Z] 
[2023-03-13T23:14:14.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2023-03-13T23:14:14.094Z] 
[2023-03-13T23:14:14.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2023-03-13T23:14:14.094Z] 
[2023-03-13T23:14:14.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2023-03-13T23:14:14.094Z] 
[2023-03-13T23:14:14.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2023-03-13T23:14:14.094Z] 
[2023-03-13T23:14:14.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-03-13T23:14:15.031Z] 
[2023-03-13T23:14:15.031Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-03-13T23:14:15.031Z] 
[2023-03-13T23:14:15.031Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testAclMethods() STARTED
[2023-03-13T23:14:15.031Z] 
[2023-03-13T23:14:15.031Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testAclMethods() PASSED
[2023-03-13T23:14:15.031Z] 
[2023-03-13T23:14:15.031Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-03-13T23:14:16.071Z] 
[2023-03-13T23:14:16.071Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-03-13T23:14:16.071Z] 
[2023-03-13T23:14:16.071Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-03-13T23:14:16.071Z] 
[2023-03-13T23:14:16.071Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-03-13T23:14:16.071Z] 
[2023-03-13T23:14:16.071Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-03-13T23:14:16.071Z] 
[2023-03-13T23:14:16.071Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-03-13T23:14:16.071Z] 
[2023-03-13T23:14:16.071Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-03-13T23:14:17.094Z] 
[2023-03-13T23:14:17.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-03-13T23:14:17.094Z] 
[2023-03-13T23:14:17.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeletePath() STARTED
[2023-03-13T23:14:17.094Z] 
[2023-03-13T23:14:17.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeletePath() PASSED
[2023-03-13T23:14:17.094Z] 
[2023-03-13T23:14:17.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetBrokerMethods() STARTED
[2023-03-13T23:14:17.094Z] 
[2023-03-13T23:14:17.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetBrokerMethods() PASSED
[2023-03-13T23:14:17.094Z] 
[2023-03-13T23:14:17.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2023-03-13T23:14:18.019Z] 
[2023-03-13T23:14:18.019Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testJuteMaxBufffer() PASSED

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-03-13 Thread Greg Harris 
Yash,

> 'm not sure I follow - are you asking about how the tests will be updated
post this change or about how upgrades will look like for clusters in
production?

Currently the AK tests have a lot of calls to, for example, new
SinkRecord(String topic, int partition, Schema keySchema, Object key,
Schema valueSchema, Object value, long kafkaOffset), a constructor without
the original T/P/O values. I assumed that for backwards compatibility these
constructors would still be usable in new runtimes.
I imagine that there are also tests in downstream projects which make use
of these constructors, whenever a Transform, Predicate, or Task is tested
without a corresponding Converter. My question was about what values are
chosen for the original T/P/O methods when these constructors are used
after an upgrade to the latest connect-api.

> There shouldn't be any difference in behavior here - the framework will
add
the original T/P/O metadata to the record after the entire transformation
chain has been applied and just before sending the record to the task for
processing. The KIP doesn't propose that transformations themselves should
also be able to retrieve original T/P/O information for a sink record.

The KIP includes this: "Note that while the record's offset can't be
modified via the standard SinkRecord::newRecord methods that SMTs are
expected to use, SinkRecord has public constructors that would allow SMTs
to return records with modified offsets. This is why the proposed changes
include a new SinkRecord::originalKafkaOffset method as well."
In order to use the new or old SinkRecord constructors outside of the
newRecord methods, SMTs will downcast the previous record and may access
the original T/P/O methods. They may or may not forward this to the next
SMT, and they may or may not use it in their own computation.
Since this is acknowledged as a possible implementation, I was just asking
about when one SMT changes the original T/P/O, what should later SMTs and
predicates see from the original T/P/O methods?
If you inject the original T/P/O only before and after the chain, SMTs
after an SMT which changes the original T/P/O will see whatever the earlier
SMT emitted. Is this intentional, or should this be avoided?
For existing SMTs use the SinkRecord constructor, either directly or via
subclasses of ConnectRecord, they will drop the original T/P/O and fall
back to the logic from question (1).

> The rejected alternative basically says that we can't do a
deterministic mapping from virtual coordinates to physical coordinates
without doing a lot of book-keeping.

I suppose there is a possible implementation of metadata book-keeping which
provides a reasonable system of virtual coordinates, it just ended up
equivalent to hydrating intermediate topics to compute a consistent record
ordering. I wasn't convinced by calling it "book-keeping" since i've seen
that phrase used to disregard much less complicated state management, and
had to see exactly where that solution becomes unreasonable.

Thanks,
Greg

On Sun, Mar 12, 2023 at 6:30 AM Yash Mayya  wrote:

> Hi Greg,
>
> Thanks for the detailed review!
>
> > What is the expected state/behavior for SinkRecords
> > which do not have original T/P/O information after the
> > upgrade? Just browsing, it appears that tests make
> > extensive use of the existing public SinkRecord
> > constructors  for both Transformations and Connectors.
>
> I'm not sure I follow - are you asking about how the tests will be updated
> post this change or about how upgrades will look like for clusters in
> production? For the latter, we won't have to worry about sink records
> without original T/P/O information at all once a cluster is fully rolled
> and we will make it (hopefully) abundantly clear that connectors need to
> account for missing original T/P/O getter methods if they expect to be
> deployed on older Connect runtimes.
>
> > What is the expected behavior for Transformation
> > implementations which do not use the newRecord
> > methods and instead use public SinkRecord constructors?
> > The KIP mentions this as a justification for the
> > originalKafkaOffset method, but if existing implementations
> > are using the existing constructors, those constructors won't
> > forward the original T/P/O information to later transforms or
> > the task.
>
> There shouldn't be any difference in behavior here - the framework will add
> the original T/P/O metadata to the record after the entire transformation
> chain has been applied and just before sending the record to the task for
> processing. The KIP doesn't propose that transformations themselves should
> also be able to retrieve original T/P/O information for a sink record.
>
> > This reasoning and the KIP design seems to imply that the
> > connector is better equipped to solve this problem than the
> > framework, but the stated reasons are not convincing for me.
>
> This was added to the KIP by the original author, but I don't think the
>

Kafka Stream Metrics (3.3 / KAFKA-13945) and Documentation

2023-03-13 Thread Neil Buesing 
I have been looking at the new metrics in KAFKA-13945 (KIP-846). The
documentation states that this metrics are procesor-node metrics

https://kafka.apache.org/documentation/#kafka_streams_node_monitoring


*kafka.streams:type=stream-processor-node-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+),topic=([-.\w]+)*

My observations is that the documentation should have a
"stream-topic-metrics" section


*kafka.streams:type=stream-topic-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+),topic=([-.\w]+)*

It looks like they are in TopicMetrics.java which was created for this
effort.

I believe this is just a minor document change adding a new section "Task
Metrics" and placing these 4 metrics in this section with the given type.

Thanks,

-Neil

Re: [DISCUSS] KIP-882: Make Kafka Connect REST API request timeouts configurable

2023-03-13 Thread Greg Harris 
Yash,

1. I think that Request-Timeout is a fine choice. I did not see any
standard header for a use-case like this, so we are free to choose a name
for ourselves.

2. While such proposals do not exist, I don't think that means that we
should exclude them from consideration. If we do, then we run the risk of
making future implementations more complicated, impossible, or awkward to
use. This is a balance of course, as we cannot let future features paralyze
us from making improvements today. We cannot hold ourselves responsible for
handling unknown unknowns, but we can at least try to plan around known
unknowns.
Applying this to a hypothetical extension of timeouts to other endpoints:
my concern was primarily about whether this design applied everywhere felt
natural or awkward. In my opinion, a request parameter would draw a lot of
outsized attention in documentation, and be the first parameter for a lot
of the endpoints, while a header could be specified as a more obscure
global modifier to all requests.

3. 4. Applying the same standard to a hypothetical asynchronous I don't
believe it makes async validation substantially more complicated, except
the "request timeout interrupts connector creation" interaction. I also do
not think that it would make it impossible to implement an async validation
scheme in the future, since the synchronization is already in place, the
timeout is just different.
My chief concern was about the clarity of the API in the future when both
configurable timeouts and asynchronous APIs exist. Will the configurable
timeouts still make sense, and will they be a feature that people will use?
Are we expecting users to hold a single REST request open for 5, 20, or 60
minutes to accommodate a connector with an excessively long validation
step? Successfully validating a request requires the caller and the network
to remain stable for the entire duration of the request to have a chance of
success, and consumes (some small) amount of resources during that time.
And what if they decide they no longer want the validation result, and wish
to cancel the operation to free resources (both connect and external
system)? There is no standard mechanism to discard or cancel an HTTP
request (not even a connection loss).

You have said that the default timeout for other endpoints is more than
satisfactory for the typical user, and I agree. If other operations
(deleting, restarting, pause/unpause, etc) had a substantial synchronous
component that would benefit from a configurable timeout, I would be more
inclined to support configurable timeouts (all at once, or validation first
and other operations later).
But as it stands, I don't think I see the value-add of configurable
timeouts once there is another solution which targets the validate call
duration specifically, and I'm concerned that configurable timeouts would
be made irrelevant in that case.

Thanks,
Greg

On Wed, Mar 8, 2023 at 7:21 AM Yash Mayya  wrote:

> Hi Greg,
>
> Thanks for the response!
>
> 1. Hm that's a fair point, and while there aren't any strict guidelines or
> rules governing whether something should be a query parameter versus a
> request header, I agree that it might be more idiomatic for a request
> timeout value to be accepted via a custom request header. What do you
> think about using a header name like "X-Request-Timeout", or maybe simply
> "Request-Timeout" if we want to take into account
> https://www.rfc-editor.org/rfc/rfc6648?
>
> 2. 3. 4. Adding asynchronous validations via new endpoints / new request
> parameters on existing endpoints is definitely an interesting idea but I'm
> not sure whether this KIP should take into consideration a hypothetical
> future proposal? If there were already such a concrete proposal, I would
> definitely agree that this KIP should take that into consideration - but
> that isn't the case today. Furthermore, I'm not sure I understand why the
> addition of request timeout configurability to the existing endpoints would
> preclude the introduction of an asynchronous validation API in the future?
>
> Thanks,
> Yash
>
> On Sat, Mar 4, 2023 at 1:13 AM Greg Harris 
> wrote:
>
> > Yash,
> >
> > 1.
> > Currently the request parameters in the REST API are individual and
> pertain
> > to just one endpoint.
> > They also change the content of the query result, or change the action
> > taken on the cluster.
> > I think that a request timeout is a property of the HTTP request more
> than
> > it is a property of the cluster query or cluster action.
> > The two solutions have very similar tradeoffs, but I'm interested in
> > whether one is more idiomatic and more obvious to users.
> >
> > 2.
> > I understand that only these three endpoints are in need of increased
> > timeouts at this time due to long connector validations.
> > From another perspective, this change is making the API more irregular
> and
> > someone in the future might choose to make it more regular by
> standardizing
> > the configurable

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1675

2023-03-13 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 451137 lines...]
[2023-03-13T20:30:26.091Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() STARTED
[2023-03-13T20:30:26.091Z] 
[2023-03-13T20:30:26.091Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() PASSED
[2023-03-13T20:30:26.091Z] 
[2023-03-13T20:30:26.091Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateTopLevelPaths() STARTED
[2023-03-13T20:30:26.091Z] 
[2023-03-13T20:30:26.091Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateTopLevelPaths() PASSED
[2023-03-13T20:30:26.091Z] 
[2023-03-13T20:30:26.091Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED
[2023-03-13T20:30:27.015Z] 
[2023-03-13T20:30:27.015Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED
[2023-03-13T20:30:27.015Z] 
[2023-03-13T20:30:27.015Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED
[2023-03-13T20:30:27.015Z] 
[2023-03-13T20:30:27.015Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2023-03-13T20:30:27.015Z] 
[2023-03-13T20:30:27.015Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testAclMethods() STARTED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testAclMethods() PASSED
[2023-03-13T20:30:28.192Z] 
[2023-03-13T20:30:28.192Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-03-13T20:30:29.282Z] 
[2023-03-13T20:30:29.282Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-03-13T20:30:29.282Z] 
[2023-03-13T20:30:29.282Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-03-13T20:30:29.282Z] 
[2023-03-13T20:30:29.282Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-03-13T20:30:29.282Z] 
[2023-03-13T20:30:29.282Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-03-13T20:30:29.282Z] 
[2023-03-13T20:30:29.282Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-03-13T20:30:29.282Z] 
[2023-03-13T20:30:29.282Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-03-13T20:30:30.208Z] 
[2023-03-13T20:30:30.208Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-03-13T20:30:30.208Z] 
[2023-03-13T20:30:30.208Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeletePath() STARTED
[2023-03-13T20:30:30.208Z] 
[2023-03-13T20:30:30.208Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest > testDeletePath() PASSED
[2023-03-13T20:30:30.208Z] 
[2023-03-13T20:30:30.208Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > KafkaZkClientTest >

[jira] [Created] (KAFKA-14805) KRaft Controller doesn't allow metadata updates before migration starts

2023-03-13 Thread David Arthur (Jira) 
David Arthur created KAFKA-14805:


 Summary: KRaft Controller doesn't allow metadata updates before 
migration starts
 Key: KAFKA-14805
 URL: https://issues.apache.org/jira/browse/KAFKA-14805
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft
Reporter: David Arthur
 Fix For: 3.5.0, 3.4.1


When starting a ZK to KRaft migration, the new KRaft quorum should not accept 
external metadata updates from things like CreateTopics or AllocateProducerIds. 
Having metadata present in the log prior to the migration can lead to undefined 
state, which is not great.

This is currently causing test failures on trunk because some producer is 
allocating a producer ID between the time the KRaft quorum starts and the 
migration starts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14255) Fetching from follower should be disallowed if fetch from follower is disabled

2023-03-13 Thread Yi Ding (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Ding reopened KAFKA-14255:
-

> Fetching from follower should be disallowed if fetch from follower is disabled
> --
>
> Key: KAFKA-14255
> URL: https://issues.apache.org/jira/browse/KAFKA-14255
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Critical
>
> There are clients out there that have implemented KIP-392 (Fetch From 
> Follower) and thus use FetchRequest >= 11. However, they have not implemented 
> KIP-320 which add the leader epoch to the FetchRequest in version 9. Without 
> KIP-320, it is not safe to fetch from the follower. If a client does it by 
> mistake – e.g. based on stale metadata – that could lead to offset out of 
> range.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14255) Fetching from follower should be disallowed if fetch from follower is disabled

2023-03-13 Thread Yi Ding (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Ding resolved KAFKA-14255.
-
Resolution: Won't Fix

> Fetching from follower should be disallowed if fetch from follower is disabled
> --
>
> Key: KAFKA-14255
> URL: https://issues.apache.org/jira/browse/KAFKA-14255
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Critical
>
> There are clients out there that have implemented KIP-392 (Fetch From 
> Follower) and thus use FetchRequest >= 11. However, they have not implemented 
> KIP-320 which add the leader epoch to the FetchRequest in version 9. Without 
> KIP-320, it is not safe to fetch from the follower. If a client does it by 
> mistake – e.g. based on stale metadata – that could lead to offset out of 
> range.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14804) Connect docs fail to build with Gradle Swagger plugin 2.2.8

2023-03-13 Thread David Arthur (Jira) 
David Arthur created KAFKA-14804:


 Summary: Connect docs fail to build with Gradle Swagger plugin 
2.2.8
 Key: KAFKA-14804
 URL: https://issues.apache.org/jira/browse/KAFKA-14804
 Project: Kafka
  Issue Type: Bug
Reporter: David Arthur


There is an incompatibility somewhere between versions 2.2.0 and 2.2.8 that 
cause the following error when building the connect docs:

{code}
Caused by: org.gradle.api.GradleException: 
io.swagger.v3.jaxrs2.integration.SwaggerLoader.setOpenAPI31(java.lang.Boolean)
at 
io.swagger.v3.plugins.gradle.tasks.ResolveTask.resolve(ResolveTask.java:458)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
at 
org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
at 
org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at 
org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at 
org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:242)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
at 
org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:227)
at 
org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:210)
at 
org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:193)
at 
org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
at 
org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:93)
at 
org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
at 
org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:57)
at 
org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:54)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at 
org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at 
org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:54)
at 
org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
at 
org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)
at 
org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)
at 
org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
at 
org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
at 
org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
at

[jira] [Created] (KAFKA-14803) topic deletion bug

2023-03-13 Thread Behavox (Jira) 
Behavox created KAFKA-14803:
---

 Summary: topic deletion bug
 Key: KAFKA-14803
 URL: https://issues.apache.org/jira/browse/KAFKA-14803
 Project: Kafka
  Issue Type: Bug
  Components: controller, replication
Affects Versions: 3.3.2
Reporter: Behavox
 Attachments: server.properties

topic deletion doesn't work as expected when attempting to delete topic(s), 
after successful deletion topic is recreated in a multi-controller environment 
with 3 controllers and ReplicationFactor: 2

How to reproduce - attempt to delete topic. Topic is removed successfully and 
recreated right after removal. Example below shows a single topic named 
example-topic. We have a total count of 17000 topics in the affected cluster.
 
Our config is attached. 


Run 1

[2023-03-10 16:16:45,625] INFO [Controller 1] Removed topic example-topic with 
ID fh_aQcc3Sf2yVBTMrltBlQ. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:19:04,722] INFO [Controller 1] Created topic example-topic with 
topic ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:16:45,730] INFO [Controller 2] Removed topic example-topic with 
ID fh_aQcc3Sf2yVBTMrltBlQ. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:19:04,851] INFO [Controller 2] Created topic example-topic with 
topic ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:16:45,837] INFO [Controller 3] Removed topic example-topic with 
ID fh_aQcc3Sf2yVBTMrltBlQ. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:19:04,833] INFO [Controller 3] Created topic example-topic with 
topic ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)

Run 2

[2023-03-10 16:20:22,469] INFO [Controller 1] Removed topic example-topic with 
ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:22:19,711] INFO [Controller 1] Created topic example-topic with 
topic ID xxlJlIe_SvqQHtfgbX2eLA. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:20:22,674] INFO [Controller 2] Removed topic example-topic with 
ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:22:20,022] INFO [Controller 2] Created topic example-topic with 
topic ID xxlJlIe_SvqQHtfgbX2eLA. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:20:22,674] INFO [Controller 3] Removed topic example-topic with 
ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:22:20,020] INFO [Controller 3] Created topic example-topic with 
topic ID xxlJlIe_SvqQHtfgbX2eLA. 
(org.apache.kafka.controller.ReplicationControlManager)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14802) topic deletion bug

2023-03-13 Thread Behavox (Jira) 
Behavox created KAFKA-14802:
---

 Summary: topic deletion bug
 Key: KAFKA-14802
 URL: https://issues.apache.org/jira/browse/KAFKA-14802
 Project: Kafka
  Issue Type: Bug
  Components: controller, replication
Affects Versions: 3.3.2
Reporter: Behavox
 Attachments: server.properties

topic deletion doesn't work as expected when attempting to delete topic(s), 
after successful deletion topic is recreated in a multi-controller environment 
with 3 controllers and ReplicationFactor: 2

How to reproduce - attempt to delete topic. Topic is removed successfully and 
recreated right after removal. Example below shows a single topic named 
example-topic. We have a total count of 17000 topics in the affected cluster.
 
Our config is attached. 


Run 1

[2023-03-10 16:16:45,625] INFO [Controller 1] Removed topic example-topic with 
ID fh_aQcc3Sf2yVBTMrltBlQ. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:19:04,722] INFO [Controller 1] Created topic example-topic with 
topic ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:16:45,730] INFO [Controller 2] Removed topic example-topic with 
ID fh_aQcc3Sf2yVBTMrltBlQ. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:19:04,851] INFO [Controller 2] Created topic example-topic with 
topic ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:16:45,837] INFO [Controller 3] Removed topic example-topic with 
ID fh_aQcc3Sf2yVBTMrltBlQ. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:19:04,833] INFO [Controller 3] Created topic example-topic with 
topic ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)

Run 2

[2023-03-10 16:20:22,469] INFO [Controller 1] Removed topic example-topic with 
ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:22:19,711] INFO [Controller 1] Created topic example-topic with 
topic ID xxlJlIe_SvqQHtfgbX2eLA. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:20:22,674] INFO [Controller 2] Removed topic example-topic with 
ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:22:20,022] INFO [Controller 2] Created topic example-topic with 
topic ID xxlJlIe_SvqQHtfgbX2eLA. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:20:22,674] INFO [Controller 3] Removed topic example-topic with 
ID a-7OZG_XQhiCatOBft-9-g. 
(org.apache.kafka.controller.ReplicationControlManager)
[2023-03-10 16:22:20,020] INFO [Controller 3] Created topic example-topic with 
topic ID xxlJlIe_SvqQHtfgbX2eLA. 
(org.apache.kafka.controller.ReplicationControlManager)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

2023-03-13 Thread Chia-Ping Tsai 
ping for more voting :_

On 2023/02/18 08:36:50 Chia-Ping Tsai wrote:
> Hi,
> 
> I'd like to start the vote on KIP-614: An new java interface to replace 
> 'kafka.common.MessageReader'
> 
> KIP-614: 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
> 
> thread: 
> https://lists.apache.org/thread.html/r6db6708f64345bb8fe0d573e05014fb790e69d501f21f855ca65619a%40%3Cdev.kafka.apache.org%3E
> 
> Cheers,
> Chia-Ping

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-13 Thread Satish Duggana 
Congratulations Chris!

On Mon, 13 Mar 2023 at 16:09, Rajini Sivaram 
wrote:

> Congratulations, Chris!
>
> Regards,
>
> Rajini
>
> On Mon, Mar 13, 2023 at 9:07 AM Bruno Cadonna  wrote:
>
> > Congrats, Chris!
> >
> > Bruno
> >
> > On 10.03.23 18:47, Guozhang Wang wrote:
> > > Congrats Chris!
> > >
> > > On Fri, Mar 10, 2023 at 8:41 AM Jeremy Custenborder
> > >  wrote:
> > >>
> > >> Congrats buddy!
> > >>
> > >> On Fri, Mar 10, 2023 at 9:28 AM Randall Hauch 
> wrote:
> > >>>
> > >>> Congratulations, Chris!
> > >>>
> > >>> Randall
> > >>>
> > >>> On Fri, Mar 10, 2023 at 9:07 AM David Arthur
> > >>>  wrote:
> > >>>
> >  Congrats, Chris!
> > 
> >  On Fri, Mar 10, 2023 at 8:27 AM Matthew Benedict de Detrich
> >   wrote:
> > 
> > > Congrats, well deserved!
> > >
> > > On Thu, 9 Mar 2023, 19:12 Jun Rao, 
> wrote:
> > >
> > >> Hi, Everyone,
> > >>
> > >> Chris Egerton has been a Kafka committer since July 2022. He has
> > been
> > > very
> > >> instrumental to the community since becoming a committer. It's my
> > > pleasure
> > >> to announce that Chris is now a member of Kafka PMC.
> > >>
> > >> Congratulations Chris!
> > >>
> > >> Jun
> > >> on behalf of Apache Kafka PMC
> > >>
> > >
> > 
> > 
> >  --
> >  -David
> > 
> >
>

Re: [DISCUSS] KIP-911: Add source tag to MirrorSourceConnector metrics

2023-03-13 Thread Mickael Maison 
Hi,

Since it's a very small KIP, if there's no other feedback I'll start a
vote in the next couple of days.

Thanks,
Mickael

On Wed, Mar 8, 2023 at 10:20 PM Mickael Maison  wrote:
>
> Hi Chris,
>
> Thanks for taking a look at the KIP.
> I've removed the mention to the update mode. It was copied from
> somewhere else and I forgot to delete it.
>
> Thanks,
> Mickael
>
> On Wed, Mar 8, 2023 at 7:26 PM Chris Egerton  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP! LGTM.
> >
> > I find the rationales for the rejected alternatives convincing, and agree
> > with the deprecation plan and opt-in behavior for pre-4.0 releases. I also
> > appreciate the historical context provided in the motivation section about
> > why we don't already use a tag for the source cluster in MM2.
> >
> > One small note: the "add.source.alias.to.metrics" property has an update
> > mode listed in the KIP. Is this necessary? AFAIK that concept only applies
> > to broker configs. Shouldn't block the KIP either way since "Read only"
> > doesn't make any promises it can't keep, mostly asking for my own
> > edification.
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Mar 7, 2023 at 10:15 AM Mickael Maison 
> > wrote:
> >
> > > Hi,
> > >
> > > I created a KIP to tag the MirrorSourceConnector metrics with the
> > > source cluster alias. Currently they only have the target cluster
> > > alias.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-911%3A+Add+source+tag+to+MirrorSourceConnector+metrics
> > >
> > > Please take a look and let me know if you have any feedback.
> > >
> > > Thanks,
> > > Mickael
> > >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1674

2023-03-13 Thread Apache Jenkins Server 
See

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-13 Thread Rajini Sivaram 
Congratulations, Chris!

Regards,

Rajini

On Mon, Mar 13, 2023 at 9:07 AM Bruno Cadonna  wrote:

> Congrats, Chris!
>
> Bruno
>
> On 10.03.23 18:47, Guozhang Wang wrote:
> > Congrats Chris!
> >
> > On Fri, Mar 10, 2023 at 8:41 AM Jeremy Custenborder
> >  wrote:
> >>
> >> Congrats buddy!
> >>
> >> On Fri, Mar 10, 2023 at 9:28 AM Randall Hauch  wrote:
> >>>
> >>> Congratulations, Chris!
> >>>
> >>> Randall
> >>>
> >>> On Fri, Mar 10, 2023 at 9:07 AM David Arthur
> >>>  wrote:
> >>>
>  Congrats, Chris!
> 
>  On Fri, Mar 10, 2023 at 8:27 AM Matthew Benedict de Detrich
>   wrote:
> 
> > Congrats, well deserved!
> >
> > On Thu, 9 Mar 2023, 19:12 Jun Rao,  wrote:
> >
> >> Hi, Everyone,
> >>
> >> Chris Egerton has been a Kafka committer since July 2022. He has
> been
> > very
> >> instrumental to the community since becoming a committer. It's my
> > pleasure
> >> to announce that Chris is now a member of Kafka PMC.
> >>
> >> Congratulations Chris!
> >>
> >> Jun
> >> on behalf of Apache Kafka PMC
> >>
> >
> 
> 
>  --
>  -David
> 
>

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-13 Thread Bruno Cadonna 

Congrats, Chris!

Bruno

On 10.03.23 18:47, Guozhang Wang wrote:

Congrats Chris!

On Fri, Mar 10, 2023 at 8:41 AM Jeremy Custenborder
 wrote:


Congrats buddy!

On Fri, Mar 10, 2023 at 9:28 AM Randall Hauch  wrote:


Congratulations, Chris!

Randall

On Fri, Mar 10, 2023 at 9:07 AM David Arthur
 wrote:


Congrats, Chris!

On Fri, Mar 10, 2023 at 8:27 AM Matthew Benedict de Detrich
 wrote:


Congrats, well deserved!

On Thu, 9 Mar 2023, 19:12 Jun Rao,  wrote:


Hi, Everyone,

Chris Egerton has been a Kafka committer since July 2022. He has been

very

instrumental to the community since becoming a committer. It's my

pleasure

to announce that Chris is now a member of Kafka PMC.

Congratulations Chris!

Jun
on behalf of Apache Kafka PMC






--
-David

[jira] [Created] (KAFKA-14801) Encoded sensitive configs are not decoded before migration

2023-03-13 Thread Akhilesh Chaganti (Jira) 
Akhilesh Chaganti created KAFKA-14801:
-

 Summary: Encoded sensitive configs are not decoded before migration
 Key: KAFKA-14801
 URL: https://issues.apache.org/jira/browse/KAFKA-14801
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Reporter: Akhilesh Chaganti






--
This message was sent by Atlassian Jira
(v8.20.10#820010)