Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1290

2022-10-12 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 505688 lines...]
[2022-10-13T04:39:18.064Z] > Task :connect:api:testJar
[2022-10-13T04:39:18.064Z] > Task :connect:api:testSrcJar
[2022-10-13T04:39:18.064Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-10-13T04:39:18.064Z] > Task :connect:api:publishToMavenLocal
[2022-10-13T04:39:19.111Z] 
[2022-10-13T04:39:19.111Z] > Task :streams:javadoc
[2022-10-13T04:39:19.111Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/processor/StreamPartitioner.java:50:
 warning - Tag @link: reference not found: 
org.apache.kafka.clients.producer.internals.DefaultPartitioner
[2022-10-13T04:39:19.111Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:19.111Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:19.111Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:20.062Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-10-13T04:39:21.281Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-10-13T04:39:21.281Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-10-13T04:39:21.282Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-10-13T04:39:21.282Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-10-13T04:39:21.282Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-10-13T04:39:21.282Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2022-10-13T04:39:21.282Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-10-13T04:39:21.282Z]

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1289

2022-10-12 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 508071 lines...]
[2022-10-13T02:11:35.115Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-10-13T02:11:36.265Z] 
[2022-10-13T02:11:36.265Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-10-13T02:11:36.265Z] 
[2022-10-13T02:11:36.265Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-10-13T02:11:58.810Z] 
[2022-10-13T02:11:58.810Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-10-13T02:11:58.810Z] 
[2022-10-13T02:11:58.810Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > shouldAddStreamThread() 
STARTED
[2022-10-13T02:12:02.186Z] 
[2022-10-13T02:12:02.187Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > shouldAddStreamThread() PASSED
[2022-10-13T02:12:02.187Z] 
[2022-10-13T02:12:02.187Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-10-13T02:12:08.096Z] 
[2022-10-13T02:12:08.096Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-10-13T02:12:08.096Z] 
[2022-10-13T02:12:08.096Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > shouldRemoveStreamThread() 
STARTED
[2022-10-13T02:12:12.843Z] 
[2022-10-13T02:12:12.843Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > shouldRemoveStreamThread() 
PASSED
[2022-10-13T02:12:12.843Z] 
[2022-10-13T02:12:12.843Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() STARTED
[2022-10-13T02:12:14.798Z] 
[2022-10-13T02:12:14.798Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() PASSED
[2022-10-13T02:12:16.912Z] 
[2022-10-13T02:12:16.912Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 STARTED
[2022-10-13T02:12:17.932Z] 
[2022-10-13T02:12:17.932Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 PASSED
[2022-10-13T02:12:17.932Z] 
[2022-10-13T02:12:17.932Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() STARTED
[2022-10-13T02:12:17.932Z] 
[2022-10-13T02:12:17.932Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() PASSED
[2022-10-13T02:12:17.932Z] 
[2022-10-13T02:12:17.932Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingTopic() STARTED
[2022-10-13T02:12:17.932Z] 
[2022-10-13T02:12:17.932Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingTopic() PASSED
[2022-10-13T02:12:17.932Z] 
[2022-10-13T02:12:17.932Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 166 > FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithInvalidCommittedOffsets() STARTED
[2022-10-13T02:12:28.614Z] 
[2022-10-13T02:12:28.614Z] > Task :core:integrationTest
[2022-10-13T02:12:28.614Z] 
[2022-10-13T02:12:28.614Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 164 > EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards() PASSED
[2022-10-13T02:12:28.614Z] 
[2022-10-13T02:12:28.614Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 164 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow() STARTED
[2022-10-13T02:12:32.508Z] 
[2022-10-13T02:12:32.508Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 164 > EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow() PASSED

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

2022-10-12 Thread José Armando García Sancio 
On Wed, Oct 12, 2022 at 3:02 PM Niket Goel  wrote:
> 1. Do we need this value to be of the order of `ms`. Is it better off being
> tunable to a minute granularity?

Hmm. The most common unit for time intervals in Kafka is milliseconds.
Very rarely does Kafka express time intervals using another unit.
There are two examples where Kafka expresses time intervals in
seconds. I would argue that even those cases should have used
milliseconds as the time unit.

We could limit the range of possible values for this configuration but
it is not clear to me that Kafka should strictly enforce a minimum in
all cases. Except for enforcing that the interval should not be a
negative number.

> 2. Somewhat related to 1 - Do we need to make any tweaks to the way we
> cleanup the metadata directory (time/size based cleanup)? My concern is if
> the metadata-dir cleanup will trigger fast enough in all cases today. e.g.
> If I were to configure the time-based snapshotting to something like every
> 10 seconds, with enough load of course, will I end up flooding the disk.?

This seems to be an implementation issue given how
RaftMetadataLogCleanerManager deletes snapshots and log segments. I
could imagine a different implementation that doesn't have this
problem. I think we should avoid as much as possible having
implementation details leak to the configuration and public APIs.

-- 
-José

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

2022-10-12 Thread José Armando García Sancio 
On Wed, Oct 12, 2022 at 3:02 PM Colin McCabe  wrote:
> Given that we already have metadata.log.max.record.bytes.between.snapshots, 
> we need to define how the two properties interact. I would expect that the 
> time-based property would take effect only if the bytes-based property did 
> not trigger. In other words, if you are regularly generating snapshots every 
> hour because of your setting for 
> metadata.log.max.record.bytes.between.snapshots, setting a time-based 
> property for every 10 hours should have no effect. But if you would have only 
> generated a snapshot every week by your 
> metadata.log.max.record.bytes.between.snapshots setting, setting a time-based 
> property for every 10 hours should result in a snapshot every 10 hours. And 
> in that case the bytes-based property is effectively ignored.

Yes. This is exactly how I was planning to implement this. I was
debating if I should go into this level of detail in the KIP or to
leave this discussion to the PR review. Either way, I updated the KIP
to provide some guidance to the implementation.

> ... it suggests that the configuration name should include "max" somewhere to 
> indicate that it is a maximum only and not necessarily the duration value 
> that will result from the sum total of all configurations. So maybe 
> metadata.log.max.snapshot.interval.ms?

Sounds good to me. I updated the name of the property to
`metadata.log.max.snapshot.interval.ms`.

-- 
-José

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

2022-10-12 Thread José Armando García Sancio 
On Wed, Oct 12, 2022 at 1:27 AM David Jacot  wrote:
> I would name the
> new property `metadata.log.snapshot.interval.ms` as `between` is
> implied by the `interval`.

I agree. I updated the KIP to use your suggestions for naming the
property `metadata.log.snapshot.interval.ms`.

-- 
-José

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

2022-10-12 Thread Colin McCabe 
On Wed, Oct 5, 2022, at 10:06, Mickael Maison wrote:
> Hi David,
>
> Thanks for starting this important KIP.
>
> I've just taken a quick look so far but I've got a couple of initial 
> questions:
>
> 1) What happens if a non KRaft compatible broker (or with
> kafka.metadata.migration.enable set to false) joins the cluster after
> the migration is triggered?
>

Hi Mickael,

Yes, I also asked this :)

I think the answer is that it's an administrative error, and the broker will 
not be able to function. Maybe we should spell out that the broker should shut 
itself down in this scenario. Obviously this will not be possible for already 
released versions, but a broker running the new 3.4 code but with something 
that disqualifies it from upgrade-from-ZK (like the wrong IBP) could shut 
itself down if it knew it was in the wrong spot.

>
> 2) In the Failure Modes section you mention a scenario where a write
> to ZK fails. What happens when the divergence limit is reached? Is
> this a fatal condition? How much divergence should we allow?
>

Yes, this is one of the complexities of supporting async mirroring to ZK. I 
don't think it's quite as bad as it sounds because we still have the ordered 
__cluster_metadata log.

I would suggest that if our backlog grows too high, we just block the kraft 
controller thread until it reduces. We really cannot afford to have an 
unbounded backlog. I would expected the performance to be comparable to the 
current controller's performance since we are doing the same stuff (less stuff 
actually, since we aren't doing reads...)

best,
Colin


> Thanks,
> Mickael
>
> On Wed, Oct 5, 2022 at 12:20 AM David Arthur  wrote:
>>
>> Hey folks, I wanted to get the ball rolling on the discussion for the
>> ZooKeeper migration KIP. This KIP details how we plan to do an online
>> migration of metadata from ZooKeeper to KRaft as well as a rolling
>> upgrade of brokers to KRaft mode.
>>
>> The general idea is to keep KRaft and ZooKeeper in sync during the
>> migration, so both types of brokers can exist simultaneously. Then,
>> once everything is migrated and updated, we can turn off ZooKeeper
>> writes.
>>
>> This is a pretty complex KIP, so please take a look :)
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration
>>
>> Thanks!
>> David

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

2022-10-12 Thread Colin McCabe 
Thanks for putting this together, David. A few questions (maybe some of them 
overlap with Jun's?)

1. Rather than having MigrationCheckRequest, can we just add a new JSON field 
to the broker registration like "kraftReady": true? After all, we are already 
going to have to read the broker registration from ZK as we discussed earlier 
(and as the KIP mentions).

2. Do we really need kafka.metadata.migration.enable? It's not clear to me what 
harm would result if we just left it out of the design. Presumably standing up 
a new KRaft controller quorum and pointing zk.connect at the old ZK cluster is 
a very intentional action without much chance that we did it accidentally.

3. We should probably be explicit that the new metadata.version the KIP 
mentions corresponds to an inter.broker.protocol of 3.4 or later (since we're 
targetting 3.4) and that using IBP 3.4 means ALWAYS forwarding. I realize 
that's implicit if you read some of the earlier KIPs, but it would be good to 
make it clear. Also presumably the controller will copy its statically 
configured value of inter.broker.protocol into the metadata.version used in the 
new log, so it all matches up. Again, probably clear to us but good to spell 
out.

4. Migration state names: should "ZooKeeper" be renamed to 
"MigrationIneligible"? Since it does mean you can't upgrade (in effect)

5. If we get a new ineligible broker registering in ZK after the migration 
begins, I assume we will just have to ignore it and keep going. This would be 
an operator error, of course, and this broker will not be able to participate 
in the cluster. It would be good to spell out that behavior a bit.

best,
Colin


On Tue, Oct 4, 2022, at 15:19, David Arthur wrote:
> Hey folks, I wanted to get the ball rolling on the discussion for the
> ZooKeeper migration KIP. This KIP details how we plan to do an online
> migration of metadata from ZooKeeper to KRaft as well as a rolling
> upgrade of brokers to KRaft mode.
>
> The general idea is to keep KRaft and ZooKeeper in sync during the
> migration, so both types of brokers can exist simultaneously. Then,
> once everything is migrated and updated, we can turn off ZooKeeper
> writes.
>
> This is a pretty complex KIP, so please take a look :)
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration
>
> Thanks!
> David

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

2022-10-12 Thread Colin McCabe 
Hi Jun,

Thanks for taking a look. I can answer some questions here because I 
collaborated on this a bit, and David is on vacation for a few days.

On Wed, Oct 12, 2022, at 14:41, Jun Rao wrote:
> Hi, David,
>
> Thanks for the KIP. A few comments below.
>
> 10. It's still not very clear to me how the KRaft controller works in the
> dual writes mode to KRaft log and ZK when the brokers still run in ZK mode.
> Does the KRaft controller run a ZK based controller in parallel or do we
> derive what needs to be written to ZK based on KRaft controller logic?

We derive what needs to be written to ZK based on KRaft controller logic.

> I am also not sure how the KRaft controller handles broker
> registration/deregistration, since brokers are still running in ZK mode and
> are not heartbeating to the KRaft controller.

The new controller will listen for broker registrations under /brokers. This is 
the only znode watch that the new controller will do.

We did consider changing how ZK-based broker registration worked, but it just 
ended up being too much work for not enough gain.

>
> 12. "A new set of nodes will be provisioned to host the controller quorum."
> I guess we don't support starting the KRaft controller quorum on existing
> brokers. It would be useful to make that clear.
>

Agreed

> 13. "Once the quorum is established and a leader is elected, the controller
> will check the state of the cluster using the MigrationCheck RPC." How does
> the quorum controller detect other brokers? Does the controller node need
> to be configured with ZK connection string? If so, it would be useful to
> document the additional configs that the quorum controller needs to set.
>

Yes, the controllers monitor ZK for broker registrations, as I mentioned above. 
So they need zk.connect and the other ZK connection configurations.

> 14. "In order to prevent further writes to ZK, the first thing the new
> KRaft quorum must do is take over leadership of the ZK controller. " The ZK
> controller processing changes to /controller update asynchronously. How
> does the KRaft controller know when the ZK controller has resigned before
> it can safely copy the ZK data?
>

This should be done through expectedControllerEpochZkVersion, just like in ZK 
mode, right? We should bump this epoch value so that any writes from the old 
controller will not go through. I agree we should spell this out in the KIP.

> 15. We have the following sentences. One says ControllerId is a random
> KRaft broker and the other says it's the active controller. Which one is
> correct?
> "UpdateMetadata: for certain metadata changes, the KRaft controller will
> need to send UpdateMetadataRequests to the ZK brokers. For the
> “ControllerId” field in this request, the controller should specify a
> random KRaft broker."
> "In the UpdateMetadataRequest sent by the KRaft controller to the ZK
> brokers, the ControllerId will point to the active controller which will be
> used for the inter-broker requests."
>

Yeah, this seems like an error to me as well. A random value is not really 
useful. Plus the text here is self-contradictory, as you pointed out.

I suspect what we should do here is add a new field, KRaftControllerId, and 
populate it with the real controller ID, and leave the old controllerId field 
as -1. A ZK-based broker that sees this can then consult its 
controller.quorum.voters configuration to see where it should send 
controller-bound RPCs. That (static) configuration lets us map between 
controller ID and host:port.

We should still keep our existing epoch logic for deciding when 
UpdateMetadataRequest / LeaderAndIsrRequests are stale, with the caveat that 
any kraft-based epoch should be treated as greater than any ZK-based epoch. 
After all, the kraft epoch is coming from the epoch of __cluster_metadata, 
whereas the ZK epoch comes from ZK.

>
> 16. "Additionally, the controller must specify if a broker in “LiveBrokers”
> is KRaft or ZK." Does that require any protocol changes to UpdateMetadata?
>

Yeah, I am also curious why the we need to care whether brokers are ZK or KRaft 
in UpdateMetadataRequest. We don't reveal this to clients, so can we just leave 
this out?

best,
Colin

> Thanks,
>
> Jun
>
> On Wed, Oct 5, 2022 at 10:07 AM Mickael Maison 
> wrote:
>
>> Hi David,
>>
>> Thanks for starting this important KIP.
>>
>> I've just taken a quick look so far but I've got a couple of initial
>> questions:
>>
>> 1) What happens if a non KRaft compatible broker (or with
>> kafka.metadata.migration.enable set to false) joins the cluster after
>> the migration is triggered?
>>
>> 2) In the Failure Modes section you mention a scenario where a write
>> to ZK fails. What happens when the divergence limit is reached? Is
>> this a fatal condition? How much divergence should we allow?
>>
>> Thanks,
>> Mickael
>>
>> On Wed, Oct 5, 2022 at 12:20 AM David Arthur  wrote:
>> >
>> > Hey folks, I wanted to get the ball rolling on the discussion for the
>> > ZooKeeper

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

2022-10-12 Thread Niket Goel 
Thanks for the KIP Jose! Adding this ability makes sense to me. Just a few
quick wonderments:
1. Do we need this value to be of the order of `ms`. Is it better off being
tunable to a minute granularity?
2. Somewhat related to 1 - Do we need to make any tweaks to the way we
cleanup the metadata directory (time/size based cleanup)? My concern is if
the metadata-dir cleanup will trigger fast enough in all cases today. e.g.
If I were to configure the time-based snapshotting to something like every
10 seconds, with enough load of course, will I end up flooding the disk.?

On Wed, Oct 12, 2022 at 1:28 AM David Jacot 
wrote:

> Hi José,
>
> Thanks for the KIP. That makes total sense. On nit, I would name the
> new property `metadata.log.snapshot.interval.ms` as `between` is
> implied by the `interval`.
>
> Best,
> David
>
> On Tue, Oct 11, 2022 at 9:16 PM José Armando García Sancio
>  wrote:
> >
> > Hey all,
> >
> > I am interested in allowing brokers and controllers in KRaft to
> > generate snapshots for the cluster metadata partition on a timely
> > basis. This would better allow Kafka users to use cluster metadata
> > snapshots as a solution for backing up the cluster's metadata.
> >
> > Let's use this thread to discuss KIP-876:
> > https://cwiki.apache.org/confluence/x/MY3GDQ
> >
> > Thanks!
> > --
> > -José
>


-- 
- Niket

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

2022-10-12 Thread Colin McCabe 
Thanks for the KIP, José.

Given that we already have metadata.log.max.record.bytes.between.snapshots, we 
need to define how the two properties interact. I would expect that the 
time-based property would take effect only if the bytes-based property did not 
trigger. In other words, if you are regularly generating snapshots every hour 
because of your setting for metadata.log.max.record.bytes.between.snapshots, 
setting a time-based property for every 10 hours should have no effect. But if 
you would have only generated a snapshot every week by your 
metadata.log.max.record.bytes.between.snapshots setting, setting a time-based 
property for every 10 hours should result in a snapshot every 10 hours. And in 
that case the bytes-based property is effectively ignored.

I think we should clarify this in the KIP to avoid confusion. Also, it suggests 
that the configuration name should include "max" somewhere to indicate that it 
is a maximum only and not necessarily the duration value that will result from 
the sum total of all configurations. So maybe 
metadata.log.max.snapshot.interval.ms?

best,
Colin

On Tue, Oct 11, 2022, at 12:15, José Armando García Sancio wrote:
> Hey all,
>
> I am interested in allowing brokers and controllers in KRaft to
> generate snapshots for the cluster metadata partition on a timely
> basis. This would better allow Kafka users to use cluster metadata
> snapshots as a solution for backing up the cluster's metadata.
>
> Let's use this thread to discuss KIP-876:
> https://cwiki.apache.org/confluence/x/MY3GDQ
>
> Thanks!
> -- 
> -José

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

2022-10-12 Thread Jun Rao 
Hi, David,

Thanks for the KIP. A few comments below.

10. It's still not very clear to me how the KRaft controller works in the
dual writes mode to KRaft log and ZK when the brokers still run in ZK mode.
Does the KRaft controller run a ZK based controller in parallel or do we
derive what needs to be written to ZK based on KRaft controller logic? I am
also not sure how the KRaft controller handles broker
registration/deregistration, since brokers are still running in ZK mode and
are not heartbeating to the KRaft controller.

11. When preparing the Cluster, each broker needs
kafka.metadata.migration.enable set to “true”, right? Could we document
that?

12. "A new set of nodes will be provisioned to host the controller quorum."
I guess we don't support starting the KRaft controller quorum on existing
brokers. It would be useful to make that clear.

13. "Once the quorum is established and a leader is elected, the controller
will check the state of the cluster using the MigrationCheck RPC." How does
the quorum controller detect other brokers? Does the controller node need
to be configured with ZK connection string? If so, it would be useful to
document the additional configs that the quorum controller needs to set.

14. "In order to prevent further writes to ZK, the first thing the new
KRaft quorum must do is take over leadership of the ZK controller. " The ZK
controller processing changes to /controller update asynchronously. How
does the KRaft controller know when the ZK controller has resigned before
it can safely copy the ZK data?

15. We have the following sentences. One says ControllerId is a random
KRaft broker and the other says it's the active controller. Which one is
correct?
"UpdateMetadata: for certain metadata changes, the KRaft controller will
need to send UpdateMetadataRequests to the ZK brokers. For the
“ControllerId” field in this request, the controller should specify a
random KRaft broker."
"In the UpdateMetadataRequest sent by the KRaft controller to the ZK
brokers, the ControllerId will point to the active controller which will be
used for the inter-broker requests."

16. "Additionally, the controller must specify if a broker in “LiveBrokers”
is KRaft or ZK." Does that require any protocol changes to UpdateMetadata?

Thanks,

Jun

On Wed, Oct 5, 2022 at 10:07 AM Mickael Maison 
wrote:

> Hi David,
>
> Thanks for starting this important KIP.
>
> I've just taken a quick look so far but I've got a couple of initial
> questions:
>
> 1) What happens if a non KRaft compatible broker (or with
> kafka.metadata.migration.enable set to false) joins the cluster after
> the migration is triggered?
>
> 2) In the Failure Modes section you mention a scenario where a write
> to ZK fails. What happens when the divergence limit is reached? Is
> this a fatal condition? How much divergence should we allow?
>
> Thanks,
> Mickael
>
> On Wed, Oct 5, 2022 at 12:20 AM David Arthur  wrote:
> >
> > Hey folks, I wanted to get the ball rolling on the discussion for the
> > ZooKeeper migration KIP. This KIP details how we plan to do an online
> > migration of metadata from ZooKeeper to KRaft as well as a rolling
> > upgrade of brokers to KRaft mode.
> >
> > The general idea is to keep KRaft and ZooKeeper in sync during the
> > migration, so both types of brokers can exist simultaneously. Then,
> > once everything is migrated and updated, we can turn off ZooKeeper
> > writes.
> >
> > This is a pretty complex KIP, so please take a look :)
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration
> >
> > Thanks!
> > David
>

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Nick Telford 
Can't wait!
+1 (non-binding)

On Wed, 12 Oct 2022, 18:02 Guozhang Wang, 
wrote:

> Hello all,
>
> I'd like to start a vote for the following KIP, aiming to improve Kafka
> Stream's restoration visibility via new metrics and callback methods:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility
>
>
> Thanks!
> -- Guozhang
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1288

2022-10-12 Thread Apache Jenkins Server 
See

[jira] [Created] (KAFKA-14295) FetchMessageConversionsPerSec meter not recorded

2022-10-12 Thread David Mao (Jira) 
David Mao created KAFKA-14295:
-

 Summary: FetchMessageConversionsPerSec meter not recorded
 Key: KAFKA-14295
 URL: https://issues.apache.org/jira/browse/KAFKA-14295
 Project: Kafka
  Issue Type: Bug
Reporter: David Mao


The broker topic metric FetchMessageConversionsPerSec doesn't get recorded on a 
fetch message conversion.

The bug is that we pass in a callback that expects a MultiRecordsSend in 
KafkaApis:
{code:java}
def updateConversionStats(send: Send): Unit = {
  send match {
case send: MultiRecordsSend if send.recordConversionStats != null =>
  send.recordConversionStats.asScala.toMap.foreach {
case (tp, stats) => updateRecordConversionStats(request, tp, stats)
  }
case _ =>
  }
} {code}
But we call this callback with a NetworkSend in the SocketServer:
{code:java}
selector.completedSends.forEach { send =>
  try {
val response = inflightResponses.remove(send.destinationId).getOrElse {
  throw new IllegalStateException(s"Send for ${send.destinationId} 
completed, but not in `inflightResponses`")
}
updateRequestMetrics(response)

// Invoke send completion callback
response.onComplete.foreach(onComplete => onComplete(send))
...{code}
Note that Selector.completedSends returns a collection of NetworkSend



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build became unstable: Kafka » Kafka Branch Builder » trunk #1287

2022-10-12 Thread Apache Jenkins Server 
See

[VOTE] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Guozhang Wang 
Hello all,

I'd like to start a vote for the following KIP, aiming to improve Kafka
Stream's restoration visibility via new metrics and callback methods:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility


Thanks!
-- Guozhang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Guozhang Wang 
I've updated the KIP doc and would start calling for a vote for this KIP
now.

On Tue, Oct 11, 2022 at 4:26 AM Nick Telford  wrote:

> Hi Guozhang,
>
> What you propose sounds good to me. Having the more detailed Task-level
> metrics at DEBUG makes sense.
>
> Regards,
>
> Nick
>


-- 
-- Guozhang

Jenkins build is back to normal : Kafka » Kafka Branch Builder » trunk #1286

2022-10-12 Thread Apache Jenkins Server 
See

[jira] [Created] (KAFKA-14294) Kafka Streams should commit transaction even no records are processed

2022-10-12 Thread Vicky Papavasileiou (Jira) 
Vicky Papavasileiou created KAFKA-14294:
---

 Summary: Kafka Streams should commit transaction even no records 
are processed
 Key: KAFKA-14294
 URL: https://issues.apache.org/jira/browse/KAFKA-14294
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Vicky Papavasileiou


Currently, if there are no records to process in the input topic, a transaction 
does not commit. If a custom punctuator code is writing to a state store (which 
is common practice) the producer gets fenced when trying to write to the 
changelog topic. This throws a TaskMigratedException and causes a rebalance. 

A better approach would be to commit a transaction even when there are no 
records processed as to allow the punctuator to make progress. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-10-12 Thread Chris Egerton 
+1 (binding)
Thanks ShunKang!

On Tue, Oct 11, 2022 at 9:26 PM Luke Chen  wrote:

> +1 from me.
> Thanks for the KIP.
>
> Luke
>
> On Fri, Sep 23, 2022 at 1:50 AM Guozhang Wang  wrote:
>
> > +1, thanks ShunKang.
> >
> > Though its proposed motivation is on consumer fetcher's deserialization,
> I
> > think adding an overloaded method with ByteBuffer would help with other
> > serde places on the client side as well.
> >
> >
> > Guozhang
> >
> > On Thu, Sep 22, 2022 at 9:41 AM ShunKang Lin 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to open the vote for KIP-863, which proposes to reduce memory
> > > allocation and memory copying in Fetcher#parseRecord(TopicPartition,
> > > RecordBatch, Record).
> > >
> > > The proposal is here:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > >
> > > Thanks to all who reviewed the proposal, and thanks in advance for
> taking
> > > the time to vote!
> > >
> > > Best,
> > > ShunKang
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-10-12 Thread Chris Egerton 
Thanks ShunKang, that's a good point about ByteBuffer::hasArray. LGTM!

On Wed, Oct 12, 2022 at 11:12 AM ShunKang Lin 
wrote:

> Hi Chris,
>
> 1. Record keys/values are duplicated from `DefaultRecordBatch#buffer`, so
> modifying key/value offsets will not modify the original ByteBuffer
> offsets. A read-only ByteBuffer calls `ByteBuffer#hasArray()` to return
> false, which means that a read-only ByteBuffer does not expose the
> underlying array, which is safer but slower when using the ByteBuffer API.
>
> 2. Good idea, I modified the KIP compatibility section, please take a look.
>
> Best,
> ShunKang
>
> Chris Egerton  于2022年10月11日周二 23:59写道：
>
> > Hi ShunKang,
> >
> > Thanks for the KIP! I have a couple thoughts:
> >
> > 1. If we pass the ByteBuffer that we're using internally for the record
> > key/value to the deserializer, it may be mutated by writing new bytes or
> > altering the position. Should we permit this, or would it make sense to
> > provide deserializers with a read-only ByteBuffer? [1]
> >
> > 2. The compatibility section should probably be fleshed out a bit further
> > to state that deserializers that wish to be compatible with older
> versions
> > of the Kafka clients library should always implement the byte array-based
> > deserialize method. We should probably also add this information to the
> > Javadocs for the new method, although this can be taken care of during PR
> > review and doesn't have to be included in the KIP itself.
> >
> > Cheers,
> >
> > Chris
> >
> > [1] -
> >
> >
> https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html#asReadOnlyBuffer--
> >
> > On Tue, Oct 11, 2022 at 8:36 AM Luke Chen  wrote:
> >
> > > Hi ShunKang,
> > >
> > > Had a quick look, I think It's a good idea.
> > > I'll check it again tomorrow, and let you know if I have any questions.
> > >
> > > Luke
> > >
> > > On Sun, Sep 25, 2022 at 3:35 PM ShunKang Lin <
> linshunkang@gmail.com>
> > > wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > When I try add method `default ByteBuffer
> serializeToByteBuffer(String
> > > > topic, Headers headers, T data)` for `ByteBufferSerializer`, I found
> > > > `ByteBufferSerializer#serialize(String, ByteBuffer)` is not correct.
> > > > Then I searched JIRA and found this:
> > > > https://issues.apache.org/jira/browse/KAFKA-4852, I made a comment
> > below
> > > > this JIRA, PTAL.
> > > >
> > > > Best,
> > > > ShunKang
> > > >
> > > > Guozhang Wang  于2022年9月20日周二 06:33写道：
> > > >
> > > > > A separate question regarding the proposed API as well: what do you
> > > think
> > > > > to also augment the serializers with ByteBuffer return type in
> order
> > to
> > > > be
> > > > > symmetric with deserializers?
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 19, 2022 at 3:32 PM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > Hello ShunKang,
> > > > > >
> > > > > > Thanks for filing the proposal, and sorry for the late reply!
> > > > > >
> > > > > > I looked over your KIP proposal and the PR, in general I think I
> > > agree
> > > > > > that adding an overloaded function with `ByteBuffer` param is
> > > > beneficial,
> > > > > > but I have a meta question regarding it's impact on Kafka
> consumer:
> > > my
> > > > > > understanding from your PR is that, we can only save memory
> > > allocations
> > > > > if
> > > > > > the key/value types happen to be ByteBuffer as well, otherwise we
> > > would
> > > > > > still do the `return deserialize(topic, headers,
> > > Utils.toArray(data));`
> > > > > > from default impls unless the user customized deserializers is
> > > > augmented
> > > > > to
> > > > > > handle ByteBuffer directly, right?
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > > > linshunkang@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi all，
> > > > > >>
> > > > > >> I'd like to start a discussion on KIP-863 which is Reduce
> > > > > >> Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> > > Consumer
> > > > > >> memory allocation by nearly 50% during fetch records.
> > > > > >>
> > > > > >> Please check
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > > > >> and https://github.com/apache/kafka/pull/12545 for more
> details.
> > > > > >>
> > > > > >> Any feedbacks and comments are welcomed.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-10-12 Thread ShunKang Lin 
Hi Chris,

1. Record keys/values are duplicated from `DefaultRecordBatch#buffer`, so
modifying key/value offsets will not modify the original ByteBuffer
offsets. A read-only ByteBuffer calls `ByteBuffer#hasArray()` to return
false, which means that a read-only ByteBuffer does not expose the
underlying array, which is safer but slower when using the ByteBuffer API.

2. Good idea, I modified the KIP compatibility section, please take a look.

Best,
ShunKang

Chris Egerton  于2022年10月11日周二 23:59写道：

> Hi ShunKang,
>
> Thanks for the KIP! I have a couple thoughts:
>
> 1. If we pass the ByteBuffer that we're using internally for the record
> key/value to the deserializer, it may be mutated by writing new bytes or
> altering the position. Should we permit this, or would it make sense to
> provide deserializers with a read-only ByteBuffer? [1]
>
> 2. The compatibility section should probably be fleshed out a bit further
> to state that deserializers that wish to be compatible with older versions
> of the Kafka clients library should always implement the byte array-based
> deserialize method. We should probably also add this information to the
> Javadocs for the new method, although this can be taken care of during PR
> review and doesn't have to be included in the KIP itself.
>
> Cheers,
>
> Chris
>
> [1] -
>
> https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html#asReadOnlyBuffer--
>
> On Tue, Oct 11, 2022 at 8:36 AM Luke Chen  wrote:
>
> > Hi ShunKang,
> >
> > Had a quick look, I think It's a good idea.
> > I'll check it again tomorrow, and let you know if I have any questions.
> >
> > Luke
> >
> > On Sun, Sep 25, 2022 at 3:35 PM ShunKang Lin 
> > wrote:
> >
> > > Hi Guozhang,
> > >
> > > When I try add method `default ByteBuffer serializeToByteBuffer(String
> > > topic, Headers headers, T data)` for `ByteBufferSerializer`, I found
> > > `ByteBufferSerializer#serialize(String, ByteBuffer)` is not correct.
> > > Then I searched JIRA and found this:
> > > https://issues.apache.org/jira/browse/KAFKA-4852, I made a comment
> below
> > > this JIRA, PTAL.
> > >
> > > Best,
> > > ShunKang
> > >
> > > Guozhang Wang  于2022年9月20日周二 06:33写道：
> > >
> > > > A separate question regarding the proposed API as well: what do you
> > think
> > > > to also augment the serializers with ByteBuffer return type in order
> to
> > > be
> > > > symmetric with deserializers?
> > > >
> > > >
> > > >
> > > > On Mon, Sep 19, 2022 at 3:32 PM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Hello ShunKang,
> > > > >
> > > > > Thanks for filing the proposal, and sorry for the late reply!
> > > > >
> > > > > I looked over your KIP proposal and the PR, in general I think I
> > agree
> > > > > that adding an overloaded function with `ByteBuffer` param is
> > > beneficial,
> > > > > but I have a meta question regarding it's impact on Kafka consumer:
> > my
> > > > > understanding from your PR is that, we can only save memory
> > allocations
> > > > if
> > > > > the key/value types happen to be ByteBuffer as well, otherwise we
> > would
> > > > > still do the `return deserialize(topic, headers,
> > Utils.toArray(data));`
> > > > > from default impls unless the user customized deserializers is
> > > augmented
> > > > to
> > > > > handle ByteBuffer directly, right?
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > > linshunkang@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi all，
> > > > >>
> > > > >> I'd like to start a discussion on KIP-863 which is Reduce
> > > > >> Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> > Consumer
> > > > >> memory allocation by nearly 50% during fetch records.
> > > > >>
> > > > >> Please check
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > > >> and https://github.com/apache/kafka/pull/12545 for more details.
> > > > >>
> > > > >> Any feedbacks and comments are welcomed.
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>

[jira] [Resolved] (KAFKA-14099) No REST API request logs in Kafka connect

2022-10-12 Thread Chris Egerton (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14099.
---
Fix Version/s: 3.4.0
 Reviewer: Chris Egerton
   Resolution: Fixed

> No REST API request logs in Kafka connect
> -
>
> Key: KAFKA-14099
> URL: https://issues.apache.org/jira/browse/KAFKA-14099
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 3.2.0
>Reporter: Alexandre Garnier
>Assignee: Alexandre Garnier
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Prior to 2.2.1, when an REST API request was performed, there was a request 
> log in the log file:
> {code:java}
> [2022-07-23 07:18:16,128] INFO 172.18.0.1 - - [23/Jul/2022:07:18:16 +] 
> "GET /connectors HTTP/1.1" 200 2 "-" "curl/7.81.0" 66 
> (org.apache.kafka.connect.runtime.rest.RestServer:62)
> {code}
> Since 2.2.1, no more request logs.
>  
> With a bisect, I found the problem comes from [PR 
> 6651|https://github.com/apache/kafka/pull/6651] to fix KAFKA-8304
> From what I understand of the problem, the ContextHandlerCollection is added 
> in the Server 
> ([https://github.com/dongjinleekr/kafka/blob/63a6130af30536d67fca5802005695a84c875b5e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java#L195])
>  before handlers are really added in the ContextHandlerCollection 
> ([https://github.com/dongjinleekr/kafka/blob/63a6130af30536d67fca5802005695a84c875b5e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java#L296]).
> I don't know the impact on other handlers, but clearly it doesn't work for 
> the RequestLogHandler.
>  
> A solution I found for the logging issue is to set the RequestLog directly in 
> the server without using an handlers:
> {code:java}
> diff --git 
> i/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
>  
> w/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
> index ab18419efc..4d09cc0e6c 100644
> --- 
> i/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
> +++ 
> w/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java
> @@ -187,6 +187,11 @@ public class RestServer {
>  public void initializeServer() {
>  log.info("Initializing REST server");
>  
> +Slf4jRequestLogWriter slf4jRequestLogWriter = new 
> Slf4jRequestLogWriter();
> +
> slf4jRequestLogWriter.setLoggerName(RestServer.class.getCanonicalName());
> +CustomRequestLog requestLog = new 
> CustomRequestLog(slf4jRequestLogWriter, CustomRequestLog.EXTENDED_NCSA_FORMAT 
> + " %{ms}T");
> +jettyServer.setRequestLog(requestLog);
> +
>  /* Needed for graceful shutdown as per `setStopTimeout` 
> documentation */
>  StatisticsHandler statsHandler = new StatisticsHandler();
>  statsHandler.setHandler(handlers);
> @@ -275,14 +280,7 @@ public class RestServer {
>  configureHttpResponsHeaderFilter(context);
>  }
>  
> -RequestLogHandler requestLogHandler = new RequestLogHandler();
> -Slf4jRequestLogWriter slf4jRequestLogWriter = new 
> Slf4jRequestLogWriter();
> -
> slf4jRequestLogWriter.setLoggerName(RestServer.class.getCanonicalName());
> -CustomRequestLog requestLog = new 
> CustomRequestLog(slf4jRequestLogWriter, CustomRequestLog.EXTENDED_NCSA_FORMAT 
> + " %{ms}T");
> -requestLogHandler.setRequestLog(requestLog);
> -
>  contextHandlers.add(new DefaultHandler());
> -contextHandlers.add(requestLogHandler);
>  
>  handlers.setHandlers(contextHandlers.toArray(new Handler[0]));
>  try {
> {code}
> Same issue raised on StackOverflow: 
> [https://stackoverflow.com/questions/67699702/no-rest-api-logs-in-kafka-connect]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request contributor access

2022-10-12 Thread Chris Egerton 
Hi Patrik,

You should be good to go.

Cheers,

Chris

On Wed, Oct 12, 2022 at 10:20 AM Patrik Marton 
wrote:

> Hi,
>
> I would like to request contributor access to the project, to be able to
> assign tickets to me.
>
> My apache jira id is pmarton
>
> Thank you!
> Patrik
>

Request contributor access

2022-10-12 Thread Patrik Marton 
Hi,

I would like to request contributor access to the project, to be able to
assign tickets to me.

My apache jira id is pmarton

Thank you!
Patrik

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-10-12 Thread ShunKang Lin 
Hi Luke,

Thanks for your comments.

Best,
ShunKang

Luke Chen  于2022年10月11日周二 20:36写道：

> Hi ShunKang,
>
> Had a quick look, I think It's a good idea.
> I'll check it again tomorrow, and let you know if I have any questions.
>
> Luke
>
> On Sun, Sep 25, 2022 at 3:35 PM ShunKang Lin 
> wrote:
>
> > Hi Guozhang,
> >
> > When I try add method `default ByteBuffer serializeToByteBuffer(String
> > topic, Headers headers, T data)` for `ByteBufferSerializer`, I found
> > `ByteBufferSerializer#serialize(String, ByteBuffer)` is not correct.
> > Then I searched JIRA and found this:
> > https://issues.apache.org/jira/browse/KAFKA-4852, I made a comment below
> > this JIRA, PTAL.
> >
> > Best,
> > ShunKang
> >
> > Guozhang Wang  于2022年9月20日周二 06:33写道：
> >
> > > A separate question regarding the proposed API as well: what do you
> think
> > > to also augment the serializers with ByteBuffer return type in order to
> > be
> > > symmetric with deserializers?
> > >
> > >
> > >
> > > On Mon, Sep 19, 2022 at 3:32 PM Guozhang Wang 
> > wrote:
> > >
> > > > Hello ShunKang,
> > > >
> > > > Thanks for filing the proposal, and sorry for the late reply!
> > > >
> > > > I looked over your KIP proposal and the PR, in general I think I
> agree
> > > > that adding an overloaded function with `ByteBuffer` param is
> > beneficial,
> > > > but I have a meta question regarding it's impact on Kafka consumer:
> my
> > > > understanding from your PR is that, we can only save memory
> allocations
> > > if
> > > > the key/value types happen to be ByteBuffer as well, otherwise we
> would
> > > > still do the `return deserialize(topic, headers,
> Utils.toArray(data));`
> > > > from default impls unless the user customized deserializers is
> > augmented
> > > to
> > > > handle ByteBuffer directly, right?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > linshunkang@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi all，
> > > >>
> > > >> I'd like to start a discussion on KIP-863 which is Reduce
> > > >> Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> Consumer
> > > >> memory allocation by nearly 50% during fetch records.
> > > >>
> > > >> Please check
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > >> and https://github.com/apache/kafka/pull/12545 for more details.
> > > >>
> > > >> Any feedbacks and comments are welcomed.
> > > >>
> > > >> Thanks.
> > > >>
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

[jira] [Created] (KAFKA-14293) Basic Auth filter should set the SecurityContext after a successful login

2022-10-12 Thread Jira 
Patrik Márton created KAFKA-14293:
-

 Summary: Basic Auth filter should set the SecurityContext after a 
successful login
 Key: KAFKA-14293
 URL: https://issues.apache.org/jira/browse/KAFKA-14293
 Project: Kafka
  Issue Type: Improvement
Reporter: Patrik Márton


Currently, the JaasBasicAuthFilter does not set the security context of the 
request after a successful login. However, this information of an authenticated 
user might be required for further processing, for example to perform 
authorization checks after the authentication.

> The filter should be extended to add the Security Context after a successful 
> login.

Another improvement would be to assign the right Priority to the filter. The 
current implementation uses the default priority, which is Priorities.USER = 
5000. This is a lower priority than for example AUTHORIZATION, which means that 
the basic auth filter would run after authorization filters.

> Assing the correct Priorities.AUTHENTICATION = 1000 priority to the filter 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

2022-10-12 Thread David Jacot 
Hi José,

Thanks for the KIP. That makes total sense. On nit, I would name the
new property `metadata.log.snapshot.interval.ms` as `between` is
implied by the `interval`.

Best,
David

On Tue, Oct 11, 2022 at 9:16 PM José Armando García Sancio
 wrote:
>
> Hey all,
>
> I am interested in allowing brokers and controllers in KRaft to
> generate snapshots for the cluster metadata partition on a timely
> basis. This would better allow Kafka users to use cluster metadata
> snapshots as a solution for backing up the cluster's metadata.
>
> Let's use this thread to discuss KIP-876:
> https://cwiki.apache.org/confluence/x/MY3GDQ
>
> Thanks!
> --
> -José