[jira] [Created] (KAFKA-14131) KafkaBasedLog#readToLogEnd() may accciedently falls into infinite loop

[Justinwins (Jira)]

Justinwins created KAFKA-14131:
--

 Summary: KafkaBasedLog#readToLogEnd() may accciedently falls into 
infinite loop
 Key: KAFKA-14131
 URL: https://issues.apache.org/jira/browse/KAFKA-14131
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Reporter: Justinwins


When a herder starts ,its KafkaOffsetBackingStore will readToLogEnd() by

DistributedHerder.herderExecutor of name "Distrubuted-connect-"  thread , e.g . 
Distrubuted-connect-28-1 , which may consume  a few minutes.

If another thread tries to shut down this herder , it will block for 
"task.shutdown.graceful.timeout.ms ' before  the 
DistributedHerder.herderExecutor is interrupted. 

And if thread in DistributedHerder.herderExecutor is interupted, 
KafkaOffsetBackingStore.readToLogEnd() will  poll(Integer.MAX_VALUE) and log "

Error polling" as  the the read  has been interrupted, then "consumer.position" 
will not advance, readToLogEnd() falls into infinite loop.

 

```

private void readToLogEnd() {
Set assignment = consumer.assignment();
Map endOffsets = readEndOffsets(assignment);
log.trace("Reading to end of log offsets {}", endOffsets);

while (!endOffsets.isEmpty()) { // this loop will never jump out 
Iterator> it = endOffsets.entrySet().iterator();
while (it.hasNext()) {
Map.Entry entry = it.next();
TopicPartition topicPartition = entry.getKey();
long endOffset = entry.getValue();
long lastConsumedOffset = consumer.position(topicPartition); // when thread was 
in interupted status ,consumer.position will not advance
if (lastConsumedOffset >= endOffset) {
log.trace("Read to end offset {} for {}", endOffset, topicPartition);
it.remove();
} else {
log.trace("Behind end offset {} for {}; last-read offset is {}",
endOffset, topicPartition, lastConsumedOffset);
poll(Integer.MAX_VALUE); // here , poll() will catch InterruptedException and 
log it without throwing it up
break;
}
}
}
}

```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14130) Reduce RackAwarenessIntegrationTest to a unit test

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-14130:
-

 Summary: Reduce RackAwarenessIntegrationTest to a unit test
 Key: KAFKA-14130
 URL: https://issues.apache.org/jira/browse/KAFKA-14130
 Project: Kafka
  Issue Type: Improvement
  Components: streams, unit tests
Reporter: Guozhang Wang


While working on KAFKA-13877, I feel it's an overkill to introduce the whole 
test class as an integration test, since all we need is to just test the 
assignor itself which could be a unit test. Running this suite with 9+ 
instances takes long time and is still vulnerable to all kinds of timing based 
flakiness. A better choice is to reduce it as a unit test, similar to 
{{HighAvailabilityStreamsPartitionAssignorTest}} that just test the behavior of 
the assignor itself, rather than creating many instances hence depend on 
various timing bombs to not explode.

The scope of this ticket is to refactor the {{RackAwarenessIntegrationTest}} 
into a {{RackAwarenessStreamsPartitionAssignorTest}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Luke Chen]

Congrats Sophie! :)

Luke

On Tue, Aug 2, 2022 at 7:56 AM Adam Bellemare 
wrote:

> Congratulations Sophie! I’m glad to see you made as a PMC member! Well
> earned.
>
> > On Aug 1, 2022, at 6:42 PM, Guozhang Wang  wrote:
> >
> > Hi everyone,
> >
> > I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> > committer since Oct. 2020 and has been contributing to the community
> > consistently, especially around Kafka Streams and Kafka java consumer.
> She
> > has also presented about Kafka Streams at Kafka Summit London this year.
> It
> > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> >
> > Congratulations, Sophie!
> >
> > -- Guozhang Wang, on behalf of Apache Kafka PMC
>

Re: [VOTE] KIP-859: Add Metadata Log Processing Error Related Metrics

[David Arthur]

Thanks, Niket.

+1 binding from me

-David

On Mon, Aug 1, 2022 at 8:15 PM Niket Goel  wrote:
>
> Hi all,
>
> I would like to start a vote on KIP-859 which adds some new metrics to KRaft 
> to allow for better visibility into log processing errors.
>
> KIP 
> —ttps://cwiki.apache.org/confluence/display/KAFKA/KIP-859%3A+Add+Metadata+Log+Processing+Error+Related+Metrics
>  
> 
> Discussion Thread — 
> https://lists.apache.org/thread/yl87h1s484yc09yjo1no46hwpbv0qkwt
>
> Thanks
> Niket
>

Re: [DISCUSS] KIP-859: Add Metadata Log Processing Error Related Metrics

[Niket Goel]

Hi all,

Thanks for the comments on this KIP. I have started a vote for this KIP here [1]

- Niket

[1] https://lists.apache.org/thread/1kqv8wjcj9w64xxy3p1qxll2ws7xqo3w 

> On Jul 27, 2022, at 11:39 AM, David Arthur 
>  wrote:
> 
> Thanks for the KIP, Niket! I definitely agree we need to surface metadata
> processing errors to the operator. I have some questions about the
> semantics of the new metrics:
> 
> 1) Does "publisher-error-count" represent the number of errors
> encountered only when loading the most recent image? Or will this value be
> the cumulative number of publisher errors since the broker started?
> 2) Same question for "listener-batch-load-error-count"
> 3) Will ForceRenounceCount be zero for non-leader controllers? Or will this
> value remain between elections and only get reset to zero upon a restart
> 
> Thanks!
> David
> 
> On Wed, Jul 27, 2022 at 2:20 PM Niket Goel 
> wrote:
> 
>> 
>> Hi all,
>> 
>> I would like to start a discussion on adding some new metrics to KRaft to
>> allow for better visibility into log processing errors.
>> 
>> KIP URL:
>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/KAFKA/KIP-859%253A%2BAdd%2BMetadata%2BLog%2BProcessing%2BError%2BRelated%2BMetrics&source=gmail-imap&ust=165955196500&usg=AOvVaw2Uzcu-JIs-OZSdfTavNjn7
>> 
>> Thanks!
>> Niket
>> 
>> 
> 
> -- 
> -David

[VOTE] KIP-859: Add Metadata Log Processing Error Related Metrics

[Niket Goel]

Hi all,

I would like to start a vote on KIP-859 which adds some new metrics to KRaft to 
allow for better visibility into log processing errors.

KIP 
—ttps://cwiki.apache.org/confluence/display/KAFKA/KIP-859%3A+Add+Metadata+Log+Processing+Error+Related+Metrics
 

Discussion Thread — 
https://lists.apache.org/thread/yl87h1s484yc09yjo1no46hwpbv0qkwt 

Thanks
Niket

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Adam Bellemare]

Congratulations Sophie! I’m glad to see you made as a PMC member! Well earned. 

> On Aug 1, 2022, at 6:42 PM, Guozhang Wang  wrote:
> 
> Hi everyone,
> 
> I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> committer since Oct. 2020 and has been contributing to the community
> consistently, especially around Kafka Streams and Kafka java consumer. She
> has also presented about Kafka Streams at Kafka Summit London this year. It
> is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> 
> Congratulations, Sophie!
> 
> -- Guozhang Wang, on behalf of Apache Kafka PMC

[ANNOUNCE] Apache Kafka 3.2.1

[David Arthur]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.2.1

This is a bugfix release with several fixes since the release of
3.2.0. A few of the major issues include:

* KAFKA-14062 OAuth client token refresh fails with SASL extensions
* KAFKA-14079 Memory leak in connectors using errors.tolerance=all
* KAFKA-14024 Cooperative rebalance regression causing clients to get stuck


All of the changes in this release can be found in the release notes:

https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:

https://kafka.apache.org/downloads#3.2.1

---


Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream of
records to one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming
the input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide,
including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.

A big thank you for the following 19 contributors to this release!

Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David
Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang, Ismael Juma,
James Hughes, Jason Gustafson, Kirk True, Lucas Bradstreet, Luke Chen,
Nicolas Guyomar, Niket Goel, Okada Haruki, Shawn Wang, Viktor
Somogyi-Vass, Walker Carlson

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/


Thank you!

Regards,
David Arthur

[ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Guozhang Wang]

Hi everyone,

I'd like to introduce our new Kafka PMC member, Sophie. She has been a
committer since Oct. 2020 and has been contributing to the community
consistently, especially around Kafka Streams and Kafka java consumer. She
has also presented about Kafka Streams at Kafka Summit London this year. It
is my pleasure to announce that Sophie agreed to join the Kafka PMC.

Congratulations, Sophie!

-- Guozhang Wang, on behalf of Apache Kafka PMC

[GitHub] [kafka-site] mumrah merged pull request #432: Update website and javadocs for 3.2.1 release

[GitBox]

mumrah merged PR #432:
URL: https://github.com/apache/kafka-site/pull/432


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #20

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 496081 lines...]
[2022-08-01T23:19:51.675Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-08-01T23:19:51.676Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-08-01T23:19:52.619Z] > Task :connect:api:testJar
[2022-08-01T23:19:52.619Z] > Task :connect:api:testSrcJar
[2022-08-01T23:19:52.619Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-08-01T23:19:52.619Z] > Task :connect:api:publishToMavenLocal
[2022-08-01T23:19:53.561Z] 
[2022-08-01T23:19:53.561Z] > Task :streams:javadoc
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-08-01T23:19:53.561Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-08-01T23:19:54.504Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-08-01T23:19:54.504Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-01T23:19:54.504Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-01T23:19:54.504Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-01T23:19:54.504Z] 
/home/jenkins/workspace/Kafka_kafka_3.3@2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-01T23:19:54.504Z] 
/home/jenkins/workspace/Kafka_kafka

[jira] [Created] (KAFKA-14129) KRaft must check manual assignments for createTopics are contiguous

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-14129:


 Summary: KRaft must check manual assignments for createTopics are 
contiguous
 Key: KAFKA-14129
 URL: https://issues.apache.org/jira/browse/KAFKA-14129
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe
Assignee: Colin McCabe


KRaft should validate that manual assignments given to createTopics are 
contiguous. In other words, they must start with partition 0, and progress 
through 1, 2, 3, etc. ZK mode does this, but KRaft mode previously did not.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] bbejeck commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1201532009

   @divijvaidya[ I found a free online JS compatibility tester 
](https://seedmanc.github.io/jscc/)
   Here is the result
   
   https://user-images.githubusercontent.com/199238/182212411-c3b6aa17-9452-4bb0-a3ae-94f7f72b073c.png";>
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] vvcephei commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

vvcephei commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1201526829

   Thanks, @bbejeck ! My suggestion is addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-731: Record Rate Limiting for Kafka Connect

[Sagar]

Also, should we look to reopen the discussion/voting thread to see if the
community's opinions have changed?

Sagar.

On Mon, Aug 1, 2022 at 10:42 PM Sagar  wrote:

> Thanks Ryanne,
>
> I am also ok either way. If you want to pick it up, great or else I can
> take it from the POC that you have. Let me know
>
> Thanks!
> Sagar.
>
> On Fri, Jul 29, 2022 at 10:16 PM Ryanne Dolan 
> wrote:
>
>> Thanks Sagar, I have a POC implementation, but I haven't seen much
>> traction
>> here yet. Happy to pick this back up or hand it off if there is interest.
>>
>> Ryanne
>>
>> On Fri, Jul 29, 2022, 9:50 AM Sagar  wrote:
>>
>> > Hey Ryan,
>> >
>> > Just curious what happened to this KIP? I see it never got enough
>> votes.. I
>> > think this could be a useful feature.
>> >
>> > Thanks!
>> > Sagar.
>> >
>> > On Sat, May 8, 2021 at 1:16 PM Dongjin Lee  wrote:
>> >
>> > > Thank you for the KIP. I need this feature.
>> > >
>> > > +1 (non-binding)
>> > >
>> > > Thanks,
>> > > Dongjin
>> > >
>> > > On Thu, Apr 29, 2021 at 9:34 AM Ryanne Dolan 
>> > > wrote:
>> > >
>> > > > Hey y'all, I'd like to start the vote on KIP-731, which enables
>> > operators
>> > > > to limit Connector throughput.
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect
>> > > >
>> > > > Thanks for your votes!
>> > > >
>> > > > Ryanne
>> > > >
>> > >
>> > >
>> > > --
>> > > *Dongjin Lee*
>> > >
>> > > *A hitchhiker in the mathematical world.*
>> > >
>> > >
>> > >
>> > > *github:  github.com/dongjinleekr
>> > > keybase:
>> > https://keybase.io/dongjinleekr
>> > > linkedin:
>> > kr.linkedin.com/in/dongjinleekr
>> > > speakerdeck:
>> > > speakerdeck.com/dongjin
>> > > *
>> > >
>> >
>>
>

Re: [VOTE] KIP-731: Record Rate Limiting for Kafka Connect

[Sagar]

Thanks Ryanne,

I am also ok either way. If you want to pick it up, great or else I can
take it from the POC that you have. Let me know

Thanks!
Sagar.

On Fri, Jul 29, 2022 at 10:16 PM Ryanne Dolan  wrote:

> Thanks Sagar, I have a POC implementation, but I haven't seen much traction
> here yet. Happy to pick this back up or hand it off if there is interest.
>
> Ryanne
>
> On Fri, Jul 29, 2022, 9:50 AM Sagar  wrote:
>
> > Hey Ryan,
> >
> > Just curious what happened to this KIP? I see it never got enough
> votes.. I
> > think this could be a useful feature.
> >
> > Thanks!
> > Sagar.
> >
> > On Sat, May 8, 2021 at 1:16 PM Dongjin Lee  wrote:
> >
> > > Thank you for the KIP. I need this feature.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Thu, Apr 29, 2021 at 9:34 AM Ryanne Dolan 
> > > wrote:
> > >
> > > > Hey y'all, I'd like to start the vote on KIP-731, which enables
> > operators
> > > > to limit Connector throughput.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect
> > > >
> > > > Thanks for your votes!
> > > >
> > > > Ryanne
> > > >
> > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
>

Last sprint to finish line: Replace EasyMock/Powermock with Mockito

[Divij Vaidya]

Hi folks

We have been trying to replace EasyMock/Powermock with Mockito
 for quite a while. This
adds complications for migrating to JDK 17 & Junit5. Significant
contributions have been made by various folks towards this goal and the
finish line is almost in sight.

Let's join forces this week and get the task done!

I and Christo(cc'ed) will be spending time converting the straggler tests
during this week.

At this stage, we are missing a shepherd to help us wrap up this task. *Could
we please solicit some code review bandwidth from a committer for this week
to help us reach the finish line?*

Current pending PR requests:
1. https://github.com/apache/kafka/pull/12459
2. https://github.com/apache/kafka/pull/12465
3. https://github.com/apache/kafka/pull/12418

Regards,
Divij Vaidya

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1111

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 500593 lines...]
[2022-08-01T16:26:07.492Z] > Task :connect:api:testJar
[2022-08-01T16:26:07.492Z] > Task :connect:api:testSrcJar
[2022-08-01T16:26:07.492Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-08-01T16:26:07.492Z] > Task :connect:api:publishToMavenLocal
[2022-08-01T16:26:09.250Z] 
[2022-08-01T16:26:09.250Z] > Task :streams:javadoc
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:09.250Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-08-01T16:26:10.187Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-08-01T16:26:10.187Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/mai

[GitHub] [kafka-site] bbejeck commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1201359633

   > Thank you for fixing this @bbejeck!
   > 
   > Two questions:
   > 
   > 1. How do we know that this code will work on all (or reasonable number 
of) browsers?
   > 2. As per the [ASF 
guideline](https://privacy.apache.org/faq/committers.html), we also need to 
need to "Make it clear that users who click the image will load a video from a 
third party." How do we achieve this?
   
   1. It's vanilla javascript, so any of today's standard browsers should 
support this
   2. I can update the text to state as much


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1201356851

   > Thanks @bbejeck.
   > 
   > I tested it locally and it looks good. Can we do the same to the Streams 
videos on https://kafka.apache.org/32/documentation/streams/ ?
   
   Hi @mimaison -yes I plan to have a PR submitted very soon (this week).  
Actually, the Streams videos go back to the `10` version, so it will cover all 
of those as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Please subscribe me for Kafka Developer mailing list

[Mickael Maison]

Hi Amit,

Subscription to the mailing lists is self served. See the steps listed
in https://kafka.apache.org/contact:
> To subscribe, send an email to dev-subscr...@kafka.apache.org. Once 
> subscribed, you can have discussion on Kafka® development by mailing to 
> dev@kafka.apache.org.

Thanks,
Mickael

On Mon, Aug 1, 2022 at 4:10 PM Amit Ramswaroop Baheti
 wrote:
>
> Hi,
>
> I will taking over the power support for Apache Kafka upstream. Kindly add me 
> to the developer subscription list.
>
> Thanks,
> Amit Baheti

Please subscribe me for Kafka Developer mailing list

[Amit Ramswaroop Baheti]

Hi,

I will taking over the power support for Apache Kafka upstream. Kindly add me 
to the developer subscription list.

Thanks,
Amit Baheti

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1110

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 488113 lines...]
[2022-08-01T13:17:36.489Z] 
[2022-08-01T13:17:36.489Z] GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart() STARTED
[2022-08-01T13:17:37.715Z] + echo Y
[2022-08-01T13:17:37.715Z] + mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java -DarchetypeVersion=3.3.0-SNAPSHOT 
-DgroupId=streams.examples -DartifactId=streams.examples -Dversion=0.1 
-Dpackage=myapps
[2022-08-01T13:17:38.560Z] 
[2022-08-01T13:17:38.560Z] GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart() PASSED
[2022-08-01T13:17:38.560Z] 
[2022-08-01T13:17:38.560Z] GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin() STARTED
[2022-08-01T13:17:39.967Z] [INFO] Scanning for projects...
[2022-08-01T13:17:39.967Z] [INFO] 
[2022-08-01T13:17:39.967Z] [INFO] --< 
org.apache.maven:standalone-pom >---
[2022-08-01T13:17:39.967Z] [INFO] Building Maven Stub Project (No POM) 1
[2022-08-01T13:17:39.967Z] [INFO] [ pom 
]-
[2022-08-01T13:17:39.967Z] [INFO] 
[2022-08-01T13:17:39.967Z] [INFO] >>> maven-archetype-plugin:3.2.1:generate 
(default-cli) > generate-sources @ standalone-pom >>>
[2022-08-01T13:17:39.967Z] [INFO] 
[2022-08-01T13:17:39.967Z] [INFO] <<< maven-archetype-plugin:3.2.1:generate 
(default-cli) < generate-sources @ standalone-pom <<<
[2022-08-01T13:17:39.967Z] [INFO] 
[2022-08-01T13:17:39.967Z] [INFO] 
[2022-08-01T13:17:39.967Z] [INFO] --- maven-archetype-plugin:3.2.1:generate 
(default-cli) @ standalone-pom ---
[2022-08-01T13:17:40.752Z] 
[2022-08-01T13:17:40.752Z] GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin() PASSED
[2022-08-01T13:17:41.728Z] [INFO] Generating project in Interactive mode
[2022-08-01T13:17:41.728Z] [WARNING] Archetype not found in any catalog. 
Falling back to central repository.
[2022-08-01T13:17:41.728Z] [WARNING] Add a repository with id 'archetype' in 
your settings.xml if archetype's repository is elsewhere.
[2022-08-01T13:17:41.728Z] [INFO] Using property: groupId = streams.examples
[2022-08-01T13:17:41.728Z] [INFO] Using property: artifactId = streams.examples
[2022-08-01T13:17:41.728Z] [INFO] Using property: version = 0.1
[2022-08-01T13:17:41.728Z] [INFO] Using property: package = myapps
[2022-08-01T13:17:41.728Z] Confirm properties configuration:
[2022-08-01T13:17:41.728Z] groupId: streams.examples
[2022-08-01T13:17:41.728Z] artifactId: streams.examples
[2022-08-01T13:17:41.728Z] version: 0.1
[2022-08-01T13:17:41.728Z] package: myapps
[2022-08-01T13:17:41.728Z]  Y: : [INFO] 

[2022-08-01T13:17:41.728Z] [INFO] Using following parameters for creating 
project from Archetype: streams-quickstart-java:3.3.0-SNAPSHOT
[2022-08-01T13:17:41.728Z] [INFO] 

[2022-08-01T13:17:41.728Z] [INFO] Parameter: groupId, Value: streams.examples
[2022-08-01T13:17:41.728Z] [INFO] Parameter: artifactId, Value: streams.examples
[2022-08-01T13:17:41.728Z] [INFO] Parameter: version, Value: 0.1
[2022-08-01T13:17:41.728Z] [INFO] Parameter: package, Value: myapps
[2022-08-01T13:17:41.728Z] [INFO] Parameter: packageInPathFormat, Value: myapps
[2022-08-01T13:17:41.728Z] [INFO] Parameter: package, Value: myapps
[2022-08-01T13:17:41.728Z] [INFO] Parameter: version, Value: 0.1
[2022-08-01T13:17:41.728Z] [INFO] Parameter: groupId, Value: streams.examples
[2022-08-01T13:17:41.728Z] [INFO] Parameter: artifactId, Value: streams.examples
[2022-08-01T13:17:41.728Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[2022-08-01T13:17:41.728Z] [INFO] 

[2022-08-01T13:17:41.728Z] [INFO] BUILD SUCCESS
[2022-08-01T13:17:41.728Z] [INFO] 

[2022-08-01T13:17:41.728Z] [INFO] Total time:  1.977 s
[2022-08-01T13:17:41.728Z] [INFO] Finished at: 2022-08-01T13:17:40Z
[2022-08-01T13:17:41.728Z] [INFO] 

[Pipeline] dir
[2022-08-01T13:17:42.245Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2022-08-01T13:17:42.752Z] 
[2022-08-01T13:17:42.752Z] GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() STARTED
[2022-08-01T13:17:44.912Z] + mvn compile
[2022-08-01T13:17:46.672Z] [INFO] Scanning for projects...
[2022-08-01T13:17:46.672Z] [INFO] 
[2022-08-01T13:17:46.672Z] [INF

[jira] [Created] (KAFKA-14128) Kafka Streams terminates on topic check

[Patrik Kleindl (Jira)]

Patrik Kleindl created KAFKA-14128:
--

 Summary: Kafka Streams terminates on topic check
 Key: KAFKA-14128
 URL: https://issues.apache.org/jira/browse/KAFKA-14128
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.0.0
 Environment: Any
Reporter: Patrik Kleindl


Our streams application shut down unexpectedly after some network issues that 
should have been easily recoverable.

Logs:

 
{code:java}
2022-07-29 13:39:37.854  INFO 25843 --- [348aefeff-admin] 
org.apache.kafka.clients.NetworkClient   : [AdminClient 
clientId=L.DII.A-b1355e4a-b909-4da1-a832-dd3348aefeff-admin] Disconnecting from 
node 3 due to request timeout.
2022-07-29 13:39:37.854  INFO 25843 --- [348aefeff-admin] 
org.apache.kafka.clients.NetworkClient   : [AdminClient 
clientId=L.DII.A-b1355e4a-b909-4da1-a832-dd3348aefeff-admin] Cancelled 
in-flight METADATA request with correlation id 985 due to node 3 being 
disconnected (elapsed time since creation: 60023ms, elapsed time since send: 
60023ms, request timeout: 3ms)
2022-07-29 13:39:37.867 ERROR 25843 --- [-StreamThread-1] 
o.a.k.s.p.i.InternalTopicManager : stream-thread [main] Unexpected 
error during topic description for 
L.DII.A-COGROUPKSTREAM-AGGREGATE-STATE-STORE-03-changelog.
Error message was: org.apache.kafka.common.errors.TimeoutException: 
Call(callName=describeTopics, deadlineMs=1659101977830, tries=1, 
nextAllowedTryMs=1659101977955) timed out at 1659101977855 after 1 attempt(s)
2022-07-29 13:39:37.869  INFO 25843 --- [-StreamThread-1] 
o.a.k.s.p.internals.StreamThread     : stream-thread 
[L.DII.A-b1355e4a-b909-4da1-a832-dd3348aefeff-StreamThread-1] State transition 
from RUNNING to PENDING_SHUTDOWN
{code}
I think the relevant code is in 
[https://github.com/apache/kafka/blob/31ff6d7f8af57e8c39061f31774a61bd1728904e/streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java#L524|https://github.com/apache/kafka/blob/31ff6d7f8af57e8c39061f31774a61bd1728904e/streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java#L523-L550]
{code:java}
topicFuture.getValue().get();{code}
without a timeout value cannot throw a TimeoutException, so the 
TimeoutException of the AdminClient will be an ExecutionException and hit the 
last else branch where the StreamsException is thrown.

Possible fix:

Use the KafkaFuture method with timeout:
{code:java}
public abstract T get(long timeout, TimeUnit unit) throws InterruptedException, 
ExecutionException,
TimeoutException;{code}
instead of 
{code:java}
public abstract T get() throws InterruptedException, ExecutionException;{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14127) KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-14127:
---

 Summary: KIP-858: Handle JBOD broker disk failure in KRaft
 Key: KAFKA-14127
 URL: https://issues.apache.org/jira/browse/KAFKA-14127
 Project: Kafka
  Issue Type: Improvement
  Components: jbod, kraft
Reporter: Igor Soarez


Tracking for KIP-858



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14095) Improve handling of sync offset failures in MirrorMaker

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14095.

Fix Version/s: 3.4.0
   Resolution: Fixed

> Improve handling of sync offset failures in MirrorMaker
> ---
>
> Key: KAFKA-14095
> URL: https://issues.apache.org/jira/browse/KAFKA-14095
> Project: Kafka
>  Issue Type: Improvement
>  Components: mirrormaker
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 3.4.0
>
>
> MirrorMaker can automatically sync offsets when sync.group.offsets.enabled is 
> set to true. However that only works if the consumer groups are not used in 
> the target cluster otherwise committing offsets fails with UNKNOWN_MEMBER_ID. 
> Currently we treat this case as an error in the admin client so it emits an 
> ERROR level log line. This is misleading as it's not an unexpected error and 
> we should treat it the same way as other group level error codes.
> In addition we should check the result of the alterConsumerGroupOffsets call 
> in MirrorMaker and provide an helpful message in case of UNKNOWN_MEMBER_ID 
> errors.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] divijvaidya commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

divijvaidya commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1201010516

   Thank you for fixing this @bbejeck!
   
   Two questions:
   1. How do we know that this code will work on all (or reasonable number of) 
browsers?
   
   2. As per the [ASF 
guideline](https://privacy.apache.org/faq/committers.html), we also need to 
need to "Make it clear that users who click the image will load a video from a 
third party." How do we achieve this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

mimaison commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1200929194

   Thanks @bbejeck. 
   
   I tested it locally and it looks good. Can we do the same to the Streams 
videos on https://kafka.apache.org/32/documentation/streams/ ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Matthew Benedict de Detrich]

Just letting everyone know that in regards to KIP-797 (which was accepted in 
this release), the PR at https://github.com/apache/kafka/pull/11478 has 
received a binding approval from Luke Chen however we are still waiting for 
review from Mickael Maison/Tom Bently since they also reviewed the PR.

Due to time pressure from the release schedule does it make sense to merge the 
PR as is since it already has the necessary approval from Luke or should we 
wait for finale reviews from Mickael/Tom as well?

Regards

--
Matthew de Detrich
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491603708037
w: aiven.io e: matthew.dedetr...@aiven.io
On 11. May 2022, 00:53 +0200, José Armando García Sancio 
, wrote:
> Hi all,
>
> I would like to volunteer for the release of Apache Kafka 3.3.0. If
> people agree, I'll start working on the release plan and update this
> thread.
>
> Thanks,
> -José