[jira] [Created] (KAFKA-15839) Review topic recreation handling in client reconciliation

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-15839:
--

 Summary: Review topic recreation handling in client reconciliation
 Key: KAFKA-15839
 URL: https://issues.apache.org/jira/browse/KAFKA-15839
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Lianet Magrans


Ensure that we properly handle topic re-creation (same name, diff topic IDs) in 
the reconciliation process (assignment cache, same assignment comparison, etc.)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15838) [Connect] ExtractField and InsertField NULL Values are replaced by default value even in NULLABLE fields

[Eric Pangiawan (Jira)]

Eric Pangiawan created KAFKA-15838:
--

 Summary: [Connect] ExtractField and InsertField NULL Values are 
replaced by default value even in NULLABLE fields
 Key: KAFKA-15838
 URL: https://issues.apache.org/jira/browse/KAFKA-15838
 Project: Kafka
  Issue Type: Bug
Reporter: Eric Pangiawan


ExtractField: Line 116-119

https://github.com/a0x8o/kafka/blob/master/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ExtractField.java#L61-L68

InsertField: Line 163 - 195

[https://github.com/a0x8o/kafka/blob/master/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/InsertField.java#L163-L195]
h1. Expect:

Value `null` is valid for an optional filed, even though the field has a 
default value.
Only when field is required, the class return default value fallback when value 
is `null`.
h1. Actual:

Always return default value if `null` was given.
h1. Example:

PostgreSQL DDL:
{code:java}
CREATE TABLE products(
    id varchar(255),
    color varchar(255),
    quantity float8
);
-- Set Default
ALTER TABLE products ALTER COLUMN quantity SET  DEFAULT 1.0; {code}
Insert A Record:
{code:java}
INSERT INTO public.products VALUES('1', 'Blue', null); {code}
Table Select *:
{code:java}
 id | color | quantity
+---+--
 1  | Blue  | {code}
Debezium Behavior when using ExtractField and InsertField class (in the event 
flattening SMT):
{code:java}
{
    "id":"1",
    "color":"Blue",
    "quantity":1.0,
    "__op":"c",
    "__ts_ms":1698127432079,
    "__source_ts_ms":1698127431679,
    "__db":"testing_db",
    "__schema":"public",
    "__table":"products",
    "__lsn":24470112,
    "__txId":957,
    "__snapshot":null,
    "__deleted":"false"
} {code}
The debezium code can be found 
[here|https://github.com/debezium/debezium/blob/2.4/debezium-core/src/main/java/io/debezium/transforms/ExtractNewRecordState.java#L116-L119]
h1. Expected Output:
{code:java}
{
    "id":"1",
    "color":"Blue",
    "quantity":null,
    "__op":"c",
    "__ts_ms":1698127432079,
    "__source_ts_ms":1698127431679,
    "__db":"testing_db",
    "__schema":"public",
    "__table":"products",
    "__lsn":24470112,
    "__txId":957,
    "__snapshot":null,
    "__deleted":"false"
}
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2386

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

[Satish Duggana]

Thanks Qichao for the KIP.

+1 (binding)

~Satish.

On Thu, 16 Nov 2023 at 02:20, Jorge Esteban Quilcate Otoya
 wrote:
>
> Qichao, thanks again for leading this proposal!
>
> +1 (non-binding)
>
> Cheers,
> Jorge.
>
> On Wed, 15 Nov 2023 at 19:17, Divij Vaidya  wrote:
>
> > +1 (binding)
> >
> > I was involved in the discussion thread for this KIP and support it in its
> > current form.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics.
> > >
> > > Please take a look here:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> > >
> > > Best,
> > > Qichao Chu
> > >
> >

[jira] [Resolved] (KAFKA-15346) Single-Key_single-timestamp IQs with versioned state stores

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15346.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Single-Key_single-timestamp IQs with versioned state stores
> ---
>
> Key: KAFKA-15346
> URL: https://issues.apache.org/jira/browse/KAFKA-15346
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Alieh Saeedi
>Assignee: Alieh Saeedi
>Priority: Major
> Fix For: 3.7.0
>
>
> [KIP-960|https://cwiki.apache.org/confluence/display/KAFKA/KIP-960%3A+Support+single-key_single-timestamp+interactive+queries+%28IQv2%29+for+versioned+state+stores]
> This ticket covers just two query types:
> *Key Queries with single timestamp:*
>  # single-key latest-value lookup
>  # single-key lookup with timestamp (upper) bound



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15837) Throw error on use of Consumer.poll(long timeout)

[Kirk True (Jira)]

Kirk True created KAFKA-15837:
-

 Summary: Throw error on use of Consumer.poll(long timeout)
 Key: KAFKA-15837
 URL: https://issues.apache.org/jira/browse/KAFKA-15837
 Project: Kafka
  Issue Type: Improvement
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True
 Fix For: 3.7.0


Per [KIP-266|https://cwiki.apache.org/confluence/x/5kiHB], the 
Consumer.poll(long timeout) method was deprecated back in 2.0.0. The method 
will now throw a KafkaException.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15836) KafkaConsumer subscribe to multiple topics does not respect max.poll.records

[Philip Nee (Jira)]

Philip Nee created KAFKA-15836:
--

 Summary: KafkaConsumer subscribe to multiple topics does not 
respect max.poll.records
 Key: KAFKA-15836
 URL: https://issues.apache.org/jira/browse/KAFKA-15836
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Philip Nee
Assignee: Kirk True


We discovered that when KafkaConsumer subscribes to multiple topics with 
max.poll.record configured.  The max.poll.record is not properly respected for 
all poll() invocation.

 

I was able to reproduce it with the AK example, here is how I ran my tests:

[https://github.com/apache/kafka/pull/14772]

 

1. start zookeeper and kafka server (or kraft mode should be fine too)

2. Run: examples/bin/java-producer-consumer-demo.sh 1000

3. Polled records > 400 will be printed to stdout

 

Here is what the program does:

The produce produces a large number of records to multiple topics.  We 
configure the consumer using a max.poll.record = 400, and subscribed to 
multiple topics.  The consumer poll, and the returned records can sometimes be 
larger than 400.

 

This is an issue in AK 3.6 but 3.5 was fine.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

[Justine Olshan]

Hey Hailey,

Thanks for the KIP.
I wonder if it would be better to either not allow both flags or if we
choose to have node take precedence, at least have a log line stating such.

Otherwise the KIP makes sense to me.

Justine

On Tue, Nov 14, 2023 at 10:17 AM Colin McCabe  wrote:

> Thanks, Hailey.
>
> +1 (binding)
>
> Colin
>
> On Mon, Nov 13, 2023, at 15:13, Hailey Ni wrote:
> > Hi Colin,
> >
> > Thank you for your review. I removed the "absolute path need to be
> > provided" line from the KIP, and will modify the code to get the absolute
> > path to the config files using some bash in the kafka-server-start file.
> > For your second question, I've added a line in the KIP: "If both
> parameters
> > are provided, the value for node-id parameter will take precedence, i.e,
> > the process with node id specified will be killed, no matter what's the
> > process role provided."
> >
> > What do you think?
> >
> > Thanks,
> > Hailey
> >
> > On Thu, Nov 9, 2023 at 4:03 PM Colin McCabe  wrote:
> >
> >> Hi Hailey,
> >>
> >> Thanks for the KIP.
> >>
> >> It feels clunky to have to pass an absolute path to the configuration
> file
> >> when starting the broker or controller. I think we should consider one
> of
> >> two alternate options:
> >>
> >> 1. Use JMXtool to examine the running kafka.Kafka processes.
> >> I believe ID is available from kafka.server, type=app-info,id=1
> (replace 1
> >> with the actual ID)
> >>
> >> Role can be deduced by the presence or absence of
> >> kafka.server,type=KafkaServer,name=BrokerState for brokers, or
> >> kafka.server,type=ControllerServer,name=ClusterId for controllers.
> >>
> >> 2. Alternately, we could inject the ID and role into the command line in
> >> kafka-server-start.sh. Basically add -Dkafka.node.id=1,
> >> -Dkafka.node.roles=broker. This would be helpful to people just
> examining
> >> the output of ps.
> >>
> >> Finally, you state that either command-line option can be given. What
> >> happens if both are given?
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Mon, Oct 23, 2023, at 12:20, Hailey Ni wrote:
> >> > Hi Ron,
> >> >
> >> > I've added the "Rejected Alternatives" section in the KIP. Thanks for
> the
> >> > comments and +1 vote!
> >> >
> >> > Thanks,
> >> > Hailey
> >> >
> >> > On Mon, Oct 23, 2023 at 6:33 AM Ron Dagostino 
> wrote:
> >> >
> >> >> Hi Hailey.  I'm +1 (binding), but could you add a "Rejected
> >> >> Alternatives" section to the KIP and mention the "--required-config "
> >> >> option that we decided against and the reason why we made the
> decision
> >> >> to reject it?  There were some other small things (dash instead of
> dot
> >> >> in the parameter names, --node-id instead of --broker-id), but
> >> >> cosmetic things like this don't warrant a mention, so I think there's
> >> >> just the one thing to document.
> >> >>
> >> >> Thanks for the KIP, and thanks for adjusting it along the way as the
> >> >> discussion moved forward.
> >> >>
> >> >> Ron
> >> >>
> >> >>
> >> >> Ron
> >> >>
> >> >> On Mon, Oct 23, 2023 at 4:00 AM Federico Valeri <
> fedeval...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > +1 (non binding)
> >> >> >
> >> >> > Thanks.
> >> >> >
> >> >> > On Mon, Oct 23, 2023 at 9:48 AM Kamal Chandraprakash
> >> >> >  wrote:
> >> >> > >
> >> >> > > +1 (non-binding). Thanks for the KIP!
> >> >> > >
> >> >> > > On Mon, Oct 23, 2023, 12:55 Hailey Ni 
> >> >> wrote:
> >> >> > >
> >> >> > > > Hi all,
> >> >> > > >
> >> >> > > > I'd like to call a vote on KIP-979 that will allow users to
> >> >> independently
> >> >> > > > stop KRaft processes.
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-979%3A+Allow+independently+stop+KRaft+processes
> >> >> > > >
> >> >> > > > Thanks,
> >> >> > > > Hailey
> >> >> > > >
> >> >>
> >>
>

[jira] [Created] (KAFKA-15835) Group commit/callbacks triggering logic

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-15835:
--

 Summary: Group commit/callbacks triggering logic
 Key: KAFKA-15835
 URL: https://issues.apache.org/jira/browse/KAFKA-15835
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Lianet Magrans


The new consumer reconciliation logic triggers a commit request, revocation 
callback and assignment callbacks sequentially to ensure that they are executed 
in that order. This means that we could require multiple iterations of the poll 
loop to complete reconciling an assignment. 

We could consider triggering them all together, to be executed in the same poll 
iteration, while still making sure that they are executed in the right order. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

[Jorge Esteban Quilcate Otoya]

Qichao, thanks again for leading this proposal!

+1 (non-binding)

Cheers,
Jorge.

On Wed, 15 Nov 2023 at 19:17, Divij Vaidya  wrote:

> +1 (binding)
>
> I was involved in the discussion thread for this KIP and support it in its
> current form.
>
> --
> Divij Vaidya
>
>
>
> On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu 
> wrote:
>
> > Hi all,
> >
> > I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics.
> >
> > Please take a look here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> >
> > Best,
> > Qichao Chu
> >
>

[jira] [Created] (KAFKA-15834) Subscribing to non-existent topic blocks StreamThread from stopping

[Greg Harris (Jira)]

Greg Harris created KAFKA-15834:
---

 Summary: Subscribing to non-existent topic blocks StreamThread 
from stopping
 Key: KAFKA-15834
 URL: https://issues.apache.org/jira/browse/KAFKA-15834
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.6.0
Reporter: Greg Harris


In 
NamedTopologyIntegrationTest#shouldContinueProcessingOtherTopologiesWhenNewTopologyHasMissingInputTopics
 a topology is created which references an input topic which does not exist. 
The test as-written passes, but the KafkaStreams#close(Duration) at the end 
times out, and leaves StreamsThreads running.

>From some cursory investigation it appears that this is happening:
1. The consumer calls the StreamsPartitionAssignor, which calls 
TaskManager#handleRebalanceStart as a side-effect
2. handleRebalanceStart sets the rebalanceInProgress flag
3. This flag is checked by StreamThread.runLoop, and causes the loop to remain 
running.
4. The consumer never calls StreamsRebalanceListener#onPartitionsAssigned, 
because the topic does not exist
5. Because no partitions are ever assigned, the 
TaskManager#handleRebalanceComplete never clears the rebalanceInProgress flag
 
This log message is printed in a tight loop while the close is ongoing and the 
consumer is being polled with zero duration:
{noformat}
[2023-11-15 11:42:43,661] WARN [Consumer 
clientId=NamedTopologyIntegrationTestshouldContinueProcessingOtherTopologiesWhenNewTopologyHasMissingInputTopics-942756f8-5213-4c44-bb6b-5f805884e026-StreamThread-1-consumer,
 
groupId=NamedTopologyIntegrationTestshouldContinueProcessingOtherTopologiesWhenNewTopologyHasMissingInputTopics]
 Received unknown topic or partition error in fetch for partition 
unique_topic_prefix-topology-1-store-repartition-0 
(org.apache.kafka.clients.consumer.internals.FetchCollector:321)
{noformat}
Practically, this means that this test leaks two StreamsThreads and the 
associated clients and sockets, and delays the completion of the test until the 
KafkaStreams#close(Duration) call times out.

Either we should change the rebalanceInProgress flag to avoid getting stuck in 
this rebalance state, or figure out a way to shut down a StreamsThread that is 
in an extended rebalance state during shutdown.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-910: Update Source offsets for Source Connectors without producing records

[Sagar]

Hey all,

Bumping this vote thread again after quite a while.

Thanks!
Sagar.

On Wed, Sep 6, 2023 at 3:58 PM Sagar  wrote:

> Hi All,
>
> Based on the latest discussion thread, it appears as if all open questions
> have been answered.
>
> Hopefully now we are in a state where we can close out on the Voting
> process.
>
> Thanks everyone for the great feedback.
>
> Thanks!
> Sagar.
>
> On Fri, Aug 18, 2023 at 9:00 AM Sagar  wrote:
>
>> Hi All,
>>
>> Bumping the voting thread again.
>>
>> Thanks!
>> Sagar.
>>
>> On Wed, Aug 2, 2023 at 4:43 PM Sagar  wrote:
>>
>>> Attaching the KIP link for reference:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
>>>
>>> Thanks!
>>> Sagar.
>>>
>>> On Wed, Aug 2, 2023 at 4:37 PM Sagar  wrote:
>>>
 Hi All,

 Calling a Vote on KIP-910 [1]. I feel we have converged to a reasonable
 design. Ofcourse I am open to any feedback/suggestions and would address
 them.

 Thanks!
 Sagar.

>>>

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

[Divij Vaidya]

+1 (binding)

I was involved in the discussion thread for this KIP and support it in its
current form.

--
Divij Vaidya



On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu  wrote:

> Hi all,
>
> I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics.
>
> Please take a look here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
>
> Best,
> Qichao Chu
>

[PR] MINOR: Sort powered by page output alphabetically [kafka-site]

[via GitHub]

dsmmcken opened a new pull request, #567:
URL: https://github.com/apache/kafka-site/pull/567

   I've tried to leave this page better than how I found it:
   
   - For fairness, output is sorted alphabetically by root domain. The listing 
was quasi alphabetical, but looks like over time people have prepended and 
appended new companies into a now random order. Rather than re-sorting it once, 
and committing I've sorted the output by root domain (ie. eng.uber.com appears 
under U) such that list will always remain fair and orderly.
   - A few logos had an excessively large file size. Anything over 100kb has 
been resized, removing 1.5mb of page weight with no loss in quality as re-sized 
image are still 2x the displayed size.
   - Fixed a trailing space
   - Added Deephaven to the list


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Apache Kafka 3.7.0 Release

[Stanislav Kozlovski]

Friendly reminder to everybody that the KIP Freeze is *exactly 7 days away*
- November 22.

A KIP must be accepted by this date in order to be considered for this
release. Note, any KIP that may not be implemented in time, or otherwise
risks heavily destabilizing the release, should be deferred.

Best,
Stan

On Fri, Nov 3, 2023 at 6:03 AM Sophie Blee-Goldman 
wrote:

> Looks great, thank you! +1
>
> On Thu, Nov 2, 2023 at 10:21 AM David Jacot 
> wrote:
>
> > +1 from me as well. Thanks, Stan!
> >
> > David
> >
> > On Thu, Nov 2, 2023 at 6:04 PM Ismael Juma  wrote:
> >
> > > Thanks Stanislav, +1
> > >
> > > Ismael
> > >
> > > On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski
> > >  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Given the discussion here and the lack of any pushback, I have
> changed
> > > the
> > > > dates of the release:
> > > > - KIP Freeze - *November 22 *(moved 4 days later)
> > > > - Feature Freeze - *December 6 *(moved 2 days earlier)
> > > > - Code Freeze - *December 20*
> > > >
> > > > If anyone has any thoughts against this proposal - please let me
> know!
> > It
> > > > would be good to settle on this early. These will be the dates we're
> > > going
> > > > with
> > > >
> > > > Best,
> > > > Stanislav
> > > >
> > > > On Thu, Oct 26, 2023 at 12:15 AM Sophie Blee-Goldman <
> > > > sop...@responsive.dev>
> > > > wrote:
> > > >
> > > > > Thanks for the response and explanations -- I think the main
> question
> > > for
> > > > > me
> > > > > was whether we intended to permanently increase the KF -- FF gap
> from
> > > the
> > > > > historical 1 week to 3 weeks? Maybe this was a conscious decision
> > and I
> > > > > just
> > > > >  missed the memo, hopefully someone else can chime in here. I'm all
> > for
> > > > > additional though. And looking around at some of the recent
> releases,
> > > it
> > > > > seems like we haven't been consistently following the "usual"
> > schedule
> > > > > since
> > > > > the 2.x releases.
> > > > >
> > > > > Anyways, my main concern was making sure to leave a full 2 weeks
> > > between
> > > > > feature freeze and code freeze, so I'm generally happy with the new
> > > > > proposal.
> > > > > Although I would still prefer to have the KIP freeze fall on a
> > > Wednesday
> > > > --
> > > > > Ismael actually brought up the same thing during the 3.5.0 release
> > > > > planning,
> > > > > so I'll just refer to his explanation for this:
> > > > >
> > > > > We typically choose a Wednesday for the various freeze dates -
> there
> > > are
> > > > > > often 1-2 day slips and it's better if that doesn't require
> people
> > > > > > working through the weekend.
> > > > > >
> > > > >
> > > > > (From this mailing list thread
> > > > >  >)
> > > > >
> > > > > Thanks for driving the release!
> > > > > Sophie
> > > > >
> > > > > On Wed, Oct 25, 2023 at 8:13 AM Stanislav Kozlovski
> > > > >  wrote:
> > > > >
> > > > > > Thanks for the thorough response, Sophie.
> > > > > >
> > > > > > - Added to the "Future Release Plan"
> > > > > >
> > > > > > > 1. Why is the KIP freeze deadline on a Saturday?
> > > > > >
> > > > > > It was simply added as a starting point - around 30 days from the
> > > > > > announcement. We can move it earlier to the 15th of November, but
> > my
> > > > > > thinking is later is better with these things - it's already
> > > aggressive
> > > > > > enough. e.g given the choice of Nov 15 vs Nov 18, I don't
> > necessarily
> > > > > see a
> > > > > > strong reason to choose 15.
> > > > > >
> > > > > > If people feel strongly about this, to make up for this, we can
> eat
> > > > into
> > > > > > the KF-FF time as I'll touch upon later, and move FF a few days
> > > earlier
> > > > > to
> > > > > > land on a Wednesday.
> > > > > >
> > > > > > This reduces the time one has to get their feature complete after
> > KF,
> > > > but
> > > > > > allows for longer time to a KIP accepted, so the KF-FF gap can be
> > > made
> > > > up
> > > > > > when developing the feature in parallel.
> > > > > >
> > > > > > > , this makes it easy for everyone to remember when the next
> > > deadline
> > > > is
> > > > > > so they can make sure to get everything in on time. I worry that
> > > > varying
> > > > > > this will catch people off guard.
> > > > > >
> > > > > > I don't see much value in optimizing the dates for ease of
> memory -
> > > > > besides
> > > > > > the KIP Freeze (which is the base date), there are only two more
> > > dates
> > > > to
> > > > > > remember that are on the wiki. More importantly, we have a
> plethora
> > > of
> > > > > > tools that can be used to set up reminders - so a contributor
> > doesn't
> > > > > > necessarily need to remember anything if they're serious about
> > > getting
> > > > > > their feature in.
> > > > > >
> > > > > > > 3. Is there a particular reason for having the feature freeze
> > > almost
> > > > a
> > > > > > full 3 weeks from the KIP freeze? ... having 3 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2384

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15833) Restrict Consumer API to be used from one thread

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15833:
--

 Summary: Restrict Consumer API to be used from one thread
 Key: KAFKA-15833
 URL: https://issues.apache.org/jira/browse/KAFKA-15833
 Project: Kafka
  Issue Type: Sub-task
Reporter: Lucas Brutschy


The legacy consumer restricts the API to be used from one thread only. This is 
not enforced in the new consumer. To avoid inconsistencies in the behavior, we 
should enforce the same restriction in the new consumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15832) Trigger reconciliation based on manager poll

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-15832:
--

 Summary: Trigger reconciliation based on manager poll
 Key: KAFKA-15832
 URL: https://issues.apache.org/jira/browse/KAFKA-15832
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Lianet Magrans


Currently the reconciliation logic on the client is triggered when a new target 
assignment is received and resolved, or when new unresolved target assignments 
are discovered in metadata.

This could be improved by triggering the reconciliation logic on each poll 
iteration, to reconcile whatever is ready to be reconciled. This would required 
changes to support poll on the MembershipManager, and integrate it with the 
current polling logic in the background thread.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15824) SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet

[Mayank Shekhar Narula (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Shekhar Narula resolved KAFKA-15824.
---
Resolution: Fixed

> SubscriptionState's maybeValidatePositionForCurrentLeader should handle 
> partition which isn't subscribed yet
> 
>
> Key: KAFKA-15824
> URL: https://issues.apache.org/jira/browse/KAFKA-15824
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Mayank Shekhar Narula
>Assignee: Mayank Shekhar Narula
>Priority: Major
> Fix For: 3.7.0
>
>
> As can be seen [here|#L453], maybeValidatePositionForCurrentLeader doesn't 
> check if partition is subscribed by checking TopicPartitionState cached is 
> null or not, as done by 
> [maybeCompleteValidation|[http://example.com|https://github.com/apache/kafka/blob/832627fc78484fdc7c8d6da8a2d20e7691dbf882/clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java#L477]].
>  So it throws IllegalStateException for a partition that is yet not 
> subscribed.
> Lack of this check writing thread-safe code w.r.t SubscriptionState class is 
> awkward. This can be seen from the example code below. For example, at line 1 
> partitionA would be in allCurrentlySubscribedTopics, but at line 2 it could 
> be removed from subscribed partitions(in a separate thread). So this forces 
> the user of this class to handle IllegalStateException which is awkward.
> {code:java}
> // Following is example code for the user of 
> SubscriptionState::maybeValidatePositionForCurrentLeader
> Set allCurrentlySubscribedTopics = 
> subscriptionState.assignedPartitions(); // line 1
> if(allCurrentlySubscribedTopics.contains(tp)) {
>      ConsumerMetadata.LeaderAndEpoch leaderAndEpoch = 
> metadata.currentLeader(tp);
>   try() {
> subscriptionState.maybeValidatePositionForCurrentLeader(apiVersions, tp, 
> leaderAndEpoch); // line 2
>   } catch (IllegalStateException e) {
>// recover from it. // line 3
>   }
> }{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-1000: List Client Metrics Configuration Resources

[Andrew Schofield]

Hi,
I’d like to start the voting for KIP-1000: List Client Metrics Configuration 
Resources.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources

Thanks,
Andrew

[jira] [Created] (KAFKA-15831) List Client Metrics Configuration Resources

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15831:


 Summary: List Client Metrics Configuration Resources
 Key: KAFKA-15831
 URL: https://issues.apache.org/jira/browse/KAFKA-15831
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Reporter: Andrew Schofield
Assignee: Andrew Schofield


This JIRA tracks the development of KIP-1000 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15653) NPE in ChunkedByteStream

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-15653.

Fix Version/s: 3.7.0
   3.6.1
   Resolution: Fixed

> NPE in ChunkedByteStream
> 
>
> Key: KAFKA-15653
> URL: https://issues.apache.org/jira/browse/KAFKA-15653
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 3.6.0
> Environment: Docker container on a Linux laptop, using the latest 
> release.
>Reporter: Travis Bischel
>Assignee: Justine Olshan
>Priority: Major
> Fix For: 3.7.0, 3.6.1
>
> Attachments: repro.sh
>
>
> When looping franz-go integration tests, I received an UNKNOWN_SERVER_ERROR 
> from producing. The broker logs for the failing request:
>  
> {noformat}
> [2023-10-19 22:29:58,160] ERROR [ReplicaManager broker=2] Error processing 
> append operation on partition 
> 2fa8995d8002fbfe68a96d783f26aa2c5efc15368bf44ed8f2ab7e24b41b9879-24 
> (kafka.server.ReplicaManager)
> java.lang.NullPointerException
>   at 
> org.apache.kafka.common.utils.ChunkedBytesStream.(ChunkedBytesStream.java:89)
>   at 
> org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:105)
>   at 
> org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
>   at 
> org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
>   at 
> org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
>   at 
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
>   at 
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
>   at kafka.log.UnifiedLog.append(UnifiedLog.scala:805)
>   at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
>   at 
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
>   at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
>   at 
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
>   at 
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
>   at 
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
>   at scala.collection.mutable.HashMap.map(HashMap.scala:35)
>   at 
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
>   at kafka.server.ReplicaManager.appendEntries$1(ReplicaManager.scala:754)
>   at 
> kafka.server.ReplicaManager.$anonfun$appendRecords$18(ReplicaManager.scala:874)
>   at 
> kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:874)
>   at 
> kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
>   at java.base/java.lang.Thread.run(Unknown Source)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [jira] [Created] (KAFKA-15829) How to build Kafka 2.7 with maven instead of gradle?

[Rain]

Unluckily, I'm going to migrate the build jdk of kafka 2.7 from jdk11 to
jdk21, while gradle doesn't support jdk21...

最红 <1559063...@qq.com.invalid> 于2023年11月15日周三 17:27写道：

> Are there any problems building kafka with gradle?
> You should take a look at the build.gradle file for building kafka
> scripts, you will see how expensive it is to migrate build tools from
> gradle to maven, and gradle is more powerful than maven builds, at least in
> my opinion :)
> So I think you need to solve network issues like VPN and HTTP PROXY.
>
>
>
>
> --原始邮件--
> 发件人:
>   "dev"
> <
> j...@apache.org;
> 发送时间:2023年11月15日(星期三) 上午10:25
> 收件人:"dev"
> 主题:[jira] [Created] (KAFKA-15829) How to build Kafka 2.7 with maven
> instead of gradle?
>
>
>
> rain.liang created KAFKA-15829:
> --
>
> 
> Summary: How to build Kafka 2.7 with maven instead of gradle?
> 
> Key: KAFKA-15829
> 
> URL: https://issues.apache.org/jira/browse/KAFKA-15829
> 
> ;
> Project: Kafka
>  Issue Type: Wish
>  Affects Versions: 2.7.2
> 
> Reporter: rain.liang
>
>
> It's difficult to upgrade the version of gradle in kafka building. Is
> there a solution to build kafka 2.7 with maven?
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)

[jira] [Created] (KAFKA-15830) Add request/response handling in KafkaApis and update metrics plugin

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15830:
-

 Summary: Add request/response handling in KafkaApis and update 
metrics plugin
 Key: KAFKA-15830
 URL: https://issues.apache.org/jira/browse/KAFKA-15830
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-997: Partition-Level Throughput Metrics

[Qichao Chu]

Hi all,

I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics.

Please take a look here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics

Best,
Qichao Chu

退订

[1]

退订