Re: [DISCUSS] Apache Kafka 3.4.0 release

[Sophie Blee-Goldman]

Hey all! It's officially *feature freeze for 3.4* so make sure you get that
feature work merged by the end of today.
After this point, only bug fixes and other work focused on stabilizing the
release should be merged to the release
branch. Also note that the *3.4 code freeze* will be in one week (*Dec 7th*)
so please make sure to stabilize and
thoroughly test any new features.

I will wait until Friday to create the release branch to allow for any
existing PRs to be merged. After this point you'll
need to cherrypick any new commits to the 3.4 branch once a PR is merged.

Finally, I've updated the list of KIPs targeted for 3.4. Please check out
the Planned KIP Content on the release
plan and let me know if there is anything missing or incorrect on there.

Cheers,
Sophie


On Wed, Nov 30, 2022 at 12:29 PM David Arthur  wrote:

> Sophie, KIP-866 has been accepted. Thanks!
>
> -David
>
> On Thu, Nov 17, 2022 at 12:21 AM Sophie Blee-Goldman
>  wrote:
> >
> > Thanks for the update Rajini, I've added this to the release page since
> it
> > looks like
> > it will pass but of course if anything changes, just let me know.
> >
> > David, I'm fine with aiming to include KIP-866 in the 3.4 release as well
> > since this
> > seems to be a critical part of the zookeeper removal/migration. Please
> let
> > me know
> > when it's been accepted
> >
> > On Wed, Nov 16, 2022 at 11:08 AM Rajini Sivaram  >
> > wrote:
> >
> > > Hi Sophie,
> > >
> > > KIP-881 has three binding votes (David Jacot, Jun and me) and one
> > > non-binding vote (Maulin). So it is good to go for 3.4.0 if there are
> no
> > > objections until the voting time of 72 hours completes on Friday.
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Wed, Nov 16, 2022 at 3:15 PM David Arthur
> > >  wrote:
> > >
> > > > Sophie, the vote for KIP-866 is underway, but there is still some
> > > > discussion happening. I'm hopeful that the vote can close this week,
> but
> > > it
> > > > may fall into next week. Can we include this KIP in 3.4?
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Tue, Nov 15, 2022 at 6:52 AM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Sophie,
> > > > >
> > > > > I was out of office and hence couldn't get voting started for
> KIP-881
> > > in
> > > > > time. I will start the vote for the KIP today. If there are
> sufficient
> > > > > votes by tomorrow (16th Nov), can we include this KIP in 3.4, even
> > > though
> > > > > voting will only complete on the 17th? It is a small KIP, so we can
> > > merge
> > > > > by feature freeze.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Thu, Nov 10, 2022 at 4:02 PM Sophie Blee-Goldman
> > > > >  wrote:
> > > > >
> > > > > > Hello again,
> > > > > >
> > > > > > This is a reminder that the KIP freeze deadline is approaching,
> all
> > > > KIPs
> > > > > > must be voted
> > > > > > and accepted by *next Wednesday* *(the 16th)*
> > > > > >
> > > > > > Keep in mind that to allow for the full voting period, this
> means you
> > > > > must
> > > > > > kick off the
> > > > > > vote for your KIP no later than* next Monday* (*the 14th*).
> > > > > >
> > > > > > The feature freeze deadline will be 2 weeks after this, so make
> sure
> > > to
> > > > > get
> > > > > > your KIPs in!
> > > > > >
> > > > > > Best,
> > > > > > Sophie
> > > > > >
> > > > > > On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman <
> > > > sop...@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey all,
> > > > > > >
> > > > > > > I've created the release page for 3.4.0 with the current plan,
> > > which
> > > > > you
> > > > > > > can find here:
> > > > > > >
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
> > > > > > >
> > > > > > > The freeze deadlines for this release are as follows:
> > > > > > >
> > > > > > > 1. KIP Freeze November 16th, 2022
> > > > > > > 2. Feature Freeze November 30th, 2022
> > > > > > > 3. Code Freeze December 7th, 2022
> > > > > > >
> > > > > > > Please take a look at the list of planned KIPs for 3.4.0 and
> let me
> > > > > know
> > > > > > > if you have any
> > > > > > > others that you are targeting in the upcoming release.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Sophie
> > > > > > >
> > > > > > > On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
> > > > > > >  wrote:
> > > > > > >
> > > > > > >> Thanks for volunteering!
> > > > > > >>
> > > > > > >> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna,  >
> > > > wrote:
> > > > > > >>
> > > > > > >> > +1
> > > > > > >> >
> > > > > > >> > Thanks Sophie!
> > > > > > >> >
> > > > > > >> > Best,
> > > > > > >> > Bruno
> > > > > > >> >
> > > > > > >> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
> > > > > > >> > > Hey all,
> > > > > > >> > >
> > > > > > >> > > I'd like to volunteer as release manager for the next
> feature
> > > > > > release,
> > > > > > >> > > which will be Apache
> > > > > > >> > > 

[jira] [Created] (KAFKA-14430) optimize: -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT

[jianbin.chen (Jira)]

jianbin.chen created KAFKA-14430:


 Summary: optimize: 
-Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT
 Key: KAFKA-14430
 URL: https://issues.apache.org/jira/browse/KAFKA-14430
 Project: Kafka
  Issue Type: Improvement
Reporter: jianbin.chen


In case the server has a firewall, exposing only 
'com.sun.management.jmxremote.port' cannot fetch metrics, and when the rmi port 
is not specified, it is randomly generated by default, should make the two 
ports consistent for metrics data reading

[https://bugs.openjdk.org/browse/JDK-8035404?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel]
[https://www.baeldung.com/jmx-ports]

_Summary of testing strategy (including rationale)_
_for the feature or bug fix. Unit and/or integration_
_tests are expected for any behaviour change and_
_system tests should be considered for larger changes._

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Colt McNealy]

Nick,

Thank you for the explanation, and also for the updated KIP. I am quite
eager for this improvement to be released as it would greatly reduce the
operational difficulties of EOS streams apps.

Two questions:

10)
>When reading records, we will use the WriteBatchWithIndex#getFromBatchAndDB
 and WriteBatchWithIndex#newIteratorWithBase utilities in order to ensure
that uncommitted writes are available to query.
Why do extra work to enable the reading of uncommitted writes during IQ?
Code complexity aside, reading uncommitted writes is, in my opinion, a
minor flaw in EOS IQ; it would be very nice to have the guarantee that,
with EOS, IQ only reads committed records. In order to avoid dirty reads,
one currently must query a standby replica (but this still doesn't fully
guarantee monotonic reads).

20) Is it also necessary to enable this optimization on ALOS stores? The
motivation of KIP-844 was mainly to reduce the need to restore state from
scratch on unclean EOS shutdowns; with ALOS it was acceptable to accept
that there may have been uncommitted writes on disk. On a side note, if you
enable this type of store on ALOS processors, the community would
definitely want to enable queries on dirty reads; otherwise users would
have to wait 30 seconds (default) to see an update.

Thank you for doing this fantastic work!
Colt McNealy
*Founder, LittleHorse.io*


On Wed, Nov 30, 2022 at 10:44 AM Nick Telford 
wrote:

> Hi everyone,
>
> I've drastically reduced the scope of this KIP to no longer include the
> StateStore management of checkpointing. This can be added as a KIP later on
> to further optimize the consistency and performance of state stores.
>
> I've also added a section discussing some of the concerns around
> concurrency, especially in the presence of Iterators. I'm thinking of
> wrapping WriteBatchWithIndex with a reference-counting copy-on-write
> implementation (that only makes a copy if there's an active iterator), but
> I'm open to suggestions.
>
> Regards,
> Nick
>
> On Mon, 28 Nov 2022 at 16:36, Nick Telford  wrote:
>
> > Hi Colt,
> >
> > I didn't do any profiling, but the 844 implementation:
> >
> >- Writes uncommitted records to a temporary RocksDB instance
> >   - Since tombstones need to be flagged, all record values are
> >   prefixed with a value/tombstone marker. This necessitates a memory
> copy.
> >- On-commit, iterates all records in this temporary instance and
> >writes them to the main RocksDB store.
> >- While iterating, the value/tombstone marker needs to be parsed and
> >the real value extracted. This necessitates another memory copy.
> >
> > My guess is that the cost of iterating the temporary RocksDB store is the
> > major factor, with the 2 extra memory copies per-Record contributing a
> > significant amount too.
> >
> > Regards,
> > Nick
> >
> > On Mon, 28 Nov 2022 at 16:12, Colt McNealy  wrote:
> >
> >> Hi all,
> >>
> >> Out of curiosity, why does the performance of the store degrade so
> >> significantly with the 844 implementation? I wouldn't be too surprised
> by
> >> a
> >> 50-60% drop (caused by each record being written twice), but 96% is
> >> extreme.
> >>
> >> The only thing I can think of which could create such a bottleneck would
> >> be
> >> that perhaps the 844 implementation deserializes and then re-serializes
> >> the
> >> store values when copying from the uncommitted to committed store, but I
> >> wasn't able to figure that out when I scanned the PR.
> >>
> >> Colt McNealy
> >> *Founder, LittleHorse.io*
> >>
> >>
> >> On Mon, Nov 28, 2022 at 7:56 AM Nick Telford 
> >> wrote:
> >>
> >> > Hi everyone,
> >> >
> >> > I've updated the KIP to resolve all the points that have been raised
> so
> >> > far, with one exception: the ALOS default commit interval of 5 minutes
> >> is
> >> > likely to cause WriteBatchWithIndex memory to grow too large.
> >> >
> >> > There's a couple of different things I can think of to solve this:
> >> >
> >> >- We already have a memory/record limit in the KIP to prevent OOM
> >> >errors. Should we choose a default value for these? My concern here
> >> is
> >> > that
> >> >anything we choose might seem rather arbitrary. We could change
> >> >its behaviour such that under ALOS, it only triggers the commit of
> >> the
> >> >StateStore, but under EOS, it triggers a commit of the Kafka
> >> > transaction.
> >> >- We could introduce a separate `checkpoint.interval.ms` to allow
> >> ALOS
> >> >to commit the StateStores more frequently than the general
> >> >commit.interval.ms? My concern here is that the semantics of this
> >> > config
> >> >would depend on the processing.mode; under ALOS it would allow more
> >> >frequently committing stores, whereas under EOS it couldn't.
> >> >
> >> > Any better ideas?
> >> >
> >> > On Wed, 23 Nov 2022 at 16:25, Nick Telford 
> >> wrote:
> >> >
> >> > > Hi Alex,
> >> > >
> >> > > Thanks for the feedback.
> >> > >
> >> > > I've updated the 

Re: [DISCUSS] KIP-890 Server Side Defense

[Artem Livshits]

Hi Justine,

I think the interesting part is not in this logic (because it tries to
figure out when UNKNOWN_PRODUCER_ID is retriable and if it's retryable,
it's definitely not fatal), but what happens when this logic doesn't return
'true' and falls through.  In the old clients it seems to be fatal, if we
keep the behavior in the new clients, I'd expect it would be fatal as well.

-Artem

On Tue, Nov 29, 2022 at 11:57 AM Justine Olshan
 wrote:

> Hi Artem and Jeff,
>
>
> Thanks for taking a look and sorry for the slow response.
>
> You both mentioned the change to handle UNKNOWN_PRODUCER_ID errors. To be
> clear — this error code will only be sent again when the client's request
> version is high enough to ensure we handle it correctly.
> The current (Java) client handles this by the following (somewhat long)
> code snippet:
>
> // An UNKNOWN_PRODUCER_ID means that we have lost the producer state on the
> broker. Depending on the log start
>
> // offset, we may want to retry these, as described for each case below. If
> none of those apply, then for the
>
> // idempotent producer, we will locally bump the epoch and reset the
> sequence numbers of in-flight batches from
>
> // sequence 0, then retry the failed batch, which should now succeed. For
> the transactional producer, allow the
>
> // batch to fail. When processing the failed batch, we will transition to
> an abortable error and set a flag
>
> // indicating that we need to bump the epoch (if supported by the broker).
>
> if (error == Errors.*UNKNOWN_PRODUCER_ID*) {
>
> if (response.logStartOffset == -1) {
>
> // We don't know the log start offset with this response. We should
> just retry the request until we get it.
>
> // The UNKNOWN_PRODUCER_ID error code was added along with the new
> ProduceResponse which includes the
>
> // logStartOffset. So the '-1' sentinel is not for backward
> compatibility. Instead, it is possible for
>
> // a broker to not know the logStartOffset at when it is returning
> the response because the partition
>
> // may have moved away from the broker from the time the error was
> initially raised to the time the
>
> // response was being constructed. In these cases, we should just
> retry the request: we are guaranteed
>
> // to eventually get a logStartOffset once things settle down.
>
> return true;
>
> }
>
>
> if (batch.sequenceHasBeenReset()) {
>
> // When the first inflight batch fails due to the truncation case,
> then the sequences of all the other
>
> // in flight batches would have been restarted from the beginning.
> However, when those responses
>
> // come back from the broker, they would also come with an
> UNKNOWN_PRODUCER_ID error. In this case, we should not
>
> // reset the sequence numbers to the beginning.
>
> return true;
>
> } else if (lastAckedOffset(batch.topicPartition).orElse(
> *NO_LAST_ACKED_SEQUENCE_NUMBER*) < response.logStartOffset) {
>
> // The head of the log has been removed, probably due to the
> retention time elapsing. In this case,
>
> // we expect to lose the producer state. For the transactional
> producer, reset the sequences of all
>
> // inflight batches to be from the beginning and retry them, so
> that the transaction does not need to
>
> // be aborted. For the idempotent producer, bump the epoch to avoid
> reusing (sequence, epoch) pairs
>
> if (isTransactional()) {
>
> txnPartitionMap.startSequencesAtBeginning(batch.topicPartition,
> this.producerIdAndEpoch);
>
> } else {
>
> requestEpochBumpForPartition(batch.topicPartition);
>
> }
>
> return true;
>
> }
>
>
> if (!isTransactional()) {
>
> // For the idempotent producer, always retry UNKNOWN_PRODUCER_ID
> errors. If the batch has the current
>
> // producer ID and epoch, request a bump of the epoch. Otherwise
> just retry the produce.
>
> requestEpochBumpForPartition(batch.topicPartition);
>
> return true;
>
> }
>
> }
>
>
> I was considering keeping this behavior — but am open to simplifying it.
>
>
>
> We are leaving changes to older clients off the table here since it caused
> many issues for clients in the past. Previously this was a fatal error and
> we didn't have the mechanisms in place to detect when this was a legitimate
> case vs some bug or gap in the protocol. Ensuring each transaction has its
> own epoch should close this gap.
>
>
>
>
> And to address Jeff's second point:
> *does the typical produce request path append records to local log along*
>
> *with the currentTxnFirstOffset information? I would like to understand*
>
> *when the field is written to disk.*
>
>
> Yes, the first produce request populates this field and writes the offset
> as part of the record batch and also to the producer state snapshot. When
> we reload the records on restart and/or 

[jira] [Created] (KAFKA-14429) Move OffsetStorageReader from storage package to source package

[Greg Harris (Jira)]

Greg Harris created KAFKA-14429:
---

 Summary: Move OffsetStorageReader from storage package to source 
package
 Key: KAFKA-14429
 URL: https://issues.apache.org/jira/browse/KAFKA-14429
 Project: Kafka
  Issue Type: Task
Reporter: Greg Harris


The OffsetStorageReader is an interface provided to source connectors. This 
does not fit with the broader context of the `storage` package, which is 
focused on sink/source-agnostic converters and serialization/deserialization.

The current interface should be deprecated and extend from the relocated 
interface in a different package.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1389

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 513956 lines...]
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] 3: Task failed with an exception.
[2022-11-30T22:08:32.598Z] ---
[2022-11-30T22:08:32.598Z] * What went wrong:
[2022-11-30T22:08:32.598Z] Execution failed for task ':tools:integrationTest'.
[2022-11-30T22:08:32.598Z] > Process 'Gradle Test Executor 145' finished with 
non-zero exit value 1
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] * Try:
[2022-11-30T22:08:32.598Z] > Run with --stacktrace option to get the stack 
trace.
[2022-11-30T22:08:32.598Z] > Run with --info or --debug option to get more log 
output.
[2022-11-30T22:08:32.598Z] > Run with --scan to get full insights.
[2022-11-30T22:08:32.598Z] 
==
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] 4: Task failed with an exception.
[2022-11-30T22:08:32.598Z] ---
[2022-11-30T22:08:32.598Z] * What went wrong:
[2022-11-30T22:08:32.598Z] Execution failed for task ':connect:mirror:unitTest'.
[2022-11-30T22:08:32.598Z] > unable to create new native thread
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] * Try:
[2022-11-30T22:08:32.598Z] > Run with --stacktrace option to get the stack 
trace.
[2022-11-30T22:08:32.598Z] > Run with --info or --debug option to get more log 
output.
[2022-11-30T22:08:32.598Z] > Run with --scan to get full insights.
[2022-11-30T22:08:32.598Z] 
==
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] 5: Task failed with an exception.
[2022-11-30T22:08:32.598Z] ---
[2022-11-30T22:08:32.598Z] * What went wrong:
[2022-11-30T22:08:32.598Z] Execution failed for task 
':streams:test-utils:unitTest'.
[2022-11-30T22:08:32.598Z] > Process 'Gradle Test Executor 97' finished with 
non-zero exit value 134
[2022-11-30T22:08:32.598Z]   This problem might be caused by incorrect test 
process configuration.
[2022-11-30T22:08:32.598Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.6/userguide/java_testing.html#sec:test_execution
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] * Try:
[2022-11-30T22:08:32.598Z] > Run with --stacktrace option to get the stack 
trace.
[2022-11-30T22:08:32.598Z] > Run with --info or --debug option to get more log 
output.
[2022-11-30T22:08:32.598Z] > Run with --scan to get full insights.
[2022-11-30T22:08:32.598Z] 
==
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] 6: Task failed with an exception.
[2022-11-30T22:08:32.598Z] ---
[2022-11-30T22:08:32.598Z] * What went wrong:
[2022-11-30T22:08:32.598Z] Execution failed for task 
':streams:streams-scala:unitTest'.
[2022-11-30T22:08:32.598Z] > Process 'Gradle Test Executor 131' finished with 
non-zero exit value 1
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] * Try:
[2022-11-30T22:08:32.598Z] > Run with --stacktrace option to get the stack 
trace.
[2022-11-30T22:08:32.598Z] > Run with --info or --debug option to get more log 
output.
[2022-11-30T22:08:32.598Z] > Run with --scan to get full insights.
[2022-11-30T22:08:32.598Z] 
==
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] 7: Task failed with an exception.
[2022-11-30T22:08:32.598Z] ---
[2022-11-30T22:08:32.598Z] * What went wrong:
[2022-11-30T22:08:32.598Z] Execution failed for task ':storage:unitTest'.
[2022-11-30T22:08:32.598Z] > Process 'Gradle Test Executor 133' finished with 
non-zero exit value 1
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] * Try:
[2022-11-30T22:08:32.598Z] > Run with --stacktrace option to get the stack 
trace.
[2022-11-30T22:08:32.598Z] > Run with --info or --debug option to get more log 
output.
[2022-11-30T22:08:32.598Z] > Run with --scan to get full insights.
[2022-11-30T22:08:32.598Z] 
==
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] 8: Task failed with an exception.
[2022-11-30T22:08:32.598Z] ---
[2022-11-30T22:08:32.598Z] * What went wrong:
[2022-11-30T22:08:32.598Z] Execution failed for task ':core:unitTest'.
[2022-11-30T22:08:32.598Z] > Process 'Gradle Test Executor 126' finished with 
non-zero exit value 1
[2022-11-30T22:08:32.598Z] 
[2022-11-30T22:08:32.598Z] * Try:
[2022-11-30T22:08:32.598Z] > Run with --stacktrace option to get the stack 
trace.
[2022-11-30T22:08:32.598Z] > Run with --info or --debug option to get more log 
output.
[2022-11-30T22:08:32.598Z] > Run with --scan to get full insights.
[2022-11-30T22:08:32.598Z] 
==

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Artem Livshits]

I think it's reasonable for practical scenarios.  Is it possible to turn
off rack awareness in the clients in case the broker selector plugin is not
compatible with rack-aware logic in the client?

-Artem

On Wed, Nov 30, 2022 at 2:46 AM Rajini Sivaram 
wrote:

> Hi Artem,
>
> Understood your concern - brokers could use a replica selector that uses
> some other client metadata other than rack id to decide the preferred read
> replica, or just use the leader. In that case, consumers wouldn't really
> benefit from aligning partition assignment on rack ids. So the question is
> whether we should make the default consumer assignors use rack ids for
> partition assignment or whether we should have different rack-aware
> assignors that can be configured when brokers use rack-aware replica
> selector. We had a similar discussion earlier in the thread (the KIP had
> originally proposed new rack-aware assignors). We decided to update the
> default assignors because:
> 1) Consumers using fetch-from-follower automatically benefit from improved
> locality, without having to update another config.
> 2) Consumers configure rack id for improved locality, so aligning on
> replica rack ids generally makes sense.
> 3) We prioritize balanced assignment over locality in the consumer, so the
> default assignors should work effectively regardless of broker's replica
> selector.
>
> Does that make sense?
>
>
> Thank you,
>
> Rajini
>
>
>
> On Tue, Nov 29, 2022 at 1:05 AM Artem Livshits
>  wrote:
>
> > I'm thinking of a case where the broker uses a plugin that does not use
> > rack ids to determine client locality, in that case the consumer might
> > arrive at the wrong decision based on rack ids.
> >
> > -Artem
> >
> > On Wed, Nov 23, 2022 at 3:54 AM Rajini Sivaram 
> > wrote:
> >
> > > Hi Artem,
> > >
> > > Thanks for reviewing the KIP. The client doesn't rely on a particular
> > > replica assignment logic in the broker. Instead, it matches the actual
> > rack
> > > assignment for partitions from the metadata with consumer racks. So
> there
> > > is no assumption in the client implementation about the use of
> > > RackAwareReplicaSelector in the broker. Did I misunderstand your
> concern?
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Tue, Nov 22, 2022 at 11:03 PM Artem Livshits
> > >  wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > Thank you for the KIP, the KIP looks good to match
> > > RackAwareReplicaSelector
> > > > behavior that is available out-of-box.  Which should probably be good
> > > > enough in practice.
> > > >
> > > > From the design perspective, though, RackAwareReplicaSelector is just
> > one
> > > > possible plugin, in theory the broker could use a plugin that
> leverages
> > > > networking information to get client locality or some other way, so
> it
> > > > seems like we're making an assumption about broker replica selection
> in
> > > the
> > > > default assignment implementation.  So I wonder if we should use a
> > > separate
> > > > plugin that would be set when RackAwareReplicaSelector is set, rather
> > > than
> > > > assume broker behavior in the client implementation.
> > > >
> > > > -Artem
> > > >
> > > > On Wed, Nov 16, 2022 at 8:08 AM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, David and Rajini,
> > > > >
> > > > > Thanks for the explanation. It makes sense to me now.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Nov 16, 2022 at 1:44 AM Rajini Sivaram <
> > > rajinisiva...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks David, that was my understanding as well.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Wed, Nov 16, 2022 at 8:08 AM David Jacot
> > > >  > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > We don't need to bump any RPC requests. The subscription is
> > > > serialized
> > > > > > > (including its version) and included as bytes in the RPCs.
> > > > > > >
> > > > > > > Best,
> > > > > > > David
> > > > > > >
> > > > > > > On Tue, Nov 15, 2022 at 11:42 PM Jun Rao
> >  > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi, Rajini,
> > > > > > > >
> > > > > > > > Thanks for the updated KIP. Just another minor comment. It
> > would
> > > be
> > > > > > > useful
> > > > > > > > to list all RPC requests whose version needs to be bumped
> > because
> > > > of
> > > > > > the
> > > > > > > > changes in ConsumerProtocolSubscription.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Tue, Nov 15, 2022 at 3:45 AM Rajini Sivaram <
> > > > > > rajinisiva...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi David,
> > > > > > > > >
> > > > > > > > > Sorry, I was out of office and hence the delay in
> responding.
> > > > > Thanks
> > > > > > > for
> > > > > > > > > reviewing the KIP and answering Viktor's question (thanks
> for
> > > the
> > > > > > > review,
> > > > > > > > > Viktor).
> > > > > > > > >
> > > > > > > > > Responses below:
> > 

[jira] [Created] (KAFKA-14428) Add Records, RPCs and Configs for KIP-866

[David Arthur (Jira)]

David Arthur created KAFKA-14428:


 Summary: Add Records, RPCs and Configs for KIP-866
 Key: KAFKA-14428
 URL: https://issues.apache.org/jira/browse/KAFKA-14428
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Arthur
Assignee: David Arthur






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14427) Add support for ZK migration transactions

[David Arthur (Jira)]

David Arthur created KAFKA-14427:


 Summary: Add support for ZK migration transactions
 Key: KAFKA-14427
 URL: https://issues.apache.org/jira/browse/KAFKA-14427
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft
Reporter: David Arthur
Assignee: David Arthur






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14426) Add documentation for Kraft limtations that have open KIPs

[Greg Harris (Jira)]

Greg Harris created KAFKA-14426:
---

 Summary: Add documentation for Kraft limtations that have open KIPs
 Key: KAFKA-14426
 URL: https://issues.apache.org/jira/browse/KAFKA-14426
 Project: Kafka
  Issue Type: Task
  Components: documentation, kraft
Reporter: Greg Harris


Currently there are a number of limitations for Kraft, which are described as 
the motivation for the following open KIPs:
* 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Voter+Changes]
* 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-856%3A+KRaft+Disk+Failure+Recovery]
* 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-650%3A+Enhance+Kafkaesque+Raft+semantics#KIP650:EnhanceKafkaesqueRaftsemantics-Pre-vote]
 

These limitations are:
* No online method of resizing the controller quorum
* No online method of recovering from controller disk loss
* No support for heterogeneous voter lists in running controller nodes
* When using a quorum size 3, there is no live-upgrade roll which is tolerant 
of a single unplanned machine failure.
* When using a quorum size >3, there is a risk of non-linearizable reads.

These are significant enough concerns for operations of a Kraft-enabled cluster 
that they should be documented as official limitations in the ops documentation.

Optionally, we may wish to provide or link to more detailed operations 
documentation about performing the offline-resize or offline-recovery stages, 
in addition to describing that such offline procedures are necessary.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.4.0 release

[David Arthur]

Sophie, KIP-866 has been accepted. Thanks!

-David

On Thu, Nov 17, 2022 at 12:21 AM Sophie Blee-Goldman
 wrote:
>
> Thanks for the update Rajini, I've added this to the release page since it
> looks like
> it will pass but of course if anything changes, just let me know.
>
> David, I'm fine with aiming to include KIP-866 in the 3.4 release as well
> since this
> seems to be a critical part of the zookeeper removal/migration. Please let
> me know
> when it's been accepted
>
> On Wed, Nov 16, 2022 at 11:08 AM Rajini Sivaram 
> wrote:
>
> > Hi Sophie,
> >
> > KIP-881 has three binding votes (David Jacot, Jun and me) and one
> > non-binding vote (Maulin). So it is good to go for 3.4.0 if there are no
> > objections until the voting time of 72 hours completes on Friday.
> >
> > Thanks,
> >
> > Rajini
> >
> > On Wed, Nov 16, 2022 at 3:15 PM David Arthur
> >  wrote:
> >
> > > Sophie, the vote for KIP-866 is underway, but there is still some
> > > discussion happening. I'm hopeful that the vote can close this week, but
> > it
> > > may fall into next week. Can we include this KIP in 3.4?
> > >
> > > Thanks,
> > > David
> > >
> > > On Tue, Nov 15, 2022 at 6:52 AM Rajini Sivaram 
> > > wrote:
> > >
> > > > Hi Sophie,
> > > >
> > > > I was out of office and hence couldn't get voting started for KIP-881
> > in
> > > > time. I will start the vote for the KIP today. If there are sufficient
> > > > votes by tomorrow (16th Nov), can we include this KIP in 3.4, even
> > though
> > > > voting will only complete on the 17th? It is a small KIP, so we can
> > merge
> > > > by feature freeze.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Thu, Nov 10, 2022 at 4:02 PM Sophie Blee-Goldman
> > > >  wrote:
> > > >
> > > > > Hello again,
> > > > >
> > > > > This is a reminder that the KIP freeze deadline is approaching, all
> > > KIPs
> > > > > must be voted
> > > > > and accepted by *next Wednesday* *(the 16th)*
> > > > >
> > > > > Keep in mind that to allow for the full voting period, this means you
> > > > must
> > > > > kick off the
> > > > > vote for your KIP no later than* next Monday* (*the 14th*).
> > > > >
> > > > > The feature freeze deadline will be 2 weeks after this, so make sure
> > to
> > > > get
> > > > > your KIPs in!
> > > > >
> > > > > Best,
> > > > > Sophie
> > > > >
> > > > > On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman <
> > > sop...@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I've created the release page for 3.4.0 with the current plan,
> > which
> > > > you
> > > > > > can find here:
> > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
> > > > > >
> > > > > > The freeze deadlines for this release are as follows:
> > > > > >
> > > > > > 1. KIP Freeze November 16th, 2022
> > > > > > 2. Feature Freeze November 30th, 2022
> > > > > > 3. Code Freeze December 7th, 2022
> > > > > >
> > > > > > Please take a look at the list of planned KIPs for 3.4.0 and let me
> > > > know
> > > > > > if you have any
> > > > > > others that you are targeting in the upcoming release.
> > > > > >
> > > > > > Cheers,
> > > > > > Sophie
> > > > > >
> > > > > > On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
> > > > > >  wrote:
> > > > > >
> > > > > >> Thanks for volunteering!
> > > > > >>
> > > > > >> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna, 
> > > wrote:
> > > > > >>
> > > > > >> > +1
> > > > > >> >
> > > > > >> > Thanks Sophie!
> > > > > >> >
> > > > > >> > Best,
> > > > > >> > Bruno
> > > > > >> >
> > > > > >> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
> > > > > >> > > Hey all,
> > > > > >> > >
> > > > > >> > > I'd like to volunteer as release manager for the next feature
> > > > > release,
> > > > > >> > > which will be Apache
> > > > > >> > > Kafka 3.4.0. If that sounds good to everyone I'll update this
> > > > thread
> > > > > >> with
> > > > > >> > > the release plan in the coming week.
> > > > > >> > >
> > > > > >> > > Cheers,
> > > > > >> > > A. Sophie Blee-Goldman
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > -David
> > >
> >



-- 
David Arthur

Re: [VOTE] KIP-866 ZooKeeper to KRaft Migration

[David Arthur]

Hey everyone, I'm going to close out the vote. There were three +1
votes from PMC members and no committer or community votes.

+1 PMC votes:
* Colin McCabe
* Jason Gustafson
* Jun Rao

With that, the vote passes. Thanks to everyone who reviewed this KIP!

Cheers,
David

On Tue, Nov 29, 2022 at 7:48 PM Jun Rao  wrote:
>
> Hi, David,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Tue, Nov 22, 2022 at 3:43 PM Jason Gustafson 
> wrote:
>
> > Thanks, +1 from me. I suspect we might be in for at least one surprise with
> > the re-implemented controller RPCs, but I agree the alternative has risks
> > as well.
> >
> > Best,
> > Jason
> >
> > On Mon, Nov 14, 2022 at 12:00 PM Colin McCabe  wrote:
> >
> > > On Fri, Nov 11, 2022, at 08:59, David Arthur wrote:
> > > > Thanks, Colin.
> > > >
> > > >> never start an upgrade without first verifying the quorum
> > configuration
> > > on the ZK-based brokers
> > > >
> > > > I agree that this is a pretty big benefit. I could imagine debugging
> > > > and fixing connection problems mid-migration would be a big pain.
> > > > Especially if you had some brokers correctly configured, and others
> > > > not.
> > > >
> > > > Adding a heartbeat raises some questions about what to do if a broker
> > > > goes into a bad state, or stops heartbeating, during a migration.
> > > > However, I think the same is true for a registration based approach,
> > > > so maybe it's not an increase in net complexity.
> > > >
> > >
> > > Hi David,
> > >
> > > Yeah. I think the goal should be for the set of heartbeaters to match the
> > > set of broker registrations under /brokers
> > >
> > > Obviously, people could add or remove brokers after the upgrade has
> > begun,
> > > but that's unavoidable, I think. We can at least ensure that at the time
> > we
> > > enter upgrade, all the brokers are ready.
> > >
> > > > I've replaced the ZK registration section with a new RPC and brief
> > > > description. Please take a look.
> > > >
> > >
> > > Thanks, David. With these changes it LGTM to me.
> > >
> > > +1 (binding)
> > >
> > > Colin
> > >
> > > > Thanks!
> > > > David
> > > >
> > > > On Wed, Nov 9, 2022 at 5:46 PM Colin McCabe 
> > wrote:
> > > >>
> > > >> Hi David,
> > > >>
> > > >> Thanks for the response. Replies inline.
> > > >>
> > > >> On Wed, Nov 9, 2022, at 08:17, David Arthur wrote:
> > > >> > Colin
> > > >> >
> > > >> >>  Maybe zk.metadata.migration.enable ?
> > > >> >
> > > >> > Done. I went with "zookeeper.metadata.migration.enable" since our
> > > >> > other ZK configs start with "zookeeper.*"
> > > >> >
> > > >> >> SImilarly, for MigrationRecord: can we rename this to
> > > ZkMigrationStateRecord? Then change MigrationState -> ZkMigrationState.
> > > >> >
> > > >> > Sure
> > > >> >
> > > >> >> With ZkMigrationStateRecord, one thing to keep in mind here is that
> > > we will eventually compact all the metadata logs into a snapshot. That
> > > snapshot will then have to keep alive the memory of the old migration. So
> > > it is not really a matter of replaying the old metadata logs (probably)
> > but
> > > a matter of checking to see what the ZkMigrationState is, which I suppose
> > > could be Optional. If it's not Optional.empty, we
> > already
> > > migrated / are migrating.
> > > >> >
> > > >> > Yea, makes sense.
> > > >> >
> > > >> >> For the /migration ZNode, is "last_update_time_ms" necessary? I
> > > thought ZK already tracked this information in the mzxid of the znode?
> > > >> >
> > > >> > Yes, Jun pointed this out previously, I missed this update in the
> > KIP.
> > > >> > Fixed now.
> > > >> >
> > > >> >> It is true that technically it is only needed in UMR, but I would
> > > still suggest including KRaftControllerId in LeaderAndIsrRequest because
> > it
> > > will make debugging much easier.
> > > >> >>
> > > >> >> I would suggest not implementing the topic deletion state machine,
> > > but just deleting topics eagerly when in migration mode. We can implement
> > > this behavior change by keying off of whether KRaftControllerId is
> > present
> > > in LeaderAndIsrRequest. On broker startup, we'll be sent a full
> > > LeaderAndIsrRequest and can delete stray partitions whose IDs are not as
> > > expected (again, this behavior change would only be for migration mode)
> > > >> >
> > > >> > Sounds good to me. Since this is somewhat of an implementation
> > detail,
> > > >> > do you think we need this included in the KIP?
> > > >>
> > > >> Yeah, maybe we don't need to go into the delete behavior here. But I
> > > think the KIP should specify that we have KRaftControllerId in both
> > > LeaderAndIsrRequest. That will allow us to implement this behavior
> > > conditionally on zk-based brokers when in dual write mode.
> > > >>
> > > >> >
> > > >> >> For existing KRaft controllers, will
> > > kafka.controller:type=KafkaController,name=MigrationState show up as 4
> > > (MigrationFinalized)? I assume this is true, but it would be good to
> > spell
> > > it out. Sorry if this is answered somewhere 

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

[Jun Rao]

Hi, Divij,

Thanks for the reply.

Point#2. My high level question is that is the new method needed for every
implementation of remote storage or just for a specific implementation. The
issues that you pointed out exist for the default implementation of RLMM as
well and so far, the default implementation hasn't found a need for a
similar new method. For public interface, ideally we want to make it more
general.

Thanks,

Jun

On Mon, Nov 21, 2022 at 7:11 AM Divij Vaidya 
wrote:

> Thank you Jun and Alex for your comments.
>
> Point#1: You are right Jun. As Alex mentioned, the "derived metadata" can
> increase the size of cached metadata by a factor of 10 but it should be ok
> to cache just the actual metadata. My point about size being a limitation
> for using cache is not valid anymore.
>
> Point#2: For a new replica, it would still have to fetch the metadata over
> the network to initiate the warm up of the cache and hence, increase the
> start time of the archival process. Please also note the repercussions of
> the warm up scan that Alex mentioned in this thread as part of #102.2.
>
> 100#: Agreed Alex. Thanks for clarifying that. My point about size being a
> limitation for using cache is not valid anymore.
>
> 101#: Alex, if I understand correctly, you are suggesting to cache the
> total size at the leader and update it on archival. This wouldn't work for
> cases when the leader restarts where we would have to make a full scan
> to update the total size entry on startup. We expect users to store data
> over longer duration in remote storage which increases the likelihood of
> leader restarts / failovers.
>
> 102#.1: I don't think that the current design accommodates the fact that
> data corruption could happen at the RLMM plugin (we don't have checksum as
> a field in metadata as part of KIP405). If data corruption occurs, w/ or
> w/o the cache, it would be a different problem to solve. I would like to
> keep this outside the scope of this KIP.
>
> 102#.2: Agree. This remains as the main concern for using the cache to
> fetch total size.
>
> Regards,
> Divij Vaidya
>
>
>
> On Fri, Nov 18, 2022 at 12:59 PM Alexandre Dupriez <
> alexandre.dupr...@gmail.com> wrote:
>
> > Hi Divij,
> >
> > Thanks for the KIP. Please find some comments based on what I read on
> > this thread so far - apologies for the repeats and the late reply.
> >
> > If I understand correctly, one of the main elements of discussion is
> > about caching in Kafka versus delegation of providing the remote size
> > of a topic-partition to the plugin.
> >
> > A few comments:
> >
> > 100. The size of the “derived metadata” which is managed by the plugin
> > to represent an rlmMetadata can indeed be close to 1 kB on average
> > depending on its own internal structure, e.g. the redundancy it
> > enforces (unfortunately resulting to duplication), additional
> > information such as checksums and primary and secondary indexable
> > keys. But indeed, the rlmMetadata is itself a lighter data structure
> > by a factor of 10. And indeed, instead of caching the “derived
> > metadata”, only the rlmMetadata could be, which should address the
> > concern regarding the memory occupancy of the cache.
> >
> > 101. I am not sure I fully understand why we would need to cache the
> > list of rlmMetadata to retain the remote size of a topic-partition.
> > Since the leader of a topic-partition is, in non-degenerated cases,
> > the only actor which can mutate the remote part of the
> > topic-partition, hence its size, it could in theory only cache the
> > size of the remote log once it has calculated it? In which case there
> > would not be any problem regarding the size of the caching strategy.
> > Did I miss something there?
> >
> > 102. There may be a few challenges to consider with caching:
> >
> > 102.1) As mentioned above, the caching strategy assumes no mutation
> > outside the lifetime of a leader. While this is true in the normal
> > course of operation, there could be accidental mutation outside of the
> > leader and a loss of consistency between the cached state and the
> > actual remote representation of the log. E.g. split-brain scenarios,
> > bugs in the plugins, bugs in external systems with mutating access on
> > the derived metadata. In the worst case, a drift between the cached
> > size and the actual size could lead to over-deleting remote data which
> > is a durability risk.
> >
> > The alternative you propose, by making the plugin the source of truth
> > w.r.t. to the size of the remote log, can make it easier to avoid
> > inconsistencies between plugin-managed metadata and the remote log
> > from the perspective of Kafka. On the other hand, plugin vendors would
> > have to implement it with the expected efficiency to have it yield
> > benefits.
> >
> > 102.2) As you mentioned, the caching strategy in Kafka would still
> > require one iteration over the list of rlmMetadata when the leadership
> > of a topic-partition is assigned to a broker, 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1388

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] bbejeck commented on pull request #462: Dream11 powered by Apache Kafka section added

[GitBox]

bbejeck commented on PR #462:
URL: https://github.com/apache/kafka-site/pull/462#issuecomment-1332641287

   Merged into `asf-site`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck merged pull request #462: Dream11 powered by Apache Kafka section added

[GitBox]

bbejeck merged PR #462:
URL: https://github.com/apache/kafka-site/pull/462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Nick Telford]

Hi everyone,

I've drastically reduced the scope of this KIP to no longer include the
StateStore management of checkpointing. This can be added as a KIP later on
to further optimize the consistency and performance of state stores.

I've also added a section discussing some of the concerns around
concurrency, especially in the presence of Iterators. I'm thinking of
wrapping WriteBatchWithIndex with a reference-counting copy-on-write
implementation (that only makes a copy if there's an active iterator), but
I'm open to suggestions.

Regards,
Nick

On Mon, 28 Nov 2022 at 16:36, Nick Telford  wrote:

> Hi Colt,
>
> I didn't do any profiling, but the 844 implementation:
>
>- Writes uncommitted records to a temporary RocksDB instance
>   - Since tombstones need to be flagged, all record values are
>   prefixed with a value/tombstone marker. This necessitates a memory copy.
>- On-commit, iterates all records in this temporary instance and
>writes them to the main RocksDB store.
>- While iterating, the value/tombstone marker needs to be parsed and
>the real value extracted. This necessitates another memory copy.
>
> My guess is that the cost of iterating the temporary RocksDB store is the
> major factor, with the 2 extra memory copies per-Record contributing a
> significant amount too.
>
> Regards,
> Nick
>
> On Mon, 28 Nov 2022 at 16:12, Colt McNealy  wrote:
>
>> Hi all,
>>
>> Out of curiosity, why does the performance of the store degrade so
>> significantly with the 844 implementation? I wouldn't be too surprised by
>> a
>> 50-60% drop (caused by each record being written twice), but 96% is
>> extreme.
>>
>> The only thing I can think of which could create such a bottleneck would
>> be
>> that perhaps the 844 implementation deserializes and then re-serializes
>> the
>> store values when copying from the uncommitted to committed store, but I
>> wasn't able to figure that out when I scanned the PR.
>>
>> Colt McNealy
>> *Founder, LittleHorse.io*
>>
>>
>> On Mon, Nov 28, 2022 at 7:56 AM Nick Telford 
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I've updated the KIP to resolve all the points that have been raised so
>> > far, with one exception: the ALOS default commit interval of 5 minutes
>> is
>> > likely to cause WriteBatchWithIndex memory to grow too large.
>> >
>> > There's a couple of different things I can think of to solve this:
>> >
>> >- We already have a memory/record limit in the KIP to prevent OOM
>> >errors. Should we choose a default value for these? My concern here
>> is
>> > that
>> >anything we choose might seem rather arbitrary. We could change
>> >its behaviour such that under ALOS, it only triggers the commit of
>> the
>> >StateStore, but under EOS, it triggers a commit of the Kafka
>> > transaction.
>> >- We could introduce a separate `checkpoint.interval.ms` to allow
>> ALOS
>> >to commit the StateStores more frequently than the general
>> >commit.interval.ms? My concern here is that the semantics of this
>> > config
>> >would depend on the processing.mode; under ALOS it would allow more
>> >frequently committing stores, whereas under EOS it couldn't.
>> >
>> > Any better ideas?
>> >
>> > On Wed, 23 Nov 2022 at 16:25, Nick Telford 
>> wrote:
>> >
>> > > Hi Alex,
>> > >
>> > > Thanks for the feedback.
>> > >
>> > > I've updated the discussion of OOM issues by describing how we'll
>> handle
>> > > it. Here's the new text:
>> > >
>> > > To mitigate this, we will automatically force a Task commit if the
>> total
>> > >> uncommitted records returned by
>> > >> StateStore#approximateNumUncommittedEntries()  exceeds a threshold,
>> > >> configured by max.uncommitted.state.entries.per.task; or the total
>> > >> memory used for buffering uncommitted records returned by
>> > >> StateStore#approximateNumUncommittedBytes() exceeds the threshold
>> > >> configured by max.uncommitted.state.bytes.per.task. This will roughly
>> > >> bound the memory required per-Task for buffering uncommitted records,
>> > >> irrespective of the commit.interval.ms, and will effectively bound
>> the
>> > >> number of records that will need to be restored in the event of a
>> > failure.
>> > >>
>> > >
>> > >
>> > > These limits will be checked in StreamTask#process and a premature
>> commit
>> > >> will be requested via Task#requestCommit().
>> > >>
>> > >
>> > >
>> > > Note that these new methods provide default implementations that
>> ensure
>> > >> existing custom stores and non-transactional stores (e.g.
>> > >> InMemoryKeyValueStore) do not force any early commits.
>> > >
>> > >
>> > > I've chosen to have the StateStore expose approximations of its buffer
>> > > size/count instead of opaquely requesting a commit in order to
>> delegate
>> > the
>> > > decision making to the Task itself. This enables Tasks to look at
>> *all*
>> > of
>> > > their StateStores, and determine whether an early commit is necessary.
>> > > Notably, it enables pre-Task thresholds, 

Re: [VOTE] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

[Hao Li]

Hi all,

Thanks for the vote. The vote passed with 4 binding votes (John, Matthias,
Sophie and Bruno).

I'll update KIP and submit a PR for this.

Thanks,
Hao

On Tue, Nov 22, 2022 at 11:08 PM Bruno Cadonna  wrote:

> Hi Hao,
>
> Thanks for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 22.11.22 10:08, Sophie Blee-Goldman wrote:
> > Hey Hao, thanks for the KIP -- I'm +1 (binding)
> >
> > On Mon, Nov 21, 2022 at 12:57 PM Matthias J. Sax 
> wrote:
> >
> >> +1 (binding)
> >>
> >> On 11/21/22 7:39 AM, John Roesler wrote:
> >>> I'm +1 (binding)
> >>>
> >>> Thanks for the KIP!
> >>> -John
> >>>
> >>> On 2022/11/17 21:06:29 Hao Li wrote:
>  Hi all,
> 
>  I would like start a vote on KIP-884:
> 
> 
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-884%3A+Add+config+to+configure+KafkaClientSupplier+in+Kafka+Streams
> 
> 
>  Thanks,
>  Hao
> 
> >>
> >
>


-- 
Thanks,
Hao

[jira] [Resolved] (KAFKA-13731) Standalone Connect workers should not require connector configs to start

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13731.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

> Standalone Connect workers should not require connector configs to start
> 
>
> Key: KAFKA-13731
> URL: https://issues.apache.org/jira/browse/KAFKA-13731
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Minor
> Fix For: 3.4.0
>
>
> In order to start a standalone Connect worker, it's currently necessary to 
> provide at least two command-line arguments:
>  # The path to the worker config file
>  # One or more paths to a connector config file
> This may be due to the now-inaccurate belief that standalone workers do not 
> support the Connect REST API, which can be used to create/reconfigure/delete 
> connectors at runtime. However, standalone mode does in fact expose the 
> Connect REST API, and since that allows for connectors to be created after 
> bringing up the worker, it is unnecessary to force users to supply at least 
> one config for a connector to be run on the worker at startup time.
> We should remove the requirement that at least one connector config file be 
> provided to standalone Connect workers at startup time. This has several 
> benefits:
>  # Easier to debug for developers investigating framework behavior, including 
> potential bugs
>  # Easier to experiment against for developers/developer evangelists who want 
> to write quickstarts that leverage standalone mode
>  # Allows for connector config files to be tracked exclusively in JSON format 
> (the REST API uses JSON for connector configs, whereas the configs passed to 
> standalone workers on startup have to be Java properties files)
>  # Helps avoid perpetuating the unfortunately common myth that standalone 
> workers do not expose a rest API
>  # Allows for standalone workers to be brought up even if there are issues 
> with connector configs (if there are validation or startup issues with 
> connector configs provided on the command line, the worker will fail to start 
> up)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14425) Automated protocol should support nullable structs

[David Jacot (Jira)]

David Jacot created KAFKA-14425:
---

 Summary: Automated protocol should support nullable structs
 Key: KAFKA-14425
 URL: https://issues.apache.org/jira/browse/KAFKA-14425
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-883: Add delete callback method to Connector API

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Thanks for your feedback Chris,

1. I think the behavior should remain the same as it is today. The worker stops 
the connector when its configuration is updated, and if the update is a 
deletion, it won't start the connector again. If an error happens during stop() 
today, the statusListener will update the backing store with a FAILED state. 
The only thing that changes on this path is that the Connector#stop() method 
will include an additional boolean parameter, so the connector knows that the 
reason it is being stopped is because of a deletion, and can perform additional 
actions if necessary. 

2. I agree; at first I thought it made sense, but after reading KIP-875 and 
finding out that connectors can use custom offsets topics to store offsets, I 
think this idea needs more refinement. There's probably a way to reuse the work 
proposed by this KIP with the "Automatically delete offsets with connectors" 
feature mentioned on the "Future work" section of KIP-875, and am happy to 
explore it more.

3. I didn't consider that. There is some asymmetry here on how the 
StandaloneHerder handles this (tasks are stopped before the connector is) and 
the DistributedHerder. One option would be not to handle this on the 
#processConnectorConfigUpdates(...) method, but instead wait for the 
RebalanceListener#onRevoked(...) callback, which already stops the revoked 
connectors and tasks synchronously. The idea would be to enhance this to check 
the configState store and, if the configuration of the revoked connector(s) is 
gone, then we can let the connector know about that fact when stopping it (by 
the aforementioned mechanism). I'll update the KIP and PR if you think it is 
worth it.

4. That's correct. As the KIP motivates, we have connectors that need to do 
some provisioning/setup when they are deployed (we run connectors for internal 
clients), and when tenants delete a connector, we don't have a clear signal 
that allows us to cleanup those resources. The goal is probably similar to 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Safely+notify+Kafka+Connect+SourceTask+is+stopped,
 just took a different approach.


From: dev@kafka.apache.org At: 11/29/22 15:31:31 UTC-5:00To:  
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API

Hi Hector,

Thanks for the KIP! Here are my initial thoughts:

1. I like the simplicity of an overloaded stop method, but there is some
asymmetry between stopping a connector and deleting one. If a connector is
stopped (for rebalance, to be reconfigured, etc.) and a failure occurs
then, the failure will be clearly visible in the REST API via, e.g., the
GET /connectors/{connector}/status endpoint. If a connector is deleted and
a failure occurs, with the current proposal, users won't have the same
level of visibility. How can we clearly surface failures caused during the
"destroy" phase of a connector's lifecycle to users?

2. I don't think that this new feature should be used to control (delete)
offsets for connectors. We're addressing that separately in KIP-875, and it
could be a source of headaches for users if they discover that some
connectors' offsets persist across deletion/recreation while others do not.
If anything, we should explicitly recommend against this kind of logic in
the Javadocs for the newly-introduced method.

3. Is it worth trying to give all of the connector's tasks a chance to shut
down before invoking "stop(true)" on the Connector? If so, any thoughts on
how we can accomplish that?

4. Just to make sure we're on the same page--this feature is not being
proposed so that connectors can try to delete the data that they've
produced (i.e., that sink connectors have written to the external system,
or that source connectors have written to Kafka), right?

Cheers,

Chris

On Thu, Nov 17, 2022 at 5:31 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Hi,
>
> I've updated the KIP with the new #stop(boolean isDeleted) overloaded
> method, and have also amended the PR and JIRA tickets. I also added a
> couple entries to the "Rejected alternatives" section with the reasons why
> I pivoted from introducing new callback methods to retrofit the existing
> one.
>
> Please let me know what your thoughts are.
>
> Cheers,
> Hector
>
> From: Hector Geraldino (BLOOMBERG/ 919 3RD A) At: 11/16/22 17:38:59
> UTC-5:00To:  dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
>
> Hi Mickael,
>
> I agree that the new STOPPED state proposed in KIP-875 will improve the
> connector lifecycle. The changes proposed in this KIP aim to cover the gap
> where connectors need to actually be deleted, but because the API doesn't
> provide any hooks, external assets are left lingering where they shouldn't.
>
> I agree that this proposal is similar to KIP-419, maybe the main
> difference is their focus on Tasks whereas KIP-833 proposes changes to the
> Connector. My 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1387

[Apache Jenkins Server]

See 

[ANNOUNCE] Call for papers: Kafka Summit London 2023

[Chris Egerton]

Hi everyone,

The call for papers (https://sessionize.com/kafka-summit-london-2023/) is
now open for Kafka Summit London 2023, and you are all welcome to submit a
talk.

We are looking for the most interesting, most informative, most advanced,
and most generally applicable talks on Apache Kafka® and the tools,
technologies, and techniques in the Kafka ecosystem.

People from all industries, backgrounds, and experience levels are
encouraged to submit!

If you have any questions about submitting, reach out to Danica Fine, the
program chair, at df...@confluent.io.

The call for papers closes on Monday, January 9 2022 at 23:59 GMT.

Thanks,

Chris

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Rajini Sivaram]

Hi Artem,

Understood your concern - brokers could use a replica selector that uses
some other client metadata other than rack id to decide the preferred read
replica, or just use the leader. In that case, consumers wouldn't really
benefit from aligning partition assignment on rack ids. So the question is
whether we should make the default consumer assignors use rack ids for
partition assignment or whether we should have different rack-aware
assignors that can be configured when brokers use rack-aware replica
selector. We had a similar discussion earlier in the thread (the KIP had
originally proposed new rack-aware assignors). We decided to update the
default assignors because:
1) Consumers using fetch-from-follower automatically benefit from improved
locality, without having to update another config.
2) Consumers configure rack id for improved locality, so aligning on
replica rack ids generally makes sense.
3) We prioritize balanced assignment over locality in the consumer, so the
default assignors should work effectively regardless of broker's replica
selector.

Does that make sense?


Thank you,

Rajini



On Tue, Nov 29, 2022 at 1:05 AM Artem Livshits
 wrote:

> I'm thinking of a case where the broker uses a plugin that does not use
> rack ids to determine client locality, in that case the consumer might
> arrive at the wrong decision based on rack ids.
>
> -Artem
>
> On Wed, Nov 23, 2022 at 3:54 AM Rajini Sivaram 
> wrote:
>
> > Hi Artem,
> >
> > Thanks for reviewing the KIP. The client doesn't rely on a particular
> > replica assignment logic in the broker. Instead, it matches the actual
> rack
> > assignment for partitions from the metadata with consumer racks. So there
> > is no assumption in the client implementation about the use of
> > RackAwareReplicaSelector in the broker. Did I misunderstand your concern?
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Tue, Nov 22, 2022 at 11:03 PM Artem Livshits
> >  wrote:
> >
> > > Hi Rajini,
> > >
> > > Thank you for the KIP, the KIP looks good to match
> > RackAwareReplicaSelector
> > > behavior that is available out-of-box.  Which should probably be good
> > > enough in practice.
> > >
> > > From the design perspective, though, RackAwareReplicaSelector is just
> one
> > > possible plugin, in theory the broker could use a plugin that leverages
> > > networking information to get client locality or some other way, so it
> > > seems like we're making an assumption about broker replica selection in
> > the
> > > default assignment implementation.  So I wonder if we should use a
> > separate
> > > plugin that would be set when RackAwareReplicaSelector is set, rather
> > than
> > > assume broker behavior in the client implementation.
> > >
> > > -Artem
> > >
> > > On Wed, Nov 16, 2022 at 8:08 AM Jun Rao 
> > wrote:
> > >
> > > > Hi, David and Rajini,
> > > >
> > > > Thanks for the explanation. It makes sense to me now.
> > > >
> > > > Jun
> > > >
> > > > On Wed, Nov 16, 2022 at 1:44 AM Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks David, that was my understanding as well.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Wed, Nov 16, 2022 at 8:08 AM David Jacot
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > We don't need to bump any RPC requests. The subscription is
> > > serialized
> > > > > > (including its version) and included as bytes in the RPCs.
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > On Tue, Nov 15, 2022 at 11:42 PM Jun Rao
>  > >
> > > > > wrote:
> > > > > > >
> > > > > > > Hi, Rajini,
> > > > > > >
> > > > > > > Thanks for the updated KIP. Just another minor comment. It
> would
> > be
> > > > > > useful
> > > > > > > to list all RPC requests whose version needs to be bumped
> because
> > > of
> > > > > the
> > > > > > > changes in ConsumerProtocolSubscription.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Tue, Nov 15, 2022 at 3:45 AM Rajini Sivaram <
> > > > > rajinisiva...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi David,
> > > > > > > >
> > > > > > > > Sorry, I was out of office and hence the delay in responding.
> > > > Thanks
> > > > > > for
> > > > > > > > reviewing the KIP and answering Viktor's question (thanks for
> > the
> > > > > > review,
> > > > > > > > Viktor).
> > > > > > > >
> > > > > > > > Responses below:
> > > > > > > > 01)  I was in two minds about adding new assignors, because
> as
> > > you
> > > > > > said,
> > > > > > > > user experience is better if assignors used racks when
> > available.
> > > > But
> > > > > > I was
> > > > > > > > a bit concerned about changing the algorithm in existing
> > > > applications
> > > > > > which
> > > > > > > > were already configuring `client.rack`. It felt less risky to
> > add
> > > > new
> > > > > > > > assignor implementations instead. But we can retain existing
> > > logic
> > > > if
> > > > > > a)
> > > > > >