Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #156

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #155

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12838) Kafka Broker - Request threads inefficiently blocking during produce

[Ryan Cabral (Jira)]

Ryan Cabral created KAFKA-12838:
---

 Summary: Kafka Broker - Request threads inefficiently blocking 
during produce
 Key: KAFKA-12838
 URL: https://issues.apache.org/jira/browse/KAFKA-12838
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 2.8.0, 2.7.0
Reporter: Ryan Cabral


Hello, I have been using Kafka brokers for a bit and have run into a problem 
with the way a kafka broker handles produce requests. If there are multiple 
producers to the same topic and partition, any request handler threads handling 
the produce for that topic and partition become blocked until all requests 
before it are done. Request handler threads for the entire broker can become 
exhausted waiting on the same partition lock, blocking requests for other 
partitions that would not have needed the same lock.

Once that starts happening, requests start to back up, queued requests can 
reach its maximum and network threads begin to be paused cascading the problem 
a bit more. Overall performance ends up being degraded. I'm not so focused on 
the cascade at the moment as I am the initial contention. Intuitively I would 
expect locking contention on a single partition to ONLY affect throughput on 
that partition and not the entire broker.

 

The append call within the request handler originates here:

[https://github.com/apache/kafka/blob/2.8.0/core/src/main/scala/kafka/server/KafkaApis.scala#L638]

Further down the stack the lock during append is created here: 
[https://github.com/apache/kafka/blob/2.8.0/core/src/main/scala/kafka/log/Log.scala#L1165]

At this point the first request will hold the lock during append and future 
requests on the same partition will block, waiting for the lock, tying up an io 
thread (request handler).

At first glance, it seems like it would make the most sense to (via config?) be 
able to funnel (produce) requests for the same partition through its own 
request queue of sorts and dispatch them such that at most one io thread is 
tied up at a time for a given partition. There are a number of reasons the lock 
could be held elsewhere too but this should at least help mitigate the issue a 
bit. I'm assuming this is easier said than done though and likely requires 
significant refactoring to properly achieve but hoping this is something that 
could end up on some sort of long term roadmap.

 

Snippet from jstack. Almost all request handlers threads (there are 256 of 
them, up from 25 to mitigate the issue) in the jstack are blocked waiting on 
the same lock due to the number of producers we have.

 
{noformat}
"data-plane-kafka-request-handler-254" #335 daemon prio=5 os_prio=0 
tid=0x7fb1c9f13000 nid=0x53f1 runnable [0x7fad35796000]
   java.lang.Thread.State: RUNNABLE
at 
org.apache.kafka.common.record.KafkaLZ4BlockOutputStream.(KafkaLZ4BlockOutputStream.java:82)
at 
org.apache.kafka.common.record.KafkaLZ4BlockOutputStream.(KafkaLZ4BlockOutputStream.java:125)
at 
org.apache.kafka.common.record.CompressionType$4.wrapForOutput(CompressionType.java:101)
at 
org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:134)
at 
org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:170)
at 
org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:508)
at 
kafka.log.LogValidator$.buildRecordsAndAssignOffsets(LogValidator.scala:500)
at 
kafka.log.LogValidator$.validateMessagesAndAssignOffsetsCompressed(LogValidator.scala:455)
at 
kafka.log.LogValidator$.validateMessagesAndAssignOffsets(LogValidator.scala:106)
at kafka.log.Log.$anonfun$append$2(Log.scala:1126)
- locked <0x0004c9a6fd60> (a java.lang.Object)
at kafka.log.Log.append(Log.scala:2387)
at kafka.log.Log.appendAsLeader(Log.scala:1050)
at 
kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1079)
at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1067)
at 
kafka.server.ReplicaManager.$anonfun$appendToLocalLog$4(ReplicaManager.scala:953)
at kafka.server.ReplicaManager$$Lambda$1078/1017241486.apply(Unknown 
Source)
at 
scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
at 
scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
at scala.collection.mutable.HashMap.map(HashMap.scala:35)
at 
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:941)
at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:621)
at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:625)
at kafka.server.KafkaApis.handle(KafkaApis.scala:137)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:74)
at java.lang.Thread.run(Thread.java:748)

[GitHub] [kafka-site] topeax opened a new pull request #355: Add Axios (https://www.axios.com/) to the list of the "Powered By ❤️"

[GitBox]

topeax opened a new pull request #355:
URL: https://github.com/apache/kafka-site/pull/355


   [Axios](https://axios.com) uses kafka to power some of our most critical 
data systems. This PR adds us to the page!
   
   @hachikuji can you take a look at this? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-12837) Process entire batch in broker metadata listener

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-12837:
--

 Summary: Process entire batch in broker metadata listener
 Key: KAFKA-12837
 URL: https://issues.apache.org/jira/browse/KAFKA-12837
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio


The currently BrokerMetadataListener process one batch at a time even thought 
it is possible for the BatchReader to contain more than one batch. This is 
functionally correct but it would required less coordination between the 
RaftIOThread and the broker metadata listener thread if the broker is changed 
to process all of the batches included in the BatchReader sent through 
handleCommit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12342) Get rid of raft/meta log shim layer

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-12342.

Resolution: Fixed

> Get rid of raft/meta log shim layer
> ---
>
> Key: KAFKA-12342
> URL: https://issues.apache.org/jira/browse/KAFKA-12342
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>  Labels: kip-500
>
> We currently use a shim to bridge the interface differences between 
> `RaftClient` and `MetaLogManager`. We need to converge the two interfaces and 
> get rid of the shim.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12543) Re-design the ownership model for snapshots

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-12543.

Resolution: Fixed

> Re-design the ownership model for snapshots
> ---
>
> Key: KAFKA-12543
> URL: https://issues.apache.org/jira/browse/KAFKA-12543
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>
> With the current implementation, {{RawSnapshotReader}} are created and closed 
> by the {{KafkaRaftClient}} as needed to satisfy {{FetchSnapshot}} requests. 
> This means that for {{FileRawSnapshotReader}} they are closed before the 
> network client has had a chance to send the bytes over the network.
> One way to fix this is to make the {{KafkaMetadataLog}} the owner of the 
> {{FileRawSnapshotReader}}. Once a {{FileRawSnapshotReader}} is created it 
> will stay open until the snapshot is deleted by 
> {{ReplicatedLog::deleteBeforeSnapshot}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Chris Egerton]

Hi Tom,

Wow, I was way off base! I was thinking that the intent of the fencible
producer was to employ it by default with 3.0, as opposed to only after the
worker-level
"exactly.once.source.enabled" property was flipped on. You are correct that
with the case you were actually describing, there would be no heightened
ACL requirements, and that it would leave room in the future for
exactly-once to be disabled on a per-connector basis (as long as all the
workers in the cluster already had "exactly.once.source.enabled" set to
"true") with no worries about breaking changes.

I agree that this is something for another KIP; even if we could squeeze it
in in time for this release, it might be a bit much for new users to take
in all at once. But I can add it to the doc as "future work" since it's a
promising idea that could prove valuable to someone who might need
per-connector granularity in the future.

Thanks for clearing things up; in retrospect your comments make a lot more
sense now, and I hope I've sufficiently addressed them by now.

PSA for you and everyone else--I plan on updating the doc next week with
the new APIs for connector-defined transaction boundaries,
user-configurable transaction boundaries (i.e., poll vs. interval vs.
connectors), and preflight checks for exactly-once validation (required vs.
requested).

Cheers,

Chris

On Fri, May 21, 2021 at 7:14 AM Tom Bentley  wrote:

> Hi Chris,
>
> Thanks for continuing to entertain some of these ideas.
>
> On Fri, May 14, 2021 at 5:06 PM Chris Egerton  >
> wrote:
>
> > [...]
> >
> That's true, but we do go from three static ACLs (write/describe on a fixed
> > transactional ID, and idempotent write on a fixed cluster) to a dynamic
> > collection of ACLs.
> >
>
> I'm not quite sure I follow, maybe I've lost track. To be clear, I was
> suggesting the use of a 'fencing producer' only in clusters with
> exactly.once.source.enabled=true where I imagined the key difference
> between the exactly once and fencing cases was how the producer was
> configured/used (transactional vs this new fencing semantic). I think the
> ACL requirements for connector producer principals would therefore be the
> same as currently described in the KIP. The same is true for the worker
> principals (which is the only breaking change you give in the KIP). So I
> don't think the fencing idea changes the backwards compatibility story
> that's already in the KIP, just allows a safe per-connector
> exactly.once=disabled option to be supported (with required as requested as
> we already discussed).
>
> But I'm wondering whether I've overlooked something.
>
> Ultimately I think it may behoove us to err on the side of reducing the
> > breaking changes here for now and saving them for 4.0 (or some later
> major
> > release), but would be interested in thoughts from you and others.
> >
>
> Difficult to answer (given I think I might be missing something).
> If there are breaking changes then I don't disagree. It's difficult to
> reason about big changes like this without some practical experience.
> If there are not, then I think we could also implement the whole
> exactly.once=disabled thing in a later KIP without additional breaking
> changes (i.e. some time in 3.x), right?
>
>
> > > Gouzhang also has a (possible) use case for a fencing-only producer (
> > https://issues.apache.org/jira/browse/KAFKA-12693), and as he points out
> > there, you should be able to get these semantics today by calling
> > initTransactions() and then just using the producer as normal (no
> > beginTransaction()/abortTransaction()/endTransaction()).
> >
> > I tested this locally and was not met with success; transactional
> producers
> > do a check right now to ensure that any calls to "KafkaProducer::send"
> > occur within a transaction (see
> >
> >
> https://github.com/apache/kafka/blob/29c55fdbbc331bbede17908ccc878953a1b15d87/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L957-L959
> > and
> >
> >
> https://github.com/apache/kafka/blob/29c55fdbbc331bbede17908ccc878953a1b15d87/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L450-L451
> > ).
> > Not a blocker, just noting that we'd have to do some legwork to make this
> > workable with the producer API.
> >
>
> Ah, sorry, I should have actually tried it rather than just taking a quick
> look at the code.
>
> Rather than remove those safety checks I suppose we'd need a way of
> distinguishing, in the config, the difference in semantics. E.g. Something
> like a fencing.id config, which was mutually exclusive with
> transactional.id.
> Likewise perhaps initFencing() alongside initTransactions() in the API. But
> I think at this point it's something for another KIP.
>
> Kind regards,
>
> Tom
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #154

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-699: Update FindCoordinator to resolve multiple Coordinators at a time

[Mickael Maison]

Hi Sanjana,

Again I'm sorry for the delay.
I've opened a draft PR which hopefully will let you make progress on
KIP-709. Let me know if you have any questions.

Thanks

On Thu, May 20, 2021 at 10:10 PM Sanjana Kaundinya  wrote:
>
> Hi Mickael,
>
> Apologies for pinging on this again, but just wondering if you got a chance 
> to make any progress on this? Would love to take a look and help with the 
> reviews if you have.
>
> Thanks,
> Sanjana
> On May 7, 2021, 3:33 AM -0700, Mickael Maison , 
> wrote:
> > Hi Sanjana,
> >
> > Sorry for the late reply.
> > I hope to have a draft PR next week.
> >
> > Thanks
> >
> > On Tue, May 4, 2021 at 11:11 PM Sanjana Kaundinya  
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Just wanted to ping the thread to inquire if you’ve made any progress on 
> > > the implementation. I was planning on doing the implementation for 
> > > KIP-709 and seeing how KIP-699 is something that’s tightly coupled and 
> > > related to KIP-709 I thought I’d ask what the status of this KIP was.
> > >
> > > Thanks,
> > > Sanjana
> > > On Apr 13, 2021, 8:44 AM -0700, Rajini Sivaram , 
> > > wrote:
> > > > Thanks Mickael!
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Apr 13, 2021 at 4:40 PM Mickael Maison 
> > > > 
> > > > wrote:
> > > >
> > > > > Thanks Rajini!
> > > > > Yes it would be good to get it into 3.0. I've got some of it
> > > > > implemented already, I'll try to get a PR out in the next couple of
> > > > > weeks.
> > > > >
> > > > > I'm closing this vote. This KIP has been accepted with 3 +1 binding
> > > > > votes from David, Tom and Rajini.
> > > > >
> > > > > On Tue, Apr 13, 2021 at 1:49 PM Rajini Sivaram 
> > > > > 
> > > > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks for the KIP!. Looks like this KIP has sufficient votes. It 
> > > > > > will be
> > > > > > good to get this into 3.0 along with KIP-709. Will you have time to 
> > > > > > work
> > > > > on
> > > > > > this? Please let us know if we can help with the implementation or
> > > > > reviews.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 18, 2021 at 4:00 PM Tom Bentley  
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I'd like to re-cast my vote as +1 (binding) now I'm a committer.
> > > > > > >
> > > > > > > Thanks again,
> > > > > > >
> > > > > > > Tom
> > > > > > >
> > > > > > > On Tue, Mar 2, 2021 at 9:46 AM David Jacot 
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the KIP, Mickael. +1 (binding)
> > > > > > > >
> > > > > > > > On Mon, Mar 1, 2021 at 11:53 AM Tom Bentley 
> > > > > > > > 
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding), thanks Mickael.
> > > > > > > > >
> > > > > > > > > On Thu, Feb 25, 2021 at 6:32 PM Mickael Maison <
> > > > > mimai...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'd like to start a vote on KIP-699 to support resolving 
> > > > > > > > > > multiple
> > > > > > > > > > coordinators at a time:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-699%3A+Update+FindCoordinator+to+resolve+multiple+Coordinators+at+a+time
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >

Re: [VOTE] KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics version config

[Sophie Blee-Goldman]

+1 (binding)

Thanks Bruno!

On Fri, May 21, 2021 at 8:06 AM Guozhang Wang  wrote:

> +1, thanks!
>
> On Fri, May 21, 2021 at 12:45 AM Bruno Cadonna  wrote:
>
> > Hi,
> >
> > I'd like to start a vote on KIP-743 that proposes to remove config value
> > 0.10.0-2.4 from Streams config built.in.metrics.version.
> >
> > https://cwiki.apache.org/confluence/x/uIfOCg
> >
> > Best,
> > Bruno
> >
>
>
> --
> -- Guozhang
>

Re: [REVIEW REQUESTED] - Pull Requests for KIP-633 and Addition of Missing Javadocs

[Israel Ekpo]

Thanks Luke for this feedback. I truly appreciate it.

There is always room for improvement and I will take this guidance into
consideration in the future.

Thanks Sophie for your initial feedback as well.

I will review these feedback and take appropriate action

On Fri, May 21, 2021 at 5:02 AM Luke Chen  wrote:

> Hi Israel,
> Thanks for submitting the PR.
> I've reviewed those 2 PRs and left some comments there.  I also see Sophie
> reviewed 1 of them. See if there are other guys available to review.
>
> But I'd like to suggest you look at the Contributing Code Changes
> <
> https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes#ContributingCodeChanges-TheReviewProcess
> >
> page first:
> *Consider identifying committers or other contributors who have worked on
> the code being changed. The easiest is to simply follow GitHub's automatic
> suggestions. You can add @username in the PR description to ping them
> immediately.*
>
> I don't think sending the review request mail to all dev group is a good
> idea*, *especially, your PR just opened less than 1 day.
>
> But still, we are glad to see your contribution to Kafka.
> Thank you very much!
>
> Luke
>
> On Fri, May 21, 2021 at 5:59 AM Israel Ekpo  wrote:
>
> > I just submitted my first two pull requests for the project today
> >
> > [KAFKA-12644] Add Missing Class-Level Javadoc to Exception Classes
> > https://github.com/apache/kafka/pull/10741
> >
> > [KAFKA-8613] Set default grace period to 0
> > KIP-633: Drop 24 hour default of grace period in Streams
> > https://github.com/apache/kafka/pull/10740
> >
> > When you have a moment, please take a look and share your feedback and
> > thoughts.
> >
> > Thanks
> >
>

[jira] [Created] (KAFKA-12836) Possible race in handling IP address change

[mlowicki (Jira)]

mlowicki created KAFKA-12836:


 Summary: Possible race in handling IP address change
 Key: KAFKA-12836
 URL: https://issues.apache.org/jira/browse/KAFKA-12836
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.4.1
Reporter: mlowicki


We've experienced few incidents where suddenly Kafka producers weren't able to 
send messages and were simply timing out. I've a repro now after some 
investigation. We're running the cluster of 3 brokers on K8s and it's enough to 
kill the underlying AWS VM using AWS Console which causes also IP change (in 
our case 10.128.132.196 → 10.128.135.90). It needs to be VM hosting broker 
which is a leader of partition (node 1).
 
Slightly before the kill there were 2 events:
 
{quote}About to close the idle connection from 10001 due to being idle for 
38545 millis
About to close the idle connection from 10002 due to being idle for 34946 
millis{quote}
 
And then client picks the node to connect to:
 
{quote}Removing node [10.128.140.202:9092|http://10.128.140.202:9092/] (id: 
10002 rack: us-east-1c) from least loaded node selection since it is neither 
ready for sending or connecting
Found least loaded connecting node 
[10.128.132.196:9092|http://10.128.132.196:9092/] (id: 1 rack: us-east-1a)
About to close the idle connection from 1 due to being idle for 30529 millis
Node 1 disconnected.
Initiating connection to node [10.128.132.196:9092|http://10.128.132.196:9092/] 
(id: 1 rack: us-east-1a) using address 
/[10.128.132.196|http://10.128.132.196/]{quote}
 
and then it repeats:
 
{quote}Found least loaded connecting node 
[10.128.132.196:9092|http://10.128.132.196:9092/] (id: 1 rack: us-east-1a)
About to close the idle connection from 1 due to being idle for 30027 millis
Node 1 disconnected.
Initiating connection to node [10.128.132.196:9092|http://10.128.132.196:9092/] 
(id: 1 rack: us-east-1a) using address 
/[10.128.132.196|http://10.128.132.196/]{quote}
 
{quote}Found least loaded connecting node 
[10.128.132.196:9092|http://10.128.132.196:9092/] (id: 1 rack: us-east-1a)
About to close the idle connection from 1 due to being idle for 30027 millis
Node 1 disconnected.
Initiating connection to node [10.128.132.196:9092|http://10.128.132.196:9092/] 
(id: 1 rack: us-east-1a) using address 
/[10.128.132.196|http://10.128.132.196/]{quote}
 
{quote}Found least loaded connecting node 
[10.128.132.196:9092|http://10.128.132.196:9092/] (id: 1 rack: us-east-1a)
About to close the idle connection from 1 due to being idle for 30027 millis
Node 1 disconnected.
Initiating connection to node [10.128.132.196:9092|http://10.128.132.196:9092/] 
(id: 1 rack: us-east-1a) using address 
/[10.128.132.196|http://10.128.132.196/]{quote}
 
Affected clients won't ever request METADATA from other brokers (10001 or 
10002) to discover new IP. It doesn't happen for all clients as some handle the 
process gracefully and those log with:
 
{quote}[log.info|http://log.info/]("Hostname for node {} changed from {} to 
{}.", id, connectionState.host(), host);{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics version config

[Guozhang Wang]

+1, thanks!

On Fri, May 21, 2021 at 12:45 AM Bruno Cadonna  wrote:

> Hi,
>
> I'd like to start a vote on KIP-743 that proposes to remove config value
> 0.10.0-2.4 from Streams config built.in.metrics.version.
>
> https://cwiki.apache.org/confluence/x/uIfOCg
>
> Best,
> Bruno
>


-- 
-- Guozhang

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #153

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12835) Topic IDs can mismatch on brokers (after interbroker protocol version update)

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-12835:
--

 Summary: Topic IDs can mismatch on brokers (after interbroker 
protocol version update)
 Key: KAFKA-12835
 URL: https://issues.apache.org/jira/browse/KAFKA-12835
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Ivan Yurchenko


We had a Kafka cluster running 2.8 version with interbroker protocol set to 
2.7. It had a number of topics and everything was fine.
Then we decided to update the interbroker protocol to 2.8 by the following 
procedure:
1. Run new brokers with the interbroker protocol set to 2.8.
2. Move the data from the old brokers to the new ones (normal partition 
reassignment API).
3. Decommission the old brokers.

At the stage 2 we had the problem: old brokers started failing on 
{{LeaderAndIsrRequest}} handling with
{code:java}
ERROR [Broker id=<...>] Topic Id in memory: <...> does not match the topic Id 
for partition <...> provided in the request: <...>. (state.change.logger)
{code}
for multiple topics. Topics were not recreated.

We checked {{partition.metadata}} files and IDs there were indeed different 
from the values in ZooKeeper. It was fixed by deleting the metadata files (and 
letting them be recreated).

 


The logs, unfortunately, didn't show anything that might point to the cause of 
the issue (or it happened longer ago than we store the logs).

We tried to reproduce this also, but no success.

If the community can point out what to check or beware of in future, it will be 
great. We'll be happy to provide additional information if needed. Thank you! 

Sorry for the ticket that might be not very actionable. We hope to at least 
rise awareness of this issue.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-737 Add canTrackSource to ReplicationPolicy

[Ryanne Dolan]

Perfect, thanks guys, will do!

On Fri, May 21, 2021, 6:30 AM Matthew de Detrich
 wrote:

> Thanks for this, I wasn't aware of the co-authored capabilities for
> git/github. In this case @Ryanne I don't think I need to split apart my
> commits, you can just add my email/name to the commit. Also make sure to
> add ivanyu (https://github.com/ivanyu) since he made the original PR/tests
> at https://github.com/apache/kafka/pull/9395, I just fixed/rebased them
> for
> the latest trunk.
>
> On Fri, May 21, 2021 at 12:48 PM Mickael Maison 
> wrote:
>
> > When merging, all commits will be squashed.
> > However, a committer can set "Co-authored-by" tags in the commit
> > message to credit all authors. Just make sure it's clearly stated in
> > the PR, so the committer is aware of it.
> >
> >
> >
> https://docs.github.com/en/github/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors
> >
> > On Fri, May 21, 2021 at 10:00 AM Matthew de Detrich
> >  wrote:
> > >
> > > Sure thing, i will split out my PR into 2 commits by EOD today so that
> > you
> > > can cherry pick the test commit in your PR.
> > >
> > > Thanks a lot for the work!
> > >
> > > On Fri, May 21, 2021 at 12:53 AM Ryanne Dolan 
> > wrote:
> > >
> > > > Matthew, I stole your integration tests to prove that point, but I
> > would
> > > > like to make sure you get credit for them in the commit history. If
> you
> > > > want to break out your tests into a separate commit, I can
> cherry-pick
> > them
> > > > into my PR.
> > > >
> > > > That would yield a PR with:
> > > > - a couple bug fixes from me in MirrorSourceConnector and
> MirrorClient
> > in
> > > > order to support IdentityReplicationPolicy
> > > > - my IdentityReplicationPolicy, based on yours and previous work, but
> > > > without the API change
> > > > - your tests
> > > >
> > > > ...which I think we shouldn't have trouble getting merged.
> > > >
> > > > We still have the open question of whether to call this
> > > > IdentityReplicationPolicy (which seems to be the general consensus)
> or
> > > > LegacyReplicationPolicy (which is what I called it back in KIP-382).
> > I'm
> > > > onboard with IdentityReplicationPolicy, but this [Discuss] thread is
> a
> > good
> > > > place to bring it up in case anyone objects.
> > > >
> > > > Ryanne
> > > >
> > > > On Thu, May 20, 2021 at 5:22 PM Matthew de Detrich
> > > >  wrote:
> > > >
> > > > > @Ryanne,
> > > > >
> > > > > I just noted you updated your PR at
> > > > >
> > > > >
> > > >
> >
> https://github.com/apache/kafka/pull/10652/files#diff-79a09517576a35906123533490ed39c0e1a9416878e284d7b71f5f4c53eeca29R31
> > > > > and I was mistaken in regards to the API changes being required. In
> > this
> > > > > case we can just use your PR instead of mine without the need for a
> > KIP.
> > > > >
> > > > > On Wed, May 19, 2021 at 11:12 AM Matthew de Detrich <
> > > > > matthew.dedetr...@aiven.io> wrote:
> > > > >
> > > > > > Hey Ryanne,
> > > > > >
> > > > > > Thanks for the reply, personally I have a slight preference for
> my
> > > > > > implementation since it doesn't require the "cheating" with the
> > > > > > remote.topic.suffix as you mentioned (this also makes my
> > > > implementation a
> > > > > > bit more clean/simple) but I am definitely not opposed to
> > adjusting to
> > > > > use
> > > > > > your method in order to avoid needing KIP (however as you stated
> in
> > > > order
> > > > > > to get all usecases we would have to create a KIP later anyways).
> > > > > >
> > > > > > What are your thoughts on seeing how the KIP works out, and if it
> > takes
> > > > > > too long or there is some issue with it we can go along with your
> > > > > > implementation as an initial step? Though this is my first time
> > doing a
> > > > > KIP
> > > > > > so I am not sure how long it typically takes to get one approved.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > On Wed, May 19, 2021 at 3:58 AM Ryanne Dolan <
> > ryannedo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hey Matthew, as you call out in the KIP there are few impls
> > floating
> > > > > >> around, including my WIP PR here:
> > > > > >>
> > > > > >> https://github.com/apache/kafka/pull/10652
> > > > > >>
> > > > > >> The tests are currently passing except for a couple asserts
> > related to
> > > > > >> failback (commented out). It appears your PR doesn't address
> > failback,
> > > > > so
> > > > > >> I
> > > > > >> think we can consider the two impls functionally equivalent,
> more
> > or
> > > > > less.
> > > > > >>
> > > > > >> However, I've cheated a bit here: I've added a
> > "remote.topic.suffix"
> > > > > >> property and have the integration tests configured to use it. So
> > I can
> > > > > get
> > > > > >> active/active replication to _mostly_ work with my impl, but
> > that's
> > > > not
> > > > > >> really a requirement per se. It's fine with me if
> > > > > >> Identity/LegacyReplicationPolicy explicitly only supports
> > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #152

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-731: Record Rate Limiting for Kafka Connect

[Tom Bentley]

Hi Ryanne,

Thanks for the KIP. I can see this would be useful.

1. Can you elaborate on the life cycle of the RateLimiter interface (in the
Javadoc)? In particular it's not clear to me how calls to accumulate() and
throttleTime() can be interleaved (I assume arbitrarily).

2. It's a bit weird that there's a separate start(Time) method in addition
to the configure() inherited from Configurable. Perhaps passing the Time to
accumulate() would be simpler than needing a two stage configuration step,
even if it would be the same instance on every call. If start() really is
needed you should document that it's called after configure().

3. Maybe including the unit in the method name, i.e. throttleTimeMs(), to
avoid any ambiguity about how the result is interpreted?

4. The metrics: Are they windowed over some time period, if so, what?

5. No metrics for batch rates?

6. It doesn't seem to be stated, but I assume the throttle time used is the
maximum of the throttleTime() returned by all the limiters.

7. The configuration uses a different mechanism than for SMTs and also
requires to add three common configs (with a risk of collision with any
connector which already defines configs with these names). I think it might
be nicer to have a consistent configuration mechanism, so for example
  rate.limiters=record,batch
  rate.limiter.record.type=RecordRateLimiter
  rate.limiter.record.limit=123
  rate.limiter.batch.type=RecordBatchRateLimiter
  rate.limiter.batch.limit=456
This means there's only a single new common config, as the others depend on
the aliases used, so further collisions can be avoided.

8. A cluster where every connector has a quota could end up being
underutilised, yet a subset of connectors could be running at their limit.
While this makes sense for the firehose problem it seems to be problematic
for the noisy neighbour case, where the spare capacity could be shared
between all the throttled tasks on the worker. While I'm not suggesting you
need to implement this as part of the KIP, maybe the API could accommodate
it being added later. Perhaps this could be as simple as using
hard.rate.limiters rather than just rate.limiters, so that
soft.rate.limiters could be added later, though maybe there are use cases
where a single limiter needs to supply both soft and hard limits.

Thanks again,

Tom

On Fri, May 14, 2021 at 6:26 PM Ryanne Dolan  wrote:

> Hey y'all, I've expanded the scope of this KIP slightly to include a
> pluggable interface, RateLimiter.
>
> After implementing this a few different ways, it's clear that the
> configuration story is actually simpler with a pluggable model.
> Out-of-the-box, we have just two configuration properties to tweak:
> record.rate.limit and record.batch.rate.limit (subj to change ofc). These
> are provided by built-in RecordRateLimiter and RecordBatchRateLimiter
> impls.
>
> From there, additional custom RateLimiters can be enabled with whatever
> configuration they need. This is essentially the same pattern taken with
> MetricsReporters and others.
>
> I had originally envisioned that the set of built-in limits would expand
> over time, eg individual put/poll/commit/flush limits. However, these can
> all be throttled adequately with the proposed API by limiting overall
> record and batch thruput.
>
> Please let me know what you think. The voting thread is open.
>
> Ryanne
>
> On Fri, Apr 9, 2021, 1:41 PM Ryanne Dolan  wrote:
>
> > Hey y'all, I'd like to draw you attention to a new KIP:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect
> >
> > Lemme know what you think. Thanks!
> >
> > Ryanne
> >
>

Re: [DISCUSS] KIP-737 Add canTrackSource to ReplicationPolicy

[Matthew de Detrich]

Thanks for this, I wasn't aware of the co-authored capabilities for
git/github. In this case @Ryanne I don't think I need to split apart my
commits, you can just add my email/name to the commit. Also make sure to
add ivanyu (https://github.com/ivanyu) since he made the original PR/tests
at https://github.com/apache/kafka/pull/9395, I just fixed/rebased them for
the latest trunk.

On Fri, May 21, 2021 at 12:48 PM Mickael Maison 
wrote:

> When merging, all commits will be squashed.
> However, a committer can set "Co-authored-by" tags in the commit
> message to credit all authors. Just make sure it's clearly stated in
> the PR, so the committer is aware of it.
>
>
> https://docs.github.com/en/github/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors
>
> On Fri, May 21, 2021 at 10:00 AM Matthew de Detrich
>  wrote:
> >
> > Sure thing, i will split out my PR into 2 commits by EOD today so that
> you
> > can cherry pick the test commit in your PR.
> >
> > Thanks a lot for the work!
> >
> > On Fri, May 21, 2021 at 12:53 AM Ryanne Dolan 
> wrote:
> >
> > > Matthew, I stole your integration tests to prove that point, but I
> would
> > > like to make sure you get credit for them in the commit history. If you
> > > want to break out your tests into a separate commit, I can cherry-pick
> them
> > > into my PR.
> > >
> > > That would yield a PR with:
> > > - a couple bug fixes from me in MirrorSourceConnector and MirrorClient
> in
> > > order to support IdentityReplicationPolicy
> > > - my IdentityReplicationPolicy, based on yours and previous work, but
> > > without the API change
> > > - your tests
> > >
> > > ...which I think we shouldn't have trouble getting merged.
> > >
> > > We still have the open question of whether to call this
> > > IdentityReplicationPolicy (which seems to be the general consensus) or
> > > LegacyReplicationPolicy (which is what I called it back in KIP-382).
> I'm
> > > onboard with IdentityReplicationPolicy, but this [Discuss] thread is a
> good
> > > place to bring it up in case anyone objects.
> > >
> > > Ryanne
> > >
> > > On Thu, May 20, 2021 at 5:22 PM Matthew de Detrich
> > >  wrote:
> > >
> > > > @Ryanne,
> > > >
> > > > I just noted you updated your PR at
> > > >
> > > >
> > >
> https://github.com/apache/kafka/pull/10652/files#diff-79a09517576a35906123533490ed39c0e1a9416878e284d7b71f5f4c53eeca29R31
> > > > and I was mistaken in regards to the API changes being required. In
> this
> > > > case we can just use your PR instead of mine without the need for a
> KIP.
> > > >
> > > > On Wed, May 19, 2021 at 11:12 AM Matthew de Detrich <
> > > > matthew.dedetr...@aiven.io> wrote:
> > > >
> > > > > Hey Ryanne,
> > > > >
> > > > > Thanks for the reply, personally I have a slight preference for my
> > > > > implementation since it doesn't require the "cheating" with the
> > > > > remote.topic.suffix as you mentioned (this also makes my
> > > implementation a
> > > > > bit more clean/simple) but I am definitely not opposed to
> adjusting to
> > > > use
> > > > > your method in order to avoid needing KIP (however as you stated in
> > > order
> > > > > to get all usecases we would have to create a KIP later anyways).
> > > > >
> > > > > What are your thoughts on seeing how the KIP works out, and if it
> takes
> > > > > too long or there is some issue with it we can go along with your
> > > > > implementation as an initial step? Though this is my first time
> doing a
> > > > KIP
> > > > > so I am not sure how long it typically takes to get one approved.
> > > > >
> > > > > Regards
> > > > >
> > > > > On Wed, May 19, 2021 at 3:58 AM Ryanne Dolan <
> ryannedo...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hey Matthew, as you call out in the KIP there are few impls
> floating
> > > > >> around, including my WIP PR here:
> > > > >>
> > > > >> https://github.com/apache/kafka/pull/10652
> > > > >>
> > > > >> The tests are currently passing except for a couple asserts
> related to
> > > > >> failback (commented out). It appears your PR doesn't address
> failback,
> > > > so
> > > > >> I
> > > > >> think we can consider the two impls functionally equivalent, more
> or
> > > > less.
> > > > >>
> > > > >> However, I've cheated a bit here: I've added a
> "remote.topic.suffix"
> > > > >> property and have the integration tests configured to use it. So
> I can
> > > > get
> > > > >> active/active replication to _mostly_ work with my impl, but
> that's
> > > not
> > > > >> really a requirement per se. It's fine with me if
> > > > >> Identity/LegacyReplicationPolicy explicitly only supports
> > > > active/passive.
> > > > >> In that case, we can drop the "remote.topic.suffix" stuff, which
> would
> > > > >> require a separate KIP anyway.
> > > > >>
> > > > >> I think that means we can take my ReplicationPolicy (minus
> > > > >> remote.topic.suffix) and your tests and get to a working state
> without
> > > > any
> > > > >> changes to the public 

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #122

[Apache Jenkins Server]

See 


Changes:

[Manikumar Reddy] KAFKA-12820: Upgrade maven-artifact dependency to resolve 
CVE-2021-26291


--
[...truncated 6.35 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Tom Bentley]

Hi Chris,

Thanks for continuing to entertain some of these ideas.

On Fri, May 14, 2021 at 5:06 PM Chris Egerton 
wrote:

> [...]
>
That's true, but we do go from three static ACLs (write/describe on a fixed
> transactional ID, and idempotent write on a fixed cluster) to a dynamic
> collection of ACLs.
>

I'm not quite sure I follow, maybe I've lost track. To be clear, I was
suggesting the use of a 'fencing producer' only in clusters with
exactly.once.source.enabled=true where I imagined the key difference
between the exactly once and fencing cases was how the producer was
configured/used (transactional vs this new fencing semantic). I think the
ACL requirements for connector producer principals would therefore be the
same as currently described in the KIP. The same is true for the worker
principals (which is the only breaking change you give in the KIP). So I
don't think the fencing idea changes the backwards compatibility story
that's already in the KIP, just allows a safe per-connector
exactly.once=disabled option to be supported (with required as requested as
we already discussed).

But I'm wondering whether I've overlooked something.

Ultimately I think it may behoove us to err on the side of reducing the
> breaking changes here for now and saving them for 4.0 (or some later major
> release), but would be interested in thoughts from you and others.
>

Difficult to answer (given I think I might be missing something).
If there are breaking changes then I don't disagree. It's difficult to
reason about big changes like this without some practical experience.
If there are not, then I think we could also implement the whole
exactly.once=disabled thing in a later KIP without additional breaking
changes (i.e. some time in 3.x), right?


> > Gouzhang also has a (possible) use case for a fencing-only producer (
> https://issues.apache.org/jira/browse/KAFKA-12693), and as he points out
> there, you should be able to get these semantics today by calling
> initTransactions() and then just using the producer as normal (no
> beginTransaction()/abortTransaction()/endTransaction()).
>
> I tested this locally and was not met with success; transactional producers
> do a check right now to ensure that any calls to "KafkaProducer::send"
> occur within a transaction (see
>
> https://github.com/apache/kafka/blob/29c55fdbbc331bbede17908ccc878953a1b15d87/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L957-L959
> and
>
> https://github.com/apache/kafka/blob/29c55fdbbc331bbede17908ccc878953a1b15d87/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L450-L451
> ).
> Not a blocker, just noting that we'd have to do some legwork to make this
> workable with the producer API.
>

Ah, sorry, I should have actually tried it rather than just taking a quick
look at the code.

Rather than remove those safety checks I suppose we'd need a way of
distinguishing, in the config, the difference in semantics. E.g. Something
like a fencing.id config, which was mutually exclusive with transactional.id.
Likewise perhaps initFencing() alongside initTransactions() in the API. But
I think at this point it's something for another KIP.

Kind regards,

Tom

Re: [DISCUSS] KIP-737 Add canTrackSource to ReplicationPolicy

[Mickael Maison]

When merging, all commits will be squashed.
However, a committer can set "Co-authored-by" tags in the commit
message to credit all authors. Just make sure it's clearly stated in
the PR, so the committer is aware of it.

https://docs.github.com/en/github/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

On Fri, May 21, 2021 at 10:00 AM Matthew de Detrich
 wrote:
>
> Sure thing, i will split out my PR into 2 commits by EOD today so that you
> can cherry pick the test commit in your PR.
>
> Thanks a lot for the work!
>
> On Fri, May 21, 2021 at 12:53 AM Ryanne Dolan  wrote:
>
> > Matthew, I stole your integration tests to prove that point, but I would
> > like to make sure you get credit for them in the commit history. If you
> > want to break out your tests into a separate commit, I can cherry-pick them
> > into my PR.
> >
> > That would yield a PR with:
> > - a couple bug fixes from me in MirrorSourceConnector and MirrorClient in
> > order to support IdentityReplicationPolicy
> > - my IdentityReplicationPolicy, based on yours and previous work, but
> > without the API change
> > - your tests
> >
> > ...which I think we shouldn't have trouble getting merged.
> >
> > We still have the open question of whether to call this
> > IdentityReplicationPolicy (which seems to be the general consensus) or
> > LegacyReplicationPolicy (which is what I called it back in KIP-382). I'm
> > onboard with IdentityReplicationPolicy, but this [Discuss] thread is a good
> > place to bring it up in case anyone objects.
> >
> > Ryanne
> >
> > On Thu, May 20, 2021 at 5:22 PM Matthew de Detrich
> >  wrote:
> >
> > > @Ryanne,
> > >
> > > I just noted you updated your PR at
> > >
> > >
> > https://github.com/apache/kafka/pull/10652/files#diff-79a09517576a35906123533490ed39c0e1a9416878e284d7b71f5f4c53eeca29R31
> > > and I was mistaken in regards to the API changes being required. In this
> > > case we can just use your PR instead of mine without the need for a KIP.
> > >
> > > On Wed, May 19, 2021 at 11:12 AM Matthew de Detrich <
> > > matthew.dedetr...@aiven.io> wrote:
> > >
> > > > Hey Ryanne,
> > > >
> > > > Thanks for the reply, personally I have a slight preference for my
> > > > implementation since it doesn't require the "cheating" with the
> > > > remote.topic.suffix as you mentioned (this also makes my
> > implementation a
> > > > bit more clean/simple) but I am definitely not opposed to adjusting to
> > > use
> > > > your method in order to avoid needing KIP (however as you stated in
> > order
> > > > to get all usecases we would have to create a KIP later anyways).
> > > >
> > > > What are your thoughts on seeing how the KIP works out, and if it takes
> > > > too long or there is some issue with it we can go along with your
> > > > implementation as an initial step? Though this is my first time doing a
> > > KIP
> > > > so I am not sure how long it typically takes to get one approved.
> > > >
> > > > Regards
> > > >
> > > > On Wed, May 19, 2021 at 3:58 AM Ryanne Dolan 
> > > > wrote:
> > > >
> > > >> Hey Matthew, as you call out in the KIP there are few impls floating
> > > >> around, including my WIP PR here:
> > > >>
> > > >> https://github.com/apache/kafka/pull/10652
> > > >>
> > > >> The tests are currently passing except for a couple asserts related to
> > > >> failback (commented out). It appears your PR doesn't address failback,
> > > so
> > > >> I
> > > >> think we can consider the two impls functionally equivalent, more or
> > > less.
> > > >>
> > > >> However, I've cheated a bit here: I've added a "remote.topic.suffix"
> > > >> property and have the integration tests configured to use it. So I can
> > > get
> > > >> active/active replication to _mostly_ work with my impl, but that's
> > not
> > > >> really a requirement per se. It's fine with me if
> > > >> Identity/LegacyReplicationPolicy explicitly only supports
> > > active/passive.
> > > >> In that case, we can drop the "remote.topic.suffix" stuff, which would
> > > >> require a separate KIP anyway.
> > > >>
> > > >> I think that means we can take my ReplicationPolicy (minus
> > > >> remote.topic.suffix) and your tests and get to a working state without
> > > any
> > > >> changes to the public interface, wdyt?
> > > >>
> > > >> Ryanne
> > > >>
> > > >> On Mon, May 10, 2021 at 6:02 PM Matthew de Detrich
> > > >>  wrote:
> > > >>
> > > >> > Hello everyone.
> > > >> >
> > > >> > I have a KIP that involves adding a public method to the
> > > >> ReplicationPolicy
> > > >> > interface called canTrackSource. The intention behind creating this
> > > >> method
> > > >> > is to implement an IdentityReplicationPolicy (aka
> > > >> LegacyReplicationPolicy)
> > > >> > which is a MirrorMaker2 ReplicationPolicy that behaves the same way
> > as
> > > >> the
> > > >> > original MirrorMaker1 ReplicationPolicy. There is already a passing
> > > >> > implementation at https://github.com/apache/kafka/pull/10648.
> > > >> >
> > > 

[jira] [Resolved] (KAFKA-12522) Cast SMT should allow null value records to pass through

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-12522.

Fix Version/s: 3.0.0
   Resolution: Fixed

> Cast SMT should allow null value records to pass through 
> -
>
> Key: KAFKA-12522
> URL: https://issues.apache.org/jira/browse/KAFKA-12522
> Project: Kafka
>  Issue Type: Bug
>Reporter: Daniel Osvath
>Assignee: Daniel Osvath
>Priority: Minor
> Fix For: 3.0.0
>
>
> The [current Cast 
> SMT|https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java]
>  fails on a null record value (or a null record key), which is problematic 
> for tombstone records. When a tombstone record reaches the transformation the 
> error below is thrown:
> {code:java}
> Caused by: org.apache.kafka.connect.errors.DataException: Only Map objects 
> supported in absence of schema for [cast types], found: null
> at 
> org.apache.kafka.connect.transforms.util.Requirements.requireMap(Requirements.java:38)
> {code}
> Null values should instead be allowed to pass through as there is no cast 
> transformation to be done, with the benefit of allowing the connector to 
> handle the tombstone records as intended. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #155

[Apache Jenkins Server]

See 


Changes:

[Manikumar Reddy] KAFKA-12820: Upgrade maven-artifact dependency to resolve 
CVE-2021-26291


--
[...truncated 3.46 MB...]
org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #151

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 2.8 #28

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12834) Remove Deprecated method MockProcessorContext#setTimestamp

[Josep Prat (Jira)]

Josep Prat created KAFKA-12834:
--

 Summary: Remove Deprecated method MockProcessorContext#setTimestamp
 Key: KAFKA-12834
 URL: https://issues.apache.org/jira/browse/KAFKA-12834
 Project: Kafka
  Issue Type: Sub-task
  Components: streams-test-utils
Reporter: Josep Prat
 Fix For: 4.0.0


Method org.apache.kafka.streams.processor.MockProcessorContext#setTimestamp was 
deprecated in 3.0.0

See KAFKA-10062 and KIP-622.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12833) Remove Deprecated methods under TopologyTestDriver

[Josep Prat (Jira)]

Josep Prat created KAFKA-12833:
--

 Summary: Remove Deprecated methods under TopologyTestDriver
 Key: KAFKA-12833
 URL: https://issues.apache.org/jira/browse/KAFKA-12833
 Project: Kafka
  Issue Type: Sub-task
  Components: streams-test-utils
Reporter: Josep Prat
 Fix For: 4.0.0


The following methods were at least deprecated in 2.8
 * 
org.apache.kafka.streams.TopologyTestDriver.KeyValueStoreFacade#init(org.apache.kafka.streams.processor.ProcessorContext,
 org.apache.kafka.streams.processor.StateStore)
 * 
org.apache.kafka.streams.TopologyTestDriver.WindowStoreFacade#init(org.apache.kafka.streams.processor.ProcessorContext,
 org.apache.kafka.streams.processor.StateStore)

 

*Disclaimer,* these methods might have been deprecated for a longer time, but 
they were definitely moved to this new "hierarchy position" in version 2.8

 

Move from standalone class to inner class was done under KAFKA-12435



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12832) Remove Deprecated methods under RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter

[Josep Prat (Jira)]

Josep Prat created KAFKA-12832:
--

 Summary: Remove Deprecated methods under 
RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter
 Key: KAFKA-12832
 URL: https://issues.apache.org/jira/browse/KAFKA-12832
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


The following methods under were deprecated in version 3.0.0
 * 
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundCompactions

 * 
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setBaseBackgroundCompactions

 * 
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundCompactions

 * 
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundFlushes

 * 
org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundFlushes

 

See KAFKA-8897 and KIP-471

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [REVIEW REQUESTED] - Pull Requests for KIP-633 and Addition of Missing Javadocs

[Luke Chen]

Hi Israel,
Thanks for submitting the PR.
I've reviewed those 2 PRs and left some comments there.  I also see Sophie
reviewed 1 of them. See if there are other guys available to review.

But I'd like to suggest you look at the Contributing Code Changes

page first:
*Consider identifying committers or other contributors who have worked on
the code being changed. The easiest is to simply follow GitHub's automatic
suggestions. You can add @username in the PR description to ping them
immediately.*

I don't think sending the review request mail to all dev group is a good
idea*, *especially, your PR just opened less than 1 day.

But still, we are glad to see your contribution to Kafka.
Thank you very much!

Luke

On Fri, May 21, 2021 at 5:59 AM Israel Ekpo  wrote:

> I just submitted my first two pull requests for the project today
>
> [KAFKA-12644] Add Missing Class-Level Javadoc to Exception Classes
> https://github.com/apache/kafka/pull/10741
>
> [KAFKA-8613] Set default grace period to 0
> KIP-633: Drop 24 hour default of grace period in Streams
> https://github.com/apache/kafka/pull/10740
>
> When you have a moment, please take a look and share your feedback and
> thoughts.
>
> Thanks
>

[jira] [Created] (KAFKA-12831) Remove Deprecated method StateStore#init

[Josep Prat (Jira)]

Josep Prat created KAFKA-12831:
--

 Summary: Remove Deprecated method StateStore#init
 Key: KAFKA-12831
 URL: https://issues.apache.org/jira/browse/KAFKA-12831
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


The method 
org.apache.kafka.streams.processor.StateStore#init(org.apache.kafka.streams.processor.ProcessorContext,
 org.apache.kafka.streams.processor.StateStore) was deprected in version 2.7

 

See KAFKA-10562 and KIP-478

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-737 Add canTrackSource to ReplicationPolicy

[Matthew de Detrich]

Sure thing, i will split out my PR into 2 commits by EOD today so that you
can cherry pick the test commit in your PR.

Thanks a lot for the work!

On Fri, May 21, 2021 at 12:53 AM Ryanne Dolan  wrote:

> Matthew, I stole your integration tests to prove that point, but I would
> like to make sure you get credit for them in the commit history. If you
> want to break out your tests into a separate commit, I can cherry-pick them
> into my PR.
>
> That would yield a PR with:
> - a couple bug fixes from me in MirrorSourceConnector and MirrorClient in
> order to support IdentityReplicationPolicy
> - my IdentityReplicationPolicy, based on yours and previous work, but
> without the API change
> - your tests
>
> ...which I think we shouldn't have trouble getting merged.
>
> We still have the open question of whether to call this
> IdentityReplicationPolicy (which seems to be the general consensus) or
> LegacyReplicationPolicy (which is what I called it back in KIP-382). I'm
> onboard with IdentityReplicationPolicy, but this [Discuss] thread is a good
> place to bring it up in case anyone objects.
>
> Ryanne
>
> On Thu, May 20, 2021 at 5:22 PM Matthew de Detrich
>  wrote:
>
> > @Ryanne,
> >
> > I just noted you updated your PR at
> >
> >
> https://github.com/apache/kafka/pull/10652/files#diff-79a09517576a35906123533490ed39c0e1a9416878e284d7b71f5f4c53eeca29R31
> > and I was mistaken in regards to the API changes being required. In this
> > case we can just use your PR instead of mine without the need for a KIP.
> >
> > On Wed, May 19, 2021 at 11:12 AM Matthew de Detrich <
> > matthew.dedetr...@aiven.io> wrote:
> >
> > > Hey Ryanne,
> > >
> > > Thanks for the reply, personally I have a slight preference for my
> > > implementation since it doesn't require the "cheating" with the
> > > remote.topic.suffix as you mentioned (this also makes my
> implementation a
> > > bit more clean/simple) but I am definitely not opposed to adjusting to
> > use
> > > your method in order to avoid needing KIP (however as you stated in
> order
> > > to get all usecases we would have to create a KIP later anyways).
> > >
> > > What are your thoughts on seeing how the KIP works out, and if it takes
> > > too long or there is some issue with it we can go along with your
> > > implementation as an initial step? Though this is my first time doing a
> > KIP
> > > so I am not sure how long it typically takes to get one approved.
> > >
> > > Regards
> > >
> > > On Wed, May 19, 2021 at 3:58 AM Ryanne Dolan 
> > > wrote:
> > >
> > >> Hey Matthew, as you call out in the KIP there are few impls floating
> > >> around, including my WIP PR here:
> > >>
> > >> https://github.com/apache/kafka/pull/10652
> > >>
> > >> The tests are currently passing except for a couple asserts related to
> > >> failback (commented out). It appears your PR doesn't address failback,
> > so
> > >> I
> > >> think we can consider the two impls functionally equivalent, more or
> > less.
> > >>
> > >> However, I've cheated a bit here: I've added a "remote.topic.suffix"
> > >> property and have the integration tests configured to use it. So I can
> > get
> > >> active/active replication to _mostly_ work with my impl, but that's
> not
> > >> really a requirement per se. It's fine with me if
> > >> Identity/LegacyReplicationPolicy explicitly only supports
> > active/passive.
> > >> In that case, we can drop the "remote.topic.suffix" stuff, which would
> > >> require a separate KIP anyway.
> > >>
> > >> I think that means we can take my ReplicationPolicy (minus
> > >> remote.topic.suffix) and your tests and get to a working state without
> > any
> > >> changes to the public interface, wdyt?
> > >>
> > >> Ryanne
> > >>
> > >> On Mon, May 10, 2021 at 6:02 PM Matthew de Detrich
> > >>  wrote:
> > >>
> > >> > Hello everyone.
> > >> >
> > >> > I have a KIP that involves adding a public method to the
> > >> ReplicationPolicy
> > >> > interface called canTrackSource. The intention behind creating this
> > >> method
> > >> > is to implement an IdentityReplicationPolicy (aka
> > >> LegacyReplicationPolicy)
> > >> > which is a MirrorMaker2 ReplicationPolicy that behaves the same way
> as
> > >> the
> > >> > original MirrorMaker1 ReplicationPolicy. There is already a passing
> > >> > implementation at https://github.com/apache/kafka/pull/10648.
> > >> >
> > >> > It would be ideal for this change (if approved) to land for Kafka
> > 3.0.0
> > >> > since there is a change to a public interface but do not there
> aren't
> > >> any
> > >> > breaking changes.
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-737%3A+Add+canTrackSource+to+ReplicationPolicy
> > >> >
> > >> > Cheers
> > >> >
> > >> > --
> > >> >
> > >> > Matthew de Detrich
> > >> >
> > >> > *Aiven Deutschland GmbH*
> > >> >
> > >> > Immanuelkirchstraße 26, 10405 Berlin
> > >> >
> > >> > Amtsgericht Charlottenburg, HRB 209739 B
> > >> >
> > >> > *m:* +491603708037
> > >> >
> > >> > *w:* 

[jira] [Created] (KAFKA-12830) Remove Deprecated constructor in TimeWindowedDeserializer and TimeWindowedSerde

[Josep Prat (Jira)]

Josep Prat created KAFKA-12830:
--

 Summary: Remove Deprecated constructor in TimeWindowedDeserializer 
and TimeWindowedSerde
 Key: KAFKA-12830
 URL: https://issues.apache.org/jira/browse/KAFKA-12830
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


The single argument constructor of the following classes were deprecated in 
version 2.8:
 * 
org.apache.kafka.streams.kstream.TimeWindowedDeserializer#TimeWindowedDeserializer(org.apache.kafka.common.serialization.Deserializer)
 * 
org.apache.kafka.streams.kstream.WindowedSerdes.TimeWindowedSerde#TimeWindowedSerde(org.apache.kafka.common.serialization.Serde)

 

See KAFKA-10366 & KAFKA-9649 and KIP-659



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12829) Remove Deprecated methods under Topology

[Josep Prat (Jira)]

Josep Prat created KAFKA-12829:
--

 Summary: Remove Deprecated methods under Topology
 Key: KAFKA-12829
 URL: https://issues.apache.org/jira/browse/KAFKA-12829
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


The following methods were deprecated in version 2.7:
 * org.apache.kafka.streams.Topology#addProcessor(java.lang.String, 
org.apache.kafka.streams.processor.ProcessorSupplier, java.lang.String...) 
 * 
org.apache.kafka.streams.Topology#addGlobalStore(org.apache.kafka.streams.state.StoreBuilder,
 java.lang.String, org.apache.kafka.common.serialization.Deserializer, 
org.apache.kafka.common.serialization.Deserializer, java.lang.String, 
java.lang.String, org.apache.kafka.streams.processor.ProcessorSupplier)
 * 
org.apache.kafka.streams.Topology#addGlobalStore(org.apache.kafka.streams.state.StoreBuilder,
 java.lang.String, org.apache.kafka.streams.processor.TimestampExtractor, 
org.apache.kafka.common.serialization.Deserializer, 
org.apache.kafka.common.serialization.Deserializer, java.lang.String, 
java.lang.String, org.apache.kafka.streams.processor.ProcessorSupplier) 

 

See KAFKA-10605 and KIP-478.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12828) Remove Deprecated methods under KeyQueryMetadata

[Josep Prat (Jira)]

Josep Prat created KAFKA-12828:
--

 Summary: Remove Deprecated methods under KeyQueryMetadata
 Key: KAFKA-12828
 URL: https://issues.apache.org/jira/browse/KAFKA-12828
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


The following methods under KeyQueryMetadata were deprecated in version 2.7
 * org.apache.kafka.streams.KeyQueryMetadata#getActiveHost

 * org.apache.kafka.streams.KeyQueryMetadata#getStandbyHosts

 * org.apache.kafka.streams.KeyQueryMetadata#getPartition

See KAFKA-10316 and KIP-648



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12827) Remove Deprecated method KafkaStreams#setUncaughtExceptionHandler

[Josep Prat (Jira)]

Josep Prat created KAFKA-12827:
--

 Summary: Remove Deprecated method 
KafkaStreams#setUncaughtExceptionHandler
 Key: KAFKA-12827
 URL: https://issues.apache.org/jira/browse/KAFKA-12827
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


++Method 
org.apache.kafka.streams.KafkaStreams#setUncaughtExceptionHandler(java.lang.Thread.UncaughtExceptionHandler)
  was deprecated in 2.8

 

See KAFKA-9331

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12826) Remove Deprecated Class Serdes (Streams)

[Josep Prat (Jira)]

Josep Prat created KAFKA-12826:
--

 Summary: Remove Deprecated Class Serdes (Streams)
 Key: KAFKA-12826
 URL: https://issues.apache.org/jira/browse/KAFKA-12826
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


Class org.apache.kafka.streams.scala.Serdes was deprecated in version 2.7

See KAFKA-10020 and KIP-616



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12825) Remove Deprecated method StreamsBuilder#addGlobalStore

[Josep Prat (Jira)]

Josep Prat created KAFKA-12825:
--

 Summary: Remove Deprecated method StreamsBuilder#addGlobalStore
 Key: KAFKA-12825
 URL: https://issues.apache.org/jira/browse/KAFKA-12825
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


Method org.apache.kafka.streams.scala.StreamsBuilder#addGlobalStore was 
deprecated in 2.7

 

See KAFKA-10379 and KIP-478



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12824) Remove Deprecated method KStream#branch

[Josep Prat (Jira)]

Josep Prat created KAFKA-12824:
--

 Summary: Remove Deprecated method KStream#branch
 Key: KAFKA-12824
 URL: https://issues.apache.org/jira/browse/KAFKA-12824
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


The method org.apache.kafka.streams.scala.kstream.KStream#branch was deprecated 
in version 2.8

 

See KAFKA-5488 and KIP-418



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12823) Remove Deprecated method KStream

[Josep Prat (Jira)]

Josep Prat created KAFKA-12823:
--

 Summary: Remove Deprecated method KStream
 Key: KAFKA-12823
 URL: https://issues.apache.org/jira/browse/KAFKA-12823
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


org.apache.kafka.streams.scala.kstream.KStream#through // 2.6



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12822) Remove Deprecated APIs of Kafka Streams in 4.0

[Josep Prat (Jira)]

Josep Prat created KAFKA-12822:
--

 Summary: Remove Deprecated APIs of Kafka Streams in 4.0
 Key: KAFKA-12822
 URL: https://issues.apache.org/jira/browse/KAFKA-12822
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Josep Prat
 Fix For: 4.0.0


This is an umbrella ticket that tries to collect all APIs under Kafka Streams 
that were deprecated after 2.5 (the current threshold for being removed in 
version 3.0.0).

Each subtask will de focusing on a specific API, so it's easy to discuss if it 
should be removed by 4.0.0 or maybe even at a later point.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics version config

[Bruno Cadonna]

Hi,

I'd like to start a vote on KIP-743 that proposes to remove config value 
0.10.0-2.4 from Streams config built.in.metrics.version.


https://cwiki.apache.org/confluence/x/uIfOCg

Best,
Bruno

Re: [DISCUSS] KIP-743: Deprecate config value 0.10.0-2.4 of Streams built-in metrics version config

[Bruno Cadonna]

Hi,

Thank you for the discussion. I am fine with removing the config value 
completely in 3.0.0. I was a bit conservative by treating the config 
value as any other config value. The points you all raise make sense and 
I will remove the config value right away. I will not remove the config, 
though, because I agree with Sophie and Guozhang that we might need it 
in future. We can remove the config in a future KIP when it is clearer 
if we still need it or not.


Next time, we should immediatley deprecate the older metrics version.

I will update the KIP and start voting.

Best,
Bruno


On 21.05.21 01:49, Sophie Blee-Goldman wrote:

I agree with John's assessment -- if I'm interpreting it correctly,
otherwise I'll claim this idea
as my own :P -- that we can just drop it in 3.0 as it was effectively
deprecated since day 1,
ie version 2.5. In addition to what John said about this not being the
default, I'll note that we
never directly referenced this config in the upgrade guide, nor do we call
out the old vs. new
metrics format in the monitoring guide. So for a user to have learned about
this new config
figure out how to configure Streams to use the old metrics format, I have
to assume they
did so by reading about it in the KIP
,
which specifically introduces this config as a way  "to
allow users having a grace period of changing their corresponding
monitoring / alerting
eco-systems", which seems to make it pretty clear that this was intended as
a temporary
config that would be removed after this "grace period"

And just to clarify, by "config" above I mostly refer to the specific
"0.10.0-2.4" value and not
the config itself. I think Guozhang is right that we may want to leverage
it again in the future
and do not/should not need to remove it now.

On Thu, May 20, 2021 at 3:57 PM Guozhang Wang  wrote:


I'm also a bit leaning towards dropping the config value in 3.0 directly.

For the config itself, I'd rather keep it a bit longer in case we may
update the metrics again (there are a couple near term projects on the
roadmap including named topology, restoration threads etc).


Guozhang

On Thu, May 20, 2021 at 8:50 AM John Roesler  wrote:


Thanks for opening this discussion, Bruno!

I wonder if it would be ok to simply drop that config
value in 3.0 instead of 4.0.

Config values are a bit different than Java APIs, since many
users would be using property files or string constants
instead of referencing the "official" constants defined in
StreamsConfig.java. As a result, we can expect the
deprecation warning on the java constant to be only
marginally effective.

Plus, the config value "latest" has been the default
already. I don't think it's a stretch for users of the
"0.10.0-2.4" config value to have seen this coming and to
have migrated to "latest" already.

If you do agree that we can treat it as having been
deprecated since 2.5, I'd further suggest that we no longer
need the "built.in.metrics.version" config at all.

WDYT?
Thanks,
John

On Thu, 2021-05-20 at 13:57 +0200, Bruno Cadonna wrote:

Hi,

I would like to propose KIP-743 to deprecate the old structures of the
built-in metrics in Streams.

https://cwiki.apache.org/confluence/x/uIfOCg

Best,
Bruno






--
-- Guozhang

[jira] [Resolved] (KAFKA-12820) Upgrade maven-artifact dependency to resolve CVE-2021-26291

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-12820.
---
Fix Version/s: 2.8.1
   2.7.2
   2.6.3
   3.0.0
   Resolution: Fixed

> Upgrade maven-artifact dependency to resolve CVE-2021-26291
> ---
>
> Key: KAFKA-12820
> URL: https://issues.apache.org/jira/browse/KAFKA-12820
> Project: Kafka
>  Issue Type: Task
>  Components: build
>Affects Versions: 2.6.1, 2.8.0, 2.7.1
>Reporter: Boojapho
>Assignee: Dongjin Lee
>Priority: Major
> Fix For: 3.0.0, 2.6.3, 2.7.2, 2.8.1
>
>
> Current Gradle builds of Kafka contain a dependency of `maven-artifact` 
> version 3.6.3, which contains CVE-2021-26291 
> ([https://nvd.nist.gov/vuln/detail/CVE-2021-26291).]  This vulnerability has 
> been fixed in Maven 3.8.1 
> ([https://maven.apache.org/docs/3.8.1/release-notes.html]).  Apache Kafka 
> should update `dependencies.gradle` to use the latest `maven-artifact` 
> library to eliminate this vulnerability.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)