[jira] [Created] (KAFKA-14059) Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
Chris Egerton created KAFKA-14059: - Summary: Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest Key: KAFKA-14059 URL: https://issues.apache.org/jira/browse/KAFKA-14059 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Chris Egerton -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14060) Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest
Chris Egerton created KAFKA-14060: - Summary: Replace EasyMock and PowerMock with Mockito in AbstractWorkerSourceTaskTest Key: KAFKA-14060 URL: https://issues.apache.org/jira/browse/KAFKA-14060 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Chris Egerton -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14058) Replace EasyMock and PowerMock with Mockito in ExactlyOnceWorkerSourceTaskTest
Chris Egerton created KAFKA-14058: - Summary: Replace EasyMock and PowerMock with Mockito in ExactlyOnceWorkerSourceTaskTest Key: KAFKA-14058 URL: https://issues.apache.org/jira/browse/KAFKA-14058 Project: Kafka Issue Type: Sub-task Components: KafkaConnect Reporter: Chris Egerton -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] KIP-851: : Add requireStable flag into ListConsumerGroupOffsetsOptions
Hello everyone, I'm ending this KIP vote as accepted with 4 binding +1s (John, Bruno, David, Luke). The wiki page has been updated and the corresponding code merged to trunk. Thanks! Guozhang On Wed, Jul 6, 2022 at 7:02 PM Luke Chen wrote: > Hi Guozhang, > > Removing the script-extention from the KIP is good to me. > +1 from me, too. > > Thank you. > Luke > > On Thu, Jul 7, 2022 at 7:39 AM Guozhang Wang wrote: > > > Hello folks, > > > > I tried to implement the script-extension along with the unit test > coverage > > on DescribeConsumerGroupTest, but it turns out more complicated than I > > anticipated due to the fact that we need to make sure the > `require-stable` > > flag is only effective for describing consumers. This also makes me > feeling > > that maybe my original thought of just adding that option for > "--describe" > > may not be comprehensive and we may need to think through under which > > action the additional option should be allowed. So I would remove this > part > > of the KIP and continue the voting thread. > > > > I've updated the KIP addressing the renaming suggestion. And I will close > > this thread as accepted with three binding votes (John, David, Bruno) if > I > > don't hear from you for more suggestions. > > > > Thanks, > > Guozhang > > > > > > > > On Mon, Jul 4, 2022 at 1:57 AM Bruno Cadonna wrote: > > > > > Hi, > > > > > > I would prefer to not include the script-extension into the KIP if you > > > you cannot commit to its implementation. I think partially implemented > > > KIPs make release management harder. If we can avoid implementing KIPs > > > partially, we should do it. > > > > > > I am +1 either way. I just wanted to bring this up. > > > > > > Best, > > > Bruno > > > > > > On 04.07.22 04:37, Luke Chen wrote: > > > > Hi Guozhang, > > > > > > > >> We can add it into this proposal though I could not commit to > > > implementing > > > > it myself with all the `DescribeConsumerGroupTest` coverage after > it's > > > > accepted, instead I could add a JIRA ticket under this KIP for others > > > who's > > > > interested to chime in. What do you think? > > > > > > > > Sounds good to me. > > > > > > > > Thank you. > > > > Luke > > > > > > > > On Mon, Jul 4, 2022 at 3:44 AM Guozhang Wang > > wrote: > > > > > > > >> Thanks for folks for your input ! > > > >> > > > >> 1) I'm happy to change the setter names to be consistent with the > > > >> topicPartition ones. I used a different name for getter from setter > > as I > > > >> remember seeing some other options differentiating function names > for > > > >> getter and setters, while some other options seem to be more on just > > > >> keeping the names the same. After getting your feedback I think it's > > > better > > > >> to do the same for both getter / setters. > > > >> > > > >> 2) For the kafka-consumer-group.sh tool, I looked at > > > >> ConsumerGroupCommand#describeGroups, and I think it's appropriate to > > add > > > >> it. I'm planning to add it in the shell tool as: > > > >> > > > >> ``` > > > >> "Require brokers to hold on returning unstable offsets (due to > pending > > > >> transactions) but retry until timeout for stably committed offsets" > > > >> "Example: --bootstrap-server localhost:9092 --describe --group > group1 > > > >> --offsets --require-stable" > > > >> ``` > > > >> > > > >> We can add it into this proposal though I could not commit to > > > implementing > > > >> it myself with all the `DescribeConsumerGroupTest` coverage after > it's > > > >> accepted, instead I could add a JIRA ticket under this KIP for > others > > > who's > > > >> interested to chime in. What do you think? > > > >> > > > >> > > > >> Guozhang > > > >> > > > >> > > > >> On Fri, Jul 1, 2022 at 1:18 AM David Jacot > > > > > > > > >> wrote: > > > >> > > > >>> Hi Guozhang, > > > >>> > > > >>> Thanks for the KIP! > > > >>> > > > >>> I agree with Luke. `requireStable` seems more consistent. > > > >>> > > > >>> Regarding the kafka-consumer-group command line tool, I wonder if > > > >>> there is real value in doing it. We don't necessarily have to add > all > > > >>> the options to it but we could if it is proven to be useful. > Anyway, > > I > > > >>> would leave it for a future KIP. > > > >>> > > > >>> +1 (binding) > > > >>> > > > >>> Best, > > > >>> David > > > >>> > > > >>> On Fri, Jul 1, 2022 at 9:47 AM Bruno Cadonna > > > wrote: > > > > > > Hi Guozhang, > > > > > > thank you for the KIP! > > > > > > I do not have strong feelings about the naming of the getter, but > I > > > >> tend > > > to agree with Luke. > > > > > > Regarding, the adaptation of the kafka-consumer-group.sh script, I > > am > > > fine if we leave that for a future KIP. > > > > > > +1 (binding) > > > > > > Best, > > > Bruno > > > > > > On 01.07.22 06:05, Luke Chen wrote: > > > > Hi Guozhang, > > > > > > > > Thanks for the KIP. > > > > Some comments: > > > > 1. I
Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol
Hi David, Thanks for sharing this KIP! Really exciting to hear how we are changing the protocol! The motivation section really made me realize how useful this change will be. I've done a first pass of the KIP, and may have more questions, but thought I'd start with a few I thought of already. - I saw some usages of topic IDs in the new protocols/records/interfaces, but wasn't sure if they were used everywhere. Are you planning on relying on topic IDs for the new protocol? - I saw the section about using a feature flag first before integrating the feature with ibp/metadata version. I understand the logic for testing with the flag, but it also seems like a bit of work to deprecate and switch to the ibp/metadata version approach. What was the reasoning behind switching the enablement mechanism? - Generally, are there implications for KRaft here? (IBP/metadata version is something that I think of) And if so, will both cluster types be supported? Thanks again to everyone who worked on this KIP! Justine On Wed, Jul 6, 2022 at 1:45 AM David Jacot wrote: > Hi all, > > I would like to start a discussion thread on KIP-848: The Next > Generation of the Consumer Rebalance Protocol. With this KIP, we aim > to make the rebalance protocol (for consumers) more reliable, more > scalable, easier to implement for clients, and easier to debug for > operators. > > The KIP is here: https://cwiki.apache.org/confluence/x/HhD1D. > > Please take a look and let me know what you think. > > Best, > David > > PS: I will be away from July 18th to August 8th. That gives you a bit > of time to read and digest this long KIP. >
[jira] [Created] (KAFKA-14057) Support dynamic reconfiguration in KRaft remote controllers
Ron Dagostino created KAFKA-14057: - Summary: Support dynamic reconfiguration in KRaft remote controllers Key: KAFKA-14057 URL: https://issues.apache.org/jira/browse/KAFKA-14057 Project: Kafka Issue Type: Task Reporter: Ron Dagostino We currently do not support dynamic reconfiguration of KRaft remote controllers. We only wire up brokers and react to metadata log changes there. We do no such wiring or reacting in a node where process.roles=controller. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14056) Test reading of old messages formats in ZK-to-KRaft upgrade test
Ron Dagostino created KAFKA-14056: - Summary: Test reading of old messages formats in ZK-to-KRaft upgrade test Key: KAFKA-14056 URL: https://issues.apache.org/jira/browse/KAFKA-14056 Project: Kafka Issue Type: Task Components: kraft Reporter: Ron Dagostino Whenever we support ZK-to-KRaft upgrade we must confirm that we can still read messages with an older message format. We can no longer write such messages as of IBP 3.0 (which is the minimum supported with KRaft), but we must still support reading such messages with KRaft. Therefore, the only way to test this would be to write the messages with a non-KRaft cluster, upgrade to KRaft, and then confirm we can read those messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] KIP-847: Add ProducerCount metrics
Hi, Artem, Thanks for the reply. It would be useful to add that clarification in the description of the metric. Other than that, the KIP looks good to me. Jun On Tue, Jul 5, 2022 at 5:57 PM Artem Livshits wrote: > I've updated the KIP to clarify that the metric reflects the total amount > of producer ids in all partitions maintained in the broker. > > -Artem > > On Thu, Jun 30, 2022 at 11:46 AM Jun Rao wrote: > > > Hi, Artem, > > > > Thanks for the reply. > > > > The memory usage on the broker is proportional to the number of > (partition, > > pid) combinations. So, I am wondering if we could have a metric that > > captures that. The proposed pid count metric doesn't fully capture that > > since each pid could be associated with a different number of partitions. > > > > Thanks, > > > > Jun > > > > On Thu, Jun 30, 2022 at 9:24 AM Justine Olshan > > > > wrote: > > > > > Hi Artem, > > > Thanks for the update to include motivation. Makes sense to me. > > > Justine > > > > > > On Wed, Jun 29, 2022 at 6:51 PM Luke Chen wrote: > > > > > > > Hi Artem, > > > > > > > > Thanks for the update. > > > > LGTM. > > > > > > > > Luke > > > > > > > > On Thu, Jun 30, 2022 at 6:51 AM Artem Livshits > > > > wrote: > > > > > > > > > Thank you for your feedback. I've updated the KIP to elaborate on > the > > > > > motivation and provide some background on producer ids and how we > > > measure > > > > > them. > > > > > > > > > > Also, after some thinking and discussing it offline with some > folks, > > I > > > > > think that we don't really need partitioner level metrics. We can > > use > > > > > existing tools to do granular debugging. I've moved partition > level > > > > > metrics to the rejected alternatives section. > > > > > > > > > > -Artem > > > > > > > > > > On Wed, Jun 29, 2022 at 1:57 AM Luke Chen > wrote: > > > > > > > > > > > Hi Artem, > > > > > > > > > > > > Could you elaborate more in the motivation section? > > > > > > I'm interested to know what kind of scenarios this metric can > > benefit > > > > > for. > > > > > > What could it bring to us when a topic partition has 100 > > > > ProducerIdCount > > > > > VS > > > > > > another topic partition has 10 ProducerIdCount? > > > > > > > > > > > > Thank you. > > > > > > Luke > > > > > > > > > > > > On Wed, Jun 29, 2022 at 6:30 AM Jun Rao > > > > > > wrote: > > > > > > > > > > > > > Hi, Artem, > > > > > > > > > > > > > > Thanks for the KIP. > > > > > > > > > > > > > > Could you explain the partition level ProducerIdCount a bit > more? > > > > Does > > > > > > that > > > > > > > reflect the number of PIDs ever produced to a partition since > the > > > > > broker > > > > > > is > > > > > > > started? Do we reduce the count after a PID expires? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > On Wed, Jun 22, 2022 at 1:08 AM David Jacot > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Artem, > > > > > > > > > > > > > > > > The KIP LGTM. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > David > > > > > > > > > > > > > > > > On Tue, Jun 21, 2022 at 9:32 PM Artem Livshits > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > If there is no other feedback I'm going to start voting in > a > > > > couple > > > > > > > days. > > > > > > > > > > > > > > > > > > -Artem > > > > > > > > > > > > > > > > > > On Fri, Jun 17, 2022 at 3:50 PM Artem Livshits < > > > > > > alivsh...@confluent.io > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thank you for your feedback. Updated the KIP and added > the > > > > > > Rejected > > > > > > > > > > Alternatives section. > > > > > > > > > > > > > > > > > > > > -Artem > > > > > > > > > > > > > > > > > > > > On Fri, Jun 17, 2022 at 1:16 PM Ismael Juma < > > > ism...@juma.me.uk > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> If we don't track them separately, then it makes sense > to > > > keep > > > > > it > > > > > > as > > > > > > > > one > > > > > > > > > >> metric. I'd probably name it ProducerIdCount in that > case. > > > > > > > > > >> > > > > > > > > > >> Ismael > > > > > > > > > >> > > > > > > > > > >> On Fri, Jun 17, 2022 at 12:04 PM Artem Livshits > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > >> > Do you propose to have 2 metrics instead of one? > Right > > > now > > > > we > > > > > > > don't > > > > > > > > > >> track > > > > > > > > > >> > if the producer id was transactional or idempotent and > > for > > > > > > metric > > > > > > > > > >> > collection we'd either have to pay the cost of > iterating > > > > over > > > > > > > > producer > > > > > > > > > >> ids > > > > > > > > > >> > (which could be a lot) or split the producer map into > 2 > > or > > > > > cache > > > > > > > the > > > > > > > > > >> > counts, which complicates the code. > > > > > > > > > >> > > > > > > > > > > >> > From the monitoring perspective, I think one metric > > should > > > > be > > >
Re: Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms
Regarding the option to integrate repair logic in "latest", I understand your concern about this approach: backward compatibility. But we should have a consensus: the problem of data loss due to expand partitions is indeed caused by kafka's own design mechanism. The user configuration "latest" may be due to the consideration of not wanting to consume from earliest when firstly deploy app, or too much lag, or consumption exceeds the maximum offset, and then consume directly from the latest. As for expanding partition, the user will definitely not want to consume from the latest, unless he clearly knows what this means. Therefore, it is necessary to solve this problem, at the same time, without causing other problems. Therefore, for the method of adding an "init.offset.reset" option, there will be a problem, that is, this configuration must be set to "earliest" to avoid this situation, but it will also cause the new group to be consumed from earliest. , which goes against the idea of consuming from the latest at the beginning (brings other problems). The same is true for the method of setting auto.offset.reset to "earliest" and seekingToEnd on new deployments: in order to avoid this case, "auto.offset.reset" has no choice but to set "earliest", when the consumption is advanced, it will also reset to the earliest, causing duplication (bringing other problems). So I think it's best to fix it in a black box to fundamentally solve it. It does not require users to perceive this problem, nor does the user's understanding of "auto.offset.reset" need to be changed, and there will be no complexity caused by redundant parameter configuration(and users doesn't necessarily know how to combine these parameters to use it). As for the compatibility issue, I think it is enough to enrich the test cases after the repair, what do you think? Matthias J. Sax mj...@apache.org写道： > I am not sure if we can/should change the behavior of existing > "latest/earliest" due to backward compatibility concerns. While I agree > that many users might not know the fine details how both behave, it > would still be a change that might break other people that do understand > the details and rely on it. > > I also agree that having both "latest" and "safe_latest" might be > difficult, as users might not know which one to choose? > > Maybe we should have two configs instead of one? `auto.offset.reset`, as > the name suggests, resets the offset automatically, and thus it's > current behavior is actually well defined and sound. -- What seems to be > missing is an `init.offset` config that is only used if there is _no_ > committed offsets, but it's not used when the consumer has a position > already (either via getting a committed offset or via seek())? > > > For the original use-case you mentioned, that you want to start from > "latest" when the app starts, but if a new partition is added you want > to start from "earliest" it seem that the right approach would be to > actually configure "earliest", and when the app is deployed for the > first time, use a `seekToEnd()` to avoid triggering auto-offset-reset? > > > Thoughts? > > > -Matthias > > > On 7/1/22 6:03 AM, hudeqi wrote: > > Thanks for your attention and reply. > > Having chatted with Guozhang Wang at KAFKA-12478 before, I came up with an > > idea similar to yours. It's just not implemented on the client side, but on > > the server side: Firstly, find out all the groups subscribed to this topic > > before extending partitions, and then let these groups commit an initial > > offset 0 for these new expanded partitions (also using adminClient). > > Finally, the real process of adding partitions is carried out. In this way, > > the problem can also be completely solved. > > > > Best, > > hudeqi > > > > Matthew Howlett m...@confluent.io.INVALID写道： > >> My first reaction also is that the proposed configuration is surely too > >> complicated. > >> > >> It seems like an ideal solution from a usability perspective (always a good > >> place to start) would be if the consumer just automatically behaved in this > >> way. To make that work: > >> 1. auto.offset.reset=latest would need to behave like > >> auto.offset.reset=earliest in the case where a consumer is in a group, and > >> is assigned a newly created partition. This might seem a bit too "magic", > >> but from the perspective of the group, I think it makes conceptual sense > >> and people wouldn't find it surprising. Also, I don't think anyone would be > >> relying on the old behavior. > >> 2. The group would need to detect the newly created partitions and > >> rebalance pretty quickly (this is not the case currently). The longer the > >> delay, the more tenuous the idea of changing the auto.offset.reset behavior > >> in this special circumstance. > >> > >> I have a feeling this approach has implementation challenges (haven't > >> thought deeply), just throwing it out there. > >> > >> > >>
Re: [DISCUSS] Apache Kafka 3.3.0 Release
Thank you, Jose. Regards, Rajini On Fri, Jul 8, 2022 at 2:43 PM José Armando García Sancio wrote: > On Fri, Jul 8, 2022 at 6:37 AM Rajini Sivaram > wrote: > > We have a PR for batched offset fetch API, which is part of KIP-709 ( > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173084258 > ) > > that was opened a year ago, but didn't get merged because we didn't have > > time to merge into 3.0: https://github.com/apache/kafka/pull/10964. The > > broker-side change of this KIP was merged into AK 3.0, so it will be good > > to include the client-side changes to make use of the batched protocol. I > > can get the PR rebased and ready to merge early next week. Since the KIP > > was approved before KIP freeze and the PR was ready before feature > freeze, > > can we include this in AK 3.3.0 please? > > Thanks for the update Rajini. Targeting the improvement for 3.3.0 > sounds good to me. Look forward to getting this admin client > improvement as part of 3.3.0. >
Re: [DISCUSS] Apache Kafka 3.3.0 Release
On Fri, Jul 8, 2022 at 6:37 AM Rajini Sivaram wrote: > We have a PR for batched offset fetch API, which is part of KIP-709 ( > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173084258) > that was opened a year ago, but didn't get merged because we didn't have > time to merge into 3.0: https://github.com/apache/kafka/pull/10964. The > broker-side change of this KIP was merged into AK 3.0, so it will be good > to include the client-side changes to make use of the batched protocol. I > can get the PR rebased and ready to merge early next week. Since the KIP > was approved before KIP freeze and the PR was ready before feature freeze, > can we include this in AK 3.3.0 please? Thanks for the update Rajini. Targeting the improvement for 3.3.0 sounds good to me. Look forward to getting this admin client improvement as part of 3.3.0.
Re: [DISCUSS] Apache Kafka 3.3.0 Release
Hi Jose, We have a PR for batched offset fetch API, which is part of KIP-709 ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173084258) that was opened a year ago, but didn't get merged because we didn't have time to merge into 3.0: https://github.com/apache/kafka/pull/10964. The broker-side change of this KIP was merged into AK 3.0, so it will be good to include the client-side changes to make use of the batched protocol. I can get the PR rebased and ready to merge early next week. Since the KIP was approved before KIP freeze and the PR was ready before feature freeze, can we include this in AK 3.3.0 please? Thank you, Rajini On Thu, Jul 7, 2022 at 3:37 PM José Armando García Sancio wrote: > On Thu, Jul 7, 2022 at 7:20 AM Sagar wrote: > > Pr for KIP-843 also got merged almost 24 days ago. There’s a follow up pr > > open for documentation changes. Could this also be added plz? > > Thank you. Added KIP-843 to the list of KIPs for 3.3.0. > > I will create a release branch for 3.3 next Monday, 11th of July. >
Re: Transactions, delivery timeout and changing transactional producer behavior
Submitted a PR with the fix: https://github.com/apache/kafka/pull/12392 In the PR I tried keeping the producer in a usable state after the forced bump. I understand that it might be the cleanest solution, but the only other option I know of is to transition into a fatal state, meaning that the producer has to be recreated after a delivery timeout. I think that is still fine compared to the out-of-order messages. Looking forward to your reviews, Daniel Dániel Urbán ezt írta (időpont: 2022. júl. 7., Cs, 12:04): > Thanks for the feedback, I created > https://issues.apache.org/jira/browse/KAFKA-14053 and started working on > a PR. > > Luke, for the workaround, we used the transaction admin tool released in > 3.0 to "abort" these hanging batches manually. > Naturally, the cluster health should be stabilized. This issue popped up > most frequently around times when some partitions went into a few minute > window of unavailability. The infinite retries on the producer side caused > a situation where the last retry was still in-flight, but the delivery > timeout was triggered on the client side. We reduced the retries and > increased the delivery timeout to avoid such situations. > Still, the issue can occur in other scenarios, for example a client > queueing up many batches in the producer buffer, and causing those batches > to spend most of the delivery timeout window in the client memory. > > Thanks, > Daniel > > Luke Chen ezt írta (időpont: 2022. júl. 7., Cs, 5:13): > >> Hi Daniel, >> >> Thanks for reporting the issue, and the investigation. >> I'm curious, so, what's your workaround for this issue? >> >> I agree with Artem, it makes sense. Please file a bug in JIRA. >> And looking forward to your PR! :) >> >> Thank you. >> Luke >> >> On Thu, Jul 7, 2022 at 3:07 AM Artem Livshits >> wrote: >> >> > Hi Daniel, >> > >> > What you say makes sense. Could you file a bug and put this info there >> so >> > that it's easier to track? >> > >> > -Artem >> > >> > On Wed, Jul 6, 2022 at 8:34 AM Dániel Urbán >> wrote: >> > >> > > Hello everyone, >> > > >> > > I've been investigating some transaction related issues in a very >> > > problematic cluster. Besides finding some interesting issues, I had >> some >> > > ideas about how transactional producer behavior could be improved. >> > > >> > > My suggestion in short is: when the transactional producer encounters >> an >> > > error which doesn't necessarily mean that the in-flight request was >> > > processed (for example a client side timeout), the producer should not >> > send >> > > an EndTxnRequest on abort, but instead it should bump the producer >> epoch. >> > > >> > > The long description about the issue I found, and how I came to the >> > > suggestion: >> > > >> > > First, the description of the issue. When I say that the cluster is >> "very >> > > problematic", I mean all kinds of different issues, be it infra (disks >> > and >> > > network) or throughput (high volume producers without fine tuning). >> > > In this cluster, Kafka transactions are widely used by many producers. >> > And >> > > in this cluster, partitions get "stuck" frequently (few times every >> > week). >> > > >> > > The exact meaning of a partition being "stuck" is this: >> > > >> > > On the client side: >> > > 1. A transactional producer sends X batches to a partition in a single >> > > transaction >> > > 2. Out of the X batches, the last few get sent, but are timed out >> thanks >> > to >> > > the delivery timeout config >> > > 3. producer.flush() is unblocked due to all batches being "finished" >> > > 4. Based on the errors reported in the producer.send() callback, >> > > producer.abortTransaction() is called >> > > 5. Then producer.close() is also invoked with a 5s timeout (this >> > > application does not reuse the producer instances optimally) >> > > 6. The transactional.id of the producer is never reused (it was >> random >> > > generated) >> > > >> > > On the partition leader side (what appears in the log segment of the >> > > partition): >> > > 1. The batches sent by the producer are all appended to the log >> > > 2. But the ABORT marker of the transaction was appended before the >> last 1 >> > > or 2 batches of the transaction >> > > >> > > On the transaction coordinator side (what appears in the transaction >> > state >> > > partition): >> > > The transactional.id is present with the Empty state. >> > > >> > > These happenings result in the following: >> > > 1. The partition leader handles the first batch after the ABORT >> marker as >> > > the first message of a new transaction of the same producer id + >> epoch. >> > > (LSO is blocked at this point) >> > > 2. The transaction coordinator is not aware of any in-progress >> > transaction >> > > of the producer, thus never aborting the transaction, not even after >> the >> > > transaction.timeout.ms passes. >> > > >> > > This is happening with Kafka 2.5 running in the cluster, producer >> > versions >> > > range between 2.0 and 2.6.
[jira] [Resolved] (KAFKA-13983) Support special character in Resource name in ACLs operation by sanitizing
[
https://issues.apache.org/jira/browse/KAFKA-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-13983.
---
Fix Version/s: 3.3.0
Reviewer: Manikumar
Resolution: Fixed
> Support special character in Resource name in ACLs operation by sanitizing
> ---
>
> Key: KAFKA-13983
> URL: https://issues.apache.org/jira/browse/KAFKA-13983
> Project: Kafka
> Issue Type: Bug
> Components: core
>Reporter: Aman Singh
>Assignee: Aman Singh
>Priority: Minor
> Fix For: 3.3.0
>
>
> Currently, resource names in ACLS can contain any special characters, but
> resource names with some special characters are not a valid zookeeper path
> entry.
> For example resource name {color:#de350b}{{test/true}} {color}is not a valid
> zookeeper path entry.
> Zookeeper will create a child node, name as {color:#de350b}{{true}}{color}
> inside the {color:#de350b}{{test}}{color} node.
> This will create two problems:-
> # If there is *one* ACL with a resource name {color:#de350b}{{test}}{color}
> it can't be deleted because if there is only one, Kafka tries to delete the
> node as well by thinking it will be empty which is not true it has the child
> node {{{color:#de350b}true{color}.}}
> # When broker restarts {color:#de350b}{{ACL cache}}{color}(which is used for
> ACL operations like describe, authorization etc) update from zookeeper and
> Kafka only looks for ACLs that are direct child nodes of resource type in
> the ACL tree.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] KIP-847
Thanks, Artem. I am still +1. On Fri, Jul 8, 2022 at 3:45 AM Artem Livshits wrote: > > Hello, > > There was an additional discussion and the KIP got changed as a result of > that. I would like to restart the vote on the updated > https://cwiki.apache.org/confluence/display/KAFKA/KIP-847%3A+Add+ProducerIdCount+metrics > . > > -Artem > > On Fri, Jun 24, 2022 at 7:49 PM Luke Chen wrote: > > > Hi Artem, > > > > Thanks for the KIP. > > +1 (binding) from me. > > > > In addition to the `ProducerIdCount` in motivation section, the KIP title > > should also be updated. > > > > Luke > > > > On Fri, Jun 24, 2022 at 8:33 PM David Jacot > > wrote: > > > > > Thanks for the KIP, Artem. > > > > > > I am +1 (binding). > > > > > > A small nit: ProducerIdCount should be used in the motivation. > > > > > > Best, > > > David > > > > > > On Thu, Jun 23, 2022 at 10:26 PM Artem Livshits > > > wrote: > > > > > > > > Hello, > > > > > > > > I'd like to start a vote on KIP-847 > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-847%3A+Add+ProducerCount+metrics > > > > > > > > -Artem > > > > >
Clarification Kafka
Dear Team, We wanted to use kafka and therefore installed on ubuntu ( Ubuntu 20.04.4 LTS ) kafka ( 3.2.0 & zoopkeeper 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a) . With this installation, we wanted to connect to mysql server installed on the same server. Is it possible? If so what document to follow ? Or We need to install and additional like confluent etc .. With warm regards Alalasundaram Saravanan
