[jira] [Created] (KAFKA-9986) Checkpointing API for State Stores

Nikolay Izhikov created KAFKA-9986: -- Summary: Checkpointing API for State Stores Key: KAFKA-9986 URL: https://issues.apache.org/jira/browse/KAFKA-9986 Project: Kafka Issue Type: New Feature

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi Tom, >> What exactly is the problem with having a huge backlog of pending >> operations? I can see that the backlog would need persisting so that the >> controller could change without losing track of the topics to be mutated, >> and the mutations would need to be submitted in batches to the

Build failed in Jenkins: kafka-2.4-jdk8 #200

See Changes: [jason] KAFKA-9669; Loosen validation of inner offsets for older message formats -- [...truncated 9.14 MB...] org.apache.kafka.streams.test.OutputVerifierTest

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi Jun, Coming back to your question regarding the differences between the token bucket algorithm and our current quota mechanism. I did some tests and they confirmed my first intuition that our current mechanism does not work well with a bursty workload. Let me try to illustrate the difference

Re: [DISCUSS] KIP-587 Suppress detailed responses for handled exceptions in security-sensitive environments

Hi Connor, I think this is really close but have one more thought. Uncaught exceptions in the REST API are different from exceptions that come about when tasks or connectors fail, and can be used for different purposes. Stack traces in 500 errors are probably only useful for the administrator of

[VOTE] KIP-606: Add Metadata Context to MetricsReporter

Hi everyone, Folks seem happy with the state of the KIP, so I'd like to start the vote for KIP-606 https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter - Xavier

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Thanks for the KIP Sophie. Getting the E2E latency is important for understanding the bottleneck of the application. A couple of questions and ideas: 1. Could you clarify the rational of picking 75, 99 and max percentiles? Normally I see cases where we use 50, 90 percentile as well in production

[jira] [Created] (KAFKA-9988) Log incorrectly reports task has failed when task takes too long to shutdown

Sanjana Kaundinya created KAFKA-9988: Summary: Log incorrectly reports task has failed when task takes too long to shutdown Key: KAFKA-9988 URL: https://issues.apache.org/jira/browse/KAFKA-9988

[jira] [Created] (KAFKA-9987) Add new cooperative assignor optimized for constant-subscription. group

Sophie Blee-Goldman created KAFKA-9987: -- Summary: Add new cooperative assignor optimized for constant-subscription. group Key: KAFKA-9987 URL: https://issues.apache.org/jira/browse/KAFKA-9987

[DISCUSS] KIP-615: add ConstrainedCooperativeStickyAssignor

Hey all, I'd like to propose adding another OOTB cooperative assignor to better meet the needs of some users who don't need to full flexibility of the existing CooperativeStickyAssignor: https://cwiki.apache.org/confluence/display/KAFKA/KIP-615%3A+add+ConstrainedCooperativeStickyAssignor I

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

Thanks Bruno! I took a look at the revised KIP and it looks good to me. Sophie On Wed, May 13, 2020 at 6:59 AM Bruno Cadonna wrote: > Hi John, > > Thank you for the feedback! > > I agree and I will change the KIP as I stated in my previous e-mail to > Guozhang. > > Best, > Bruno > > On Tue,

[jira] [Resolved] (KAFKA-8989) Embedded broker could not be reached in unit test

[ https://issues.apache.org/jira/browse/KAFKA-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-8989. Resolution: Won't Fix > Embedded broker could not be reached in unit test >

[jira] [Resolved] (KAFKA-9897) Flaky Test StoreQueryIntegrationTest#shouldQuerySpecificActivePartitionStores

[ https://issues.apache.org/jira/browse/KAFKA-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9897. -- Resolution: Fixed > Flaky Test

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

Hi David and Alexandre, Thanks so much for your feedback! Here are my answers: 1. Yes, we have seen several cases of clients that create a new connection per produce/consume request. One hypothesis is someone who is used to connection pooling may accidentally write a Kafka client that creates a

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

1. I felt that 50% was not a particularly useful gauge for this specific metric, as it's presumably most useful at putting an *upper *bound on the latency you can reasonably expect to see. I chose percentiles that would hopefully give a good sense of what *most* records will experience, and what

[jira] [Created] (KAFKA-9990) Supporting transactions in tiered storage

Satish Duggana created KAFKA-9990: - Summary: Supporting transactions in tiered storage Key: KAFKA-9990 URL: https://issues.apache.org/jira/browse/KAFKA-9990 Project: Kafka Issue Type:

[jira] [Resolved] (KAFKA-9465) Enclose consumer call with catching InvalidOffsetException

[ https://issues.apache.org/jira/browse/KAFKA-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-9465. Resolution: Not A Problem This was fixed implicitly via some code refactoring. > Enclose

[jira] [Created] (KAFKA-9991) Flaky Test KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled

Sophie Blee-Goldman created KAFKA-9991: -- Summary: Flaky Test KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled Key: KAFKA-9991 URL:

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Thanks Sophie, I hope this isn't too nit-picky, but is there a reason to choose "avg" instead of "mean"? Maybe this is too paranoid, and I might be oversensitive because of the mistake I just made earlier, but it strikes me that "avg" is actually ambiguous, as it refers to a family of statistics,

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Hello all, and thanks for the KIP, Sophie, Just some comments on the discussion so far: B2/G1: In principle, it shouldn't matter whether we report "spans" or "end-to-end" latency. But in practice, some of the spans are pretty difficult to really measure (like time spent waiting in the topics, or

Re: [DISCUSS] KIP-614: Add Prefix Scan support for State Stores

Not to derail this KIP discussion, but to leave a few notes on some of the RocksDB points that have come up: Someone actually merged some long overdue performance improvements to the RocksJava implementation (the PR was opened back in 2017! yikes). I haven't looked into the prefix seek API

Subscribe to Kafka dev mailing list

发自我的iPhone

[jira] [Resolved] (KAFKA-9688) kafka-topic.sh should show KIP-455 adding and removing replicas

[ https://issues.apache.org/jira/browse/KAFKA-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9688. - Fix Version/s: 2.5.0 Reviewer: Colin McCabe Resolution: Fixed > kafka-topic.sh

Build failed in Jenkins: kafka-trunk-jdk11 #1451

See Changes: -- Started by an SCM change Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on H30 (ubuntu) in workspace

Re: [DISCUSS] KIP-615: add ConstrainedCooperativeStickyAssignor

Putting this to rest as one of the shortest-lived KIPs, I've decided to just improve the existing CooperativeStickyAssignor rather than introduce an entire new one. Since this will not involve changing any public APIs there's no need for a KIP anymore. Let me know if you have any questions or

[jira] [Resolved] (KAFKA-9850) Move KStream#repartition operator validation during Topology build process

[ https://issues.apache.org/jira/browse/KAFKA-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-9850. Fix Version/s: 2.6.0 Resolution: Fixed > Move KStream#repartition operator

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Yeah, the specific reason was just to align with the current metrics. Is it better to conform than to be right? History has a lot to say on that matter but I'm not sure how much of it applies to the fine details of metrics naming :P More seriously, I figured if people are looking at this metric

Re: [DISCUSS] KIP-614: Add Prefix Scan support for State Stores

Thanks Adam, Sagar. I read your PR as well the rocksDB reference, and I have a few quick questions: 1. In your code I saw you did not specifically overwrite any rocksDB configs like `useFixedLengthPrefixExtractor`. Also, by comparing the `RocksDBPrefixIterator` and `RocksDBRangeIterator`

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Oh boy, I never miss an opportunity to embarrass myself. I guess the mean seems more interesting to me than the median, but neither are as interesting as the higher percentiles (99th and max). Min isn’t really important for any SLAs, but it does round out the mental picture of the

[jira] [Resolved] (KAFKA-9966) Flaky Test EosBetaUpgradeIntegrationTest#shouldUpgradeFromEosAlphaToEosBeta

[ https://issues.apache.org/jira/browse/KAFKA-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-9966. Fix Version/s: 2.6.0 Resolution: Fixed > Flaky Test

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Thanks Sophie for the KIP, a few quick thoughts: 1) The end-to-end latency includes both the processing latency of the task and the latency spent sitting in intermediate topics. I have a similar feeling as Boyang mentioned above that the latency metric of a task A actually measures the latency of

[jira] [Created] (KAFKA-9989) StreamsUpgradeTest.test_metadata_upgrade could not guarantee all processor gets assigned task

Boyang Chen created KAFKA-9989: -- Summary: StreamsUpgradeTest.test_metadata_upgrade could not guarantee all processor gets assigned task Key: KAFKA-9989 URL: https://issues.apache.org/jira/browse/KAFKA-9989

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

I updated the KIP to add a new broker configuration to limit connection creation rate per IP: max.connection.creation.rate.per.ip. Once the limit is reached for a particular IP address, the broker will reject the connection from that IP (close the connection it accepted) and continue rejecting

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

G1: I was considering it as the "end-to-end latency *up* to the specific task" but I'm happy with "record-staleness" if that drives the point home better. So it's the "staleness of the record when it is received by that task" -- will update the KIP B1/J: I'm struggling to imagine a case where the

Re: [VOTE] KIP-606: Add Metadata Context to MetricsReporter

+1 (binding) Thanks for the proposal, Xavier. On Wed, May 13, 2020 at 11:54 AM Xavier Léauté wrote: > Hi everyone, > > Folks seem happy with the state of the KIP, so I'd like to start the vote > for KIP-606 > >

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

Hi Bruno, Sounds good to me. I think I'm just a bit more curious to see its impact on performance: as long as we have one INFO level rocksDB metrics, then we'd have to turn on the scheduled rocksdb metrics recorder whereas previously, we can decide to not turn on the recorder at all if all are

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

Alright, I can get behind adding the min metric for the sake of pretty graphs (and trivial computation). I'm still on the fence regarding the mean (or 50th percentile) but I can see how users might expect it and find it a bit disorienting not to have. So the updated proposed metrics are -

Jenkins build is back to normal : kafka-trunk-jdk8 #4526

See

Jenkins build is back to normal : kafka-trunk-jdk11 #1452

See

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Hi Rajini, Thanks for the comments. > I think > they started off as connection timeouts but now include authentication time > as well. Have we considered using similar configs for this case? The new config I proposed is focusing on the connections to unreachable servers. The timeout count

Re: [DISCUSS] KIP-585: Conditional SMT

Hi Tom. Thanks for the KIP. I like how the proposal has ended up to be and I think it describes a practical approach. I have to say that, for a moment, earlier in the discussion I thought we were leaning a bit towards an unconventional mini assembly language based on java properties. The

Re: [VOTE] KIP-437: Custom replacement for MaskField SMT

I think this improvement makes total sense. It's interesting that it didn't accompany the initial version of this transformation. +1 (binding) Konstantine On Wed, May 6, 2020 at 2:03 PM Randall Hauch wrote: > Thanks for starting the vote, Yu. > > +1 (binding) > > Randall > > On Sat, Dec 21,

Build failed in Jenkins: kafka-trunk-jdk8 #4527

See Changes: [github] MINOR: Add a duplicate() method to Message classes (#8556) [github] KAFKA-9850 Move KStream#repartition operator validation during Topolo… [github] MINOR: add option to rebuild source for

Re: [VOTE] KIP-577: Allow HTTP Response Headers Configured for Kafka Connect

Makes sense to allow users to comply with their requirements without taking on the maintenance cost of keeping up with new headers across different versions. Thanks for the KIP Jeff. +1 (binding) Konstantine On Tue, May 12, 2020 at 3:13 AM Manikumar wrote: > +1 (binding) > > Thanks for the

[VOTE] KIP 585: Filter and conditional SMTs

Hi, I'd like to start a vote on KIP-585: Filter and conditional SMTs https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs Those involved in the discussion seem to be positively disposed to the idea, but in the absence of any committer participation it's been

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi David, Thanks for the explanation and confirmation that evolving the APIs is not off the table in the longer term. Kind regards, Tom

[jira] [Resolved] (KAFKA-9798) Flaky test: org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldAllowConcurrentAccesses

[ https://issues.apache.org/jira/browse/KAFKA-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9798. -- Resolution: Not A Problem > Flaky test: >

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

Hi Sophie, Thank you for your feedback! Please find my comments inline. On Mon, May 11, 2020 at 9:29 PM Sophie Blee-Goldman wrote: > > Hey Bruno, > > Thanks for the KIP! I have one high-level concern, which is that we should > consider > reporting these metrics on the per-store level rather

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

Hi John, Thank you for the feedback! I agree and I will change the KIP as I stated in my previous e-mail to Guozhang. Best, Bruno On Tue, May 12, 2020 at 3:07 AM John Roesler wrote: > > Thanks, all. > > If you don’t mind, I’ll pitch in a few cents’ worth. > > In my life I’ve generally found

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

Hi Guozhang, The recording level is set on sensor level. At the moment, I do not see any reason why we could not set store-level metrics to INFO. So I would propose putting the metrics on store-level and setting the recording level to INFO in the KIP. If it does not work due to technical issues,

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Hi Cheng, Thanks for the KIP, sounds like a good improvement. A couple of comments: 1) We currently have client connection timeouts on the broker with configs named `xxx.socket.timeout.ms` (e.g. controller.socket.timeout.ms). I think they started off as connection timeouts but now include