[jira] [Resolved] (KAFKA-8606) Provide a method to fetch committed offsets for a collection of TopicPartition

2024-01-30 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-8606. - Fix Version/s: 2.4.0 Resolution: Fixed Method added in AK 2.4. > Provid

[jira] [Created] (KAFKA-16184) Flaky test - testAlterReplicaLogDirs(String).quorum=kraft – kafka.api.PlaintextAdminIntegrationTest

2024-01-22 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16184: Summary: Flaky test - testAlterReplicaLogDirs(String).quorum=kraft – kafka.api.PlaintextAdminIntegrationTest Key: KAFKA-16184 URL: https://issues.apache.org/jira/browse

[jira] [Created] (KAFKA-16183) Flaky test - testMetricsDuringTopicCreateDelete(String).quorum=zk – kafka.integration.MetricsDuringTopicCreationDeletionTest

2024-01-22 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16183: Summary: Flaky test - testMetricsDuringTopicCreateDelete(String).quorum=zk – kafka.integration.MetricsDuringTopicCreationDeletionTest Key: KAFKA-16183 URL: https

[jira] [Created] (KAFKA-16182) Flaky test - testClientInstanceId() - org.apache.kafka.clients.admin.KafkaAdminClientTest

2024-01-22 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16182: Summary: Flaky test - testClientInstanceId() - org.apache.kafka.clients.admin.KafkaAdminClientTest Key: KAFKA-16182 URL: https://issues.apache.org/jira/browse/KAFKA-16182

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-22 Thread Andrew Schofield
+1 (non-binding) Thanks, Andrew > On 22 Jan 2024, at 07:29, Federico Valeri wrote: > > +1 (non binding) > > Thanks. > > On Mon, Jan 22, 2024 at 7:03 AM Luke Chen wrote: >> >> Hi Ziming, >> >> +1(binding) from me. >> >> Thanks. >> Luke >> >> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash

[jira] [Created] (KAFKA-16137) ListClientMetricsResourcesResponse definition is missing field descriptions

2024-01-15 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16137: Summary: ListClientMetricsResourcesResponse definition is missing field descriptions Key: KAFKA-16137 URL: https://issues.apache.org/jira/browse/KAFKA-16137

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-12 Thread Andrew Schofield
I also agree with 2 & 2 with reasoning along the same lines as Artem. Thanks, Andrew > On 12 Jan 2024, at 09:15, Federico Valeri wrote: > > On Thu, Jan 11, 2024 at 10:43 PM Artem Livshits > wrote: >> >> Hi Proven, >> >> I'd say that we should do 2 & 2. The idea is that for small features th

[jira] [Created] (KAFKA-16104) Enable additional PlaintextConsumerTest tests for new consumer

2024-01-09 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16104: Summary: Enable additional PlaintextConsumerTest tests for new consumer Key: KAFKA-16104 URL: https://issues.apache.org/jira/browse/KAFKA-16104 Project

[jira] [Created] (KAFKA-16100) Consistent handling of timeouts and responses for new consumer ApplicationEvents

2024-01-09 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16100: Summary: Consistent handling of timeouts and responses for new consumer ApplicationEvents Key: KAFKA-16100 URL: https://issues.apache.org/jira/browse/KAFKA-16100

[jira] [Created] (KAFKA-16099) Handle timeouts for AsyncKafkaConsumer.commitSync

2024-01-09 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16099: Summary: Handle timeouts for AsyncKafkaConsumer.commitSync Key: KAFKA-16099 URL: https://issues.apache.org/jira/browse/KAFKA-16099 Project: Kafka

[jira] [Created] (KAFKA-16092) Queues for Kafka

2024-01-08 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16092: Summary: Queues for Kafka Key: KAFKA-16092 URL: https://issues.apache.org/jira/browse/KAFKA-16092 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] Kafka distributed transaction support

2024-01-02 Thread Andrew Schofield
Hi Rowland, KIP-939 https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC introduces two-phase commit, but not XA support. I believe the KIP is expected to be delivered in Kafka 4.0 later this year. The way that it works and the use cases that it’s intended fo

Re: [VOTE] KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

2023-12-19 Thread Andrew Schofield
Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 14 Dec 2023, at 13:41, Mickael Maison wrote: > > +1 (binding) > Thanks for the KIP > > Mickael > > On Wed, Dec 13, 2023 at 4:01 PM Chris Egerton wrote: >> >> Hi Tina, >> >> Thanks for the KIP! +1 (binding) >> >> Cheers, >> >> Chris >> >>

[jira] [Resolved] (KAFKA-15775) Implement listTopics() and partitionFor() for the AsyncKafkaConsumer

2023-12-19 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15775. -- Resolution: Fixed > Implement listTopics() and partitionFor() for

Re: [VOTE] KIP-714: Client metrics and observability

2023-12-09 Thread Andrew Schofield
configuration resources”. The subscriptions are derived from the configuration resources, but they are not the same thing. Thanks, Andrew > On 16 Oct 2023, at 09:18, Andrew Schofield > wrote: > > The vote for KIP-714 has now concluded and the KIP is APPROVED. > > The votes are: > B

[jira] [Resolved] (KAFKA-15986) New consumer group protocol integration test failures

2023-12-08 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15986. -- Resolution: Fixed > New consumer group protocol integration test failu

[jira] [Created] (KAFKA-15986) New consumer group protocol integration test failures

2023-12-07 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15986: Summary: New consumer group protocol integration test failures Key: KAFKA-15986 URL: https://issues.apache.org/jira/browse/KAFKA-15986 Project: Kafka

[jira] [Resolved] (KAFKA-15932) Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")

2023-12-06 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15932. -- Resolution: Fixed > Flaky test - PlaintextConsumerTest.testSeek("kraft

[jira] [Created] (KAFKA-15978) New consumer sends OffsetCommit with empty member ID

2023-12-06 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15978: Summary: New consumer sends OffsetCommit with empty member ID Key: KAFKA-15978 URL: https://issues.apache.org/jira/browse/KAFKA-15978 Project: Kafka

[jira] [Resolved] (KAFKA-15533) Ensure HeartbeatRequestManager only send out some fields once

2023-12-04 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15533. -- Resolution: Resolved Already resolved > Ensure HeartbeatRequestManager only send

[jira] [Created] (KAFKA-15971) Re-enable consumer integration tests for new consumer

2023-12-04 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15971: Summary: Re-enable consumer integration tests for new consumer Key: KAFKA-15971 URL: https://issues.apache.org/jira/browse/KAFKA-15971 Project: Kafka

[jira] [Resolved] (KAFKA-15966) Test failure - ClassCastException in ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy

2023-12-03 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15966. -- Resolution: Duplicate Just fixed by https://github.com/apache/kafka/pull/14895

[jira] [Resolved] (KAFKA-15953) Refactor polling delays

2023-12-03 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15953. -- Resolution: Fixed > Refactor polling del

[jira] [Created] (KAFKA-15966) Test failure - ClassCastException in ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy

2023-12-03 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15966: Summary: Test failure - ClassCastException in ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy Key: KAFKA-15966 URL: https://issues.apache.org

[jira] [Created] (KAFKA-15953) Refactor polling delays

2023-12-01 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15953: Summary: Refactor polling delays Key: KAFKA-15953 URL: https://issues.apache.org/jira/browse/KAFKA-15953 Project: Kafka Issue Type: Sub-task

[jira] [Resolved] (KAFKA-15890) Consumer.poll with long timeout unaware of assigned partitions

2023-12-01 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15890. -- Resolution: Fixed > Consumer.poll with long timeout unaware of assigned partiti

[jira] [Resolved] (KAFKA-15842) Correct handling of KafkaConsumer.committed for new consumer

2023-12-01 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15842. -- Resolution: Fixed > Correct handling of KafkaConsumer.committed for new consu

[jira] [Created] (KAFKA-15945) Flaky test - testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest

2023-11-29 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15945: Summary: Flaky test - testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest Key: KAFKA-15945 URL: https

[jira] [Created] (KAFKA-15944) Flaky test - verifyStore[cache=false, log=true, supplier=ROCKS_KV, kind=DSL] – org.apache.kafka.streams.integration.PositionRestartIntegrationTest

2023-11-29 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15944: Summary: Flaky test - verifyStore[cache=false, log=true, supplier=ROCKS_KV, kind=DSL] – org.apache.kafka.streams.integration.PositionRestartIntegrationTest Key: KAFKA-15944

[jira] [Created] (KAFKA-15943) Flaky test - testHighWaterMarkAfterPartitionReassignment(String).quorum=kraft – org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest

2023-11-29 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15943: Summary: Flaky test - testHighWaterMarkAfterPartitionReassignment(String).quorum=kraft – org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest Key: KAFKA-15943

[jira] [Created] (KAFKA-15932) Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")

2023-11-29 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15932: Summary: Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer") Key: KAFKA-15932 URL: https://issues.apache.org/jir

[jira] [Resolved] (KAFKA-15840) Correct initialization of ConsumerGroupHeartbeat by client

2023-11-23 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15840. -- Resolution: Duplicate > Correct initialization of ConsumerGroupHeartbeat by cli

[jira] [Created] (KAFKA-15890) kafka-console-consumer.sh problems with new consumer

2023-11-23 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15890: Summary: kafka-console-consumer.sh problems with new consumer Key: KAFKA-15890 URL: https://issues.apache.org/jira/browse/KAFKA-15890 Project: Kafka

[jira] [Created] (KAFKA-15871) Implement kafka-client-metrics.sh tool

2023-11-21 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15871: Summary: Implement kafka-client-metrics.sh tool Key: KAFKA-15871 URL: https://issues.apache.org/jira/browse/KAFKA-15871 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-15837) Throw error on use of Consumer.poll(long timeout)

2023-11-21 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15837. -- Resolution: Fixed > Throw error on use of Consumer.poll(long time

Re: [VOTE] KIP-1000: List Client Metrics Configuration Resources

2023-11-20 Thread Andrew Schofield
ew, >> >> Thanks for the KIP. +1 >> >> Jun >> >> On Thu, Nov 16, 2023 at 9:12 AM Andrew Schofield < >> andrew_schofield_j...@outlook.com> wrote: >> >>> Hi Apoorv, >>> Thanks for your vote. >>> >>> Initially, I put

Re: [DISCUSS] KIP-1000: List Client Metrics Configuration Resources

2023-11-18 Thread Andrew Schofield
t-metrics.sh --bootstrap-server $BROKERS --describe > kafka-configs.sh --bootstrap-server $BROKERS --describe --entity-type > client-metrics > > Should we use --list instead to be more consistent with other tools like > (kafka-topics)? > > Thanks, > > Jun &g

Re: [DISCUSS] KIP-1000: List Client Metrics Configuration Resources

2023-11-17 Thread Andrew Schofield
for the KIP. Just one comment. >> >> Should we extend ConfigCommand or add a new tool to list client metrics? >> >> Thanks, >> >> Jun >> >> On Tue, Nov 7, 2023 at 9:42 AM Andrew Schofield < >> andrew_schofield_j...@outlook.com>

Re: [DISCUSS] KIP-1000: List Client Metrics Configuration Resources

2023-11-16 Thread Andrew Schofield
Rao wrote: > > Hi, Andrew, > > Thanks for the KIP. Just one comment. > > Should we extend ConfigCommand or add a new tool to list client metrics? > > Thanks, > > Jun > > On Tue, Nov 7, 2023 at 9:42 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wro

Re: [VOTE] KIP-1000: List Client Metrics Configuration Resources

2023-11-16 Thread Andrew Schofield
istener in the new RPC. > As the client-metrics resource is not supported in Zookeeper mode hence > shouldn't we disallow ListClientMetricsResourcesRequest for > Zookeper in the APIVersion request itself? > > +1(non-binding) > > Regards, > Apoorv Mittal > +44 77216

[jira] [Created] (KAFKA-15842) Correct handling of KafkaConsumer.committed for new consumer

2023-11-16 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15842: Summary: Correct handling of KafkaConsumer.committed for new consumer Key: KAFKA-15842 URL: https://issues.apache.org/jira/browse/KAFKA-15842 Project: Kafka

[jira] [Created] (KAFKA-15840) Correct initialization of ConsumerGroupHeartbeat by client

2023-11-16 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15840: Summary: Correct initialization of ConsumerGroupHeartbeat by client Key: KAFKA-15840 URL: https://issues.apache.org/jira/browse/KAFKA-15840 Project: Kafka

[VOTE] KIP-1000: List Client Metrics Configuration Resources

2023-11-15 Thread Andrew Schofield
Hi, I’d like to start the voting for KIP-1000: List Client Metrics Configuration Resources. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources Thanks, Andrew

[jira] [Created] (KAFKA-15831) List Client Metrics Configuration Resources

2023-11-15 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-15831: Summary: List Client Metrics Configuration Resources Key: KAFKA-15831 URL: https://issues.apache.org/jira/browse/KAFKA-15831 Project: Kafka Issue

Re: [DISCUSS] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

2023-11-07 Thread Andrew Schofield
Hi Artem, I think you make a very good point. This also looks to me like it deserves a version bump for the request. Andrew > On 8 Nov 2023, at 04:12, Artem Livshits > wrote: > > Hi Raman, > > Thank you for the KIP. I think using the tagged field > in DescribeTransactionsResponse should be go

[DISCUSS] KIP-1000: List Client Metrics Configuration Resources

2023-11-07 Thread Andrew Schofield
Hi, I would like to start discussion of a small KIP which fills a gap in the administration of client metrics configuration. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources Thanks, Andrew

Re: [VOTE] KIP-714: Client metrics and observability

2023-10-16 Thread Andrew Schofield
ote: >>>>>>> >>>>>>> Hey Andrew - >>>>>>> >>>>>>> +1 but I don't have a binding vote! >>>>>>> >>>>>>> It took me a while to go through the KIP. Here are so

Re: KIP-991: Allow DropHeaders SMT to drop headers by wildcard/regexp

2023-10-15 Thread Andrew Schofield
Hi Roman, Thanks for the KIP. I think it’s an interesting idea, but I think the KIP document needs some more details added before it’s ready for review. For example, here’s a KIP in the same area which was delivered in an earlier version of Kafka. I think this is a good KIP to copy for a suitabl

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-13 Thread Andrew Schofield
> listener. > > Jun > > On Thu, Oct 12, 2023 at 11:42 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi Jun, >> Thanks for your comments. >> >> 130. As Matthias described, and I am adding to the KIP, the >> `KafkaStreams#c

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-12 Thread Andrew Schofield
aStreams initialization? >> 131. Why does globalConsumerInstanceId() return Optional while >> other consumer instances don't return Optional? >> 132. ClientMetricsSubscriptionRequestCount: Do we need this since we have a >> set of generic metrics >> (kafka.network:type=RequestMetrics

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-12 Thread Andrew Schofield
ct 11, 2023 at 1:47 PM Matthias J. Sax wrote: > >> Thanks! >> >> On 10/10/23 11:31 PM, Andrew Schofield wrote: >>> Matthias, >>> Yes, I think that’s a sensible way forward and the interface you propose >> looks good. I’ll update the KIP accordingly.

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-12 Thread Andrew Schofield
very request type? >>>>> >>>>> Thanks, >>>>> >>>>> Jun >>>>> >>>>> On Wed, Oct 11, 2023 at 1:47 PM Matthias J. Sax >>>> wrote: >>>>> >>>>>> Thanks! >>>>>> >>>>>

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-10-11 Thread Andrew Schofield
Hi Jack, Thanks for your comments. I have added a new section on Log Retention which describes the behaviour of the SPSO as the LSO advances. That makes total sense and was an omission from the KIP. I have added the other ideas as potential future work. I do like the idea of having the SPSO inf

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-10 Thread Andrew Schofield
p restoreConsumerInstanceIds(); >Map producerInstanceIds(); > } > > For the `threadKey`, we would use some pattern like this: > > [Stream|StateUpdater]Thread- > > > Would this work from your POV? > > > > -Matthias > > > On 10/9/23 2:15 AM, A

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-09 Thread Andrew Schofield
`Future` because collection > all the `instanceId` might be a blocking all on each client? I have already a > few idea how it could be implemented but I don't think it must be discussed > on the KIP, as it's an implementation detail. > > Thoughts? > > > -Matthia

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-06 Thread Andrew Schofield
? It set via > StreamsConfig, we would forward it to all clients automatically. > > > > > -Matthias > > > On 9/29/23 5:45 PM, David Jacot wrote: >> Hi Andrew, >> Thanks for driving this one. I haven't read all the KIP yet but I already >> have an initi

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-03 Thread Andrew Schofield
n the consumer threading > refactor which is underway)". If I understand this correctly, it means > that KIP-714 won't work if the "old consumer" is used. Am I correct? > > Cheers, > David > > > On Fri, Sep 22, 2023 at 12:18 PM Andrew Schofield < >

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-09-22 Thread Andrew Schofield
hink the client should actively > search for a broker that supports this RPC? As previously mentioned, the > broker uses the leastLoadedNode to find its first connection (am > I correct?), and what if that broker doesn't support the metric push? > > P > > On Fri,

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-09-22 Thread Andrew Schofield
> correct? > > Thanks, > Kirk > >> On Sep 21, 2023, at 7:42 AM, Andrew Schofield >> wrote: >> >> Hi Jun, >> Thanks for your comments. I’ve updated the KIP to clarify where necessary. >> >> 110. Yes, agree. The motivation section mentions t

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-09-21 Thread Andrew Schofield
sRequest periodically to preserve the current > behavior. > > 120. GetTelemetrySubscriptionsResponseV0 and PushTelemetryRequestV0: Could > we list error codes for each? > > 121. "ClientTelemetryReceiver.ClientTelemetryReceiver This method may be > called from the request

Re: [VOTE] KIP-714: Client metrics and observability

2023-09-12 Thread Andrew Schofield
is already a busy channel, which leads to connection timeout and > subsequently disconnection. > > Thanks, > P > > On Fri, Sep 8, 2023 at 4:15 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Bumping the voting thread for KIP-714. >> >

Re: [VOTE] KIP-714: Client metrics and observability

2023-09-08 Thread Andrew Schofield
Bumping the voting thread for KIP-714. So far, we have: Non-binding +2 (Milind and Kirk), non-binding -1 (Ryanne) Thanks, Andrew > On 4 Aug 2023, at 09:45, Andrew Schofield wrote: > > Hi, > After almost 2 1/2 years in the making, I would like to call a vote for > KI

Re: [VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

2023-08-30 Thread Andrew Schofield
Thanks for the KIP. Looks good to me. +1 (non-binding). Andrew > On 30 Aug 2023, at 18:07, Hector Geraldino (BLOOMBERG/ 919 3RD A) > wrote: > > This makes sense to me, +1 (non-binding) > > From: dev@kafka.apache.org At: 08/30/23 02:58:59 UTC-4:00To: > dev@kafka.apache.org > Subject: [VOTE] K

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-08-25 Thread Andrew Schofield
remove or limit configured > metrics subscriptions. " How would a user know that the observed load was > due to handling metrics requests? > > 8. If I understand correctly, when the configured metrics reporter does not > implement the new interface the client would still follow the d

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-08-10 Thread Andrew Schofield
r way than the client side metrics > to identify this problem. > > Doguscan > > On Fri, 4 Aug 2023 at 15:33, Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi Doguscan, >> Thanks for your comments. I’m glad to hear you’re interested in

Re: [VOTE] KIP-940: Broker extension point for validating record contents at produce time

2023-08-07 Thread Andrew Schofield
Hi Adrian, Thanks for the KIP. Looks like a nice improvement. +1 (non-binding) Thanks, Andrew > On 2 Aug 2023, at 12:33, Adrian Preston wrote: > > Hello all, > > > > Edo and I would like to call for a vote on KIP-940: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-940%3A+Broker+exten

Re: [DISCUSS] Cluster-wide disablement of Tiered Storage

2023-08-04 Thread Andrew Schofield
Hi Christo, I agree with you. Option 4.1 without a KIP seems like an acceptable starting point for something which will be relatively rare, provided that it’s easy for the user to get a list of the topics that have to be deleted before they can successfully start the broker with TS turned off.

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-08-04 Thread Andrew Schofield
f aggregated data is too not helpful in the world of > telemetry 😊 > > 6. UnsupportedCompressionType: Shall we fallback to non-compression mode in > that case? I think compression is nice to have, but non-compressed > telemetry data is valuable as well. Especially for low throughpu

[VOTE] KIP-714: Client metrics and observability

2023-08-04 Thread Andrew Schofield
Hi, After almost 2 1/2 years in the making, I would like to call a vote for KIP-714 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability). This KIP aims to improve monitoring and troubleshooting of client performance by enabling clients to push metrics t

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

2023-08-03 Thread Andrew Schofield
way to let client side >>> partitioning logic be present as well. Anyways, I am good on this point. >>> 2) The example provided does seem achievable by simply attaching the >>> partition number in the ProducerRecord. I guess if we can't find any >>> further example

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-07-31 Thread Andrew Schofield
e good > to just log the INVALID_RECORD as an error, and treat the error as fatal > for the client (only fatal in terms of sending the metrics, the client can > keep functioning otherwise). What do you think? > > Thanks > Milind > > On Mon, Jul 24, 2023 at 8:18 PM Andrew Schofi

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

2023-07-28 Thread Andrew Schofield
> we should ignore the keys in this case (similar to the effect of > setting *partitioner.ignore.keys > *config to false) and document it appropriately? > > Let me know what you think. > > Thanks! > Sagar. > > > On Thu, Jul 27, 2023 at 9:41 PM Andrew Schofield <

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

2023-07-27 Thread Andrew Schofield
Hi Jack, Thanks for the KIP. I have a few concerns about the idea. 1) I think that while a client-side partitioner seems like a neat idea and it’s an established part of Kafka, it’s one of the things which makes Kafka clients quite complicated. Just as KIP-848 is moving from client-side assignor

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

2023-07-25 Thread Andrew Schofield
Thanks for the KIP. As you say, not that controversial. +1 (non-binding) Thanks, Andrew > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) > wrote: > > Hi everyone, > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect) have > a limited scope and shouldn'

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-07-24 Thread Andrew Schofield
n it's been obtained already? Or will a new ClientInstanceId be > returned without an error? > > Thanks! > > On Tue, Jun 13, 2023 at 8:38 PM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi, >> I would like to start a new discussion thre

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

2023-07-13 Thread Andrew Schofield
", "ignorable": > true, "default": "null", > "about": "The rack of the leader, or null if it has not been > assigned to a rack." } >]} > > Andrew > > 6. I wonder if non-Kafka clients might benefit from not bu

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

2023-07-13 Thread Andrew Schofield
Hi José, Thanks. Sounds good. Andrew > On 13 Jul 2023, at 16:45, José Armando García Sancio > wrote: > > Hi Andrew, > > On Thu, Jul 13, 2023 at 8:35 AM Andrew Schofield > wrote: >> I have a question about José’s comment (2). I can see that it’s possible for &

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

2023-07-13 Thread Andrew Schofield
Hi, Thanks for the KIP, Mayank. I have a question about José’s comment (2). I can see that it’s possible for multiple partitions to change leadership to the same broker/node and it’s wasteful to repeat all of the connection information for each topic-partition. But, I think it’s important to kn

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-07-11 Thread Andrew Schofield
or the KIP -- interesting read. I have some questions: >> 101. "The calls to KafkaConsumer.acknowledge(ConsumerRecord, >> AcknowledgeType) must be >> issued in the order in which the records appear in the ConsumerRecords >> object, which will >> be in order of inc

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-07-11 Thread Andrew Schofield
have any effect > on the share consumers? > > 104. How will the consumer quota work? Will it be similar to the existing > consumer quota mechanism? > > -- > Kamal > > On Wed, Jun 7, 2023 at 9:17 PM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >&

Re: Offsets: consumption and production in rollback

2023-06-29 Thread Andrew Schofield
Compaction? > https://kafka.apache.org/documentation/#design_compactionbasics > > Furthermore, is there a way to force the cleanup of transactions? Could this > potentially help address the issue? > > Cordially, > Henry > > > De: Andrew Schofield > En

Re: Offsets: consumption and production in rollback

2023-06-28 Thread Andrew Schofield
fsets in case of transaction rollbacks? It's possible that there's a > logic aspect I'm not fully grasping. > > Perhaps I need to utilize the internal control records and their offsets. > Could you point me in the right direction for their documentation? > > Thank y

Fwd: Offsets: consumption and production in rollback

2023-06-27 Thread Andrew Schofield
Hi Henry, Thanks for your message. Kafka transactions are a bit unusual. If you produce a message inside a transaction, it is assigned an offset on a topic-partition before the transaction even commits. That offset is not “revoked” if the transaction rolls back. This is why the consumer has the

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

2023-06-26 Thread Andrew Schofield
Hi Edo, Thanks for the KIP. Looks like a useful improvement. I have some comments/questions. 4) For a new interface, I wonder whether it would be better to use TopicIdPartition rather than TopicPartition. Topic IDs are gradually spreading across the public interfaces for Kafka. 5) The new topi

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-06-25 Thread Andrew Schofield
:47, Kirk True wrote: > > Hi Andrew, > > > >> On Jun 13, 2023, at 8:06 AM, Andrew Schofield >> wrote: >> >> Hi, >> I would like to start a new discussion thread on KIP-714: Client metrics and >> observability. >> >> https://cwiki

Re: [VOTE] KIP-937: Improve Message Timestamp Validation

2023-06-20 Thread Andrew Schofield
+1 (non-binding). Thanks, Andrew > On 19 Jun 2023, at 11:42, Divij Vaidya wrote: > > This KIP solves a real operational pain point for the administrator of > Kafka cluster. > > +1 (binding) > > -- > Divij Vaidya > > > > On Sun, Jun 18, 2023 at 5:09 AM Kirk True wrote: > >> +1 (non-binding) >> >

Re: [ANNOUNCE] New committer: Divij Vaidya

2023-06-13 Thread Andrew Schofield
Congratulations. Always good to see new committers. > On 13 Jun 2023, at 17:15, Mickael Maison wrote: > > Congratulations! > > On Tue, Jun 13, 2023 at 6:05 PM Yash Mayya wrote: >> >> Congratulations Divij! >> >> On Tue, Jun 13, 2023 at 9:20 PM Bruno Cadonna wrote: >> >>> Hi all, >>> >>> The PMC

[DISCUSS] KIP-714: Client metrics and observability

2023-06-13 Thread Andrew Schofield
Hi, I would like to start a new discussion thread on KIP-714: Client metrics and observability. https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability I have edited the proposal significantly to reduce the scope. The overall mechanism for client metric sub

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-06-07 Thread Andrew Schofield
e-groups. I've seen use-cases where the > LSO being blocked for an extended period of time caused huge lag for > traditional read_committed consumers, which could be completely avoided by > share-groups. > > Thanks, > Daniel > > Andrew Schofield ezt írta (időpont: >

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-06-07 Thread Andrew Schofield
; With this limitation (SPEO not moving beyond LSO), are you trying to avoid > handling the complexity of some kind of a "pending" state for the > uncommitted in-flight messages? > > Thanks, > Daniel > > Andrew Schofield ezt írta (időpont: > 2023. jún. 7., Sze, 16

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-06-07 Thread Andrew Schofield
guarantees into account, does this > mean that in queues, share-group consumers will be able to consume > committed records AFTER the LSO? > Thanks, > Daniel > > Andrew Schofield ezt írta (időpont: > 2023. jún. 2., P, 10:39): > >> Hi Daniel, >> Thanks for your quest

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-06-02 Thread Andrew Schofield
"skip X, fetch Y batches" fetch requests. > This solution would need changes in the Fetch API to allow such batch based > addresses, but I assume that fetch protocol changes will be needed > regardless of the specific solution. > > Thanks, > Daniel > > Andrew Sch

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-05-30 Thread Andrew Schofield
of > batches to ensure, that it will never be "bored"? > > Adam > >> On 30 May 2023, at 15:25, Andrew Schofield wrote: >> >> Hi Adam, >> Thanks for your question. >> >> With a share group, each fetch is able to grab available records from a

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-05-30 Thread Andrew Schofield
tor between > share and consumer groups (the other one being persisting state of messages). > And maybe the exact way we’d want to approach this would, to a certain > degree, dictate the design of the queueing system? > > Best, > Adam Warski > > On 2023/05/15 11:55:14 An

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-05-30 Thread Andrew Schofield
; number of delivery attempts for a record delivered to a share group. >>> But the actual delivery count may be more than this number as the >>> leader may fail updating the delivery count as leader or consumer may >>> fail and more delivery attempts may be made later. I

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-05-30 Thread Andrew Schofield
may be the > minimum number of delivery attempts instead of the maximum delivery > attempts. > > Thanks, > Satish. > > > On Wed, 24 May 2023 at 21:26, Andrew Schofield > wrote: >> >> Hi Stanislav, >> Thanks for your email. You bring up some interesti

Re: [DISCUSS] KIP-932: Queues for Kafka

2023-05-24 Thread Andrew Schofield
el like SSO and SEO may be bad acronyms. Googling "Kafka SEO" is bound to return weird results. What do we think about the tradeoff of using more-unique acronyms (like SGEO, SSGO) at the expense of one extra letter? Again - thanks for working on this! I think it's a great initia

[DISCUSS] KIP-932: Queues for Kafka

2023-05-15 Thread Andrew Schofield
Hi, I would like to start a discussion thread on KIP-932: Queues for Kafka. This KIP proposes an alternative to consumer groups to enable cooperative consumption by consumers without partition assignment. You end up with queue semantics on top of regular Kafka topics, with per-message acknowledg

Re: [VOTE] KIP-610: Error Reporting in Sink Connectors

2020-05-19 Thread Andrew Schofield
+1 (non-binding) This is now looking very nice. Andrew Schofield On 19/05/2020, 16:11, "Randall Hauch" wrote: Thank you, Aakash, for putting together this KIP and shepherding the discussion. Also, many thanks to all those that participated in the very active discu

Re: [VOTE] KIP 585: Filter and conditional SMTs

2020-05-15 Thread Andrew Schofield
+1 (non-binding) Thanks for the KIP. This will be very useful. Andrew Schofield On 13/05/2020, 10:14, "Tom Bentley" wrote: Hi, I'd like to start a vote on KIP-585: Filter and conditional SMTs https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+

<    1   2   3   4   >