[jira] [Created] (KAFKA-15678) [Tiered Storage] Stall remote reads with long-spanning transactions

2023-10-24 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-15678: - Summary: [Tiered Storage] Stall remote reads with long-spanning transactions Key: KAFKA-15678 URL: https://issues.apache.org/jira/browse/KAFKA-15678

[jira] [Created] (KAFKA-15490) Invalid path provided to the log failure channel upon I/O error when writing broker metadata checkpoint

2023-09-23 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-15490: - Summary: Invalid path provided to the log failure channel upon I/O error when writing broker metadata checkpoint Key: KAFKA-15490 URL: https://issues.apache.org/jira

[jira] [Created] (KAFKA-15486) Include NIO exceptions as I/O exceptions to be part of disk failure handling

2023-09-22 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-15486: - Summary: Include NIO exceptions as I/O exceptions to be part of disk failure handling Key: KAFKA-15486 URL: https://issues.apache.org/jira/browse/KAFKA-15486

[jira] [Created] (KAFKA-15038) Use topic id/name mapping from the Metadata cache in RLM

2023-05-30 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-15038: - Summary: Use topic id/name mapping from the Metadata cache in RLM Key: KAFKA-15038 URL: https://issues.apache.org/jira/browse/KAFKA-15038 Project: Kafka

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

2023-05-25 Thread Alexandre Dupriez
Hi, Igor, Thanks for the excellent, thorough and very comprehensive KIP. Although not directly in scope of the KIP, but related to it, I would have the following question about a potential future work on disk degradation. Today, what characterises as a disk failure in Kafka is an I/O exception

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

2023-05-11 Thread Alexandre Dupriez
Hi, Luke, Thanks for your reply. 102. Whether such a replica could become a leader depends on what the end-user wants to use it for and what tradeoffs they wish to make down the line. There are cases, for instance with heterogeneous or interregional networks, where the difference in latency

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

2023-05-10 Thread Alexandre Dupriez
Hi, Luke, Thanks for the KIP. It clearly highlights the tradeoff between latency and durability and proposes an approach relaxing a durability constraint to provide lower ingestion latency. Please find a few comments/questions. 100. The KIP makes one statement which may be considered critical:

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

2023-04-18 Thread Alexandre Dupriez
risk > is as big as the risk of running a third-party code (i.e. the RSM plugin). > The cluster admin must make the decision if they trust it. > To mitigate this risk and put it under control, the RSM plugin > implementations could document what custom metadata they use and estimate

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

2023-04-17 Thread Alexandre Dupriez
Hi Ivan, Thank you for the KIP. I think the KIP clearly explains the need for out-of-band metadata authored and used by an implementation of the remote storage manager. One question I would have is, what is the benefit of adding these custom metadata in the rlmMetadata rather than letting the

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

2023-04-12 Thread Alexandre Dupriez
Hi Divij, Thanks for the follow-up. A few comments/questions. 100. The stated motivation to increase the number of partitions above 50 is scale. Are we sure that 50 partitions is not enough to cover all valid use cases? An upper bound in the range 1 to 10 MB/s of ingress per partition gives 50

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

2023-04-04 Thread Alexandre Dupriez
Hi Christo, Thanks for the KIP. Apologies for the delayed review. At a high-level, I am not sure if the KIP really solves the problem it intends to. More specifically, the KIP mentions that once a broker is restarted and the group coordinator becomes aware of the new partition count of the

Re: [VOTE] KIP-915: Txn and Group Coordinator Downgrade Foundation

2023-03-30 Thread Alexandre Dupriez
Thanks for the KIP and clarifications, Jeff. +1 (non binding) Le jeu. 30 mars 2023 à 14:41, David Jacot a écrit : > > Thanks for the KIP, Jeff. > > +1 (binding) > > On Wed, Mar 29, 2023 at 4:48 PM Jeff Kim > wrote: > > > Hi all, > > > > I would like to start the vote on KIP-915: Txn and Group

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

2023-03-28 Thread Alexandre Dupriez
supported version. Does this > answer your question? > > 101. The future versioning scheme needs to be backward compatible with > older coordinators. Wouldn't segregating into 2 versions > be incompatible? > > Thanks, > Jeff > > On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

2023-03-28 Thread Alexandre Dupriez
Hi Jeff, Team, Thank you for the KIP. This is a very interesting approach. I feel it is simpler than the described alternative although it comes with tradeoffs, thanks for highlighting those. If I may, I would like to share two naive questions. 100. The KIP mentions that records will be

[jira] [Created] (KAFKA-14852) Propagate Topic Ids to the Group Coordinator for Offset Fetch

2023-03-27 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14852: - Summary: Propagate Topic Ids to the Group Coordinator for Offset Fetch Key: KAFKA-14852 URL: https://issues.apache.org/jira/browse/KAFKA-14852 Project

[jira] [Created] (KAFKA-14845) Broker ZNode creation can fail due to a session ID unknown to the broker

2023-03-24 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14845: - Summary: Broker ZNode creation can fail due to a session ID unknown to the broker Key: KAFKA-14845 URL: https://issues.apache.org/jira/browse/KAFKA-14845

Re: [VOTE] KIP-906 Tools migration guidelines

2023-03-15 Thread Alexandre Dupriez
Hi, Frederico, Thanks for the KIP. Non-binding +1. Thanks, Alexandre Le mer. 15 mars 2023 à 08:28, Luke Chen a écrit : > > +1 from me. > > Luke > > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri > wrote: > > > Hi everyone, > > > > I'd like to start the vote on KIP-906 Tools migration

[jira] [Created] (KAFKA-14806) Add connection timeout in PlaintextSender used by SelectorTests

2023-03-14 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14806: - Summary: Add connection timeout in PlaintextSender used by SelectorTests Key: KAFKA-14806 URL: https://issues.apache.org/jira/browse/KAFKA-14806 Project

[jira] [Created] (KAFKA-14793) Propagate topic ids to the group coordinator

2023-03-08 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14793: - Summary: Propagate topic ids to the group coordinator Key: KAFKA-14793 URL: https://issues.apache.org/jira/browse/KAFKA-14793 Project: Kafka Issue

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

2023-03-07 Thread Alexandre Dupriez
+1 (non-binding). Thanks for driving this. Le mar. 7 mars 2023 à 08:47, Federico Valeri a écrit : > > + 1 (non binding) for the latest revision of this KIP (RecordReader). > > Thanks > Fede > > On Sat, Feb 25, 2023 at 7:04 AM Luke Chen wrote: > > > > +1 from me. > > > > Thank you > > Luke > >

[jira] [Created] (KAFKA-14780) Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic

2023-03-06 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14780: - Summary: Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic Key: KAFKA-14780 URL: https://issues.apache.org/jira/browse/KAFKA-14780

[jira] [Created] (KAFKA-14779) Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic

2023-03-06 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14779: - Summary: Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic Key: KAFKA-14779 URL: https://issues.apache.org/jira/browse/KAFKA-14779

[jira] [Created] (KAFKA-14777) Add support of topic id for OffsetCommitRequests in CommitRequestManager

2023-03-05 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14777: - Summary: Add support of topic id for OffsetCommitRequests in CommitRequestManager Key: KAFKA-14777 URL: https://issues.apache.org/jira/browse/KAFKA-14777

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

2023-02-22 Thread Alexandre Dupriez
Hi Chia-Ping, Thanks for your answer. Apologies I should have been clearer in the previous message. What I meant is, is there a plan to use the SPI more broadly inside the Kafka codebase? The question arises because the interface exposes a close() method which is never invoked by the

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

2023-02-20 Thread Alexandre Dupriez
Hi Chia-Ping, Thank you for the KIP and apologies for missing it earlier. A minor comment. The SPI is currently used exclusively for the ConsoleProducer. However, it exposes high-level methods which hint at it being a generic component. What is the actual scope of the SPI inside the Kafka

[jira] [Resolved] (KAFKA-8752) Ensure plugin classes are instantiable when discovering plugins

2023-02-20 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8752. -- Resolution: Not A Problem > Ensure plugin classes are instantiable when discover

Re: [VOTE] KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

2023-02-10 Thread Alexandre Dupriez
+1 (non-binding). Le ven. 10 févr. 2023 à 19:02, Calvin Liu a écrit : > > Hi all, > > I'd like to call for a vote on KIP-903, which proposes a fix to the broker > reboot data loss KAFKA-14139 > > It changes the Fetch and AlterPartition requests

Re: [DISCUSS] KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

2023-02-09 Thread Alexandre Dupriez
st version is derived from the > > metadata version/IBP. As we add a new fetch version, we need to add a new > > metadata version/IBP as well. > > > > 08. Regarding KRaft vs ZK, I don't have a strong opinion. ZK is on the way > > out so not doing it seems fine. If we

Re: [DISCUSS] KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

2023-02-08 Thread Alexandre Dupriez
Hi, Calvin, Thanks for the KIP and fast follow-up. A few questions. 100. The scenario illustrated in the KIP involves a controller movement. Is this really required? Cannot the scenario occur with a similar stale AlterPartition request and the same controller throughout? 101. In the case where

Re: [DISCUSS] Tiered-Storage: Implement RocksdbBasedMetadataCache for TopicBased RLMM

2023-01-26 Thread Alexandre Dupriez
Hi, hzh0425, Thank you for raising the question. There have been discussions about using RocksDB as a RLMM in the community, one of which can be found on the dev list [1]. What is the size of the metadata set in your use case? Have you considered using an external data store to abstract metadata

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-11-18 Thread Alexandre Dupriez
Hi Divij, Thanks for the KIP. Please find some comments based on what I read on this thread so far - apologies for the repeats and the late reply. If I understand correctly, one of the main elements of discussion is about caching in Kafka versus delegation of providing the remote size of a

[jira] [Created] (KAFKA-14190) Corruption of Topic IDs with pre-2.8.0 admin clients

2022-08-30 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-14190: - Summary: Corruption of Topic IDs with pre-2.8.0 admin clients Key: KAFKA-14190 URL: https://issues.apache.org/jira/browse/KAFKA-14190 Project: Kafka

Re: Recover partitions from a failed broker leader which is the only replica in isr.

2021-05-16 Thread Alexandre Dupriez
Hi JD, You can enable unclean leader election (disabled by default) to use ex-ISR followers as fallbacks for new leaders - but be mindful of data loss. Thanks, Alexandre Le mar. 10 nov. 2020 à 06:33, JD Zheng a écrit : > > Hi, everyone, > > Not sure if this is the right place to ask this

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

2021-05-16 Thread Alexandre Dupriez
Hello Nakamura, Thanks for proposing this change. I can see how the blocking behaviour can be a problem when integrating with reactive frameworks such as Akka. One of the questions I would have is how you would handle back pressure and avoid memory exhaustion when the producer's buffer is full

Broker-level min ISR for __consumer_offsets

2021-03-26 Thread Alexandre Dupriez
Hi, Community, I wanted to reach out about the following scenario. It was most likely discussed before and a long time ago: I apologise in advance for the repeat. I am customising broker-level replication settings of an internal topic (in this case, the consumer offsets topic) to accommodate the

Re: Gradle error - aggregatedJavadoc depending on "compileJava"

2021-03-24 Thread Alexandre Dupriez
> > On 2021/03/10 14:07:08, Alexandre Dupriez wrote: > > Hi Community, > > > > I tried to build Kafka from trunk on my environment today (2021, March > > 10th) and it failed with the following Gradle error at the beginning > > of the build, while Gradle con

Re: Gradle error - aggregatedJavadoc depending on "compileJava"

2021-03-10 Thread Alexandre Dupriez
? That works for me. Also `./gradlew clean releaseTarGz` > works for me. > > Ismael > > On Wed, Mar 10, 2021 at 8:42 AM Ismael Juma wrote: > > > Interesting, I didn't have the same problem. I'll try to reproduce. > > > > Ismael > > > > On Wed, Mar 10,

Gradle error - aggregatedJavadoc depending on "compileJava"

2021-03-10 Thread Alexandre Dupriez
Hi Community, I tried to build Kafka from trunk on my environment today (2021, March 10th) and it failed with the following Gradle error at the beginning of the build, while Gradle configures project from build.gradle: "Could not get unknown property 'compileJava' for root project '' of type

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-08-20 Thread Alexandre Dupriez
Hi Jun, Many thanks for your initiative. If you like, I am happy to attend at the time you suggested. Many thanks, Alexandre Le mer. 19 août 2020 à 22:00, Harsha Ch a écrit : > Hi Jun, > Thanks. This will help a lot. Tuesday will work for us. > -Harsha > > > On Wed, Aug 19, 2020 at

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-06-12 Thread Alexandre Dupriez
make all ZK writes happen on the > > > > > > controller. > > > > > > > > > > > > Of course, this is just my take on it. > > > > > > > > > > > > Kind regards, > > > > > > > > > > &g

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-06-06 Thread Alexandre Dupriez
gt;>>>>>> the rate of 1/sec. It can cost $0.5/sec, which is roughly $40K per > > >>>>>>> > > >>>>>>> day. > > >>>>>>> > > >>>>>>> I want to note here, that no reasonably dura

Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers

2020-05-19 Thread Alexandre Dupriez
+1 (non-binding) Thank you for the KIP! Le mar. 19 mai 2020 à 07:57, David Jacot a écrit : > > +1 (non-binding) > > Thanks for the KIP, Anna! > > On Tue, May 19, 2020 at 7:12 AM Satish Duggana > wrote: > > > +1 (non-binding) > > Thanks Anna for the nice feature to control the connection

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

2020-05-19 Thread Alexandre Dupriez
>>> > > > > >>> > > > Your second concern makes sense -- rejecting the connection right > >>> away > >>> > > will > >>> > > > likely cause a new connection from the same client. I am concerned > >>>

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-05-17 Thread Alexandre Dupriez
; the S3 implementation, if one of the multiple uploads fails [4]), > >> > >> The S3 implementation is going to change, but it's worth clarifying > >> > >> anyway. > >> > >> The segment log file is being uploaded after S3 has acked up

[jira] [Resolved] (KAFKA-6959) Any impact we foresee if we upgrade Linux version or move to VM instead of physical Linux server

2020-05-16 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-6959. -- Resolution: Fixed > Any impact we foresee if we upgrade Linux version or move to

[jira] [Resolved] (KAFKA-8916) Unreliable kafka-reassign-partitions.sh affecting performance

2020-05-16 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8916. -- Resolution: Invalid Closing this as there is no bug or development required. Please

[jira] [Resolved] (KAFKA-8815) Kafka broker blocked on I/O primitive

2020-05-16 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8815. -- Resolution: Not A Problem System failure. Not related to Kafka. > Kafka bro

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

2020-05-16 Thread Alexandre Dupriez
t; > > while > > > > controlling queue size may mitigate the issue for some of the > > workloads, > > > it > > > > does not guarantee that. Plus, if we want to limit how many connections > > > are > > > > created per IP, the queue size approach

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

2020-05-12 Thread Alexandre Dupriez
Hello, Thank you for the KIP. I experienced in the past genuine broker brownouts due to connection storms consuming most of the CPU available on the server and I think it is useful to protect against it. I tend to share the questions asked in points 2 and 4 from David. Is there still a risk of

[jira] [Resolved] (KAFKA-9549) Local storage implementations for RSM which can be used in tests

2020-04-27 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-9549. -- Resolution: Fixed > Local storage implementations for RSM which can be used in te

Re: [VOTE] KIP-551: Expose disk read and write metrics

2020-04-17 Thread Alexandre Dupriez
Hello all, Apologies for the excavation. +1 (non-binding). One of the advantage I see having the metrics exposed by Kafka, as opposed to a system monitoring tool, is that it can be refined to provide a segregated view on the consumption of disk resources, for instance factored by topic or

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-09 Thread Alexandre Dupriez
Hi Gokul, Thanks for the KIP. >From what I understand, the objective of the new configuration is to protect a cluster from an overload driven by an excessive number of partitions independently from the load handled on the partitions themselves. As such, the approach uncouples the data-path load

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

2020-03-31 Thread Alexandre Dupriez
+1 (non binding). Thanks. Le mar. 31 mars 2020 à 14:24, M. Manna a écrit : > > +1 (binding). > > Thanks for the KIP. > > On Tue, 31 Mar 2020 at 14:17, Paolo Moriello > wrote: > > > Hello, > > > > Thanks to everybody who has given feedback. I've incorporated the > > suggestions and think that

TLS failures and map I/O faults

2020-03-27 Thread Alexandre Dupriez
Dear community, I recently faced an unexpected type of failures in the middle of an incident related to the exhaustion of memory-map handles on a Kafka broker. The use case is as follows - a broker, not overloaded, manages enough indexes to reach the limit on mmap count per process. This leads

[jira] [Resolved] (KAFKA-7958) Transactions are broken with kubernetes hosted brokers

2020-03-20 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-7958. -- Fix Version/s: 2.1.1 Resolution: Fixed > Transactions are bro

Re: Question about log flusher real frequency

2020-03-09 Thread Alexandre Dupriez
Hi Fares, On Linux kernels, you can use the property "dirty_writeback_centisecs" [1] to configure the period between executions of kswapd, which does this "sync" job. The period is usually set to 30 seconds. There are few exceptions where Kafka explicitly forces a sync (via the force() method

Re: [DISCUSS] KAFKA-4680: min.insync.replica can be set > replication factor

2020-03-03 Thread Alexandre Dupriez
Thanks Paolo for taking care of this. I think option (3) would be the closest to iso-functionality. I wonder how useful it could be to always add the replication factor of the topic in the error message [1] versus returning a modified one fitted for this specific case? [1]

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

2020-02-18 Thread Alexandre Dupriez
transport layer > protocols have different headers. In order for the "new handshake" protocol > to work, it would have to work in the same namespace as those headers, > which will require careful tuning of handshake constants. > > Any thoughts from the community on

[jira] [Created] (KAFKA-9565) Implementation of Tiered Storage SPI to integrate with S3

2020-02-18 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-9565: Summary: Implementation of Tiered Storage SPI to integrate with S3 Key: KAFKA-9565 URL: https://issues.apache.org/jira/browse/KAFKA-9565 Project: Kafka

[jira] [Created] (KAFKA-9564) Integration Tests for Tiered Storage

2020-02-17 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-9564: Summary: Integration Tests for Tiered Storage Key: KAFKA-9564 URL: https://issues.apache.org/jira/browse/KAFKA-9564 Project: Kafka Issue Type: Sub

Possible to create Scrum board under Kafka project in JIRA?

2020-02-14 Thread Alexandre Dupriez
Good morning, Would it be possible to allow the the Apache Kafka project in JIRA to be included in a new Scrum board? I can see there is already a Kanban board for Cloudera and tried to create a Scrum board for Tiered-Storage but don't have the permissions to include Apache Kafka. Thank you,

[jira] [Created] (KAFKA-9555) Topic-based implementation for the RLMM

2020-02-14 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-9555: Summary: Topic-based implementation for the RLMM Key: KAFKA-9555 URL: https://issues.apache.org/jira/browse/KAFKA-9555 Project: Kafka Issue Type

[jira] [Created] (KAFKA-9554) Define the SPI for Tiered Storage framework

2020-02-14 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-9554: Summary: Define the SPI for Tiered Storage framework Key: KAFKA-9554 URL: https://issues.apache.org/jira/browse/KAFKA-9554 Project: Kafka Issue Type

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2019-12-19 Thread Alexandre Dupriez
Hi Jun, Thank you for the feedback. I am trying to understand how a push-based approach would work. In order for the metadata to be propagated (under the assumption you stated), would you plan to add a new API in Kafka to allow the metadata store to send them directly to the brokers? Thanks,

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2019-12-18 Thread Alexandre Dupriez
Hi all, A.1 As commented on PR 7561, S3 consistency model [1][2] implies RSM cannot relies solely on S3 APIs to guarantee the expected strong consistency. The proposed implementation [3] would need to be updated to take this into account. Let’s talk more about this. A.2 Contract for the RSM API

Re: [VOTE] KIP-544: Make metrics exposed via JMX configurable

2019-11-09 Thread Alexandre Dupriez
+1 (non-binding) Le ven. 8 nov. 2019 à 20:21, Bill Bejeck a écrit : > Thanks for the KIP, +1 (binding). > > -Bill > > On Fri, Nov 8, 2019 at 1:28 PM Gwen Shapira wrote: > > > +1 (binding) > > > > On Thu, Nov 7, 2019 at 11:17 AM Xavier Léauté > wrote: > > > > > > Hi everyone, > > > > > > I'd

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

2019-11-08 Thread Alexandre Dupriez
Hello, This can be very handy when dealing with large numbers of partitions on a broker. I was recently experimenting with a third-party monitoring framework which provides a JMX collector [1] with the same mechanism to filter out the JMX beans retrieved from Kafka. When running a couple of

Re: UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

2019-08-15 Thread Alexandre Dupriez
-James > > Sent from my iPhone > > > On Aug 13, 2019, at 1:17 AM, Alexandre Dupriez < > alexandre.dupr...@gmail.com> wrote: > > > > Hello all, > > > > We run into a scenario where we had misconfigured the replication factor > > and the minimum

[jira] [Resolved] (KAFKA-8695) Metrics UnderReplicated and UnderMinIsr are diverging when configuration is inconsistent

2019-08-14 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8695. -- Resolution: Duplicate De-duplicating in favor of  [KAFKA-4680|http://issues.apache.org

UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

2019-08-13 Thread Alexandre Dupriez
Hello all, We run into a scenario where we had misconfigured the replication factor and the minimum in-sync replicas count in such a way that the replication factor (either default or defined at the topic level) is strictly lower than the property min.insync.replicas. We observed broker metrics

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

2019-08-06 Thread Alexandre Dupriez
KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM Le jeu. 1 août 2019 à 09:42, Alexandre Dupriez a écrit : > Thanks Gokul and Stanislav for your answers. > > I went through the PR 5940 [1]. Indeed Gokul, your reasoning echoes the &

[jira] [Created] (KAFKA-8752) Ensure plugin classes are instantiable when discovering plugins

2019-08-05 Thread Alexandre Dupriez (JIRA)
Alexandre Dupriez created KAFKA-8752: Summary: Ensure plugin classes are instantiable when discovering plugins Key: KAFKA-8752 URL: https://issues.apache.org/jira/browse/KAFKA-8752 Project: Kafka

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

2019-08-01 Thread Alexandre Dupriez
et accepted by the > > community. > > > > In the absence of a fool-proof way to do a handhskake across all security > > protocols (plaintext, SSL, other future ones), I think that the proposed > > KIP provides a good solution. Therefore, you have my +1. (Not sur

[DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

2019-07-28 Thread Alexandre Dupriez
Hello, I have created the KIP-498 to propose a minor addition to the producer configuration properties, in order to protect against OOM when the response

Re: JIRA and KIP contributor permissions

2019-07-23 Thread Alexandre Dupriez
the list of contributors in JIRA, so you can self-assign > ticket. However, I did not find any corresponding wiki. Note, that both > are independent accounts and you might need to create a wiki account > first (and share you ID so we can grant write permission). > > > -Matthias > > On 7

JIRA and KIP contributor permissions

2019-07-22 Thread Alexandre Dupriez
Hello Community, In order to start contributing to Apache Kafka project, could I please request contributor access to JIRA and be granted write permissions to the Kafka wiki? JIRA username: adupriez Committer email: alexandre.dupr...@amazon.com Thank you in advance, Alexandre

[jira] [Created] (KAFKA-8695) Metrics UnderReplicated and UnderMinSir are diverging when configuration is inconsistent

2019-07-22 Thread Alexandre Dupriez (JIRA)
Alexandre Dupriez created KAFKA-8695: Summary: Metrics UnderReplicated and UnderMinSir are diverging when configuration is inconsistent Key: KAFKA-8695 URL: https://issues.apache.org/jira/browse/KAFKA-8695

Metrics UnderReplicated and UnderMinSir are diverging when configuration is inconsistent

2019-07-22 Thread Alexandre Dupriez
Hello Community, I noticed Kafka allows the replication factor of a topic and "min.insync.replicas" to be set such that "min.insync.replicas" > the topic's replication factor. As a consequences, the JMX beans kafka.cluster:type=Partition,name=UnderReplicated and