Re: [DISCUSS] KIP-405 + KAFKA-7739 - Implementation of Tiered Storage Integration with Azure Storage

Hi Israel, Linkedin is interested in evaluating KIP-405 for HDFS and S3 in the short term and Azure Blob Storage in the long run. You may already know that Linkedin is migrating to Azure . We think that Blobs will provide us with the

[jira] [Created] (KAFKA-6701) synchronize Log modification between delete cleanup and async delete

Sumant Tambe created KAFKA-6701: --- Summary: synchronize Log modification between delete cleanup and async delete Key: KAFKA-6701 URL: https://issues.apache.org/jira/browse/KAFKA-6701 Project: Kafka

[jira] [Created] (KAFKA-5886) Implement KIP-91

Sumant Tambe created KAFKA-5886: --- Summary: Implement KIP-91 Key: KAFKA-5886 URL: https://issues.apache.org/jira/browse/KAFKA-5886 Project: Kafka Issue Type: Improvement Components

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

nds or not) . > > On Mon, Sep 11, 2017 at 10:06 AM, Sumant Tambe <suta...@gmail.com> wrote: > > > @Jun, Until we make idempotent producer the default (kip-185), this kip > is > > sensitive to retries. I.e., we expire batches either delivery.timeout.ms > > passes o

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

gt; > > expiry. > > > > > > The compatibility impact of such changes can remain in the > compatibility > > > section. Also, I agree about keeping your "reordering" text although it > > > seems like the wiki wasn't updated to match what you posted in

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

ty > section. Also, I agree about keeping your "reordering" text although it > seems like the wiki wasn't updated to match what you posted in the > discussion thread. > > Ismael > > On Fri, Sep 8, 2017 at 6:06 AM, Sumant Tambe <suta...@gmail.com> wrote: > >

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

patibility section, your note on > 'max.in.flight.requests.per.connection == 5' resulting in out of order > delivery is true irrespective of these changes. As such, I don't think it > should be mentioned in the context of this KIP. > > Thanks, > Apurva > > On Thu, Sep 7, 2

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

ld make sense to > > apply > > > exponential backoff, we could also consider that. So, I'm not sure > > retries > > > has much use apart from compatibility and the retries=0 case (for now). > > > > > > Ismael > > > > > > On Wed, Sep 6, 201

[VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

Hi all, I would like to open the vote for KIP-91: https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer Thank you all for your input on the kip so far. Regards, Sumant

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

include the batching time. > > > > For retries, probably we can just default it to MAX_INT? > > > > Thanks, > > > > Jun > > > > > > On Wed, Sep 6, 2017 at 10:26 AM, Sumant Tambe <suta...@gmail.com> wrote: > > > > > 120 seconds

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

s a log warning). > > 3. We should mention that we will not cancel in flight requests until the > request timeout even though we'll expire the batch early if needed. > > I think we should start the vote tomorrow so that we have a chance of > hitting the KIP freeze for 1.0.0. >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

OK though because > > > currently > > > > > when TimeoutException is thrown, there is no guarantee whether the > > > > messages > > > > > are delivered or not. > > > > > > > > > > Thanks, > &g

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

t; > Now we are introducing another case: > > > > > > 3. A batch is in retry because we expired an in-flight request > > before > > > > it > > > > > > hits request.timeout.ms. > > > > > > > > > > > > The

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

control delay in > >>> the > >>> accumulator. It seems the batch.expiry.ms approach will have the same > >>> issue. If you start the clock when a batch is initialized, you may > expire > >>> some messages in the same batch early than batch.expiry.ms. If you &g

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

t multiple times and new issues keep coming back. Ideally, > this time, we want to have a solution that covers all cases, even though > that requires a bit more work. > > Thanks, > > Jun > > > On Fri, Aug 11, 2017 at 12:30 PM, Sumant Tambe <suta...@gmail.com> wrote: > >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

e.get can use this timeout value. Thoughts? Regards, Sumant On 11 August 2017 at 07:50, Sumant Tambe <suta...@gmail.com> wrote: > > Thanks for the KIP. Nice documentation on all current issues with the >> timeout. > > For the KIP writeup, all credit goes to Joel Koshy

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

ore intuitive to the > end user. > > Does that sound reasonable to you? > > Thanks, > > Jun > > > On Wed, Aug 9, 2017 at 10:03 PM, Sumant Tambe <suta...@gmail.com> wrote: > > > On Wed, Aug 9, 2017 at 1:28 PM Apurva Mehta <apu...@confluent.io>

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

On Wed, Aug 9, 2017 at 1:28 PM Apurva Mehta wrote: > > > There seems to be no relationship with cluster metadata availability or > > > staleness. Expiry is just based on the time since the batch has been > > ready. > > > Please correct me if I am wrong. > > > > > > > I was

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Responses inline. > > However, one thing which has not come out of the JIRA discussion is the > > > actual use cases for batch expiry. > > > > There are two usecases I can think of for batch expiry mechanism > > irrespective of how we try to bound the time (batch.expiry.ms or > >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

icPartition, broker-id, an app may provide differentiated service based on a topic name or availability zone of a broker-id. KIP-91 does not propose anything like that. It's a very niche usecase though. Regards, Sumant > > > > > > On Thu, Aug 3, 2017 at 4:35 PM, Sumant Tambe <su

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

KIP. Just a quick comment. Can you list the alternatives > mentioned in the JIRA discussion in the rejected alternatives section? > > -Jason > > On Thu, Aug 3, 2017 at 3:09 PM, Sumant Tambe <suta...@gmail.com> wrote: > > > Hi all, > > > > KIP-91 [1] is another attemp

[DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Hi all, KIP-91 [1] is another attempt to get better control on producer side timeouts. In essence we're proposing a new config named batch.expiry.ms that will cause batches in the accumulator to expire after the configured timeout. Recently, the discussion on KAFKA-5621 [2] has shed new light

[jira] [Created] (KAFKA-4395) KafkaConfig and LogConfig should not have static initialization order dependencies

Sumant Tambe created KAFKA-4395: --- Summary: KafkaConfig and LogConfig should not have static initialization order dependencies Key: KAFKA-4395 URL: https://issues.apache.org/jira/browse/KAFKA-4395

[jira] [Updated] (KAFKA-4395) KafkaConfig and LogConfig should not have static initialization order dependencies

[ https://issues.apache.org/jira/browse/KAFKA-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4395: Affects Version/s: 0.10.0.1 > KafkaConfig and LogConfig should not have static initialization or

[jira] [Assigned] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe reassigned KAFKA-4089: --- Assignee: Sumant Tambe (was: Dong Lin) > KafkaProducer raises Batch Expired except

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when producer

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when producer

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when producer

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when producer

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}}) ejects

[jira] [Created] (KAFKA-4089) KafkaProducer raises Batch Expired exception

Sumant Tambe created KAFKA-4089: --- Summary: KafkaProducer raises Batch Expired exception Key: KAFKA-4089 URL: https://issues.apache.org/jira/browse/KAFKA-4089 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1911) Log deletion on stopping replicas should be async

[ https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392877#comment-15392877 ] Sumant Tambe commented on KAFKA-1911: - Hi [~mgharat], I would like to take your patch over. I've