Re: [VOTE] KIP-237: More Controller Health Metrics

t; > >> On Wed, Jan 3, 2018 at 10:48 PM, Jun Rao <j...@confluent.io> wrote: > >> > >> > Hi, Dong, > >> > > >> > Thanks for the KIP. +1 > >> > > >> > Jun > >> > > >> > On Mon, Dec 18, 2017 at 2:

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

and reset > the offset based on the offset reset policy in the consumer. This seems > harder to do with a global metadata version. > > Jun > > > > On Mon, Dec 25, 2017 at 6:56 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > This i

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jun, This is a very good example. After thinking through this in detail, I agree that we need to commit offset with leader epoch in order to address this example. I think the remaining question is how to address the scenario that the topic is deleted and re-created. One possible solution is

Re: [VOTE] KIP-237: More Controller Health Metrics

Bump up the thread so that we can have these sensors to monitor our Kafka service sooner. On Mon, Dec 18, 2017 at 2:03 PM, Dong Lin <lindon...@gmail.com> wrote: > Hi all, > > Since there are no more outstanding comments, I would like to start voting > thread for KIP-237: https:

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

t in those cases if we remember the epoch when the data is fetched. > > Jun > > > > On Mon, Dec 18, 2017 at 1:58 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > Thanks much for your comments. These are very thoughtful ideas. Please >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

think you're saying that depending on the bug, in the worst case, you > > may have to downgrade the client. I think that's fair. Note that one > > advantage of making this a fatal error is that we'll be more likely to > hit > > unexpected edge cases in system tests. > >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

ier version. > > > Not sure I follow this. Wouldn't we just restart the consumer? That would > cause it to fetch the previous committed offset and then fetch the correct > metadata. > > Thanks, > Jason > > On Tue, Dec 19, 2017 at 10:36 AM, Dong Lin <lindon...@gmail.com

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

l a > bit less intuitive as well since it is rather difficult to explain the edge > case it is protecting. That said, we've hit other scenarios where being > able to detect stale metadata in the client would be helpful, so I think it > might be worth the tradeoff. > > -Jason > >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

-broker RPCs and the message format > itself, but just wanted to mention the fact that good encapsulation applies > to the client request API as well. > > Thanks, > Jason > > On Mon, Dec 18, 2017 at 1:58 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun

[VOTE] KIP-237: More Controller Health Metrics

Hi all, Since there are no more outstanding comments, I would like to start voting thread for KIP-237: https://cwiki.apache.org/confluence/display/KAFKA/KIP-237%3A+More+Controller+Health+Metrics The KIP proposes to add a few more metrics to help monitor Kafka Controller health. Thanks, Dong

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

than the leader_epoch = -1 in the OffetFetchResponse. Thus we don't need specific procedure for upgrades due to this change in the offset topic schema. By "upgrade nodes", do you mean the sentences we need to include in the upgrade.html in the PR later? > > Jun > > > On

Re: [DISCUSS] KIP-237: More Controller Health Metrics

queue, > but it would be good for Jun and Onur to confirm this. > > One minor comment: > > 1. For metric 1 and 2 in the KIP, do we want the type to be > ControllerEventManager or should it be ControllerStats like many other > Controller metrics? > > Ismael > > On Thu, Dec 7,

[VOTE] KIP-232: Detect outdated metadata using per-partition leaderEpoch field

Hi all, Since there are no more outstanding comments, I would like to start voting thread for KIP-232: https://cwiki.apache.org/confluence/display/KAFKA/KIP-232%3A+Detect+outdated+metadata+using+per-partition+leaderEpoch+field The KIP will fix an issue which can cause consumer to either lose

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

n't get any new metadata. > > We can potentially solve this problem by requiring some kind of regular > heartbeats between the controller and the broker. This may need some more > thoughts. So, it's probably fine to leave this to another KIP in the > future. > > Thanks, > >

Re: [DISCUSS] KIP-228 Negative record timestamp support

> >> > >> > >> On 12/5/17 1:24 PM, Konstantin Chukhlomin wrote: > >>> Hi Dong, > >>> > >>> Currently we are storing historical timestamp in the message. > >>> > >>> What we are trying to achieve is

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jun, I have updated the KIP based on our discussion. Thanks! Dong On Sat, Dec 9, 2017 at 10:12 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jun, > > Thanks much for your comments. Given that client needs to de-serialize the > metadata anyway, the extra overhead o

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

> detecting outdated metadata in a separate KIP in the future if you didn't > intend to address it in this KIP. > > Thanks, > > Jun > > > On Sat, Dec 9, 2017 at 10:12 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > Thanks much

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

t; > sessions, > > > > and > > > > > > > > > generally > > > > > > > > > > >>> tries to hide that information from the upper layers > > of the > > > > > > c

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

orrelation > > > ID > > > > > is not > > > > > > > > > > unique > > > > > > > > > > > >>> (and has no meaning) outside the context of that > > > single TCP > >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

; is not guaranteed to fetch metadata from every broker within some bounded > period of time. Maybe this is out of the scope of your KIP. But one idea is > force the consumer to refresh metadata from the controller periodically. > > Jun > > > On Thu, Dec 7, 2017 at 11:25 AM, Dong Lin &

Re: [DISCUSS] KIP-237: More Controller Health Metrics

wing metric, it seems that RequestRate > is better than EventRate. > > kafka.controller:type=ControllerChannelManager,name= > EventRateAndQueueTimeMs, > brokerId=someId > > Jun > > On Wed, Dec 6, 2017 at 6:21 PM, Dong Lin <lindon...@gmail.com> wrote: > > >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

> this, we need to change the fetch request protocol and the offset commit > api, which requires some more thoughts. > > Thanks, > > Jun > > > On Wed, Dec 6, 2017 at 10:57 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Bump up the thread. > > &

Re: [DISCUSS] KIP-237: More Controller Health Metrics

Thanks for the comment. I have updated the KIP to explain the value of these metrics. On Wed, Dec 6, 2017 at 6:38 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Thanks for the KIP. > > Can you add description for EventRateAndQueueTimeMs ? > > Cheers > > On Wed, Dec 6

[jira] [Resolved] (KAFKA-5878) Add sensor for queue size of the controller-event-thread

[ https://issues.apache.org/jira/browse/KAFKA-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-5878. - Resolution: Duplicate This metric will be added in https://issues.apache.org/jira/browse/KAFKA-3473

[DISCUSS] KIP-237: More Controller Health Metrics

Hi all, I have created KIP-237: More Controller Health Metrics https://cwiki.apache.org/confluence/display/KAFKA/KIP-237%3A+More+Controller+Health+Metrics . The KIP proposes to add a few more metrics to help monitor Kafka Controller health. Feedback and suggestions are welcome! Thanks, Dong

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

. > Plus, as Ismael mentioned, we already implement something very similar for > the TopicMetadataRequest API (when metadata of all topics is requested). > > Thanks again for reviewing the KIP closely and sharing your feedback. > > Regards. > --Vahid > > > > > Fro

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

they will have a reason to verify and speak with the admin > about > > > their usecase. > > > > > > > > > moreover, i think without this added capability it makes it > > > difficult/impossible to accurately use broker defaults for topics. > right

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Bump up the thread. It will be great to have more comments on whether we should do it or whether there is better way to address the motivation of this KIP. On Mon, Dec 4, 2017 at 3:09 PM, Dong Lin <lindon...@gmail.com> wrote: > I don't have an interesting rejected alternative soluti

Re: [DISCUSS] KIP-228 Negative record timestamp support

e perfectly valid > timestamps, and I can’t see any reason why Kafka shouldn’t support them - I > don’t think using `Long.MIN_VALUE` instead of -1 would necessarily add > complexity here? > > > Thanks, > Boerge. > > > > > On 2017-12-05, at 21:36, Dong Lin <lindon.

Re: [DISCUSS] KIP-228 Negative record timestamp support

e-kafka-new-york-times/ < > https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/> > has more information on the use case. > > > Thanks, > Boerge. > > > -- > > Boerge Svingen > Director of Engineering > The New York Times > > > >

[jira] [Reopened] (KAFKA-5878) Add sensor for queue size of the controller-event-thread

[ https://issues.apache.org/jira/browse/KAFKA-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin reopened KAFKA-5878: - > Add sensor for queue size of the controller-event-thr

[jira] [Resolved] (KAFKA-5878) Add sensor for queue size of the controller-event-thread

[ https://issues.apache.org/jira/browse/KAFKA-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-5878. - Resolution: Won't Fix > Add sensor for queue size of the controller-event-thr

[jira] [Resolved] (KAFKA-6285) OffsetCommitRequest should have read-after-write logic

[ https://issues.apache.org/jira/browse/KAFKA-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-6285. - Resolution: Duplicate Duplicate of KAFKA-6285 > OffsetCommitRequest should have read-after-write lo

Re: [DISCUSS] KIP-228 Negative record timestamp support

as a timestamp which publisher can set. > > Thanks, > Konstantin > > > On Dec 5, 2017, at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > Hey Konstantin, > > > > Thanks for the KIP. I have a few questions below. > > > > Strictly s

Re: [DISCUSS] KIP-228 Negative record timestamp support

Hey Konstantin, Thanks for the KIP. I have a few questions below. Strictly speaking Kafka actually allows you to store historical data. And user are free to encode arbitrary timestamp field in their Kafka message. For example, your Kafka message can currently have Json or Avro format and you can

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

our concerns. If I did not, or if I missed anything, > please let me know. Thanks. > > Regards. > --Vahid > > > > > From: Dong Lin <lindon...@gmail.com> > To: dev@kafka.apache.org > Date: 12/04/2017 01:43 PM > Subject:Re: [DISCUSS] KIP-231: Im

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

wrote: > It is clearer now. > > I noticed that Rejected Alternatives section is empty. > Have you considered any alternative ? > > Cheers > > On Mon, Dec 4, 2017 at 1:07 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Ted, thanks for catching this. I have updated the sen

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

+1 (non-binding) On Wed, Nov 29, 2017 at 7:05 PM, Hu Xi wrote: > Hi all, > > As I didn't see any further discussion around this KIP, I'd like to start > voting. > > KIP documentation: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

<dan.norw...@gmail.com> wrote: > updated again :) > > by having users always set all configs you lose the ability to use the > broker defaults as intended, since topic configs are overlaid. example in > the kip doc. > > dan > > On Mon, Dec 4, 2017 at 11:47 AM, Dong

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

I forgot another question. Can you provide a use-case where a user wants to list all groups for which he/she has the Describe access? Thanks, Dong On Mon, Dec 4, 2017 at 1:40 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Vahid, > > Thanks for the KIP. If I understand the you

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Hey Vahid, Thanks for the KIP. If I understand the you correctly, you want client to be able to list all the groups for which it currently has the describe access. As of now the ListGroupRequest does not allow user to specify the group. If user does not have the Describe Cluster access,

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

is the same but the controller_metadata_epoch > > Can you update the above sentence so that the intention is clearer ? > > Thanks > > On Fri, Dec 1, 2017 at 6:33 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hi all, > > > > I have created KIP-232: Detect outdated metada

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

essages in your topic in an incorrect/invalid state if you > do this. > > thanks, > dan > > On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Dan, > > > > Thanks for the KIP. Can you help me understand the motivation by &

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks for the KIP. Can you help me understand the motivation by providing a use-case that can not be easily completed without this KIP? It seems that most users will simply create the topic without worrying about the default configs. If a user has specific requirement for the default

[DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi all, I have created KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field: https://cwiki.apache.org/confluence/display/KAFKA/KIP-232%3A+Detect+outdated+metadata+by+adding+ControllerMetadataEpoch+field . The KIP proposes to add fields in MetadataResponse and

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Thu, Nov 30, 2017 at 9:37 AM, Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Nov 29, 2017, at 18:59, Dong Lin wrote: > > Hey Colin, > > > > Thanks much for the update. I have a few questions below: > > > > 1. I am not very sure that we need Fetch

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, Thanks much for the update. I have a few questions below: 1. I am not very sure that we need Fetch Session Epoch. It seems that Fetch Session Epoch is only needed to help leader distinguish between "a full fetch request" and "a full fetch request and request a new incremental fetch

[jira] [Created] (KAFKA-6285) OffsetCommitRequest should have read-after-write logic

Dong Lin created KAFKA-6285: --- Summary: OffsetCommitRequest should have read-after-write logic Key: KAFKA-6285 URL: https://issues.apache.org/jira/browse/KAFKA-6285 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-229: DeleteGroups API

Hey Vahid, Thanks for the KIP! This is certainly a useful one and users have been asking about the ability to delete group from the Kafka offset topic in my past experience. It seems that the protocol of the new request/response should probably include more fields fields. For example, it may be

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, On Mon, Nov 27, 2017 at 2:36 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Sat, Nov 25, 2017, at 21:25, Dong Lin wrote: > > Hey Colin, > > > > Thanks for the reply. Please see my comments inline. > > > > On Sat, Nov 25, 2017 at 3:33 PM, Co

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, Thanks for the reply. Please see my comments inline. On Sat, Nov 25, 2017 at 3:33 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Fri, Nov 24, 2017, at 22:06, Dong Lin wrote: > > Hey Colin, > > > > Thanks for the reply! Please see my comment inline. &

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, Thanks for the reply! Please see my comment inline. On Fri, Nov 24, 2017 at 9:39 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Thu, Nov 23, 2017, at 18:35, Dong Lin wrote: > > Hey Colin, > > > > Thanks for the KIP! This is definitely useful when there a

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, Thanks for the KIP! This is definitely useful when there are many idle partitions in the clusters. Just in case it is useful, I will provide some number here. We observe that for a clsuter that have around 2.5k partitions per broker, the ProduceRequestTotal time average value is

[jira] [Created] (KAFKA-6262) Consumer should not uses metadata that is older than the existing metadata

Dong Lin created KAFKA-6262: --- Summary: Consumer should not uses metadata that is older than the existing metadata Key: KAFKA-6262 URL: https://issues.apache.org/jira/browse/KAFKA-6262 Project: Kafka

[jira] [Created] (KAFKA-6258) SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data

Dong Lin created KAFKA-6258: --- Summary: SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data Key: KAFKA-6258 URL: https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-6230) KafkaConsumer.poll(0) should not block forever if coordinator is not available

Dong Lin created KAFKA-6230: --- Summary: KafkaConsumer.poll(0) should not block forever if coordinator is not available Key: KAFKA-6230 URL: https://issues.apache.org/jira/browse/KAFKA-6230 Project: Kafka

Re: Can someone update this KIP's status to "accepted"?

Thanks. I have updated it to be accepted. On Wed, Nov 8, 2017 at 1:33 PM, Jeff Widman wrote: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+ > UnderMinIsrPartitionCount+and+per-partition+UnderMinIsr+metrics > > the JIRA shows it was shipped in 1.0, but the

[jira] [Created] (KAFKA-6183) Broker should send OffsetCommitResponse only after it has written offset to cache

Dong Lin created KAFKA-6183: --- Summary: Broker should send OffsetCommitResponse only after it has written offset to cache Key: KAFKA-6183 URL: https://issues.apache.org/jira/browse/KAFKA-6183 Project: Kafka

[jira] [Resolved] (KAFKA-6172) Cache lastEntry in TimeIndex to avoid unnecessary disk access

[ https://issues.apache.org/jira/browse/KAFKA-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-6172. - Resolution: Fixed > Cache lastEntry in TimeIndex to avoid unnecessary disk acc

Re: [ANNOUNCE] New committer: Onur Karaman

Congratulations Onur! On Mon, Nov 6, 2017 at 9:24 AM, Jun Rao wrote: > Hi, everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur > Karaman. > > Onur's most significant work is the improvement of Kafka controller, which > is the brain of a

[jira] [Resolved] (KAFKA-3908) Set SendBufferSize for socket used by Processor

[ https://issues.apache.org/jira/browse/KAFKA-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-3908. - Resolution: Not A Problem > Set SendBufferSize for socket used by Proces

[jira] [Resolved] (KAFKA-6078) Investigate failure of ReassignPartitionsClusterTest.shouldExpandCluster

[ https://issues.apache.org/jira/browse/KAFKA-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-6078. - Resolution: Cannot Reproduce > Investigate failure of ReassignPartitionsClusterTest.shouldExpandClus

[jira] [Resolved] (KAFKA-5759) Allow user to specify relative path as log directory

[ https://issues.apache.org/jira/browse/KAFKA-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-5759. - Resolution: Fixed > Allow user to specify relative path as log direct

[jira] [Created] (KAFKA-6175) AbstractIndex should cache index file to avoid unnecessary disk access during resize()

Dong Lin created KAFKA-6175: --- Summary: AbstractIndex should cache index file to avoid unnecessary disk access during resize() Key: KAFKA-6175 URL: https://issues.apache.org/jira/browse/KAFKA-6175 Project

[jira] [Created] (KAFKA-6172) Cache lastEntry in TimeIndex to avoid unnecessary disk access

Dong Lin created KAFKA-6172: --- Summary: Cache lastEntry in TimeIndex to avoid unnecessary disk access Key: KAFKA-6172 URL: https://issues.apache.org/jira/browse/KAFKA-6172 Project: Kafka Issue

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

ameters. Also, we'd end up with 2 > methods in AdminClient. > > Ismael > > > On Wed, Oct 18, 2017 at 1:42 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Ismael, > > > > To clarify, I think you are saying that we should replace > > "deleteRecords(M

[jira] [Created] (KAFKA-6078) Investigate failure of ReassignPartitionsClusterTest.shouldExpandCluster

Dong Lin created KAFKA-6078: --- Summary: Investigate failure of ReassignPartitionsClusterTest.shouldExpandCluster Key: KAFKA-6078 URL: https://issues.apache.org/jira/browse/KAFKA-6078 Project: Kafka

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

t; additional options, not for defining the core behaviour. > > Ismael > > On Wed, Oct 18, 2017 at 12:27 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Colin, > > > > I have also thought about deleteRecordsBeforeOffset so that we can keep > the > >

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hey Colin, I have also thought about deleteRecordsBeforeOffset so that we can keep the name consistent with the existing API in the Scala AdminClient. But then I think it may be better to be able to specify in DeleteRecordsOptions whether the deletion is before/after timestamp or offset. By doing

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

Thanks for the KIP. +1 (non-binding) On Wed, Oct 11, 2017 at 2:27 AM, Ted Yu wrote: > +1 > > On Mon, Oct 2, 2017 at 10:51 PM, Paolo Patierno > wrote: > > > Hi all, > > > > I didn't see any further discussion around this KIP, so I'd like to start > > the

[jira] [Created] (KAFKA-6045) All access to log should fail if log is closed

Dong Lin created KAFKA-6045: --- Summary: All access to log should fail if log is closed Key: KAFKA-6045 URL: https://issues.apache.org/jira/browse/KAFKA-6045 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-5877) Controller should only update reassignment znode if there is change in the reassignment data

[ https://issues.apache.org/jira/browse/KAFKA-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-5877. - Resolution: Fixed > Controller should only update reassignment znode if there is cha

[jira] [Created] (KAFKA-5995) Rename AlterReplicaDir to AlterReplicaDirs

Dong Lin created KAFKA-5995: --- Summary: Rename AlterReplicaDir to AlterReplicaDirs Key: KAFKA-5995 URL: https://issues.apache.org/jira/browse/KAFKA-5995 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

+1 (non-binding) On Mon, Sep 18, 2017 at 1:04 PM, Ted Yu wrote: > +1 > > On Mon, Sep 18, 2017 at 9:19 AM, Paolo Patierno > wrote: > > > Hi devs, > > > > > > I'd like to start a discussion around adding the delete records > operation, > > already

[jira] [Created] (KAFKA-5879) Controller should read the latest IsrChangeNotification znodes when handling IsrChangeNotification event

Dong Lin created KAFKA-5879: --- Summary: Controller should read the latest IsrChangeNotification znodes when handling IsrChangeNotification event Key: KAFKA-5879 URL: https://issues.apache.org/jira/browse/KAFKA-5879

[jira] [Created] (KAFKA-5878) Add sensor for queue size of the controller-event-thread

Dong Lin created KAFKA-5878: --- Summary: Add sensor for queue size of the controller-event-thread Key: KAFKA-5878 URL: https://issues.apache.org/jira/browse/KAFKA-5878 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-5877) Controller should only update reassignment znode if there is change in the reassignment data

Dong Lin created KAFKA-5877: --- Summary: Controller should only update reassignment znode if there is change in the reassignment data Key: KAFKA-5877 URL: https://issues.apache.org/jira/browse/KAFKA-5877

[jira] [Resolved] (KAFKA-5845) KafkaController should send LeaderAndIsrRequest to broker which starts very soon after shutdown

[ https://issues.apache.org/jira/browse/KAFKA-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-5845. - Resolution: Won't Fix > KafkaController should send LeaderAndIsrRequest to broker which starts v

[jira] [Created] (KAFKA-5864) ReplicaFetcherThread.processPartitionData() should throw KafkaStorageException if replica is offline

Dong Lin created KAFKA-5864: --- Summary: ReplicaFetcherThread.processPartitionData() should throw KafkaStorageException if replica is offline Key: KAFKA-5864 URL: https://issues.apache.org/jira/browse/KAFKA-5864

[jira] [Created] (KAFKA-5845) KafkaController should send LeaderAndIsrRequest to brokers which starts very soon after shutdown

Dong Lin created KAFKA-5845: --- Summary: KafkaController should send LeaderAndIsrRequest to brokers which starts very soon after shutdown Key: KAFKA-5845 URL: https://issues.apache.org/jira/browse/KAFKA-5845

[jira] [Created] (KAFKA-5843) Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true

Dong Lin created KAFKA-5843: --- Summary: Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true Key: KAFKA-5843 URL: https://issues.apache.org/jira/browse/KAFKA-5843 Project: Kafka

[jira] [Created] (KAFKA-5829) Speedup broker startup after unclean shutdown by reducing unnecessary snapshot files deletion

Dong Lin created KAFKA-5829: --- Summary: Speedup broker startup after unclean shutdown by reducing unnecessary snapshot files deletion Key: KAFKA-5829 URL: https://issues.apache.org/jira/browse/KAFKA-5829

[jira] [Created] (KAFKA-5812) Represent logDir with case class where absolute path is required

Dong Lin created KAFKA-5812: --- Summary: Represent logDir with case class where absolute path is required Key: KAFKA-5812 URL: https://issues.apache.org/jira/browse/KAFKA-5812 Project: Kafka Issue

[jira] [Created] (KAFKA-5767) Kafka server should halt if IBP < 1.0.0 and there is log directory failure

Dong Lin created KAFKA-5767: --- Summary: Kafka server should halt if IBP < 1.0.0 and there is log directory failure Key: KAFKA-5767 URL: https://issues.apache.org/jira/browse/KAFKA-5767 Project: Ka

[jira] [Created] (KAFKA-5759) Allow user to specify relative path as log directory

Dong Lin created KAFKA-5759: --- Summary: Allow user to specify relative path as log directory Key: KAFKA-5759 URL: https://issues.apache.org/jira/browse/KAFKA-5759 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

es sense. > > > > > > In your early email, I thought you were proposing to have > > > PartitionReassignmentRequest > > > dealing with both inter and intra broker data movement (i.e., include > log > > > dirs in the request). Then, I am not sure

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

gt; > > > > As Dong says, this doesn't work for the intra-broker case: > > > > > > Note that we can not calculate lag as max(0, HW - LEO) > > >> because we still need the difference between two lags to measure the > > >> progress of intra-broker r

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Hey Apurva, Thanks for the KIP. I have read through the KIP and the prior discussion in this thread. I have three concerns that are related to Becket's comments: - Is it true that, as Becket has mentioned, producer.flush() may block infinitely if retries=MAX_INT? This seems like a possible

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

HW. So, as long as a replica is in sync, the lag is always > 0. > > So, I was suggesting to return lag instead of LEO in DescribeDirsResponse > for each replica. I am not sure if we need to return HW though. > > Thanks, > > Jun > > On Wed, Aug 9, 2017 at 5:01 PM, Dong Lin &

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

seconds. I also assumed that 10 seconds is probably not a big deal given the typical time length of the reassignment. Thanks, Dong On Wed, Aug 9, 2017 at 4:40 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jun, > > If I understand you right, you are suggesting that, i

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

o the > user. > > Thanks, > > Jun > > > > On Tue, Aug 8, 2017 at 6:26 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Jun, > > > > Thanks for the comment! > > > > Yes, it should work. The tool can send request to any broker and broke

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

tead > return the lag in offset per replica. This way, the status can probably be > reported more reliably. > > Thanks, > > Jun > > On Tue, Aug 8, 2017 at 11:23 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Tom, > > > > Thanks for the quick rep

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Tom, Thanks for the quick reply. Please see my comment inline. On Tue, Aug 8, 2017 at 11:06 AM, Tom Bentley wrote: > Hi Dong, > > Replies inline, as usual > > > As I originally envisaged it, KIP-179's support for reassigning > partitions > > > > would have

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks for your reply. Yes, my original idea is that user can continue to collect the static information for reassignment as they are doing now. It is the status quo. I agree it can be beneficial to have a tool in Kafka to collect other information that may be needed for reassignment so that user

[jira] [Created] (KAFKA-5710) KafkaAdminClient should remove inflight call correctly after response is received

Dong Lin created KAFKA-5710: --- Summary: KafkaAdminClient should remove inflight call correctly after response is received Key: KAFKA-5710 URL: https://issues.apache.org/jira/browse/KAFKA-5710 Project: Kafka

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

device IO > for example) between the disks on a broker, so maybe including the total > disk capacity is only of limited use. > > Cheers, > > Tom > > On 7 August 2017 at 17:54, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Tom, > > > > Good question. We have

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Ismael, Thanks much for your comments. Please see my reply inline. On Mon, Aug 7, 2017 at 5:28 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Dong, > > Thanks for the explanation. Comments inline. > > On Fri, Aug 4, 2017 at 6:47 PM, Dong Lin <lindon...@gmail.com>

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

nformation to have to help figure out that certain > assignments are impossible, for instance. Is there a reason you've left > this out? > > Cheers, > > Tom > > On 4 August 2017 at 18:47, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Ismael, > > > >

Re: [DISCUSS] KIP-178: Size-based log directory selection strategy

selection strategy > > > Dong, yes, many thanks for the comments from the second mail. Will take > some time to figure out an algorithm to better handle the situation you > mentioned. Thanks again. > > > > 发件人: Dong Lin <lindon...@gmail.com> &g

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

on the > rationale for the design choices that were made and rejected. > > 3. Should changeReplicaDir be alterReplicaDir? We have used `alter` for > other methods. > > Thanks, > Ismael > > > > > On Fri, Aug 4, 2017 at 5:37 AM, Dong Lin <lindon...@gmail.com>

<    1   2   3   4   5   6   7   8   9   10   >