Re: [DISCUSS] KIP-17 - Add HighwaterMarkOffset to OffsetFetchResponse

2015-03-26 Thread Todd Palino
I agree with Jun here, that it would make it easier to do lag checking. However, for individual checks it's really not that much trouble to do the second request. If you're doing a lot of lag checking (like every consumer and every topic) where the scale would start to make a difference, I would

Re: [DISCUSS] KIP-17 - Add HighwaterMarkOffset to OffsetFetchResponse

2015-03-26 Thread Jun Rao
Grant, In addition to FetchRequest, currently we have another way to get the high watermark through OffsetRequest ( https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetRequest). OffsetRequest is a read-only request and is much lighter

[DISCUSS] KIP-17 - Add HighwaterMarkOffset to OffsetFetchResponse

2015-03-24 Thread Grant Henke
Here is an initial proposal to add HighwaterMarkOffset to the OffsetFetchResponse: https://cwiki.apache.org/confluence/display/KAFKA/KIP-17+-+Add+HighwaterMarkOffset+to+OffsetFetchResponse I can add a jira and more implementation details if the initial proposal has interest. Thanks, Grant --