Errors related to Kafka API checks
Hi, there, I have a problem when I routinely run the following command line. In addition, I am experiencing process hangs on one Broker at irregular intervals, but I do not know the relevance of the error content. Has anyone had the same problem to help me? ○Kafka Broker Server Information ・3-unit cluster configuration ・Apache kafka version 3.2.1 ○Commandline bin/kafka-broker-api-versions.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 ○Error ERROR admin-client-network-thread exited (kafka.admin.BrokerApiVersionsCommand$AdminClient) org.apache.kafka.common.errors.DisconnectException: NetworkClient is no longer active, state is CLOSED Best regards Sharp Odashima
[jira] [Resolved] (KAFKA-13958) Expose logdirs total and usable space via Kafka API
[ https://issues.apache.org/jira/browse/KAFKA-13958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-13958. Fix Version/s: 3.3.0 Resolution: Fixed > Expose logdirs total and usable space via Kafka API > --- > > Key: KAFKA-13958 > URL: https://issues.apache.org/jira/browse/KAFKA-13958 > Project: Kafka > Issue Type: Bug >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > Fix For: 3.3.0 > > > JIRA for KIP-827: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API -- This message was sent by Atlassian Jira (v8.20.7#820007)
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Cong, Some of use cases I have in mind are more around validation that an operation was successful. - Let's say you trigger a reassignment to even out disk usage on some brokers. Your tool will submit the reassignment and wait for completion using the AlterPartitionReassignments and ListPartitionReassignments APIs respectively. Once it is complete you may want to check the new disk usage on your brokers to validate the reassignment did what you wanted. In that case you'd only call DescribeLogDirs once. - A similar example is when resizing volumes. You typically will query repetitively the API that handles the volume resize and at the end make a single call to DescribeLogDirs to validate brokers indeed picked up the updated volumes. Thanks, Mickael On Fri, Jun 3, 2022 at 9:08 PM Cong Ding wrote: > > Thank you, Mickael. One more question: are you imaging these > tooling/automation to call this API at a very low frequency? since > high-frequency calls to this API are prohibitively expensive. Can you > give some examples of low-frequency call use cases? I can think of > some high-frequency call use cases which are valid in this case, but I > had a hard time coming up with low-frequency call use cases. > > The one you give in the KIP is validating whether disk resize > operations have been completed. However, this looks like a > high-frequency call use case to me because we need to keep monitoring > disk usage before and after resizing. > > Cong > > On Fri, Jun 3, 2022 at 5:22 AM Mickael Maison > wrote: > > > > Hi Cong, > > > > Maybe some people can do without this KIP. > > But in many cases, especially around tooling and automation, it's > > useful to be able to retrieve disk utilization values via the Kafka > > API rather than interfacing with a metrics system. > > > > Does that clarify the motivation? > > > > Thanks, > > Mickael > > > > On Wed, Jun 1, 2022 at 7:10 PM Cong Ding wrote: > > > > > > Thanks for the explanation. I think the question is that if we have disk > > > utilization in our environment, what is the use case for KIP-827? The disk > > > utilization in our environment can already do the job. Is there anything I > > > missed? > > > > > > Thanks, > > > Cong > > > > > > On Tue, May 31, 2022 at 2:57 AM Mickael Maison > > > wrote: > > > > > > > Hi Cong, > > > > > > > > Kafka does not expose disk utilization metrics. This is something you > > > > need to provide in your environment. You definitively should have a > > > > mechanism for exposing metrics from your Kafka broker hosts and you > > > > should absolutely monitor disk usage and have appropriate alerts. > > > > > > > > Thanks, > > > > Mickael > > > > > > > > On Thu, May 26, 2022 at 7:34 PM Jun Rao > > > > wrote: > > > > > > > > > > Hi, Igor, > > > > > > > > > > Thanks for the reply. > > > > > > > > > > I agree that this KIP could be useful for improving the tool for > > > > > moving > > > > > data across disks. It would be useful to clarify on the main > > > > > motivation > > > > of > > > > > the KIP. Also, DescribeLogDirsResponse already includes the size of > > > > > each > > > > > partition on a disk. So, it seems that UsableBytes is redundant since > > > > it's > > > > > derivable. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > This can also be quite useful to make better use of existing > > > > functionality > > > > > > in the Kafka API — moving replicas between log directories via > > > > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also > > > > > > available > > > > the > > > > > > caller can make better decisions using the same API. It means a more > > > > > > consistent way of interacting with Kafka to manage replicas > > > > > > locations > > > > > > within a broker without having to correlate Kafka metrics with > > > > information > > > > > > from the Kafka API. > > > > > > > > > > > > --
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Thank you, Mickael. One more question: are you imaging these
tooling/automation to call this API at a very low frequency? since
high-frequency calls to this API are prohibitively expensive. Can you
give some examples of low-frequency call use cases? I can think of
some high-frequency call use cases which are valid in this case, but I
had a hard time coming up with low-frequency call use cases.
The one you give in the KIP is validating whether disk resize
operations have been completed. However, this looks like a
high-frequency call use case to me because we need to keep monitoring
disk usage before and after resizing.
Cong
On Fri, Jun 3, 2022 at 5:22 AM Mickael Maison wrote:
>
> Hi Cong,
>
> Maybe some people can do without this KIP.
> But in many cases, especially around tooling and automation, it's
> useful to be able to retrieve disk utilization values via the Kafka
> API rather than interfacing with a metrics system.
>
> Does that clarify the motivation?
>
> Thanks,
> Mickael
>
> On Wed, Jun 1, 2022 at 7:10 PM Cong Ding wrote:
> >
> > Thanks for the explanation. I think the question is that if we have disk
> > utilization in our environment, what is the use case for KIP-827? The disk
> > utilization in our environment can already do the job. Is there anything I
> > missed?
> >
> > Thanks,
> > Cong
> >
> > On Tue, May 31, 2022 at 2:57 AM Mickael Maison
> > wrote:
> >
> > > Hi Cong,
> > >
> > > Kafka does not expose disk utilization metrics. This is something you
> > > need to provide in your environment. You definitively should have a
> > > mechanism for exposing metrics from your Kafka broker hosts and you
> > > should absolutely monitor disk usage and have appropriate alerts.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, May 26, 2022 at 7:34 PM Jun Rao wrote:
> > > >
> > > > Hi, Igor,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > I agree that this KIP could be useful for improving the tool for moving
> > > > data across disks. It would be useful to clarify on the main motivation
> > > of
> > > > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > > > partition on a disk. So, it seems that UsableBytes is redundant since
> > > it's
> > > > derivable.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > This can also be quite useful to make better use of existing
> > > functionality
> > > > > in the Kafka API — moving replicas between log directories via
> > > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> > > the
> > > > > caller can make better decisions using the same API. It means a more
> > > > > consistent way of interacting with Kafka to manage replicas locations
> > > > > within a broker without having to correlate Kafka metrics with
> > > information
> > > > > from the Kafka API.
> > > > >
> > > > > --
> > > > > Igor
> > > > >
> > > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > > Hi, Mickael,
> > > > > >
> > > > > > Thanks for the KIP. Since this is mostly for monitoring and
> > > alerting,
> > > > > > could we expose them as metrics instead of as part of the API? We
> > > already
> > > > > > have a size metric per log. Perhaps we could extend that to add
> > > > > used/total
> > > > > > metrics per disk?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello Mikael,
> > > > > >>
> > > > > >> Thanks for the KIP.
> > > > > >>
> > > > > >> I see that the API response contains some information about each
> > > > > partition.
> > > > > >> ```
> > > > > >> { "name": "PartitionSize", "type": "int64&quo
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Cong,
Maybe some people can do without this KIP.
But in many cases, especially around tooling and automation, it's
useful to be able to retrieve disk utilization values via the Kafka
API rather than interfacing with a metrics system.
Does that clarify the motivation?
Thanks,
Mickael
On Wed, Jun 1, 2022 at 7:10 PM Cong Ding wrote:
>
> Thanks for the explanation. I think the question is that if we have disk
> utilization in our environment, what is the use case for KIP-827? The disk
> utilization in our environment can already do the job. Is there anything I
> missed?
>
> Thanks,
> Cong
>
> On Tue, May 31, 2022 at 2:57 AM Mickael Maison
> wrote:
>
> > Hi Cong,
> >
> > Kafka does not expose disk utilization metrics. This is something you
> > need to provide in your environment. You definitively should have a
> > mechanism for exposing metrics from your Kafka broker hosts and you
> > should absolutely monitor disk usage and have appropriate alerts.
> >
> > Thanks,
> > Mickael
> >
> > On Thu, May 26, 2022 at 7:34 PM Jun Rao wrote:
> > >
> > > Hi, Igor,
> > >
> > > Thanks for the reply.
> > >
> > > I agree that this KIP could be useful for improving the tool for moving
> > > data across disks. It would be useful to clarify on the main motivation
> > of
> > > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > > partition on a disk. So, it seems that UsableBytes is redundant since
> > it's
> > > derivable.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> > >
> > > > Hi,
> > > >
> > > > This can also be quite useful to make better use of existing
> > functionality
> > > > in the Kafka API — moving replicas between log directories via
> > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> > the
> > > > caller can make better decisions using the same API. It means a more
> > > > consistent way of interacting with Kafka to manage replicas locations
> > > > within a broker without having to correlate Kafka metrics with
> > information
> > > > from the Kafka API.
> > > >
> > > > --
> > > > Igor
> > > >
> > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > Hi, Mickael,
> > > > >
> > > > > Thanks for the KIP. Since this is mostly for monitoring and
> > alerting,
> > > > > could we expose them as metrics instead of as part of the API? We
> > already
> > > > > have a size metric per log. Perhaps we could extend that to add
> > > > used/total
> > > > > metrics per disk?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > > > >
> > > > > wrote:
> > > > >
> > > > >> Hello Mikael,
> > > > >>
> > > > >> Thanks for the KIP.
> > > > >>
> > > > >> I see that the API response contains some information about each
> > > > partition.
> > > > >> ```
> > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > >> "about": "The size of the log segments in this partition in
> > bytes." }
> > > > >> ```
> > > > >> Can this be summed up to provide a used space in a `log.dir`
> > > > >> This will also be specific to a `log.dir` (for the case where
> > multiple
> > > > >> log.dir are hosted on the same underlying device)
> > > > >>
> > > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> >
> > > > >> wrote:
> > > > >> >
> > > > >> > Hey Mickael,
> > > > >> >
> > > > >> > Great KIP!
> > > > >> >
> > > > >> > I have one question:
> > > > >> >
> > > > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > > > change
> > > > >> > should not
> > > > >> > significantly affect the latency of this API." and &qu
[jira] [Created] (KAFKA-13958) Expose logdirs total and usable space via Kafka API
Mickael Maison created KAFKA-13958: -- Summary: Expose logdirs total and usable space via Kafka API Key: KAFKA-13958 URL: https://issues.apache.org/jira/browse/KAFKA-13958 Project: Kafka Issue Type: Bug Reporter: Mickael Maison Assignee: Mickael Maison JIRA for KIP-827: https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API -- This message was sent by Atlassian Jira (v8.20.7#820007)
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Thanks for the explanation. I think the question is that if we have disk
utilization in our environment, what is the use case for KIP-827? The disk
utilization in our environment can already do the job. Is there anything I
missed?
Thanks,
Cong
On Tue, May 31, 2022 at 2:57 AM Mickael Maison
wrote:
> Hi Cong,
>
> Kafka does not expose disk utilization metrics. This is something you
> need to provide in your environment. You definitively should have a
> mechanism for exposing metrics from your Kafka broker hosts and you
> should absolutely monitor disk usage and have appropriate alerts.
>
> Thanks,
> Mickael
>
> On Thu, May 26, 2022 at 7:34 PM Jun Rao wrote:
> >
> > Hi, Igor,
> >
> > Thanks for the reply.
> >
> > I agree that this KIP could be useful for improving the tool for moving
> > data across disks. It would be useful to clarify on the main motivation
> of
> > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > partition on a disk. So, it seems that UsableBytes is redundant since
> it's
> > derivable.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> >
> > > Hi,
> > >
> > > This can also be quite useful to make better use of existing
> functionality
> > > in the Kafka API — moving replicas between log directories via
> > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> the
> > > caller can make better decisions using the same API. It means a more
> > > consistent way of interacting with Kafka to manage replicas locations
> > > within a broker without having to correlate Kafka metrics with
> information
> > > from the Kafka API.
> > >
> > > --
> > > Igor
> > >
> > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > Hi, Mickael,
> > > >
> > > > Thanks for the KIP. Since this is mostly for monitoring and
> alerting,
> > > > could we expose them as metrics instead of as part of the API? We
> already
> > > > have a size metric per log. Perhaps we could extend that to add
> > > used/total
> > > > metrics per disk?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > >
> > > > wrote:
> > > >
> > > >> Hello Mikael,
> > > >>
> > > >> Thanks for the KIP.
> > > >>
> > > >> I see that the API response contains some information about each
> > > partition.
> > > >> ```
> > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > >> "about": "The size of the log segments in this partition in
> bytes." }
> > > >> ```
> > > >> Can this be summed up to provide a used space in a `log.dir`
> > > >> This will also be specific to a `log.dir` (for the case where
> multiple
> > > >> log.dir are hosted on the same underlying device)
> > > >>
> > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
>
> > > >> wrote:
> > > >> >
> > > >> > Hey Mickael,
> > > >> >
> > > >> > Great KIP!
> > > >> >
> > > >> > I have one question:
> > > >> >
> > > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > > change
> > > >> > should not
> > > >> > significantly affect the latency of this API." and "That would
> allow
> > > to
> > > >> > easily validate whether disk operations (like a resize), or topic
> > > >> deletion
> > > >> > (log deletion only happen after a short delay) have completed." I
> > > wonder
> > > >> if
> > > >> > there is an existing metric/API that can allow administrators to
> > > >> determine
> > > >> > whether we need to resize? If administrators use this API to
> determine
> > > >> > whether we need a resize, would this API become a high-volume
> API? I
> > > >> > understand we don't want this API to be a high-volume one because
> the
> > > API
> > > >> > is already costly by returning `"name": "Topics"`.
> > > >> >
> > > >> > Cong
> > > >> >
> > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > I wrote a small KIP to expose the total and usable space of
> logdirs
> > > >> > > via the DescribeLogDirs API:
> > > >> > >
> > > >> > >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > >> > >
> > > >> > > Please take a look and let me know if you have any feedback.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Mickael
> > > >> > >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best Regards,
> > > >> Raman Verma
> > > >>
> > >
>
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, Thanks for the votes and feedback! The vote passes with: - 4 +1 (binding) votes from Luke, Tom, Jun and John - 3 +1 (non-binding) votes from Igor, Divij and Federico Thanks, Mickael On Wed, Jun 1, 2022 at 3:31 AM John Roesler wrote: > > Thanks for the KIP Mickael, > > I'm +1 (binding) > > -John > > On Tue, May 31, 2022, at 18:48, Jun Rao wrote: > > Hi, Mickael, > > > > Thanks for the KIP. +1 > > > > Jun > > > > On Wed, May 25, 2022 at 7:54 AM Tom Bentley wrote: > > > >> Hi Mickael, > >> > >> Thanks for the KIP! +1 (binding). > >> > >> Kind regards, > >> > >> Tom > >> > >> On Thu, 19 May 2022 at 11:28, Federico Valeri > >> wrote: > >> > >> > Thanks Mickael. > >> > > >> > +1 (non binding) > >> > > >> > On Wed, May 18, 2022 at 11:08 AM Divij Vaidya > >> > wrote: > >> > > > >> > > +1 non binding. > >> > > > >> > > Divij Vaidya > >> > > > >> > > > >> > > > >> > > On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: > >> > > > >> > > > Thanks for this KIP Mickael. > >> > > > > >> > > > +1 non binding > >> > > > > >> > > > -- > >> > > > Igor > >> > > > > >> > > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > >> > > > > Hi Mickael, > >> > > > > > >> > > > > +1 (binding) from me. > >> > > > > Thanks for the KIP! > >> > > > > > >> > > > > Luke > >> > > > > > >> > > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison < > >> > mickael.mai...@gmail.com > >> > > > > > >> > > > > wrote: > >> > > > > > >> > > > >> Hi, > >> > > > >> > >> > > > >> I'd like to start a vote on KIP-827. It proposes exposing the > >> total > >> > > > >> and usable space of logdirs > >> > > > >> via the DescribeLogDirs API: > >> > > > >> > >> > > > >> > >> > > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > >> > > > >> > >> > > > >> Thanks, > >> > > > >> Mickael > >> > > > >> > >> > > > > >> > > >> > > >>
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Thanks for the KIP Mickael, I'm +1 (binding) -John On Tue, May 31, 2022, at 18:48, Jun Rao wrote: > Hi, Mickael, > > Thanks for the KIP. +1 > > Jun > > On Wed, May 25, 2022 at 7:54 AM Tom Bentley wrote: > >> Hi Mickael, >> >> Thanks for the KIP! +1 (binding). >> >> Kind regards, >> >> Tom >> >> On Thu, 19 May 2022 at 11:28, Federico Valeri >> wrote: >> >> > Thanks Mickael. >> > >> > +1 (non binding) >> > >> > On Wed, May 18, 2022 at 11:08 AM Divij Vaidya >> > wrote: >> > > >> > > +1 non binding. >> > > >> > > Divij Vaidya >> > > >> > > >> > > >> > > On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: >> > > >> > > > Thanks for this KIP Mickael. >> > > > >> > > > +1 non binding >> > > > >> > > > -- >> > > > Igor >> > > > >> > > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: >> > > > > Hi Mickael, >> > > > > >> > > > > +1 (binding) from me. >> > > > > Thanks for the KIP! >> > > > > >> > > > > Luke >> > > > > >> > > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison < >> > mickael.mai...@gmail.com >> > > > > >> > > > > wrote: >> > > > > >> > > > >> Hi, >> > > > >> >> > > > >> I'd like to start a vote on KIP-827. It proposes exposing the >> total >> > > > >> and usable space of logdirs >> > > > >> via the DescribeLogDirs API: >> > > > >> >> > > > >> >> > > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API >> > > > >> >> > > > >> Thanks, >> > > > >> Mickael >> > > > >> >> > > > >> > >> > >>
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, Mickael, Thanks for the KIP. +1 Jun On Wed, May 25, 2022 at 7:54 AM Tom Bentley wrote: > Hi Mickael, > > Thanks for the KIP! +1 (binding). > > Kind regards, > > Tom > > On Thu, 19 May 2022 at 11:28, Federico Valeri > wrote: > > > Thanks Mickael. > > > > +1 (non binding) > > > > On Wed, May 18, 2022 at 11:08 AM Divij Vaidya > > wrote: > > > > > > +1 non binding. > > > > > > Divij Vaidya > > > > > > > > > > > > On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: > > > > > > > Thanks for this KIP Mickael. > > > > > > > > +1 non binding > > > > > > > > -- > > > > Igor > > > > > > > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > > > > > Hi Mickael, > > > > > > > > > > +1 (binding) from me. > > > > > Thanks for the KIP! > > > > > > > > > > Luke > > > > > > > > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison < > > mickael.mai...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> I'd like to start a vote on KIP-827. It proposes exposing the > total > > > > >> and usable space of logdirs > > > > >> via the DescribeLogDirs API: > > > > >> > > > > >> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > >> > > > > >> Thanks, > > > > >> Mickael > > > > >> > > > > > > > > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, Mickael,
Thanks for the explanation. The KIP looks to me now.
Jun
On Tue, May 31, 2022 at 6:44 AM Mickael Maison
wrote:
> Hi Jun,
>
> Igor answered to your question.
> Users should rely on their host metrics to monitor disk usage. But
> with tooling and automation it's sometimes not ideal to retrieve
> values from metrics. So exposing disk usage via the Kafka API will
> simplify coordinating disk operations.
>
> I've updated the motivation section to make it clear why being able to
> retrieve these values via the API rather than metrics is interesting.
>
> Thanks,
> Mickael
>
> On Tue, May 31, 2022 at 3:38 PM Mickael Maison
> wrote:
> >
> > Hi Raman,
> >
> > Unfortunately the replica size only includes the log files and it does
> > not include indexes or other metadata files. Obviously any extra non
> > Kafka files are also not included either.
> >
> > For these reasons, I decided to have a separate field with the actual
> > usable space reported by the volume.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Tue, May 31, 2022 at 11:57 AM Mickael Maison
> > wrote:
> > >
> > > Hi Cong,
> > >
> > > Kafka does not expose disk utilization metrics. This is something you
> > > need to provide in your environment. You definitively should have a
> > > mechanism for exposing metrics from your Kafka broker hosts and you
> > > should absolutely monitor disk usage and have appropriate alerts.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, May 26, 2022 at 7:34 PM Jun Rao
> wrote:
> > > >
> > > > Hi, Igor,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > I agree that this KIP could be useful for improving the tool for
> moving
> > > > data across disks. It would be useful to clarify on the main
> motivation of
> > > > the KIP. Also, DescribeLogDirsResponse already includes the size of
> each
> > > > partition on a disk. So, it seems that UsableBytes is redundant
> since it's
> > > > derivable.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > This can also be quite useful to make better use of existing
> functionality
> > > > > in the Kafka API — moving replicas between log directories via
> > > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also
> available the
> > > > > caller can make better decisions using the same API. It means a
> more
> > > > > consistent way of interacting with Kafka to manage replicas
> locations
> > > > > within a broker without having to correlate Kafka metrics with
> information
> > > > > from the Kafka API.
> > > > >
> > > > > --
> > > > > Igor
> > > > >
> > > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > > Hi, Mickael,
> > > > > >
> > > > > > Thanks for the KIP. Since this is mostly for monitoring and
> alerting,
> > > > > > could we expose them as metrics instead of as part of the API?
> We already
> > > > > > have a size metric per log. Perhaps we could extend that to add
> > > > > used/total
> > > > > > metrics per disk?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello Mikael,
> > > > > >>
> > > > > >> Thanks for the KIP.
> > > > > >>
> > > > > >> I see that the API response contains some information about each
> > > > > partition.
> > > > > >> ```
> > > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > > >> "about": "The size of the log segments in this partition in
> bytes." }
> > > > > >> ```
> > > > > >> Can this be summed up to provide a used space in a `log.dir`
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Jun,
Igor answered to your question.
Users should rely on their host metrics to monitor disk usage. But
with tooling and automation it's sometimes not ideal to retrieve
values from metrics. So exposing disk usage via the Kafka API will
simplify coordinating disk operations.
I've updated the motivation section to make it clear why being able to
retrieve these values via the API rather than metrics is interesting.
Thanks,
Mickael
On Tue, May 31, 2022 at 3:38 PM Mickael Maison wrote:
>
> Hi Raman,
>
> Unfortunately the replica size only includes the log files and it does
> not include indexes or other metadata files. Obviously any extra non
> Kafka files are also not included either.
>
> For these reasons, I decided to have a separate field with the actual
> usable space reported by the volume.
>
> Thanks,
> Mickael
>
>
>
> On Tue, May 31, 2022 at 11:57 AM Mickael Maison
> wrote:
> >
> > Hi Cong,
> >
> > Kafka does not expose disk utilization metrics. This is something you
> > need to provide in your environment. You definitively should have a
> > mechanism for exposing metrics from your Kafka broker hosts and you
> > should absolutely monitor disk usage and have appropriate alerts.
> >
> > Thanks,
> > Mickael
> >
> > On Thu, May 26, 2022 at 7:34 PM Jun Rao wrote:
> > >
> > > Hi, Igor,
> > >
> > > Thanks for the reply.
> > >
> > > I agree that this KIP could be useful for improving the tool for moving
> > > data across disks. It would be useful to clarify on the main motivation of
> > > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > > partition on a disk. So, it seems that UsableBytes is redundant since it's
> > > derivable.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> > >
> > > > Hi,
> > > >
> > > > This can also be quite useful to make better use of existing
> > > > functionality
> > > > in the Kafka API — moving replicas between log directories via
> > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> > > > the
> > > > caller can make better decisions using the same API. It means a more
> > > > consistent way of interacting with Kafka to manage replicas locations
> > > > within a broker without having to correlate Kafka metrics with
> > > > information
> > > > from the Kafka API.
> > > >
> > > > --
> > > > Igor
> > > >
> > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > Hi, Mickael,
> > > > >
> > > > > Thanks for the KIP. Since this is mostly for monitoring and alerting,
> > > > > could we expose them as metrics instead of as part of the API? We
> > > > > already
> > > > > have a size metric per log. Perhaps we could extend that to add
> > > > used/total
> > > > > metrics per disk?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > > > > > > >
> > > > > wrote:
> > > > >
> > > > >> Hello Mikael,
> > > > >>
> > > > >> Thanks for the KIP.
> > > > >>
> > > > >> I see that the API response contains some information about each
> > > > partition.
> > > > >> ```
> > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > >> "about": "The size of the log segments in this partition in
> > > > >> bytes." }
> > > > >> ```
> > > > >> Can this be summed up to provide a used space in a `log.dir`
> > > > >> This will also be specific to a `log.dir` (for the case where
> > > > >> multiple
> > > > >> log.dir are hosted on the same underlying device)
> > > > >>
> > > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> > > > >>
> > > > >> wrote:
> > > > >> >
> > > > >> > Hey Mickael,
> > > > >> >
> > > > >> > Great KIP!
> > > > >&
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Raman,
Unfortunately the replica size only includes the log files and it does
not include indexes or other metadata files. Obviously any extra non
Kafka files are also not included either.
For these reasons, I decided to have a separate field with the actual
usable space reported by the volume.
Thanks,
Mickael
On Tue, May 31, 2022 at 11:57 AM Mickael Maison
wrote:
>
> Hi Cong,
>
> Kafka does not expose disk utilization metrics. This is something you
> need to provide in your environment. You definitively should have a
> mechanism for exposing metrics from your Kafka broker hosts and you
> should absolutely monitor disk usage and have appropriate alerts.
>
> Thanks,
> Mickael
>
> On Thu, May 26, 2022 at 7:34 PM Jun Rao wrote:
> >
> > Hi, Igor,
> >
> > Thanks for the reply.
> >
> > I agree that this KIP could be useful for improving the tool for moving
> > data across disks. It would be useful to clarify on the main motivation of
> > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > partition on a disk. So, it seems that UsableBytes is redundant since it's
> > derivable.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> >
> > > Hi,
> > >
> > > This can also be quite useful to make better use of existing functionality
> > > in the Kafka API — moving replicas between log directories via
> > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available the
> > > caller can make better decisions using the same API. It means a more
> > > consistent way of interacting with Kafka to manage replicas locations
> > > within a broker without having to correlate Kafka metrics with information
> > > from the Kafka API.
> > >
> > > --
> > > Igor
> > >
> > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > Hi, Mickael,
> > > >
> > > > Thanks for the KIP. Since this is mostly for monitoring and alerting,
> > > > could we expose them as metrics instead of as part of the API? We
> > > > already
> > > > have a size metric per log. Perhaps we could extend that to add
> > > used/total
> > > > metrics per disk?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > > > > >
> > > > wrote:
> > > >
> > > >> Hello Mikael,
> > > >>
> > > >> Thanks for the KIP.
> > > >>
> > > >> I see that the API response contains some information about each
> > > partition.
> > > >> ```
> > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > >> "about": "The size of the log segments in this partition in bytes." }
> > > >> ```
> > > >> Can this be summed up to provide a used space in a `log.dir`
> > > >> This will also be specific to a `log.dir` (for the case where multiple
> > > >> log.dir are hosted on the same underlying device)
> > > >>
> > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> > > >> wrote:
> > > >> >
> > > >> > Hey Mickael,
> > > >> >
> > > >> > Great KIP!
> > > >> >
> > > >> > I have one question:
> > > >> >
> > > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > > change
> > > >> > should not
> > > >> > significantly affect the latency of this API." and "That would allow
> > > to
> > > >> > easily validate whether disk operations (like a resize), or topic
> > > >> deletion
> > > >> > (log deletion only happen after a short delay) have completed." I
> > > wonder
> > > >> if
> > > >> > there is an existing metric/API that can allow administrators to
> > > >> determine
> > > >> > whether we need to resize? If administrators use this API to
> > > >> > determine
> > > >> > whether we need a resize, would this API become a high-volume API? I
> > > >> > understand we don't want this API to be a high-volume one because the
> > > API
> > > >> > is already costly by returning `"name": "Topics"`.
> > > >> >
> > > >> > Cong
> > > >> >
> > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > I wrote a small KIP to expose the total and usable space of logdirs
> > > >> > > via the DescribeLogDirs API:
> > > >> > >
> > > >> > >
> > > >>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > >> > >
> > > >> > > Please take a look and let me know if you have any feedback.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Mickael
> > > >> > >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best Regards,
> > > >> Raman Verma
> > > >>
> > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Cong,
Kafka does not expose disk utilization metrics. This is something you
need to provide in your environment. You definitively should have a
mechanism for exposing metrics from your Kafka broker hosts and you
should absolutely monitor disk usage and have appropriate alerts.
Thanks,
Mickael
On Thu, May 26, 2022 at 7:34 PM Jun Rao wrote:
>
> Hi, Igor,
>
> Thanks for the reply.
>
> I agree that this KIP could be useful for improving the tool for moving
> data across disks. It would be useful to clarify on the main motivation of
> the KIP. Also, DescribeLogDirsResponse already includes the size of each
> partition on a disk. So, it seems that UsableBytes is redundant since it's
> derivable.
>
> Thanks,
>
> Jun
>
> On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
>
> > Hi,
> >
> > This can also be quite useful to make better use of existing functionality
> > in the Kafka API — moving replicas between log directories via
> > ALTER_REPLICA_LOG_DIRS. If usable space information is also available the
> > caller can make better decisions using the same API. It means a more
> > consistent way of interacting with Kafka to manage replicas locations
> > within a broker without having to correlate Kafka metrics with information
> > from the Kafka API.
> >
> > --
> > Igor
> >
> > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > Hi, Mickael,
> > >
> > > Thanks for the KIP. Since this is mostly for monitoring and alerting,
> > > could we expose them as metrics instead of as part of the API? We already
> > > have a size metric per log. Perhaps we could extend that to add
> > used/total
> > > metrics per disk?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, May 19, 2022 at 10:21 PM Raman Verma > >
> > > wrote:
> > >
> > >> Hello Mikael,
> > >>
> > >> Thanks for the KIP.
> > >>
> > >> I see that the API response contains some information about each
> > partition.
> > >> ```
> > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > >> "about": "The size of the log segments in this partition in bytes." }
> > >> ```
> > >> Can this be summed up to provide a used space in a `log.dir`
> > >> This will also be specific to a `log.dir` (for the case where multiple
> > >> log.dir are hosted on the same underlying device)
> > >>
> > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> > >> wrote:
> > >> >
> > >> > Hey Mickael,
> > >> >
> > >> > Great KIP!
> > >> >
> > >> > I have one question:
> > >> >
> > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > change
> > >> > should not
> > >> > significantly affect the latency of this API." and "That would allow
> > to
> > >> > easily validate whether disk operations (like a resize), or topic
> > >> deletion
> > >> > (log deletion only happen after a short delay) have completed." I
> > wonder
> > >> if
> > >> > there is an existing metric/API that can allow administrators to
> > >> determine
> > >> > whether we need to resize? If administrators use this API to determine
> > >> > whether we need a resize, would this API become a high-volume API? I
> > >> > understand we don't want this API to be a high-volume one because the
> > API
> > >> > is already costly by returning `"name": "Topics"`.
> > >> >
> > >> > Cong
> > >> >
> > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I wrote a small KIP to expose the total and usable space of logdirs
> > >> > > via the DescribeLogDirs API:
> > >> > >
> > >> > >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > >> > >
> > >> > > Please take a look and let me know if you have any feedback.
> > >> > >
> > >> > > Thanks,
> > >> > > Mickael
> > >> > >
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards,
> > >> Raman Verma
> > >>
> >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, Igor,
Thanks for the reply.
I agree that this KIP could be useful for improving the tool for moving
data across disks. It would be useful to clarify on the main motivation of
the KIP. Also, DescribeLogDirsResponse already includes the size of each
partition on a disk. So, it seems that UsableBytes is redundant since it's
derivable.
Thanks,
Jun
On Thu, May 26, 2022 at 3:30 AM Igor Soarez wrote:
> Hi,
>
> This can also be quite useful to make better use of existing functionality
> in the Kafka API — moving replicas between log directories via
> ALTER_REPLICA_LOG_DIRS. If usable space information is also available the
> caller can make better decisions using the same API. It means a more
> consistent way of interacting with Kafka to manage replicas locations
> within a broker without having to correlate Kafka metrics with information
> from the Kafka API.
>
> --
> Igor
>
> On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > Hi, Mickael,
> >
> > Thanks for the KIP. Since this is mostly for monitoring and alerting,
> > could we expose them as metrics instead of as part of the API? We already
> > have a size metric per log. Perhaps we could extend that to add
> used/total
> > metrics per disk?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 19, 2022 at 10:21 PM Raman Verma >
> > wrote:
> >
> >> Hello Mikael,
> >>
> >> Thanks for the KIP.
> >>
> >> I see that the API response contains some information about each
> partition.
> >> ```
> >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> >> "about": "The size of the log segments in this partition in bytes." }
> >> ```
> >> Can this be summed up to provide a used space in a `log.dir`
> >> This will also be specific to a `log.dir` (for the case where multiple
> >> log.dir are hosted on the same underlying device)
> >>
> >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> >> wrote:
> >> >
> >> > Hey Mickael,
> >> >
> >> > Great KIP!
> >> >
> >> > I have one question:
> >> >
> >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> change
> >> > should not
> >> > significantly affect the latency of this API." and "That would allow
> to
> >> > easily validate whether disk operations (like a resize), or topic
> >> deletion
> >> > (log deletion only happen after a short delay) have completed." I
> wonder
> >> if
> >> > there is an existing metric/API that can allow administrators to
> >> determine
> >> > whether we need to resize? If administrators use this API to determine
> >> > whether we need a resize, would this API become a high-volume API? I
> >> > understand we don't want this API to be a high-volume one because the
> API
> >> > is already costly by returning `"name": "Topics"`.
> >> >
> >> > Cong
> >> >
> >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I wrote a small KIP to expose the total and usable space of logdirs
> >> > > via the DescribeLogDirs API:
> >> > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> >> > >
> >> > > Please take a look and let me know if you have any feedback.
> >> > >
> >> > > Thanks,
> >> > > Mickael
> >> > >
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Raman Verma
> >>
>
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi,
This can also be quite useful to make better use of existing functionality in
the Kafka API — moving replicas between log directories via
ALTER_REPLICA_LOG_DIRS. If usable space information is also available the
caller can make better decisions using the same API. It means a more consistent
way of interacting with Kafka to manage replicas locations within a broker
without having to correlate Kafka metrics with information from the Kafka API.
--
Igor
On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> Hi, Mickael,
>
> Thanks for the KIP. Since this is mostly for monitoring and alerting,
> could we expose them as metrics instead of as part of the API? We already
> have a size metric per log. Perhaps we could extend that to add used/total
> metrics per disk?
>
> Thanks,
>
> Jun
>
> On Thu, May 19, 2022 at 10:21 PM Raman Verma
> wrote:
>
>> Hello Mikael,
>>
>> Thanks for the KIP.
>>
>> I see that the API response contains some information about each partition.
>> ```
>> { "name": "PartitionSize", "type": "int64", "versions": "0+",
>> "about": "The size of the log segments in this partition in bytes." }
>> ```
>> Can this be summed up to provide a used space in a `log.dir`
>> This will also be specific to a `log.dir` (for the case where multiple
>> log.dir are hosted on the same underlying device)
>>
>> On Thu, May 19, 2022 at 10:21 AM Cong Ding
>> wrote:
>> >
>> > Hey Mickael,
>> >
>> > Great KIP!
>> >
>> > I have one question:
>> >
>> > You mentioned "DescribeLogDirs is usually a low volume API. This change
>> > should not
>> > significantly affect the latency of this API." and "That would allow to
>> > easily validate whether disk operations (like a resize), or topic
>> deletion
>> > (log deletion only happen after a short delay) have completed." I wonder
>> if
>> > there is an existing metric/API that can allow administrators to
>> determine
>> > whether we need to resize? If administrators use this API to determine
>> > whether we need a resize, would this API become a high-volume API? I
>> > understand we don't want this API to be a high-volume one because the API
>> > is already costly by returning `"name": "Topics"`.
>> >
>> > Cong
>> >
>> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I wrote a small KIP to expose the total and usable space of logdirs
>> > > via the DescribeLogDirs API:
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
>> > >
>> > > Please take a look and let me know if you have any feedback.
>> > >
>> > > Thanks,
>> > > Mickael
>> > >
>>
>>
>>
>> --
>> Best Regards,
>> Raman Verma
>>
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, Mickael,
Thanks for the KIP. Since this is mostly for monitoring and alerting,
could we expose them as metrics instead of as part of the API? We already
have a size metric per log. Perhaps we could extend that to add used/total
metrics per disk?
Thanks,
Jun
On Thu, May 19, 2022 at 10:21 PM Raman Verma
wrote:
> Hello Mikael,
>
> Thanks for the KIP.
>
> I see that the API response contains some information about each partition.
> ```
> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> "about": "The size of the log segments in this partition in bytes." }
> ```
> Can this be summed up to provide a used space in a `log.dir`
> This will also be specific to a `log.dir` (for the case where multiple
> log.dir are hosted on the same underlying device)
>
> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> wrote:
> >
> > Hey Mickael,
> >
> > Great KIP!
> >
> > I have one question:
> >
> > You mentioned "DescribeLogDirs is usually a low volume API. This change
> > should not
> > significantly affect the latency of this API." and "That would allow to
> > easily validate whether disk operations (like a resize), or topic
> deletion
> > (log deletion only happen after a short delay) have completed." I wonder
> if
> > there is an existing metric/API that can allow administrators to
> determine
> > whether we need to resize? If administrators use this API to determine
> > whether we need a resize, would this API become a high-volume API? I
> > understand we don't want this API to be a high-volume one because the API
> > is already costly by returning `"name": "Topics"`.
> >
> > Cong
> >
> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison
> > wrote:
> >
> > > Hi,
> > >
> > > I wrote a small KIP to expose the total and usable space of logdirs
> > > via the DescribeLogDirs API:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > >
> > > Please take a look and let me know if you have any feedback.
> > >
> > > Thanks,
> > > Mickael
> > >
>
>
>
> --
> Best Regards,
> Raman Verma
>
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael, Thanks for the KIP! +1 (binding). Kind regards, Tom On Thu, 19 May 2022 at 11:28, Federico Valeri wrote: > Thanks Mickael. > > +1 (non binding) > > On Wed, May 18, 2022 at 11:08 AM Divij Vaidya > wrote: > > > > +1 non binding. > > > > Divij Vaidya > > > > > > > > On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: > > > > > Thanks for this KIP Mickael. > > > > > > +1 non binding > > > > > > -- > > > Igor > > > > > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > > > > Hi Mickael, > > > > > > > > +1 (binding) from me. > > > > Thanks for the KIP! > > > > > > > > Luke > > > > > > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison < > mickael.mai...@gmail.com > > > > > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> I'd like to start a vote on KIP-827. It proposes exposing the total > > > >> and usable space of logdirs > > > >> via the DescribeLogDirs API: > > > >> > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > >> > > > >> Thanks, > > > >> Mickael > > > >> > > > > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hello Mikael,
Thanks for the KIP.
I see that the API response contains some information about each partition.
```
{ "name": "PartitionSize", "type": "int64", "versions": "0+",
"about": "The size of the log segments in this partition in bytes." }
```
Can this be summed up to provide a used space in a `log.dir`
This will also be specific to a `log.dir` (for the case where multiple
log.dir are hosted on the same underlying device)
On Thu, May 19, 2022 at 10:21 AM Cong Ding wrote:
>
> Hey Mickael,
>
> Great KIP!
>
> I have one question:
>
> You mentioned "DescribeLogDirs is usually a low volume API. This change
> should not
> significantly affect the latency of this API." and "That would allow to
> easily validate whether disk operations (like a resize), or topic deletion
> (log deletion only happen after a short delay) have completed." I wonder if
> there is an existing metric/API that can allow administrators to determine
> whether we need to resize? If administrators use this API to determine
> whether we need a resize, would this API become a high-volume API? I
> understand we don't want this API to be a high-volume one because the API
> is already costly by returning `"name": "Topics"`.
>
> Cong
>
> On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison
> wrote:
>
> > Hi,
> >
> > I wrote a small KIP to expose the total and usable space of logdirs
> > via the DescribeLogDirs API:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> >
> > Please take a look and let me know if you have any feedback.
> >
> > Thanks,
> > Mickael
> >
--
Best Regards,
Raman Verma
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hey Mickael, Great KIP! I have one question: You mentioned "DescribeLogDirs is usually a low volume API. This change should not significantly affect the latency of this API." and "That would allow to easily validate whether disk operations (like a resize), or topic deletion (log deletion only happen after a short delay) have completed." I wonder if there is an existing metric/API that can allow administrators to determine whether we need to resize? If administrators use this API to determine whether we need a resize, would this API become a high-volume API? I understand we don't want this API to be a high-volume one because the API is already costly by returning `"name": "Topics"`. Cong On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison wrote: > Hi, > > I wrote a small KIP to expose the total and usable space of logdirs > via the DescribeLogDirs API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > Please take a look and let me know if you have any feedback. > > Thanks, > Mickael >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Ismael, 1. I'm fine dropping "Space" from the field name, I think the names are clear enough, I've updated the KIP. 2. These values are properties of the volume each log directory is into. If you have multiple log directories in the same volume, they will both return the usable and total size of that volume. I'm not sure if it's something many people do but I've clarified the sizes are from the underlying volumes in the KIP. Thanks, Mickael On Tue, May 17, 2022 at 5:46 PM Ismael Juma wrote: > > Hi Mickael, > > Thanks for the KIP. Two questions: > > 1. Is `space` redundant? is `totalBytes` and `usableBytes` a more concise > description of the same thing? > 2. Is usable space a property of the log directory? What if you have > multiple log directories in the same underlying OS partition? > > Ismael > > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison > wrote: > > > Hi, > > > > I wrote a small KIP to expose the total and usable space of logdirs > > via the DescribeLogDirs API: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > Please take a look and let me know if you have any feedback. > > > > Thanks, > > Mickael > >
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Thanks Mickael. +1 (non binding) On Wed, May 18, 2022 at 11:08 AM Divij Vaidya wrote: > > +1 non binding. > > Divij Vaidya > > > > On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: > > > Thanks for this KIP Mickael. > > > > +1 non binding > > > > -- > > Igor > > > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > > > Hi Mickael, > > > > > > +1 (binding) from me. > > > Thanks for the KIP! > > > > > > Luke > > > > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison > > > > > wrote: > > > > > >> Hi, > > >> > > >> I'd like to start a vote on KIP-827. It proposes exposing the total > > >> and usable space of logdirs > > >> via the DescribeLogDirs API: > > >> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > >> > > >> Thanks, > > >> Mickael > > >> > >
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
+1 non binding. Divij Vaidya On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: > Thanks for this KIP Mickael. > > +1 non binding > > -- > Igor > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > > Hi Mickael, > > > > +1 (binding) from me. > > Thanks for the KIP! > > > > Luke > > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison > > > wrote: > > > >> Hi, > >> > >> I'd like to start a vote on KIP-827. It proposes exposing the total > >> and usable space of logdirs > >> via the DescribeLogDirs API: > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > >> > >> Thanks, > >> Mickael > >> >
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Thanks for this KIP Mickael. +1 non binding -- Igor On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > Hi Mickael, > > +1 (binding) from me. > Thanks for the KIP! > > Luke > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison > wrote: > >> Hi, >> >> I'd like to start a vote on KIP-827. It proposes exposing the total >> and usable space of logdirs >> via the DescribeLogDirs API: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API >> >> Thanks, >> Mickael >>
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael, Thanks for the KIP. Two questions: 1. Is `space` redundant? is `totalBytes` and `usableBytes` a more concise description of the same thing? 2. Is usable space a property of the log directory? What if you have multiple log directories in the same underlying OS partition? Ismael On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison wrote: > Hi, > > I wrote a small KIP to expose the total and usable space of logdirs > via the DescribeLogDirs API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > Please take a look and let me know if you have any feedback. > > Thanks, > Mickael >
Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael, +1 (binding) from me. Thanks for the KIP! Luke On Tue, May 17, 2022 at 9:30 PM Mickael Maison wrote: > Hi, > > I'd like to start a vote on KIP-827. It proposes exposing the total > and usable space of logdirs > via the DescribeLogDirs API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > Thanks, > Mickael >
[VOTE] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, I'd like to start a vote on KIP-827. It proposes exposing the total and usable space of logdirs via the DescribeLogDirs API: https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API Thanks, Mickael
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Thanks for addressing my comments Mickael. No more comments/suggestions from my side. LGTM. Divij Vaidya On Tue, May 10, 2022 at 6:10 PM Mickael Maison wrote: > Hi Colin, > > Thanks for the suggestion. > > I guess there are pros and cons with both methods. In my mind I'm > expecting these values to always be there in the long run (once people > have upgraded to brokers that support this feature). So I thought > having a primitive directly may be nicer to use in the future. On the > other hand OptionalLong avoids exposing magic values to users and > makes error cases (brokers that support the feature but hit an error) > a bit nicer. > > We seem to already be using Optional fields in a few *Result classes > so I'll follow your suggestion. As a bonus that also resolves the > naming questions from Divij! > > I've updated the KIP accordingly. > > Thanks, > Mickael > > On Wed, May 4, 2022 at 2:52 PM Igor Soarez wrote: > > > > Hi Mickael, > > > > Thanks for writing this KIP. This would be a very useful improvement! > > > > -- > > Igor > > > > On Thu, Apr 7, 2022, at 10:16 AM, Mickael Maison wrote: > > > Hi, > > > > > > I wrote a small KIP to expose the total and usable space of logdirs > > > via the DescribeLogDirs API: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > > > Please take a look and let me know if you have any feedback. > > > > > > Thanks, > > > Mickael >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Colin, Thanks for the suggestion. I guess there are pros and cons with both methods. In my mind I'm expecting these values to always be there in the long run (once people have upgraded to brokers that support this feature). So I thought having a primitive directly may be nicer to use in the future. On the other hand OptionalLong avoids exposing magic values to users and makes error cases (brokers that support the feature but hit an error) a bit nicer. We seem to already be using Optional fields in a few *Result classes so I'll follow your suggestion. As a bonus that also resolves the naming questions from Divij! I've updated the KIP accordingly. Thanks, Mickael On Wed, May 4, 2022 at 2:52 PM Igor Soarez wrote: > > Hi Mickael, > > Thanks for writing this KIP. This would be a very useful improvement! > > -- > Igor > > On Thu, Apr 7, 2022, at 10:16 AM, Mickael Maison wrote: > > Hi, > > > > I wrote a small KIP to expose the total and usable space of logdirs > > via the DescribeLogDirs API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > Please take a look and let me know if you have any feedback. > > > > Thanks, > > Mickael
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael, Thanks for writing this KIP. This would be a very useful improvement! -- Igor On Thu, Apr 7, 2022, at 10:16 AM, Mickael Maison wrote: > Hi, > > I wrote a small KIP to expose the total and usable space of logdirs > via the DescribeLogDirs API: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > Please take a look and let me know if you have any feedback. > > Thanks, > Mickael
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael, Thanks for the KIP. In the API, I would suggest using an OptionalLong rather than a "magic value" of -1. best, Colin On Thu, Apr 7, 2022, at 02:16, Mickael Maison wrote: > Hi, > > I wrote a small KIP to expose the total and usable space of logdirs > via the DescribeLogDirs API: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > Please take a look and let me know if you have any feedback. > > Thanks, > Mickael
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
ave a few lingering > > > questions/comments. > > > > > > > > > > > > > > *Reg#1* Understood. I checked and the underlying system call is > > > > statvfs > > > > > > for > > > > > > > unix systems which should be ok to call here. > > > > > > > *Reg#2* Fair point. I checked again and yes, log.dir always > means > > > > local > > > > > > > storage even when tiered storage is enabled. > > > > > > > *Reg#3* The rationale for adding these new (size) fields to the > > > > > > > `DescribeLogDirs` is to allow the administrator to monitor or > > > perhaps > > > > > > take > > > > > > > automated action based on results. Doesn't monitoring the > number of > > > > file > > > > > > > descriptors fall in the same category of use cases? I am > assuming > > > > that we > > > > > > > want to add the size information in the API response because > JVM > > > > makes it > > > > > > > possible to get this information in a platform agnostic manner > > > which > > > > is > > > > > > not > > > > > > > true for open file descriptors, correct? > > > > > > > *Reg#4* Agree. > > > > > > > *New#5*: As an FYI, Java FileStore API breaks on large storage > > > sizes. > > > > > > See: > > > > > > > https://bugs.openjdk.java.net/browse/JDK-8162520. > ElasticSearch > > > has > > > > been > > > > > > > hit by these limitations in the past. For JDK 11, you will > probably > > > > have > > > > > > to > > > > > > > add defensive checks such as > > > > > > > > > > > > > > > > > > > > > https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148 > > > > > > . > > > > > > > The documentation of the API mentioned in KIP will also be > modified > > > > to > > > > > > > account for this edge case. > > > > > > > *New#6*: Can you please provide an example where the return for > > > these > > > > > > APIs > > > > > > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API > will > > > > > > definitely > > > > > > > return results (else it throws an IOException)? I would propose > > > that > > > > we > > > > > > get > > > > > > > rid of default since JVM guarantees that this would work on all > > > > > > platforms. > > > > > > > If it doesn't then it's a bug and should be uncovered via an > > > > exception. > > > > > > > > > > > > > > Also, I would like to volunteer to code review (of course, it > would > > > > be > > > > > > > non-binding) your implementation once this KIP is approved. > > > > > > > > > > > > > > Regards, > > > > > > > Divij Vaidya > > > > > > > > > > > > > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison < > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > > > Thanks for taking a look! > > > > > > > > > > > > > > > > 1. In order to retrieve the sizes, the plan is to use > > > > getTotalSpace() > > > > > > > > and getUsableSpace() from java.nio.file.FileStore. The > > > > implementations > > > > > > > > may vary depending on the filesystem but these calls > typically > > > > don't > > > > > > > > depend on the size of storage but instead just return > metadata > > > the > > > > > > > > filesystem maintains. > > > > > > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. > As > > > > far as > > > > > > > > I understand brokers will still have local log dirs and > remote >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
gt; > *New#5*: As an FYI, Java FileStore API breaks on large storage > > sizes. > > > > > See: > > > > > > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch > > has > > > been > > > > > > hit by these limitations in the past. For JDK 11, you will probably > > > have > > > > > to > > > > > > add defensive checks such as > > > > > > > > > > > > > > > > https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148 > > > > > . > > > > > > The documentation of the API mentioned in KIP will also be modified > > > to > > > > > > account for this edge case. > > > > > > *New#6*: Can you please provide an example where the return for > > these > > > > > APIs > > > > > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will > > > > > definitely > > > > > > return results (else it throws an IOException)? I would propose > > that > > > we > > > > > get > > > > > > rid of default since JVM guarantees that this would work on all > > > > > platforms. > > > > > > If it doesn't then it's a bug and should be uncovered via an > > > exception. > > > > > > > > > > > > Also, I would like to volunteer to code review (of course, it would > > > be > > > > > > non-binding) your implementation once this KIP is approved. > > > > > > > > > > > > Regards, > > > > > > Divij Vaidya > > > > > > > > > > > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison < > > > mickael.mai...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > > > Thanks for taking a look! > > > > > > > > > > > > > > 1. In order to retrieve the sizes, the plan is to use > > > getTotalSpace() > > > > > > > and getUsableSpace() from java.nio.file.FileStore. The > > > implementations > > > > > > > may vary depending on the filesystem but these calls typically > > > don't > > > > > > > depend on the size of storage but instead just return metadata > > the > > > > > > > filesystem maintains. > > > > > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As > > > far as > > > > > > > I understand brokers will still have local log dirs and remote > > > volumes > > > > > > > are not counted as log dirs. KIP-405 does not mention updating > > the > > > > > > > DescribeLogDirs API. So I don't think this KIP needs to do > > anything > > > > > > > special to be compatible with KIP-405. On the other hand, I > > wonder > > > if > > > > > > > KIP-405 should update DescribeLogDirs to provide details about > > the > > > > > > > location of replicas. > > > > > > > 3. Counting files can be a slow operation as it requires > > exploring > > > all > > > > > > > paths recursively to find all files. Administrators should > > > > > > > definitively monitor file descriptors via host metrics but I'm > > not > > > > > > > sure it's something we want to expose via the Kafka API. As > > > mentioned > > > > > > > it could be slow to compute and files are not really a Kafka > > > concept. > > > > > > > 4. DescribeLogDirs is usually a low volume API. This change > > should > > > not > > > > > > > significantly affect the latency of this API. > > > > > > > > > > > > > > Thanks, > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya < > > > divijvaidy...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Mickael > > > > > > > > > > > > > > > > Thanks for starting this. It is a very useful feature. > > > > > > > > > > > > > > > > Some initial thoughts (I am new to Kafka so please excuse if > > > these > > > > > are > > > > > > > > naive suggestions): > > > > > > > > 1. What is the impact on latency of the DescribeLogDirs API due > > > to > > > > > this > > > > > > > > change? Would calculating the totalSpace from each logdir be a > > > > > bottleneck > > > > > > > > for the API? What if we are talking about a large storage size > > > in the > > > > > > > order > > > > > > > > of hundred (or tens) of GBs? > > > > > > > > 2. How does this fit in with RemoteStorage (KIP-405)? I think > > > > > integration > > > > > > > > with KIP-405 is worth discussing in the scope of this KIP. My > > > > > > > > recommendation will be to add a new API in the RLMM > > > > > > > > (RemoteLogMetadataManager) called GetLogSize() and leave it > > upto > > > the > > > > > > > remote > > > > > > > > storage to perform a concrete implementation for this > > > > > > > > interface. DescribeLogDirs could call this interface internally > > > to > > > > > > > provide > > > > > > > > the relevant information. > > > > > > > > 3. Do you think adding the number of files in the directory as > > > part > > > > > of > > > > > > > the > > > > > > > > API response will be useful as well? e.g. a use case where this > > > > > > > information > > > > > > > > will be useful is to monitor/alarm the situations when the > > > number of > > > > > > > files > > > > > > > > are dangerously reaching the max value of file descriptors > > > > > configured at > > > > > > > > the OS. > > > > > > > > 4. Please add an API latency perf test as part of the release > > > > > criteria > > > > > > > for > > > > > > > > this change. We want to avoid regression. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison < > > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I wrote a small KIP to expose the total and usable space of > > > logdirs > > > > > > > > > via the DescribeLogDirs API: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > > > > > > > > > > > > > > > Please take a look and let me know if you have any feedback. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
vide an example where the return for > these > > > > APIs > > > > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will > > > > definitely > > > > > return results (else it throws an IOException)? I would propose > that > > we > > > > get > > > > > rid of default since JVM guarantees that this would work on all > > > > platforms. > > > > > If it doesn't then it's a bug and should be uncovered via an > > exception. > > > > > > > > > > Also, I would like to volunteer to code review (of course, it would > > be > > > > > non-binding) your implementation once this KIP is approved. > > > > > > > > > > Regards, > > > > > Divij Vaidya > > > > > > > > > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison < > > mickael.mai...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hi Divij, > > > > > > > > > > > > Thanks for taking a look! > > > > > > > > > > > > 1. In order to retrieve the sizes, the plan is to use > > getTotalSpace() > > > > > > and getUsableSpace() from java.nio.file.FileStore. The > > implementations > > > > > > may vary depending on the filesystem but these calls typically > > don't > > > > > > depend on the size of storage but instead just return metadata > the > > > > > > filesystem maintains. > > > > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As > > far as > > > > > > I understand brokers will still have local log dirs and remote > > volumes > > > > > > are not counted as log dirs. KIP-405 does not mention updating > the > > > > > > DescribeLogDirs API. So I don't think this KIP needs to do > anything > > > > > > special to be compatible with KIP-405. On the other hand, I > wonder > > if > > > > > > KIP-405 should update DescribeLogDirs to provide details about > the > > > > > > location of replicas. > > > > > > 3. Counting files can be a slow operation as it requires > exploring > > all > > > > > > paths recursively to find all files. Administrators should > > > > > > definitively monitor file descriptors via host metrics but I'm > not > > > > > > sure it's something we want to expose via the Kafka API. As > > mentioned > > > > > > it could be slow to compute and files are not really a Kafka > > concept. > > > > > > 4. DescribeLogDirs is usually a low volume API. This change > should > > not > > > > > > significantly affect the latency of this API. > > > > > > > > > > > > Thanks, > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya < > > divijvaidy...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > Hi Mickael > > > > > > > > > > > > > > Thanks for starting this. It is a very useful feature. > > > > > > > > > > > > > > Some initial thoughts (I am new to Kafka so please excuse if > > these > > > > are > > > > > > > naive suggestions): > > > > > > > 1. What is the impact on latency of the DescribeLogDirs API due > > to > > > > this > > > > > > > change? Would calculating the totalSpace from each logdir be a > > > > bottleneck > > > > > > > for the API? What if we are talking about a large storage size > > in the > > > > > > order > > > > > > > of hundred (or tens) of GBs? > > > > > > > 2. How does this fit in with RemoteStorage (KIP-405)? I think > > > > integration > > > > > > > with KIP-405 is worth discussing in the scope of this KIP. My > > > > > > > recommendation will be to add a new API in the RLMM > > > > > > > (RemoteLogMetadataManager) called GetLogSize() and leave it > upto > > the > > > > > > remote > > > > > > > storage to perform a concrete implementation for this > > > > > > > interface. DescribeLogDirs could call this interface internally > > to > > > > > > provide > > > > > > > the relevant information. > > > > > > > 3. Do you think adding the number of files in the directory as > > part > > > > of > > > > > > the > > > > > > > API response will be useful as well? e.g. a use case where this > > > > > > information > > > > > > > will be useful is to monitor/alarm the situations when the > > number of > > > > > > files > > > > > > > are dangerously reaching the max value of file descriptors > > > > configured at > > > > > > > the OS. > > > > > > > 4. Please add an API latency perf test as part of the release > > > > criteria > > > > > > for > > > > > > > this change. We want to avoid regression. > > > > > > > > > > > > > > Regards, > > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison < > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I wrote a small KIP to expose the total and usable space of > > logdirs > > > > > > > > via the DescribeLogDirs API: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > > > > > > > > > > > > > Please take a look and let me know if you have any feedback. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
kael.mai...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi Divij, > > > > > > > > > > Thanks for taking a look! > > > > > > > > > > 1. In order to retrieve the sizes, the plan is to use > getTotalSpace() > > > > > and getUsableSpace() from java.nio.file.FileStore. The > implementations > > > > > may vary depending on the filesystem but these calls typically > don't > > > > > depend on the size of storage but instead just return metadata the > > > > > filesystem maintains. > > > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As > far as > > > > > I understand brokers will still have local log dirs and remote > volumes > > > > > are not counted as log dirs. KIP-405 does not mention updating the > > > > > DescribeLogDirs API. So I don't think this KIP needs to do anything > > > > > special to be compatible with KIP-405. On the other hand, I wonder > if > > > > > KIP-405 should update DescribeLogDirs to provide details about the > > > > > location of replicas. > > > > > 3. Counting files can be a slow operation as it requires exploring > all > > > > > paths recursively to find all files. Administrators should > > > > > definitively monitor file descriptors via host metrics but I'm not > > > > > sure it's something we want to expose via the Kafka API. As > mentioned > > > > > it could be slow to compute and files are not really a Kafka > concept. > > > > > 4. DescribeLogDirs is usually a low volume API. This change should > not > > > > > significantly affect the latency of this API. > > > > > > > > > > Thanks, > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya < > divijvaidy...@gmail.com> > > > > > wrote: > > > > > > > > > > > > Hi Mickael > > > > > > > > > > > > Thanks for starting this. It is a very useful feature. > > > > > > > > > > > > Some initial thoughts (I am new to Kafka so please excuse if > these > > > are > > > > > > naive suggestions): > > > > > > 1. What is the impact on latency of the DescribeLogDirs API due > to > > > this > > > > > > change? Would calculating the totalSpace from each logdir be a > > > bottleneck > > > > > > for the API? What if we are talking about a large storage size > in the > > > > > order > > > > > > of hundred (or tens) of GBs? > > > > > > 2. How does this fit in with RemoteStorage (KIP-405)? I think > > > integration > > > > > > with KIP-405 is worth discussing in the scope of this KIP. My > > > > > > recommendation will be to add a new API in the RLMM > > > > > > (RemoteLogMetadataManager) called GetLogSize() and leave it upto > the > > > > > remote > > > > > > storage to perform a concrete implementation for this > > > > > > interface. DescribeLogDirs could call this interface internally > to > > > > > provide > > > > > > the relevant information. > > > > > > 3. Do you think adding the number of files in the directory as > part > > > of > > > > > the > > > > > > API response will be useful as well? e.g. a use case where this > > > > > information > > > > > > will be useful is to monitor/alarm the situations when the > number of > > > > > files > > > > > > are dangerously reaching the max value of file descriptors > > > configured at > > > > > > the OS. > > > > > > 4. Please add an API latency perf test as part of the release > > > criteria > > > > > for > > > > > > this change. We want to avoid regression. > > > > > > > > > > > > Regards, > > > > > > Divij Vaidya > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison < > > > mickael.mai...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I wrote a small KIP to expose the total and usable space of > logdirs > > > > > > > via the DescribeLogDirs API: > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > > > > > > > > > > > Please take a look and let me know if you have any feedback. > > > > > > > > > > > > > > Thanks, > > > > > > > Mickael > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Luke, 7. I've updated the KIP to clarify these sizes are in bytes. Thanks, Mickael On Fri, Apr 15, 2022 at 12:16 PM Luke Chen wrote: > > Hi Mickael, > > Thanks for the KIP! > This is a good improvement. > > (3) +1 for not adding the number of files in the directory. Counting the > file numbers should be slow. > (7) Could you make the fields clear in `DescribeLogDirsResponse`, to > mention the returned number is size in Byte (or not?) > > Thank you. > Luke > > On Fri, Apr 15, 2022 at 5:27 PM Mickael Maison > wrote: > > > Hi, > > > > Thanks for the feedback. > > > > 3. Yes that's right. Also the number of file descriptors is really not > > a property of log directories. Administrators typically tracked that > > count per process and for the whole operating system. > > > > 5. That's a good point, I've updated the KIP to mention sizes will be > > capped to Long.MAX_VALUE even if the actual storage is larger. > > > > 6. Brokers would never return UNKNOWN_SPACE. When new clients query > > older brokers via the admin API, the admin client will use > > UNKNOWN_SPACE to indicate these values weren't provided by brokers. > > > > Thanks, > > Mickael > > > > On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya > > wrote: > > > > > > Thanks for replying. I still have a few lingering questions/comments. > > > > > > *Reg#1* Understood. I checked and the underlying system call is statvfs > > for > > > unix systems which should be ok to call here. > > > *Reg#2* Fair point. I checked again and yes, log.dir always means local > > > storage even when tiered storage is enabled. > > > *Reg#3* The rationale for adding these new (size) fields to the > > > `DescribeLogDirs` is to allow the administrator to monitor or perhaps > > take > > > automated action based on results. Doesn't monitoring the number of file > > > descriptors fall in the same category of use cases? I am assuming that we > > > want to add the size information in the API response because JVM makes it > > > possible to get this information in a platform agnostic manner which is > > not > > > true for open file descriptors, correct? > > > *Reg#4* Agree. > > > *New#5*: As an FYI, Java FileStore API breaks on large storage sizes. > > See: > > > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been > > > hit by these limitations in the past. For JDK 11, you will probably have > > to > > > add defensive checks such as > > > > > https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148 > > . > > > The documentation of the API mentioned in KIP will also be modified to > > > account for this edge case. > > > *New#6*: Can you please provide an example where the return for these > > APIs > > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will > > definitely > > > return results (else it throws an IOException)? I would propose that we > > get > > > rid of default since JVM guarantees that this would work on all > > platforms. > > > If it doesn't then it's a bug and should be uncovered via an exception. > > > > > > Also, I would like to volunteer to code review (of course, it would be > > > non-binding) your implementation once this KIP is approved. > > > > > > Regards, > > > Divij Vaidya > > > > > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison > > > > > wrote: > > > > > > > Hi Divij, > > > > > > > > Thanks for taking a look! > > > > > > > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace() > > > > and getUsableSpace() from java.nio.file.FileStore. The implementations > > > > may vary depending on the filesystem but these calls typically don't > > > > depend on the size of storage but instead just return metadata the > > > > filesystem maintains. > > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as > > > > I understand brokers will still have local log dirs and remote volumes > > > > are not counted as log dirs. KIP-405 does not mention updating the > > > > DescribeLogDirs API. So I don't think this KIP needs to do anything > > > > special to be compatible with KIP-405. On the
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael, Thanks for the KIP! This is a good improvement. (3) +1 for not adding the number of files in the directory. Counting the file numbers should be slow. (7) Could you make the fields clear in `DescribeLogDirsResponse`, to mention the returned number is size in Byte (or not?) Thank you. Luke On Fri, Apr 15, 2022 at 5:27 PM Mickael Maison wrote: > Hi, > > Thanks for the feedback. > > 3. Yes that's right. Also the number of file descriptors is really not > a property of log directories. Administrators typically tracked that > count per process and for the whole operating system. > > 5. That's a good point, I've updated the KIP to mention sizes will be > capped to Long.MAX_VALUE even if the actual storage is larger. > > 6. Brokers would never return UNKNOWN_SPACE. When new clients query > older brokers via the admin API, the admin client will use > UNKNOWN_SPACE to indicate these values weren't provided by brokers. > > Thanks, > Mickael > > On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya > wrote: > > > > Thanks for replying. I still have a few lingering questions/comments. > > > > *Reg#1* Understood. I checked and the underlying system call is statvfs > for > > unix systems which should be ok to call here. > > *Reg#2* Fair point. I checked again and yes, log.dir always means local > > storage even when tiered storage is enabled. > > *Reg#3* The rationale for adding these new (size) fields to the > > `DescribeLogDirs` is to allow the administrator to monitor or perhaps > take > > automated action based on results. Doesn't monitoring the number of file > > descriptors fall in the same category of use cases? I am assuming that we > > want to add the size information in the API response because JVM makes it > > possible to get this information in a platform agnostic manner which is > not > > true for open file descriptors, correct? > > *Reg#4* Agree. > > *New#5*: As an FYI, Java FileStore API breaks on large storage sizes. > See: > > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been > > hit by these limitations in the past. For JDK 11, you will probably have > to > > add defensive checks such as > > > https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148 > . > > The documentation of the API mentioned in KIP will also be modified to > > account for this edge case. > > *New#6*: Can you please provide an example where the return for these > APIs > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will > definitely > > return results (else it throws an IOException)? I would propose that we > get > > rid of default since JVM guarantees that this would work on all > platforms. > > If it doesn't then it's a bug and should be uncovered via an exception. > > > > Also, I would like to volunteer to code review (of course, it would be > > non-binding) your implementation once this KIP is approved. > > > > Regards, > > Divij Vaidya > > > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison > > > wrote: > > > > > Hi Divij, > > > > > > Thanks for taking a look! > > > > > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace() > > > and getUsableSpace() from java.nio.file.FileStore. The implementations > > > may vary depending on the filesystem but these calls typically don't > > > depend on the size of storage but instead just return metadata the > > > filesystem maintains. > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as > > > I understand brokers will still have local log dirs and remote volumes > > > are not counted as log dirs. KIP-405 does not mention updating the > > > DescribeLogDirs API. So I don't think this KIP needs to do anything > > > special to be compatible with KIP-405. On the other hand, I wonder if > > > KIP-405 should update DescribeLogDirs to provide details about the > > > location of replicas. > > > 3. Counting files can be a slow operation as it requires exploring all > > > paths recursively to find all files. Administrators should > > > definitively monitor file descriptors via host metrics but I'm not > > > sure it's something we want to expose via the Kafka API. As mentioned > > > it could be slow to compute and files are not really a Kafka concept. > > > 4. DescribeLogDirs is usually a low volume API. This change should not > > &g
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, Thanks for the feedback. 3. Yes that's right. Also the number of file descriptors is really not a property of log directories. Administrators typically tracked that count per process and for the whole operating system. 5. That's a good point, I've updated the KIP to mention sizes will be capped to Long.MAX_VALUE even if the actual storage is larger. 6. Brokers would never return UNKNOWN_SPACE. When new clients query older brokers via the admin API, the admin client will use UNKNOWN_SPACE to indicate these values weren't provided by brokers. Thanks, Mickael On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya wrote: > > Thanks for replying. I still have a few lingering questions/comments. > > *Reg#1* Understood. I checked and the underlying system call is statvfs for > unix systems which should be ok to call here. > *Reg#2* Fair point. I checked again and yes, log.dir always means local > storage even when tiered storage is enabled. > *Reg#3* The rationale for adding these new (size) fields to the > `DescribeLogDirs` is to allow the administrator to monitor or perhaps take > automated action based on results. Doesn't monitoring the number of file > descriptors fall in the same category of use cases? I am assuming that we > want to add the size information in the API response because JVM makes it > possible to get this information in a platform agnostic manner which is not > true for open file descriptors, correct? > *Reg#4* Agree. > *New#5*: As an FYI, Java FileStore API breaks on large storage sizes. See: > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been > hit by these limitations in the past. For JDK 11, you will probably have to > add defensive checks such as > https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148. > The documentation of the API mentioned in KIP will also be modified to > account for this edge case. > *New#6*: Can you please provide an example where the return for these APIs > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will definitely > return results (else it throws an IOException)? I would propose that we get > rid of default since JVM guarantees that this would work on all platforms. > If it doesn't then it's a bug and should be uncovered via an exception. > > Also, I would like to volunteer to code review (of course, it would be > non-binding) your implementation once this KIP is approved. > > Regards, > Divij Vaidya > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison > wrote: > > > Hi Divij, > > > > Thanks for taking a look! > > > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace() > > and getUsableSpace() from java.nio.file.FileStore. The implementations > > may vary depending on the filesystem but these calls typically don't > > depend on the size of storage but instead just return metadata the > > filesystem maintains. > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as > > I understand brokers will still have local log dirs and remote volumes > > are not counted as log dirs. KIP-405 does not mention updating the > > DescribeLogDirs API. So I don't think this KIP needs to do anything > > special to be compatible with KIP-405. On the other hand, I wonder if > > KIP-405 should update DescribeLogDirs to provide details about the > > location of replicas. > > 3. Counting files can be a slow operation as it requires exploring all > > paths recursively to find all files. Administrators should > > definitively monitor file descriptors via host metrics but I'm not > > sure it's something we want to expose via the Kafka API. As mentioned > > it could be slow to compute and files are not really a Kafka concept. > > 4. DescribeLogDirs is usually a low volume API. This change should not > > significantly affect the latency of this API. > > > > Thanks, > > Mickael > > > > > > > > > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya > > wrote: > > > > > > Hi Mickael > > > > > > Thanks for starting this. It is a very useful feature. > > > > > > Some initial thoughts (I am new to Kafka so please excuse if these are > > > naive suggestions): > > > 1. What is the impact on latency of the DescribeLogDirs API due to this > > > change? Would calculating the totalSpace from each logdir be a bottleneck > > > for the API? What if we are talking about a large storage size in the > > order > > > of hundred (or tens) of GBs? > > > 2. How d
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Thanks for replying. I still have a few lingering questions/comments. *Reg#1* Understood. I checked and the underlying system call is statvfs for unix systems which should be ok to call here. *Reg#2* Fair point. I checked again and yes, log.dir always means local storage even when tiered storage is enabled. *Reg#3* The rationale for adding these new (size) fields to the `DescribeLogDirs` is to allow the administrator to monitor or perhaps take automated action based on results. Doesn't monitoring the number of file descriptors fall in the same category of use cases? I am assuming that we want to add the size information in the API response because JVM makes it possible to get this information in a platform agnostic manner which is not true for open file descriptors, correct? *Reg#4* Agree. *New#5*: As an FYI, Java FileStore API breaks on large storage sizes. See: https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been hit by these limitations in the past. For JDK 11, you will probably have to add defensive checks such as https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148. The documentation of the API mentioned in KIP will also be modified to account for this edge case. *New#6*: Can you please provide an example where the return for these APIs would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will definitely return results (else it throws an IOException)? I would propose that we get rid of default since JVM guarantees that this would work on all platforms. If it doesn't then it's a bug and should be uncovered via an exception. Also, I would like to volunteer to code review (of course, it would be non-binding) your implementation once this KIP is approved. Regards, Divij Vaidya On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison wrote: > Hi Divij, > > Thanks for taking a look! > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace() > and getUsableSpace() from java.nio.file.FileStore. The implementations > may vary depending on the filesystem but these calls typically don't > depend on the size of storage but instead just return metadata the > filesystem maintains. > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as > I understand brokers will still have local log dirs and remote volumes > are not counted as log dirs. KIP-405 does not mention updating the > DescribeLogDirs API. So I don't think this KIP needs to do anything > special to be compatible with KIP-405. On the other hand, I wonder if > KIP-405 should update DescribeLogDirs to provide details about the > location of replicas. > 3. Counting files can be a slow operation as it requires exploring all > paths recursively to find all files. Administrators should > definitively monitor file descriptors via host metrics but I'm not > sure it's something we want to expose via the Kafka API. As mentioned > it could be slow to compute and files are not really a Kafka concept. > 4. DescribeLogDirs is usually a low volume API. This change should not > significantly affect the latency of this API. > > Thanks, > Mickael > > > > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya > wrote: > > > > Hi Mickael > > > > Thanks for starting this. It is a very useful feature. > > > > Some initial thoughts (I am new to Kafka so please excuse if these are > > naive suggestions): > > 1. What is the impact on latency of the DescribeLogDirs API due to this > > change? Would calculating the totalSpace from each logdir be a bottleneck > > for the API? What if we are talking about a large storage size in the > order > > of hundred (or tens) of GBs? > > 2. How does this fit in with RemoteStorage (KIP-405)? I think integration > > with KIP-405 is worth discussing in the scope of this KIP. My > > recommendation will be to add a new API in the RLMM > > (RemoteLogMetadataManager) called GetLogSize() and leave it upto the > remote > > storage to perform a concrete implementation for this > > interface. DescribeLogDirs could call this interface internally to > provide > > the relevant information. > > 3. Do you think adding the number of files in the directory as part of > the > > API response will be useful as well? e.g. a use case where this > information > > will be useful is to monitor/alarm the situations when the number of > files > > are dangerously reaching the max value of file descriptors configured at > > the OS. > > 4. Please add an API latency perf test as part of the release criteria > for > > this change. We want to avoid regression. > > > > Regards, > > Divij Vaid
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Divij, Thanks for taking a look! 1. In order to retrieve the sizes, the plan is to use getTotalSpace() and getUsableSpace() from java.nio.file.FileStore. The implementations may vary depending on the filesystem but these calls typically don't depend on the size of storage but instead just return metadata the filesystem maintains. 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as I understand brokers will still have local log dirs and remote volumes are not counted as log dirs. KIP-405 does not mention updating the DescribeLogDirs API. So I don't think this KIP needs to do anything special to be compatible with KIP-405. On the other hand, I wonder if KIP-405 should update DescribeLogDirs to provide details about the location of replicas. 3. Counting files can be a slow operation as it requires exploring all paths recursively to find all files. Administrators should definitively monitor file descriptors via host metrics but I'm not sure it's something we want to expose via the Kafka API. As mentioned it could be slow to compute and files are not really a Kafka concept. 4. DescribeLogDirs is usually a low volume API. This change should not significantly affect the latency of this API. Thanks, Mickael On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya wrote: > > Hi Mickael > > Thanks for starting this. It is a very useful feature. > > Some initial thoughts (I am new to Kafka so please excuse if these are > naive suggestions): > 1. What is the impact on latency of the DescribeLogDirs API due to this > change? Would calculating the totalSpace from each logdir be a bottleneck > for the API? What if we are talking about a large storage size in the order > of hundred (or tens) of GBs? > 2. How does this fit in with RemoteStorage (KIP-405)? I think integration > with KIP-405 is worth discussing in the scope of this KIP. My > recommendation will be to add a new API in the RLMM > (RemoteLogMetadataManager) called GetLogSize() and leave it upto the remote > storage to perform a concrete implementation for this > interface. DescribeLogDirs could call this interface internally to provide > the relevant information. > 3. Do you think adding the number of files in the directory as part of the > API response will be useful as well? e.g. a use case where this information > will be useful is to monitor/alarm the situations when the number of files > are dangerously reaching the max value of file descriptors configured at > the OS. > 4. Please add an API latency perf test as part of the release criteria for > this change. We want to avoid regression. > > Regards, > Divij Vaidya > > > > On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison > wrote: > > > Hi, > > > > I wrote a small KIP to expose the total and usable space of logdirs > > via the DescribeLogDirs API: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > > > Please take a look and let me know if you have any feedback. > > > > Thanks, > > Mickael > >
Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi Mickael Thanks for starting this. It is a very useful feature. Some initial thoughts (I am new to Kafka so please excuse if these are naive suggestions): 1. What is the impact on latency of the DescribeLogDirs API due to this change? Would calculating the totalSpace from each logdir be a bottleneck for the API? What if we are talking about a large storage size in the order of hundred (or tens) of GBs? 2. How does this fit in with RemoteStorage (KIP-405)? I think integration with KIP-405 is worth discussing in the scope of this KIP. My recommendation will be to add a new API in the RLMM (RemoteLogMetadataManager) called GetLogSize() and leave it upto the remote storage to perform a concrete implementation for this interface. DescribeLogDirs could call this interface internally to provide the relevant information. 3. Do you think adding the number of files in the directory as part of the API response will be useful as well? e.g. a use case where this information will be useful is to monitor/alarm the situations when the number of files are dangerously reaching the max value of file descriptors configured at the OS. 4. Please add an API latency perf test as part of the release criteria for this change. We want to avoid regression. Regards, Divij Vaidya On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison wrote: > Hi, > > I wrote a small KIP to expose the total and usable space of logdirs > via the DescribeLogDirs API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API > > Please take a look and let me know if you have any feedback. > > Thanks, > Mickael >
[DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API
Hi, I wrote a small KIP to expose the total and usable space of logdirs via the DescribeLogDirs API: https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API Please take a look and let me know if you have any feedback. Thanks, Mickael
????,????????????kafka-API??????????????????.
,??API,??,. ?? : KStreamBuilder??,??from??. ??kafka 10.0..,??,??. : http://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/streams/KafkaStreams.html
[jira] [Resolved] (KAFKA-7723) Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api
[
https://issues.apache.org/jira/browse/KAFKA-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
laomei resolved KAFKA-7723.
---
Resolution: Duplicate
> Kafka Connect support override worker kafka api configuration with connector
> configuration that post by rest api
>
>
> Key: KAFKA-7723
> URL: https://issues.apache.org/jira/browse/KAFKA-7723
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: laomei
>Priority: Minor
> Labels: needs-kip
>
> I'm using kafka sink connect; "auto.offset.reset" is set in
> connect-distributed*.properties;
> It works for all connector which in one worker; So the consumer will poll
> records from latest or earliest; I can not control the auto.offset.reset in
> connector configs post with rest api;
> So I think is necessary to override worker kafka api configs with connector
> configs;
> Like this
> {code:java}
> {
> "name": "test",
> "config": {
> "consumer.auto.offset.reset": "latest",
> "consumer.xxx"
> "connector.class": "com.laomei.sis.solr.SolrConnector",
> "tasks.max": "1",
> "poll.interval.ms": "100",
> "connect.timeout.ms": "6",
> "topics": "test"
> }
> }
> {code}
> We can override kafka consumer auto offset reset in sink connector;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7723) Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api
laomei created KAFKA-7723:
-
Summary: Kafka Connect support override worker kafka api
configuration with connector configuration that post by rest api
Key: KAFKA-7723
URL: https://issues.apache.org/jira/browse/KAFKA-7723
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: laomei
I'm using kafka sink connect; "auto.offset.reset" is set in
connect-distributed*.properties;
It works for all connector which in one worker; So the consumer will poll
records from latest or earliest; I can not control the auto.offset.reset in
connector configs post with rest api;
So I think is necessary to override worker kafka api configs with connector
configs;
Like this
{code:java}
{
"name": "test",
"config": {
"consumer.auto.offset.reset": "latest",
"consumer.xxx"
"connector.class": "com.laomei.sis.solr.SolrConnector",
"tasks.max": "1",
"poll.interval.ms": "100",
"connect.timeout.ms": "6",
"topics": "test"
}
}
{code}
We can override kafka consumer auto offset reset in sink connector;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1583: -- Resolution: Fixed Fix Version/s: (was: 0.9.0) 0.8.3 Status: Resolved (was: Patch Available) Thanks for the patch and your patience! Just checked in to trunk. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.8.3 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch, > KAFKA-1583_2014-10-28_15:09:30.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188577#comment-14188577 ] Joel Koshy commented on KAFKA-1583: --- Ok cool - I will re-review the latest patch and should be able to get that checked in today. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch, > KAFKA-1583_2014-10-28_15:09:30.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187960#comment-14187960 ] Jun Rao commented on KAFKA-1583: [~jjkoshy], you can commit the patch once you are done with the review. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch, > KAFKA-1583_2014-10-28_15:09:30.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187571#comment-14187571 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch, > KAFKA-1583_2014-10-28_15:09:30.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-28_15:09:30.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch, > KAFKA-1583_2014-10-28_15:09:30.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187118#comment-14187118 ] Joel Koshy commented on KAFKA-1583: --- Thanks for the updated patch. I will do this today. [~junrao] do you also want to take another look? > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180899#comment-14180899 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-22_18:52:52.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch, KAFKA-1583_2014-10-22_18:52:52.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180037#comment-14180037 ] Guozhang Wang commented on KAFKA-1583: -- Yes, the system tests without the offset management test passes. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179460#comment-14179460 ] Joel Koshy commented on KAFKA-1583: --- I posted review comments, mostly minor. Did you get a chance to run the system tests with this patch? > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175763#comment-14175763 ] Joel Koshy commented on KAFKA-1583: --- Reviewing is taking a bit longer than expected - I'm about half-way through, so hopefully should be done on Monday. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175239#comment-14175239 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-17_09:56:33.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch, > KAFKA-1583_2014-10-17_09:56:33.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174709#comment-14174709 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-16_21:15:40.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch, KAFKA-1583_2014-10-16_21:15:40.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1583: -- Reviewer: Joel Koshy (was: Jun Rao) > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173215#comment-14173215 ] Joel Koshy commented on KAFKA-1583: --- Yes please. I will be able to do this on Friday. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173210#comment-14173210 ] Jun Rao commented on KAFKA-1583: Just reviewed it. Joel, Do you want to take another look before committing this? Thanks, > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171998#comment-14171998 ] Guozhang Wang commented on KAFKA-1583: -- Bump, [~junrao] could you take a look now? > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-13_19:41:58.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170405#comment-14170405 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch, > KAFKA-1583_2014-10-13_19:41:58.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167187#comment-14167187 ] Guozhang Wang commented on KAFKA-1583: -- Sure, will do that asap. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167184#comment-14167184 ] Jun Rao commented on KAFKA-1583: Guozhang, Now that the 0.8.2 branch is cut, do you want to rebase the patch to latest trunk? Thanks, > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141318#comment-14141318 ] Joel Koshy commented on KAFKA-1583: --- Yes I intend to review this within the next couple days. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141316#comment-14141316 ] Guozhang Wang commented on KAFKA-1583: -- That is fine. I will wait then. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141310#comment-14141310 ] Jun Rao commented on KAFKA-1583: Hi, Guozhang, Yes, since this is a relatively large patch, I am thinking of committing it after the 0.8.2 branch is cut. I think we should be able to cut the branch in a week or so. Hopefully there won't be too many changes in the next week to make the rebase more complicated. Will this work for you? Thanks, > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141242#comment-14141242 ] Guozhang Wang commented on KAFKA-1583: -- [~junrao] [~jjkoshy] Could you give this a look? Since this is a relatively large patch rebasing it multiple times with other new commits is quite some work :P. > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Rao updated KAFKA-1620:
---
Resolution: Fixed
Fix Version/s: 0.8.2
Status: Resolved (was: Patch Available)
Thanks for patch v2. +1 and committed to trunk.
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Fix For: 0.8.2
>
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch,
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136817#comment-14136817
]
Anton Karamanov commented on KAFKA-1620:
Yes, it is actually one of the primary motives for this issue, since it allows
to abstract from specific requests/responses on a low level. Without a common
class any generic collection of messages would have to have a type of
{{Collection[Any]}} reducing type safety and making it harder to control the
flow of requests/responses.
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch,
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136754#comment-14136754
]
Jun Rao commented on KAFKA-1620:
Thanks for the new patch. Similarly, do we need to make RequestOrResponse
public since it's just an abstract base class?
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch,
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Karamanov updated KAFKA-1620:
---
Attachment: 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
Not really. I just thought it might become useful at some point as well. I took
a closer look and now realize that it's related to internal messages, so
there's no actual need to make them public.
Here's updated and rebased
[patch|^0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch].
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch,
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134959#comment-14134959
]
Jun Rao commented on KAFKA-1620:
Thanks for the patch. Is there a particular reason that you need to make
GenericResponseAndHeader and GenericRequestAndHeader public?
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1583: - Reviewer: Jun Rao > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123664#comment-14123664 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-09-05_14:55:38.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch, KAFKA-1583_2014-09-05_14:55:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-09-05_14:08:36.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123581#comment-14123581 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch, > KAFKA-1583_2014-09-05_14:08:36.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118696#comment-14118696 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-09-02_13:37:47.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch, KAFKA-1583_2014-09-02_13:37:47.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117812#comment-14117812 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/diff/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-09-01_18:07:42.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch, > KAFKA-1583_2014-09-01_18:07:42.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jira] [Updated] (KAFKA-1620) Make kafka api protocol implementation public
I'm curiosity why Kafka don't implementation protocol by protocol buffer or
any other tools . It's good to use by other language
2014-09-01 22:48 GMT+08:00 Anton Karamanov (JIRA) :
>
> [
> https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Anton Karamanov updated KAFKA-1620:
> ---
> Reviewer: Jun Rao
> Assignee: Anton Karamanov
> Status: Patch Available (was: Open)
>
> > Make kafka api protocol implementation public
> > -
> >
> > Key: KAFKA-1620
> > URL: https://issues.apache.org/jira/browse/KAFKA-1620
> > Project: Kafka
> > Issue Type: Improvement
> >Reporter: Anton Karamanov
> >Assignee: Anton Karamanov
> > Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
> >
> >
> > Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly
> on top of Akka TCP), and don't seem to contain any actuall internal logic
> of Kafka. Therefore it seems like a nice idea to make them public.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
--
long is the way and hard that out of Hell leads up to light
[jira] [Updated] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Karamanov updated KAFKA-1620:
---
Reviewer: Jun Rao
Assignee: Anton Karamanov
Status: Patch Available (was: Open)
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1620) Make kafka api protocol implementation public
[
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Karamanov updated KAFKA-1620:
---
Attachment: 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
Here's a small
[patch|^0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch] to
address the issue.
> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
> Issue Type: Improvement
>Reporter: Anton Karamanov
> Attachments:
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to
> {{kafka}} package. Those classes would be extremely usefull for writing
> custom clients (we're using Scala with Akka and implementing one directly on
> top of Akka TCP), and don't seem to contain any actuall internal logic of
> Kafka. Therefore it seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-1620) Make kafka api protocol implementation public
Anton Karamanov created KAFKA-1620:
--
Summary: Make kafka api protocol implementation public
Key: KAFKA-1620
URL: https://issues.apache.org/jira/browse/KAFKA-1620
Project: Kafka
Issue Type: Improvement
Reporter: Anton Karamanov
Some of the classes which implement Kafka api protocol, such as
{{RequestOrResponse}} and {{FetchRequest}} are defined as private to {{kafka}}
package. Those classes would be extremely usefull for writing custom clients
(we're using Scala with Akka and implementing one directly on top of Akka TCP),
and don't seem to contain any actuall internal logic of Kafka. Therefore it
seems like a nice idea to make them public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-08-27_09:44:50.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112421#comment-14112421 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch, KAFKA-1583_2014-08-27_09:44:50.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105739#comment-14105739 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-08-21_11:30:34.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch, > KAFKA-1583_2014-08-21_11:30:34.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104560#comment-14104560 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.org/r/24676/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-08-20_13:54:38.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Status: Patch Available (was: Open) > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch, KAFKA-1583_2014-08-20_13:54:38.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096287#comment-14096287 ] Guozhang Wang commented on KAFKA-1583: -- Created reviewboard https://reviews.apache.org/r/24676/ against branch origin/trunk > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1583) Kafka API Refactoring
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583.patch > Kafka API Refactoring > - > > Key: KAFKA-1583 > URL: https://issues.apache.org/jira/browse/KAFKA-1583 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 0.9.0 > > Attachments: KAFKA-1583.patch > > > This is the next step of KAFKA-1430. Details can be found at this page: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1583) Kafka API Refactoring
Guozhang Wang created KAFKA-1583: Summary: Kafka API Refactoring Key: KAFKA-1583 URL: https://issues.apache.org/jira/browse/KAFKA-1583 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.9.0 This is the next step of KAFKA-1430. Details can be found at this page: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+API+Refactoring -- This message was sent by Atlassian JIRA (v6.2#6252)
