Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

[Tom Bentley]

Hi Hu,

I wonder whether changing, or configuring a size-balancing strategy would
be sufficient for all users. I would expect that users might want to take
other factors into account. For example, with KIP-113, balancing IO across
the disks might also be a factor, in addition to balancing free space: An
allocation that best optimises for free space might make one disc much for
heavily utilised than another, increasing latency.

Also, an alternative to putting this logic on the server side would be
providing the necessary information via the AdminClient to allow replica
assignments to be calculated off-server. This has a couple of benefits:

1. Finding optimal or near-optimal solutions to this kind of "knapsack
problem" might well involve compute-intensive algorithms. It would be
better to not run such computations on the server if possible.
2. It's not one-size-fits-all: It would let people devise the best
algorithm for their particular needs.

If this is an alternative you've rejected, it would be good to explain why
in the rejected alternatives section.

On 9 August 2017 at 03:38, Hu Xi <huxi...@hotmail.com> wrote:

> Hi Lin,
>
>
> Yes, it is a problem since it's not easy for us to predict the possible
> disk spaces each partition occupies in the future. How about the algorithm
> below:
>
>
>   1.  Introduce a data structure to maintain the current free spaces for
> each log directory in `log.dirs`. Say, a ConcurrentHashMap<String,
> AtomicLong> named logFreeDiskMap
>   2.  Initialize and update `logFreeDiskMap` before batch invoking
> `nextLogDir`. Invoke it before `makeLeaders` for instance.
>   3.  Every time we call `nextLogDir`, check:
>  *   If `log.retention.bytes` or  topic-level `retention.bytes` is
> set(> 0), then it's to say user does want to control the total disk spaces
> for that partition. So we select the directory with most disk spaces from
> `logFreeDiskMap` and subtract `retention.bytes` from the original value.
>  *   If `log.retention.bytes` or  topic-level `retention.bytes` is not
> set, meaning user does not care about the disk or he/she does not know how
> much the total disk spaces the partition occupies, then we turn back to the
> current strategy: selecting directory with least partitions
>
> The algorithm should be working well with the situation you mentioned,
> namely large number of partitions in a single LeaderAndIsr request.
> However, I have to admit that it's poor for the situation when consecutive
> LeaderAndIsr requests come from the controller.
>
> Any comments are welcomed.
>
> 
> 发件人: Dong Lin <lindon...@gmail.com>
> 发送时间: 2017年8月5日 2:00
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Hu,
>
> I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move
> 100 replicas to a broker. The the controller will send LeaderAndIsrRequest
> asking this broker to be the follower of these 100 partitions. While it is
> true that the broker will create replicas sequentially, but they will be
> created in a very short period of time (e.g. 2 seconds) and thus the
> replicas will be put in the same log directory that has the most free space
> at the time this broker receives the LeaderAndIsrRequest. Do you think this
> is a problem?
>
> Dong
>
>
> On Thu, Aug 3, 2017 at 7:36 PM, Hu Xi <huxi...@hotmail.com> wrote:
>
> > Hi Dong, some thoughts on your second mail. Since currently logs for
> > multiple partitions are created sequentially not in parallel, it's
> probably
> > okay for us to simply select the directory with most disk spaces in a
> > single round of `nextLogDir` calling. which can be guaranteed to lead to
> > extreme skew. Does it make any senses?
> >
> >
> > 
> > 发件人: Hu Xi <huxi...@hotmail.com>
> > 发送时间: 2017年8月3日 16:51
> > 收件人: dev@kafka.apache.org
> > 主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection
> strategy
> >
> >
> > Dong, yes, many thanks for the comments from the second mail. Will take
> > some time to figure out an algorithm to better handle the situation you
> > mentioned. Thanks again.
> >
> >
> > 
> > 发件人: Dong Lin <lindon...@gmail.com>
> > 发送时间: 2017年8月3日 12:07
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection
> strategy
> >
> > Hu, I think this is worth discussion even if it doesn't require new
> config.
> > Could you also read my second email?
> >
> > On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <huxi...@hotmail.com> wrote:
> 

答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

[Hu Xi]

Hi Lin,


Yes, it is a problem since it's not easy for us to predict the possible disk 
spaces each partition occupies in the future. How about the algorithm below:


  1.  Introduce a data structure to maintain the current free spaces for each 
log directory in `log.dirs`. Say, a ConcurrentHashMap<String, AtomicLong> named 
logFreeDiskMap
  2.  Initialize and update `logFreeDiskMap` before batch invoking 
`nextLogDir`. Invoke it before `makeLeaders` for instance.
  3.  Every time we call `nextLogDir`, check:
 *   If `log.retention.bytes` or  topic-level `retention.bytes` is set(> 
0), then it's to say user does want to control the total disk spaces for that 
partition. So we select the directory with most disk spaces from 
`logFreeDiskMap` and subtract `retention.bytes` from the original value.
 *   If `log.retention.bytes` or  topic-level `retention.bytes` is not set, 
meaning user does not care about the disk or he/she does not know how much the 
total disk spaces the partition occupies, then we turn back to the current 
strategy: selecting directory with least partitions

The algorithm should be working well with the situation you mentioned, namely 
large number of partitions in a single LeaderAndIsr request. However, I have to 
admit that it's poor for the situation when consecutive LeaderAndIsr requests 
come from the controller.

Any comments are welcomed.


发件人: Dong Lin <lindon...@gmail.com>
发送时间: 2017年8月5日 2:00
收件人: dev@kafka.apache.org
主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hey Hu,

I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move
100 replicas to a broker. The the controller will send LeaderAndIsrRequest
asking this broker to be the follower of these 100 partitions. While it is
true that the broker will create replicas sequentially, but they will be
created in a very short period of time (e.g. 2 seconds) and thus the
replicas will be put in the same log directory that has the most free space
at the time this broker receives the LeaderAndIsrRequest. Do you think this
is a problem?

Dong


On Thu, Aug 3, 2017 at 7:36 PM, Hu Xi <huxi...@hotmail.com> wrote:

> Hi Dong, some thoughts on your second mail. Since currently logs for
> multiple partitions are created sequentially not in parallel, it's probably
> okay for us to simply select the directory with most disk spaces in a
> single round of `nextLogDir` calling. which can be guaranteed to lead to
> extreme skew. Does it make any senses?
>
>
> 
> 发件人: Hu Xi <huxi...@hotmail.com>
> 发送时间: 2017年8月3日 16:51
> 收件人: dev@kafka.apache.org
> 主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
>
> Dong, yes, many thanks for the comments from the second mail. Will take
> some time to figure out an algorithm to better handle the situation you
> mentioned. Thanks again.
>
>
> 
> 发件人: Dong Lin <lindon...@gmail.com>
> 发送时间: 2017年8月3日 12:07
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hu, I think this is worth discussion even if it doesn't require new config.
> Could you also read my second email?
>
> On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <huxi...@hotmail.com> wrote:
>
> > Thanks Dong,  do you mean it is more like a naive improvement and no KIP
> > is needed  then?
> >
> > 
> > 发件人: Dong Lin <lindon...@gmail.com>
> > 发送时间: 2017年8月3日 9:10
> > 收件人: dev@kafka.apache.org
> > 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> > Hey Xu,
> >
> > Thanks for the KIP. This is a very good idea to select log directory
> based
> > on the free disk space. Do you think we can simply simply change the
> > implementation to select log directory based on the free disk space
> instead
> > of adding a new config? Or is there any good reason that user will want
> to
> > select log directory with the least partition number instead of the one
> > with the most free disk space?
> >
> > Thanks,
> > Dong
> >
> >
> > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <huxi...@hotmail.com> wrote:
> >
> > > Hi all, how do you think of this KIP? Any comments are welcomed.
> > >
> > >
> > > 
> > > 发件人: Hu Xi <huxi...@hotmail.com>
> > > 发送时间: 2017年7月18日 15:21
> > > 收件人: dev@kafka.apache.org
> > > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> > >
> > >
> > > Hi all,
> > >
> > >  KIP-178 is created for a discussion on how LogManager selects log
> > > directory. In this KIP, a new strategy is introduced to allow for the
> > real
> > > disk spaces for each directories. Be free to drop your comments here.
> > > Thanks.
> > >
> >
>

答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

[Hu Xi]

Dong, yes, many thanks for the comments from the second mail. Will take some 
time to figure out an algorithm to better handle the situation you mentioned. 
Thanks again.



发件人: Dong Lin <lindon...@gmail.com>
发送时间: 2017年8月3日 12:07
收件人: dev@kafka.apache.org
主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hu, I think this is worth discussion even if it doesn't require new config.
Could you also read my second email?

On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <huxi...@hotmail.com> wrote:

> Thanks Dong,  do you mean it is more like a naive improvement and no KIP
> is needed  then?
>
> 
> 发件人: Dong Lin <lindon...@gmail.com>
> 发送时间: 2017年8月3日 9:10
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <huxi...@hotmail.com> wrote:
>
> > Hi all, how do you think of this KIP? Any comments are welcomed.
> >
> >
> > 
> > 发件人: Hu Xi <huxi...@hotmail.com>
> > 发送时间: 2017年7月18日 15:21
> > 收件人: dev@kafka.apache.org
> > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> >
> > Hi all,
> >
> >  KIP-178 is created for a discussion on how LogManager selects log
> > directory. In this KIP, a new strategy is introduced to allow for the
> real
> > disk spaces for each directories. Be free to drop your comments here.
> > Thanks.
> >
>

Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

[Dong Lin]

Hu, I think this is worth discussion even if it doesn't require new config.
Could you also read my second email?

On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi  wrote:

> Thanks Dong,  do you mean it is more like a naive improvement and no KIP
> is needed  then?
>
> 
> 发件人: Dong Lin 
> 发送时间: 2017年8月3日 9:10
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi  wrote:
>
> > Hi all, how do you think of this KIP? Any comments are welcomed.
> >
> >
> > 
> > 发件人: Hu Xi 
> > 发送时间: 2017年7月18日 15:21
> > 收件人: dev@kafka.apache.org
> > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> >
> > Hi all,
> >
> >  KIP-178 is created for a discussion on how LogManager selects log
> > directory. In this KIP, a new strategy is introduced to allow for the
> real
> > disk spaces for each directories. Be free to drop your comments here.
> > Thanks.
> >
>

答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

[Hu Xi]

Thanks Dong,  do you mean it is more like a naive improvement and no KIP is 
needed  then?


发件人: Dong Lin 
发送时间: 2017年8月3日 9:10
收件人: dev@kafka.apache.org
主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hey Xu,

Thanks for the KIP. This is a very good idea to select log directory based
on the free disk space. Do you think we can simply simply change the
implementation to select log directory based on the free disk space instead
of adding a new config? Or is there any good reason that user will want to
select log directory with the least partition number instead of the one
with the most free disk space?

Thanks,
Dong


On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi  wrote:

> Hi all, how do you think of this KIP? Any comments are welcomed.
>
>
> 
> 发件人: Hu Xi 
> 发送时间: 2017年7月18日 15:21
> 收件人: dev@kafka.apache.org
> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
>
> Hi all,
>
>  KIP-178 is created for a discussion on how LogManager selects log
> directory. In this KIP, a new strategy is introduced to allow for the real
> disk spaces for each directories. Be free to drop your comments here.
> Thanks.
>