[
https://issues.apache.org/jira/browse/KAFKA-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huxihx updated KAFKA-5560:
--------------------------
Description:
Currently, log manager chooses a directory configured in `log.dirs` by
calculating the number partitions in each directory and then choosing the one
with the fewest partitions. But in some real production scenarios where data
volumes of partitions are not even, some disks nearly become full whereas the
others have a lot of spaces which lead to a poor data distribution.
We should offer a new strategy to users to have log manager honor the real disk
free spaces and choose the directory with the most disk space. Maybe a new
broker configuration parameter is needed, `log.directory.strategy` for
instance. A new KIP is created to track this issue:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-178%3A+Size-based+log+directory+selection+strategy
Does it make sense?
was:
Currently, log manager chooses a directory configured in `log.dirs` by
calculating the number partitions in each directory and then choosing the one
with the fewest partitions. But in some real production scenarios where data
volumes of partitions are not even, some disks nearly become full whereas the
others have a lot of spaces which lead to a poor data distribution.
We should offer a new strategy to users to have log manager honor the real disk
free spaces and choose the directory with the most disk space. Maybe a new
broker configuration parameter is needed, `log.directory.strategy` for
instance. Perhaps this needs a new KIP also.
Does it make sense?
> LogManager should be able to create new logs based on free disk space
> ---------------------------------------------------------------------
>
> Key: KAFKA-5560
> URL: https://issues.apache.org/jira/browse/KAFKA-5560
> Project: Kafka
> Issue Type: Improvement
> Components: log
> Affects Versions: 0.11.0.0
> Reporter: huxihx
> Labels: kips
>
> Currently, log manager chooses a directory configured in `log.dirs` by
> calculating the number partitions in each directory and then choosing the one
> with the fewest partitions. But in some real production scenarios where data
> volumes of partitions are not even, some disks nearly become full whereas the
> others have a lot of spaces which lead to a poor data distribution.
> We should offer a new strategy to users to have log manager honor the real
> disk free spaces and choose the directory with the most disk space. Maybe a
> new broker configuration parameter is needed, `log.directory.strategy` for
> instance. A new KIP is created to track this issue:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-178%3A+Size-based+log+directory+selection+strategy
> Does it make sense?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
