[
https://issues.apache.org/jira/browse/KAFKA-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuan Po Tseng updated KAFKA-19264:
----------------------------------
Description:
The fallback mechanism was first introduced in
[KIP-950|https://cwiki.apache.org/confluence/x/joqzDw]. According to the
proposal, if no thread values are set for
{{remote.log.manager.copier.thread.pool.size}} and
{{{}remote.log.manager.expiration.thread.pool.size{}}}, these two configs would
default to using the value of {{{}remote.log.manager.thread.pool.size{}}}.
As quoted from the KIP:
{quote}If no thread values are set for the two new configurations presented
later on in the document we will default to using the same number of threads in
each pool as detailed by remote.log.manager.thread.pool.size.
{quote}
This fallback behavior was implemented in
[https://github.com/apache/kafka/commit/84ab3b9a5c4930f5ae047df088e38c456c7cde54].
However, this approach was abandoned in
[KIP-1030|https://cwiki.apache.org/confluence/x/FAqpEQ], where the default
values for {{copier}} and {{expiration}} thread pool sizes were changed from
{{-1}} to {{{}10{}}}. The related commit can be found in
[https://github.com/apache/kafka/commit/3b1bd3812e48d488e4b6b53a9085d6552e8adf02].
Additionally, both {{remote.log.manager.copier.thread.pool.size}} and
{{remote.log.manager.expiration.thread.pool.size}} now include a configuration
validator that enforces a minimum value of {{{}1{}}}. This means the fallback
mechanism should be removed from
[RemoteLogManagerConfig.java|https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java]
to align with the new defaults and validation.
In short, RemoteLogManagerConfig should apply the following changes:
{code:java}
public int remoteLogManagerCopierThreadPoolSize() {
- int size =
config.getInt(REMOTE_LOG_MANAGER_COPIER_THREAD_POOL_SIZE_PROP);
- return size == -1 ? remoteLogManagerThreadPoolSize() : size;
+ return config.getInt(REMOTE_LOG_MANAGER_COPIER_THREAD_POOL_SIZE_PROP);
}
public int remoteLogManagerExpirationThreadPoolSize() {
- int size =
config.getInt(REMOTE_LOG_MANAGER_EXPIRATION_THREAD_POOL_SIZE_PROP);
- return size == -1 ? remoteLogManagerThreadPoolSize() : size;
+ return
config.getInt(REMOTE_LOG_MANAGER_EXPIRATION_THREAD_POOL_SIZE_PROP);
}{code}
was:
The fallback mechanism was first introduced in
[KIP-950|https://cwiki.apache.org/confluence/x/joqzDw]. According to the
proposal, if no thread values are set for
{{remote.log.manager.copier.thread.pool.size}} and
{{{}remote.log.manager.expiration.thread.pool.size{}}}, these two configs would
default to using the value of {{{}remote.log.manager.thread.pool.size{}}}.
As quoted from the KIP:
{quote}If no thread values are set for the two new configurations presented
later on in the document we will default to using the same number of threads in
each pool as detailed by remote.log.manager.thread.pool.size.
{quote}
This fallback behavior was implemented in
[https://github.com/apache/kafka/commit/84ab3b9a5c4930f5ae047df088e38c456c7cde54].
However, this approach was abandoned in
[KIP-1030|https://cwiki.apache.org/confluence/x/FAqpEQ], where the default
values for {{copier}} and {{expiration}} thread pool sizes were changed from
{{-1}} to {{{}10{}}}. The related commit can be found in
[https://github.com/apache/kafka/commit/3b1bd3812e48d488e4b6b53a9085d6552e8adf02].
Additionally, both {{remote.log.manager.copier.thread.pool.size}} and
{{remote.log.manager.expiration.thread.pool.size}} now include a configuration
validator that enforces a minimum value of {{{}1{}}}. This means the fallback
mechanism should be removed from
[{{RemoteLogManagerConfig.java}}|https://issues.apache.org/jira/issues/RemoteLogManagerConfig.java]
to align with the new defaults and validation.
In short, RemoteLogManagerConfig should apply the following changes:
{code:java}
public int remoteLogManagerCopierThreadPoolSize() {
- int size =
config.getInt(REMOTE_LOG_MANAGER_COPIER_THREAD_POOL_SIZE_PROP);
- return size == -1 ? remoteLogManagerThreadPoolSize() : size;
+ return config.getInt(REMOTE_LOG_MANAGER_COPIER_THREAD_POOL_SIZE_PROP);
}
public int remoteLogManagerExpirationThreadPoolSize() {
- int size =
config.getInt(REMOTE_LOG_MANAGER_EXPIRATION_THREAD_POOL_SIZE_PROP);
- return size == -1 ? remoteLogManagerThreadPoolSize() : size;
+ return
config.getInt(REMOTE_LOG_MANAGER_EXPIRATION_THREAD_POOL_SIZE_PROP);
}{code}
> Remove fallback for thread pool sizes in RemoteLogManagerConfig
> ---------------------------------------------------------------
>
> Key: KAFKA-19264
> URL: https://issues.apache.org/jira/browse/KAFKA-19264
> Project: Kafka
> Issue Type: Task
> Components: Tiered-Storage
> Reporter: Kuan Po Tseng
> Assignee: Kuan Po Tseng
> Priority: Minor
>
> The fallback mechanism was first introduced in
> [KIP-950|https://cwiki.apache.org/confluence/x/joqzDw]. According to the
> proposal, if no thread values are set for
> {{remote.log.manager.copier.thread.pool.size}} and
> {{{}remote.log.manager.expiration.thread.pool.size{}}}, these two configs
> would default to using the value of
> {{{}remote.log.manager.thread.pool.size{}}}.
> As quoted from the KIP:
> {quote}If no thread values are set for the two new configurations presented
> later on in the document we will default to using the same number of threads
> in each pool as detailed by remote.log.manager.thread.pool.size.
> {quote}
> This fallback behavior was implemented in
> [https://github.com/apache/kafka/commit/84ab3b9a5c4930f5ae047df088e38c456c7cde54].
> However, this approach was abandoned in
> [KIP-1030|https://cwiki.apache.org/confluence/x/FAqpEQ], where the default
> values for {{copier}} and {{expiration}} thread pool sizes were changed from
> {{-1}} to {{{}10{}}}. The related commit can be found in
> [https://github.com/apache/kafka/commit/3b1bd3812e48d488e4b6b53a9085d6552e8adf02].
> Additionally, both {{remote.log.manager.copier.thread.pool.size}} and
> {{remote.log.manager.expiration.thread.pool.size}} now include a
> configuration validator that enforces a minimum value of {{{}1{}}}. This
> means the fallback mechanism should be removed from
> [RemoteLogManagerConfig.java|https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java]
> to align with the new defaults and validation.
> In short, RemoteLogManagerConfig should apply the following changes:
> {code:java}
> public int remoteLogManagerCopierThreadPoolSize() {
> - int size =
> config.getInt(REMOTE_LOG_MANAGER_COPIER_THREAD_POOL_SIZE_PROP);
> - return size == -1 ? remoteLogManagerThreadPoolSize() : size;
> + return
> config.getInt(REMOTE_LOG_MANAGER_COPIER_THREAD_POOL_SIZE_PROP);
> }
> public int remoteLogManagerExpirationThreadPoolSize() {
> - int size =
> config.getInt(REMOTE_LOG_MANAGER_EXPIRATION_THREAD_POOL_SIZE_PROP);
> - return size == -1 ? remoteLogManagerThreadPoolSize() : size;
> + return
> config.getInt(REMOTE_LOG_MANAGER_EXPIRATION_THREAD_POOL_SIZE_PROP);
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
