[
https://issues.apache.org/jira/browse/HDDS-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Meng-Shuan Tsai updated HDDS-16374:
-----------------------------------
Description:
h2. Problem
This is a bug. Datanode `BlockDeletingService` starts with one interval key,
then rereads a different, unaaliased key on dynamic reconfiguration.
* Startup (`OzoneContainer`) uses
`hdds.datanode.block.deleting.service.interval` via
`DatanodeConfiguration.getBlockDeletionInterval()`.
* Reconfiguration (`HddsDatanodeService` registration +
`BlockDeletingService.updateAndRestart`) uses
`ozone.block.deleting.service.interval`.
Timeout and workers already share one key on both paths. Only interval is
split. Both keys default to 60s, so a default-only cluster never shows the
mismatch.
HDDS-4367 split OM / SCM / Datanode deletion intervals on purpose. OM still
uses `ozone.block.deleting.service.interval`. The later Datanode
reconfiguration path (HDDS-11513) registered and reread the old shared key, so
runtime restart no longer follows the datanode-specific contract.
h2. Root cause
`HddsDatanodeService` registers `OZONE_BLOCK_DELETING_SERVICE_INTERVAL`
(`ozone.block.deleting.service.interval`). The complete callback treats a
change to workers, timeout, or that old interval key as a full restart, and
`updateAndRestart` always rereads the old key:
{code:java}
long newInterval = ozoneConf.getTimeDuration(
OZONE_BLOCK_DELETING_SERVICE_INTERVAL,
OZONE_BLOCK_DELETING_SERVICE_INTERVAL_DEFAULT,
TimeUnit.SECONDS);
// ...
setInterval(newInterval, TimeUnit.SECONDS);{code}
`hdds.datanode.block.deleting.service.interval` is not `reconfigurable = true`,
so it is not in the DataNode reconfigurable set. There is no `DeprecationDelta`
between the two keys; setting one does not fill the other.
h2. Impact
After an unrelated workers/timeout reconfig, the deletion scheduler can
silently leave the operator-configured datanode interval. Scope is Datanode
`BlockDeletingService` only. OM still owns
`ozone.block.deleting.service.interval`.
was:
*Problem*
This is a bug. Datanode `BlockDeletingService` starts with one interval key,
then rereads a different, unaaliased key on dynamic reconfiguration.
* Startup (`OzoneContainer`) uses
`hdds.datanode.block.deleting.service.interval` via
`DatanodeConfiguration.getBlockDeletionInterval()`.
* Reconfiguration (`HddsDatanodeService` registration +
`BlockDeletingService.updateAndRestart`) uses
`ozone.block.deleting.service.interval`.
Timeout and workers already share one key on both paths. Only interval is
split. Both keys default to 60s, so a default-only cluster never shows the
mismatch.
HDDS-4367 split OM / SCM / Datanode deletion intervals on purpose. OM still
uses `ozone.block.deleting.service.interval`. The later Datanode
reconfiguration path (HDDS-11513) registered and reread the old shared key, so
runtime restart no longer follows the datanode-specific contract.
*Root cause*
`HddsDatanodeService` registers `OZONE_BLOCK_DELETING_SERVICE_INTERVAL`
(`ozone.block.deleting.service.interval`). The complete callback treats a
change to workers, timeout, or that old interval key as a full restart, and
`updateAndRestart` always rereads the old key:
```java
long newInterval = ozoneConf.getTimeDuration(
OZONE_BLOCK_DELETING_SERVICE_INTERVAL,
OZONE_BLOCK_DELETING_SERVICE_INTERVAL_DEFAULT,
TimeUnit.SECONDS);
// ...
setInterval(newInterval, TimeUnit.SECONDS);
```
`hdds.datanode.block.deleting.service.interval` is not `reconfigurable = true`,
so it is not in the DataNode reconfigurable set. There is no `DeprecationDelta`
between the two keys; setting one does not fill the other.
*Impact*
After an unrelated workers/timeout reconfig, the deletion scheduler can
silently leave the operator-configured datanode interval. Scope is Datanode
`BlockDeletingService` only. OM still owns
`ozone.block.deleting.service.interval`.
> Datanode BlockDeletingService reconfiguration uses a different interval key
> than startup
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-16374
> URL: https://issues.apache.org/jira/browse/HDDS-16374
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Meng-Shuan Tsai
> Assignee: Meng-Shuan Tsai
> Priority: Major
>
> h2. Problem
> This is a bug. Datanode `BlockDeletingService` starts with one interval key,
> then rereads a different, unaaliased key on dynamic reconfiguration.
> * Startup (`OzoneContainer`) uses
> `hdds.datanode.block.deleting.service.interval` via
> `DatanodeConfiguration.getBlockDeletionInterval()`.
> * Reconfiguration (`HddsDatanodeService` registration +
> `BlockDeletingService.updateAndRestart`) uses
> `ozone.block.deleting.service.interval`.
> Timeout and workers already share one key on both paths. Only interval is
> split. Both keys default to 60s, so a default-only cluster never shows the
> mismatch.
> HDDS-4367 split OM / SCM / Datanode deletion intervals on purpose. OM still
> uses `ozone.block.deleting.service.interval`. The later Datanode
> reconfiguration path (HDDS-11513) registered and reread the old shared key,
> so runtime restart no longer follows the datanode-specific contract.
> h2. Root cause
> `HddsDatanodeService` registers `OZONE_BLOCK_DELETING_SERVICE_INTERVAL`
> (`ozone.block.deleting.service.interval`). The complete callback treats a
> change to workers, timeout, or that old interval key as a full restart, and
> `updateAndRestart` always rereads the old key:
> {code:java}
> long newInterval = ozoneConf.getTimeDuration(
> OZONE_BLOCK_DELETING_SERVICE_INTERVAL,
> OZONE_BLOCK_DELETING_SERVICE_INTERVAL_DEFAULT,
> TimeUnit.SECONDS);
> // ...
> setInterval(newInterval, TimeUnit.SECONDS);{code}
> `hdds.datanode.block.deleting.service.interval` is not `reconfigurable =
> true`, so it is not in the DataNode reconfigurable set. There is no
> `DeprecationDelta` between the two keys; setting one does not fill the other.
> h2. Impact
> After an unrelated workers/timeout reconfig, the deletion scheduler can
> silently leave the operator-configured datanode interval. Scope is Datanode
> `BlockDeletingService` only. OM still owns
> `ozone.block.deleting.service.interval`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
