[
https://issues.apache.org/jira/browse/HBASE-30363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-30363:
--------------------------------------
Description:
h2. Description
{{LruAdaptiveBlockCache}} is currently implemented as a specialized subclass of
{{LruBlockCache}}.
Most of its implementation and runtime behavior are inherited directly from
{{LruBlockCache}}.
Its distinguishing behavior is an adaptive admission mechanism which reduces
the percentage of
data blocks admitted to the cache when sustained eviction pressure is detected,
and gradually
restores admission when that pressure decreases.
With the pluggable block cache architecture, these responsibilities should be
separated:
* local LRU storage and eviction belong to {{LruCacheEngine}};
* admission and placement decisions belong to {{CachePlacementAdmissionPolicy}}.
Therefore, the native representation of AdaptiveLRU should not require a
separate
{{AdaptiveLruCacheEngine}}. It should instead be composed from:
{code}
LruCacheEngine
+
AdaptiveLruCachePlacementAdmissionPolicy
{code}
The adaptive policy requires feedback about capacity-driven cache evictions in
order to adjust
future admission decisions. This ticket therefore also adds eviction
notification support to
{{CachePlacementAdmissionPolicy}}.
h2. Proposed Architecture
The resulting cache path is:
{code}
TopologyBackedCacheAccessService
|
+-- CacheTopology
| |
| `-- LruCacheEngine
|
`-- AdaptiveLruCachePlacementAdmissionPolicy
{code}
{{LruCacheEngine}} remains responsible for:
* block storage;
* LRU replacement;
* cache priority handling;
* capacity management;
* eviction execution;
* reference counting;
* cache size and element accounting;
* cache statistics.
{{AdaptiveLruCachePlacementAdmissionPolicy}} is responsible for:
* tracking cache eviction pressure;
* maintaining the current data-block admission level;
* adjusting admission based on sustained eviction activity;
* applying the adaptive admission decision to future cache writes.
This keeps engine behavior independent from higher-level placement and
admission policy.
h2. Eviction Feedback
Add a default no-op capacity-driven eviction notification to
{{CachePlacementAdmissionPolicy}}, for example:
{code:java}
default void onEviction(BlockCacheKey cacheKey, Cacheable block, CacheTier
sourceTier,
CacheTopologyView topologyView) {
}
{code}
The notification allows stateful placement/admission policies to adapt based on
cache pressure
without requiring cache engines to know about policy implementations.
{{TopologyBackedCacheAccessService}} should forward capacity-driven
{{CacheEngine}} eviction notifications to the active
{{CachePlacementAdmissionPolicy}}.
The notification should represent cache-pressure eviction rather than explicit
administrative
eviction.
The existing default policy remains unaffected because the callback is a no-op
by default.
h2. Adaptive Policy
Introduce {{AdaptiveLruCachePlacementAdmissionPolicy}}.
The adaptive algorithm should preserve the existing {{LruAdaptiveBlockCache}}
behavior as closely
as possible.
The policy should own only the adaptive state and decision logic currently
unique to
{{LruAdaptiveBlockCache}}, including:
* current data-block admission percentage;
* eviction counters;
* eviction-volume tracking;
* measurement-window state;
* heavy-eviction thresholds;
* admission reduction and recovery logic.
The existing algorithm should be migrated without unrelated behavioral changes.
The policy should reuse or delegate normal HBase cache placement behavior where
possible rather
than duplicating the complete default placement implementation.
h2. Topology Semantics
{{CachePlacementAdmissionPolicy}} is a cache-graph-level policy. Cache engines
must not contain
their own independent placement/admission policies.
Adaptive admission therefore needs to respect the active topology.
For a single-tier topology:
{code}
metadata block -> SINGLE
admitted data block -> SINGLE
rejected data block -> not cached
{code}
For a tiered topology, adaptive rejection from L1 must not automatically imply
rejection from the
entire cache hierarchy.
For example, an L1 adaptive decision may result in:
{code}
data block -> do not place in L1
-> L2 may still receive the block according to topology/policy
semantics
{code}
The implementation must therefore avoid introducing conflicting policy
decisions inside individual
{{CacheEngine}} instances.
h2. Legacy Compatibility
The legacy {{LruAdaptiveBlockCache}} cannot be removed as part of this ticket.
Legacy cache construction still uses:
{code}
BlockCacheFactory -> BlockCache
{code}
while the new architecture uses:
{code}
CacheAccessService
-> CacheTopology
-> CacheEngine
-> CachePlacementAdmissionPolicy
{code}
The legacy {{BlockCacheFactory}} path should therefore continue to construct
{{LruAdaptiveBlockCache}} where a {{BlockCache}} instance is required.
The native {{CacheAccessService}} path should use:
{code}
LruCacheEngine
+
AdaptiveLruCachePlacementAdmissionPolicy
{code}
{{LruAdaptiveBlockCache}} can be retired later after remaining
Region/RegionServer cache ownership
and other legacy {{BlockCache}} consumers are migrated to
{{CacheAccessService}}.
h2. Tests
Add tests covering:
* delivery of capacity-driven engine eviction notifications to
{{CachePlacementAdmissionPolicy}};
* default policy behavior remaining unchanged when eviction notifications are
received;
* adaptive eviction-pressure accounting;
* reduction of data-block admission under sustained heavy eviction;
* recovery of data-block admission when eviction pressure decreases;
* metadata-block behavior;
* data-block admission behavior matching existing AdaptiveLRU semantics;
* single-tier AdaptiveLRU composition using {{LruCacheEngine}};
* tiered behavior where an adaptive L1 decision does not incorrectly reject a
block from the
entire cache hierarchy;
* native cache construction selecting {{LruCacheEngine}} together with
{{AdaptiveLruCachePlacementAdmissionPolicy}};
* preservation of the legacy {{LruAdaptiveBlockCache}} construction path where
{{BlockCache}} is still required.
h2. Acceptance Criteria
* AdaptiveLRU can be represented natively as {{LruCacheEngine}} plus
{{AdaptiveLruCachePlacementAdmissionPolicy}}.
* No {{AdaptiveLruCacheEngine}} implementation is introduced.
* {{CachePlacementAdmissionPolicy}} can receive capacity-driven cache eviction
feedback.
* Existing policies that do not require eviction feedback are unaffected.
* Adaptive admission behavior is driven by eviction feedback from the cache
engine.
* LRU storage and replacement logic is not duplicated in the adaptive policy.
* Adaptive admission respects single-tier and tiered cache topology semantics.
* The existing AdaptiveLRU configuration continues to work through the legacy
{{BlockCacheFactory}} path.
* No existing {{BlockCache}} consumer is forced to migrate as part of this
ticket.
* Tests verify both eviction-feedback propagation and AdaptiveLRU-specific
behavior.
was:
* Add LruAdaptiveBlockCacheEngine.
* Preserve adaptive admission/eviction behavior.
* Expose CacheEngine diagnostics.
* Add compatibility tests against existing LruAdaptiveBlockCache behavior.
> Migrate LruAdaptiveBlockCache to LruCacheEngine and adaptive
> placement/admission policy
> ---------------------------------------------------------------------------------------
>
> Key: HBASE-30363
> URL: https://issues.apache.org/jira/browse/HBASE-30363
> Project: HBase
> Issue Type: New Feature
> Components: BlockCache
> Reporter: Vladimir Rodionov
> Assignee: Vladimir Rodionov
> Priority: Major
> Fix For: 4.0.0-alpha-1
>
>
> h2. Description
> {{LruAdaptiveBlockCache}} is currently implemented as a specialized subclass
> of
> {{LruBlockCache}}.
> Most of its implementation and runtime behavior are inherited directly from
> {{LruBlockCache}}.
> Its distinguishing behavior is an adaptive admission mechanism which reduces
> the percentage of
> data blocks admitted to the cache when sustained eviction pressure is
> detected, and gradually
> restores admission when that pressure decreases.
> With the pluggable block cache architecture, these responsibilities should be
> separated:
> * local LRU storage and eviction belong to {{LruCacheEngine}};
> * admission and placement decisions belong to
> {{CachePlacementAdmissionPolicy}}.
> Therefore, the native representation of AdaptiveLRU should not require a
> separate
> {{AdaptiveLruCacheEngine}}. It should instead be composed from:
> {code}
> LruCacheEngine
> +
> AdaptiveLruCachePlacementAdmissionPolicy
> {code}
> The adaptive policy requires feedback about capacity-driven cache evictions
> in order to adjust
> future admission decisions. This ticket therefore also adds eviction
> notification support to
> {{CachePlacementAdmissionPolicy}}.
> h2. Proposed Architecture
> The resulting cache path is:
> {code}
> TopologyBackedCacheAccessService
> |
> +-- CacheTopology
> | |
> | `-- LruCacheEngine
> |
> `-- AdaptiveLruCachePlacementAdmissionPolicy
> {code}
> {{LruCacheEngine}} remains responsible for:
> * block storage;
> * LRU replacement;
> * cache priority handling;
> * capacity management;
> * eviction execution;
> * reference counting;
> * cache size and element accounting;
> * cache statistics.
> {{AdaptiveLruCachePlacementAdmissionPolicy}} is responsible for:
> * tracking cache eviction pressure;
> * maintaining the current data-block admission level;
> * adjusting admission based on sustained eviction activity;
> * applying the adaptive admission decision to future cache writes.
> This keeps engine behavior independent from higher-level placement and
> admission policy.
> h2. Eviction Feedback
> Add a default no-op capacity-driven eviction notification to
> {{CachePlacementAdmissionPolicy}}, for example:
> {code:java}
> default void onEviction(BlockCacheKey cacheKey, Cacheable block, CacheTier
> sourceTier,
> CacheTopologyView topologyView) {
> }
> {code}
> The notification allows stateful placement/admission policies to adapt based
> on cache pressure
> without requiring cache engines to know about policy implementations.
> {{TopologyBackedCacheAccessService}} should forward capacity-driven
> {{CacheEngine}} eviction notifications to the active
> {{CachePlacementAdmissionPolicy}}.
> The notification should represent cache-pressure eviction rather than
> explicit administrative
> eviction.
> The existing default policy remains unaffected because the callback is a
> no-op by default.
> h2. Adaptive Policy
> Introduce {{AdaptiveLruCachePlacementAdmissionPolicy}}.
> The adaptive algorithm should preserve the existing {{LruAdaptiveBlockCache}}
> behavior as closely
> as possible.
> The policy should own only the adaptive state and decision logic currently
> unique to
> {{LruAdaptiveBlockCache}}, including:
> * current data-block admission percentage;
> * eviction counters;
> * eviction-volume tracking;
> * measurement-window state;
> * heavy-eviction thresholds;
> * admission reduction and recovery logic.
> The existing algorithm should be migrated without unrelated behavioral
> changes.
> The policy should reuse or delegate normal HBase cache placement behavior
> where possible rather
> than duplicating the complete default placement implementation.
> h2. Topology Semantics
> {{CachePlacementAdmissionPolicy}} is a cache-graph-level policy. Cache
> engines must not contain
> their own independent placement/admission policies.
> Adaptive admission therefore needs to respect the active topology.
> For a single-tier topology:
> {code}
> metadata block -> SINGLE
> admitted data block -> SINGLE
> rejected data block -> not cached
> {code}
> For a tiered topology, adaptive rejection from L1 must not automatically
> imply rejection from the
> entire cache hierarchy.
> For example, an L1 adaptive decision may result in:
> {code}
> data block -> do not place in L1
> -> L2 may still receive the block according to topology/policy
> semantics
> {code}
> The implementation must therefore avoid introducing conflicting policy
> decisions inside individual
> {{CacheEngine}} instances.
> h2. Legacy Compatibility
> The legacy {{LruAdaptiveBlockCache}} cannot be removed as part of this ticket.
> Legacy cache construction still uses:
> {code}
> BlockCacheFactory -> BlockCache
> {code}
> while the new architecture uses:
> {code}
> CacheAccessService
> -> CacheTopology
> -> CacheEngine
> -> CachePlacementAdmissionPolicy
> {code}
> The legacy {{BlockCacheFactory}} path should therefore continue to construct
> {{LruAdaptiveBlockCache}} where a {{BlockCache}} instance is required.
> The native {{CacheAccessService}} path should use:
> {code}
> LruCacheEngine
> +
> AdaptiveLruCachePlacementAdmissionPolicy
> {code}
> {{LruAdaptiveBlockCache}} can be retired later after remaining
> Region/RegionServer cache ownership
> and other legacy {{BlockCache}} consumers are migrated to
> {{CacheAccessService}}.
> h2. Tests
> Add tests covering:
> * delivery of capacity-driven engine eviction notifications to
> {{CachePlacementAdmissionPolicy}};
> * default policy behavior remaining unchanged when eviction notifications are
> received;
> * adaptive eviction-pressure accounting;
> * reduction of data-block admission under sustained heavy eviction;
> * recovery of data-block admission when eviction pressure decreases;
> * metadata-block behavior;
> * data-block admission behavior matching existing AdaptiveLRU semantics;
> * single-tier AdaptiveLRU composition using {{LruCacheEngine}};
> * tiered behavior where an adaptive L1 decision does not incorrectly reject a
> block from the
> entire cache hierarchy;
> * native cache construction selecting {{LruCacheEngine}} together with
> {{AdaptiveLruCachePlacementAdmissionPolicy}};
> * preservation of the legacy {{LruAdaptiveBlockCache}} construction path where
> {{BlockCache}} is still required.
> h2. Acceptance Criteria
> * AdaptiveLRU can be represented natively as {{LruCacheEngine}} plus
> {{AdaptiveLruCachePlacementAdmissionPolicy}}.
> * No {{AdaptiveLruCacheEngine}} implementation is introduced.
> * {{CachePlacementAdmissionPolicy}} can receive capacity-driven cache
> eviction feedback.
> * Existing policies that do not require eviction feedback are unaffected.
> * Adaptive admission behavior is driven by eviction feedback from the cache
> engine.
> * LRU storage and replacement logic is not duplicated in the adaptive policy.
> * Adaptive admission respects single-tier and tiered cache topology semantics.
> * The existing AdaptiveLRU configuration continues to work through the legacy
> {{BlockCacheFactory}} path.
> * No existing {{BlockCache}} consumer is forced to migrate as part of this
> ticket.
> * Tests verify both eviction-feedback propagation and AdaptiveLRU-specific
> behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
