Ichsan Said created PHOENIX-8001:
------------------------------------
Summary: Support specifying a subset of composite key columns for
SALT_BUCKETS hash computation
Key: PHOENIX-8001
URL: https://issues.apache.org/jira/browse/PHOENIX-8001
Project: Phoenix
Issue Type: New Feature
Components: core
Affects Versions: 5.4.0
Reporter: Ichsan Said
h2. Background: The Salt Bucket Trade-off
Salt buckets in Apache Phoenix transparently prepend a 1-byte hash to every
HBase
row key, distributing writes across N pre-split regions to prevent write
hotspots.
The hash is computed from the entire encoded composite primary key.
This design makes an implicit assumption: that writes should be distributed as
randomly as possible across all buckets. While this is optimal for pure write
throughput, it forces a fundamental trade-off that users currently cannot
control.
h2. The Three Access Pattern Archetypes
Consider a table with PK (tenant_id, user_id, event_ts) and SALT_BUCKETS = 16.
*Archetype 1: Write-heavy, point-lookup reads (current behavior is optimal)*
Use case: High-volume event ingestion, reads are always by full PK.
With current salting: writes spread across 16 buckets, point lookups compute
the exact salt byte. No problem here.
*Archetype 2: Write-heavy, range-scan reads on salt columns prefix (current
behavior is suboptimal)*
Use case: IoT / time-series ingestion combined with queries like
"show me all events for user X in the last hour".
With current salting: writes spread across 16 buckets, but range scans on
(tenant_id, user_id) must fan out to all 16 buckets. Read amplification = 16x.
With SALT_BUCKETS_COLUMNS = 'tenant_id,user_id': writes still spread across
buckets by salt columns value, and range scans on those salt columns hit
exactly 1 bucket.
*Archetype 3: Moderate write, heavily skewed read on salt columns (new
capability)*
Use case: Multi-tenant SaaS where one salt columns value drives 80% of traffic.
With current salting: writes spread, but all reads for the dominant salt
columns
value still fan out to all buckets. With SALT_BUCKETS_COLUMNS = 'tenant_id':
reads for any salt columns value hit exactly 1 bucket, and load is spread
across
all buckets by salt columns value.
h2. Flexible Bucket Isolation
Beyond access pattern optimization, SALT_BUCKETS_COLUMNS enables a new class
of operational capability: deterministic data placement by salt columns value.
With the current implementation, the bucket a salt columns value lands in is
opaque.
It is the result of hashing the entire composite key, which means operators
cannot
predict or reason about where data for a specific salt columns value resides
without
computing the hash manually.
When SALT_BUCKETS_COLUMNS is specified, the bucket assignment becomes
deterministic
and transparent. Given a known salt columns value, the destination bucket is
predictable at schema design time. This unlocks several capabilities that are
not possible today:
*Data isolation per salt columns value*
All data sharing the same salt columns value is guaranteed to reside in the
same
bucket, and therefore the same HBase region (at least initially, before region
splits).
This makes per-salt-columns-value operations feasible at the infrastructure
level:
selective cache eviction, backup and restore, compaction scheduling, and
targeted
bulk operations without touching unrelated data.
*Hotspot isolation*
When a small number of salt columns values drive disproportionate traffic, the
operator can identify exactly which bucket is hot and apply targeted
remediation
such as region server rebalancing or dedicated region server assignment.
Because
bucket assignment is deterministic based on the salt columns value, tracing a
hot
bucket back to the responsible salt columns value is a simple hash computation.
With full-key hashing, a single salt columns value is spread across many
buckets
depending on the other PK columns, so there is no direct mapping between a hot
bucket and the salt columns value causing it.
*Compliance and security isolation*
In regulated environments where data from different salt columns value classes
must not share the same physical storage boundary, deterministic bucket
assignment
provides a foundation for enforcing such guarantees at the schema level. While
HBase does not natively enforce security at the region boundary, deterministic
placement makes it tractable to build operational policies around it.
It is worth noting that this flexibility comes with a responsibility: if the
chosen
salt columns have very low cardinality relative to SALT_BUCKETS, some buckets
may
receive significantly more data than others. For example, using
SALT_BUCKETS_COLUMNS = 'tenant_id' with only 3 distinct salt columns values and
SALT_BUCKETS = 16 means 13 buckets will be empty. Users must size their salt
columns
cardinality against their bucket count to achieve the desired distribution
properties.
h2. Root Cause
The current implementation in SaltingUtil.getSaltingByte() always hashes
buf[1..size-1], the entire encoded PK, with no mechanism to specify
which portion of the key should drive bucket assignment. This prevents users
from making an informed trade-off between:
* Write distribution uniformity
* Scan locality for prefix-based queries
* Data co-location for salt columns value workloads
* Region-level data isolation by salt columns value
h2. Current Workarounds and Their Limitations
Users currently work around this limitation through:
1. *Manual key design* reordering PK columns to put the desired salt columns
first, then relying on non-salted distribution. This loses the ability to
use SALT_BUCKETS entirely and requires application-level key management.
2. *Accepting fan-out scans* building application logic that issues N parallel
scans (one per bucket) and merges results. This pushes infrastructure
concerns
into application code and increases client-side complexity.
3. *Secondary indexes* creating an index on the salt columns to avoid fan-out.
This doubles write load and storage, and introduces index maintenance
overhead.
None of these approaches are satisfactory for production workloads at scale.
h2. Proposed Solution
Add SALT_BUCKETS_COLUMNS as an optional CREATE TABLE property. When specified,
only the listed PK columns are used as input to the salt byte hash. When
omitted,
behavior is identical to current (full row key hash), fully backward compatible.
{code:sql}
-- hash(tenant_id, user_id, event_ts): maximum write spread,
-- but range scans on salt columns must touch all 16 buckets
CREATE TABLE events (...) SALT_BUCKETS = 16;
-- hash(tenant_id, user_id): co-locate by salt columns value,
-- range scans on salt columns touch exactly 1 bucket
CREATE TABLE events (...) SALT_BUCKETS = 16, SALT_BUCKETS_COLUMNS =
'tenant_id,user_id';
-- hash(tenant_id): co-locate by salt columns value,
-- all data for a salt columns value is in 1 bucket
CREATE TABLE events (...) SALT_BUCKETS = 16, SALT_BUCKETS_COLUMNS = 'tenant_id';
{code}
h2. Backward Compatibility
When SALT_BUCKETS_COLUMNS is not specified, SaltingUtil.getSaltingByte() is
called
with the full row key bytes, identical to current behavior. Existing tables are
unaffected. New tables without SALT_BUCKETS_COLUMNS behave identically to today.
h2. Scale Impact
At production scale (10B+ rows, 100+ nodes, multi-tenant deployments):
* Eliminates up to 16x read amplification for salt columns scoped range scans
* Reduces cross-region RPC overhead for prefix queries proportional to
SALT_BUCKETS
* Enables data isolation by salt columns value without schema redesign
* Improves cache hit rates on RegionServer block cache for salt columns value
workloads
* Allows capacity planners to predict and control data distribution by salt
columns value
h2. Guide Posts: Open Question for Committer Review
Based on our initial reading of the code, guide posts appear to store actual
HBase
row keys including the already-computed salt byte, and BaseResultIterators
reads
those stored bytes directly without re-hashing. If this understanding is
correct,
changing which columns drive the hash at write time would not affect statistics
collection, guide post storage, or parallel scan boundary detection.
However, we recognize that the full interaction between the salt bucket
algorithm
and guide post mechanics, particularly around synthetic bucket boundary
insertion
in ScanRanges.intersectScan() and the splitPostfix computation in
getParallelScans(),
may have subtleties that are not apparent from reading the code alone. We would
appreciate committer review of whether SALT_BUCKETS_COLUMNS introduces any edge
cases in guide post-based scan planning that we may have missed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
