Andrew Kyle Purtell created PHOENIX-8043:
--------------------------------------------
Summary: Global index scans exceeding the pread threshold can
degrade into a cascade of small RPCs
Key: PHOENIX-8043
URL: https://issues.apache.org/jira/browse/PHOENIX-8043
Project: Phoenix
Issue Type: Bug
Components: core
Affects Versions: 5.3.2
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
Fix For: 5.4.0, 5.3.3
Global index scans that exceed the HBase pread byte threshold
({{{}hbase.storescanner.pread.max.bytes{}}}) can degrade into single row RPC
cascades, multiplying network round trips from a handful of RPCs into thousands
of unnecessary RPCs. This bug affects covered index scans with client side
limits, residual WHERE clauses on included columns, long scan tails following
read repair operations, and uncovered index scans with row limits.
HBase manages scan read modes through its {{Shipper}} interface. When
{{RSRpcServices}} processes a scan request, it invokes {{{}preScannerOpen{}}},
instantiates the core {{{}RegionScannerImpl{}}}, and retains that instance as
the sole RPC {{{}Shipper{}}}. Store scanners start in pread mode under
{{{}ReadType.DEFAULT{}}}. Once a store scanner reads beyond the configured
pread byte limit, it signals the scan context to return immediately
({{{}returnImmediately = true{}}}) at cell boundaries. This early termination
is intentional, for yielding execution back to the RPC layer so that
{{RSRpcServices}} can call {{{}shipped(){}}}, which in turn triggers
{{trySwitchToStreamRead()}} to transition subsequent reads to streaming I/O.
Phoenix breaks this contract by replacing the underlying scanner after HBase
has already bound the RPC shipper. During the initial {{next()}} call,
{{GlobalIndexScanner.init()}} evaluates scan constraints. If a client
{{PageFilter}} is present, it strips the filter to enforce the row limit
internally. If residual filters exist, it wraps them in an
{{{}UnverifiedRowFilter{}}}. In both scenarios as well as during read-repair
rewinds it closes the core scanner opened by HBase and instantiates a private
replacement scanner. The {{UncoveredGlobalIndexRegionScanner}} constructor
performs an identical substitution, stripping {{PageFilter}} so that unverified
index rows do not prematurely consume the client limit.
Because HBase retains a reference only to the discarded initial scanner,
{{shipped()}} is never dispatched to the replacement scanners or their
underlying store scanners. The replacement scanners remain permanently locked
in pread mode. Once the pread byte threshold is crossed, every evaluated cell
asserts {{{}returnImmediately{}}}, propagating through
{{PhoenixScannerContext}} to the RPC engine and terminating the RPC after
yielding a single row.
The appropriate, imho, architectural fix here is to shift scan rewrites into
{{preScannerOpen}} hooks within {{GlobalIndexChecker}} and
{{{}BaseScannerRegionObserver{}}}. Phoenix can rewrite the scanner hierarchy
before HBase instantiates {{{}RegionScannerImpl{}}}, and so when the scanner is
created it is already properly configured and does not need to be closed and
replaced. Rebuilding scanner chains mid-flight requires careful partitioning of
filter state.
Also, scanners opened during read repairs cannot rely on HBase's outer
{{shipped()}} hook. For these instances, {{GlobalIndexChecker}} manages its own
lazy pread to streaming transitions.
After resolving the above described performance issue, several concurrency
issues related to metrics emerged. The fixes are trivial and obvious. The task
metrics container ({{{}dataTableScanMetrics{}}}), an unsynchronized
{{{}ArrayList{}}}, should be replaced with a mt-safe structure like
{{{}ConcurrentLinkedQueue{}}}. Related state flags should be made volatile to
guarantee cross-thread visibility across concurrent data retrieval workers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
