[
https://issues.apache.org/jira/browse/PHOENIX-8046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-8046:
-----------------------------------------
Description:
{{ALTER INDEX ... CONSISTENCY}} writes the new consistency only to the index
header row in {{{}SYSTEM.CATALOG{}}}. The parent table's metadata does not
change, so other clients and the region servers keep a cached parent with the
old index definition. Writers then keep sending index maintainers with the old
consistency. The DDL client's own cache can also lose the index entirely, so
its writes skip index maintenance. The fix writes {{INDEX_CONSISTENCY}} and
{{LAST_DDL_TIMESTAMP}} for both the index and the parent in one commit. It then
clears the server side metadata cache for both tables, and the DDL client
evicts its own cached copies. Every client then gets the change through normal
{{{}LAST_DDL_TIMESTAMP validation{}}}. This does not depend on
{{{}phoenix.metadata.invalidate.cache.enabled{}}}.
When the switch reaches every writer, the {{ALTER}} must also coordinate with
the eventual index consumers, in both directions. For {{STRONG}} to
{{{}EVENTUAL{}}}, the {{ALTER}} now creates the CDC stream and CDC index first,
while the index is still {{{}STRONG{}}}. Only after that does it publish
{{{}EVENTUAL{}}}. So any writer that sees {{EVENTUAL}} already produces CDC
records for its writes. Writes during the transition can get both synchronous
maintenance and a CDC record, which is idempotent. For {{EVENTUAL}} to
{{{}STRONG{}}}, a consumer no longer discards CDC events only because the index
is now {{{}STRONG{}}}. It applies events to the index up to a cutoff: the
index's {{{}LAST_DDL_TIMESTAMP{}}}, plus the larger {{UPDATE_CACHE_FREQUENCY}}
of the data table and the index, plus one minute. The extra minute covers
in-flight batches and clock skew. This lets clients with stale caches, which
still send EVENTUAL maintainers, finish their writes. Replay produces the same
index cells at the same timestamps, so it is idempotent.
A consumer now starts whenever the table has an active CDC index, and it no
longer requires an {{EVENTUAL}} index to exist. So a region created by a split
or merge after the switch still replays the unconsumed events of its parent
regions. Also, when a consumer decides whether it has work to do, it reads the
table definition from {{SYSTEM.CATALOG}} and does not trust the region server's
metadata cache. Before this change, a consumer on a moved or reopened region
could read a stale definition and exit, or treat an {{EVENTUAL}} index as
{{STRONG}} and skip its events.
was:
Modifying an index's consistency model via {{ALTER INDEX ... CONSISTENCY}} can
silently desynchronize client metadata caches from the underlying schema. A
client connection can cache the parent data table without the altered index
attached. Subsequent write paths fail to maintain the index, and query planners
ignore it. Because the index metadata remains marked as {{{}ACTIVE{}}}, direct
reads against it yield missing or stale data.
Updating consistency modifies only the index header row, leaving the parent
table's {{PTable}} and timestamp unchanged. Immediately following the write,
the client invalidates its local cache by invoking
{{PMetaDataImpl.removeTable}} with the parent table's name and
{{{}LATEST_TIMESTAMP{}}}. Rather than dropping the parent entry, the client
cache attempts to rebuild the parent definition in place with the index
stripped out, preserving the original parent timestamp. Because this timestamp
still matches the unmodified server record, subsequent schema resolutions
conclude the client's cached parent is current. The client remains pinned to an
incomplete table definition. Explicitly resolving the index by name could
temporarily reattach it. However, a concurrent lookup of the parent table would
overwrite that repair with the earlier incorrect snapshot.
To guarantee cache coherence, the invalidation model requires two coordinated
changes:
* *Client Cache Invalidation:* When {{PMetaDataImpl.removeTable}} is invoked
with {{{}LATEST_TIMESTAMP{}}}, it must evict the parent table entry entirely.
Forcing an eviction ensures the next schema resolution refetches the complete
parent definition from the server, preventing stale in-memory snapshots from
perpetuating partial states. When an explicit timestamp is supplied, the
existing selective removal behavior is preserved.
* *Server Cache Coherence:* The ALTER path must proactively purge the parent
data table from the server side metadata cache. Because the server caches data
tables alongside their associated index descriptors, evicting the parent
prevents the server from serving stale descriptors to subsequent client lookups.
> ALTER INDEX ... CONSISTENCY coordination fixes
> ----------------------------------------------
>
> Key: PHOENIX-8046
> URL: https://issues.apache.org/jira/browse/PHOENIX-8046
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.3.2
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 5.4.0, 5.3.3
>
>
> {{ALTER INDEX ... CONSISTENCY}} writes the new consistency only to the index
> header row in {{{}SYSTEM.CATALOG{}}}. The parent table's metadata does not
> change, so other clients and the region servers keep a cached parent with the
> old index definition. Writers then keep sending index maintainers with the
> old consistency. The DDL client's own cache can also lose the index entirely,
> so its writes skip index maintenance. The fix writes {{INDEX_CONSISTENCY}}
> and {{LAST_DDL_TIMESTAMP}} for both the index and the parent in one commit.
> It then clears the server side metadata cache for both tables, and the DDL
> client evicts its own cached copies. Every client then gets the change
> through normal {{{}LAST_DDL_TIMESTAMP validation{}}}. This does not depend on
> {{{}phoenix.metadata.invalidate.cache.enabled{}}}.
> When the switch reaches every writer, the {{ALTER}} must also coordinate with
> the eventual index consumers, in both directions. For {{STRONG}} to
> {{{}EVENTUAL{}}}, the {{ALTER}} now creates the CDC stream and CDC index
> first, while the index is still {{{}STRONG{}}}. Only after that does it
> publish {{{}EVENTUAL{}}}. So any writer that sees {{EVENTUAL}} already
> produces CDC records for its writes. Writes during the transition can get
> both synchronous maintenance and a CDC record, which is idempotent. For
> {{EVENTUAL}} to {{{}STRONG{}}}, a consumer no longer discards CDC events only
> because the index is now {{{}STRONG{}}}. It applies events to the index up to
> a cutoff: the index's {{{}LAST_DDL_TIMESTAMP{}}}, plus the larger
> {{UPDATE_CACHE_FREQUENCY}} of the data table and the index, plus one minute.
> The extra minute covers in-flight batches and clock skew. This lets clients
> with stale caches, which still send EVENTUAL maintainers, finish their
> writes. Replay produces the same index cells at the same timestamps, so it is
> idempotent.
> A consumer now starts whenever the table has an active CDC index, and it no
> longer requires an {{EVENTUAL}} index to exist. So a region created by a
> split or merge after the switch still replays the unconsumed events of its
> parent regions. Also, when a consumer decides whether it has work to do, it
> reads the table definition from {{SYSTEM.CATALOG}} and does not trust the
> region server's metadata cache. Before this change, a consumer on a moved or
> reopened region could read a stale definition and exit, or treat an
> {{EVENTUAL}} index as {{STRONG}} and skip its events.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
