[
https://issues.apache.org/jira/browse/PHOENIX-8000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Roy updated PHOENIX-8000:
---------------------------------
Description:
Description
Summary
A SYSTEM table configured with a conditional TTL never has that TTL applied
during compaction. Expired rows are hidden at read time (masking) but are never
physically removed from the HFiles, so they accumulate indefinitely.
SYSTEM.CDC_STREAM — which is created with a conditional TTL to expire
closed-partition rows after a minimum age — is the concrete instance where this
shows up.
Root cause
In CompactionScanner.NonPartitionedTableTTLTracker, SYSTEM tables are
unconditionally routed to the HBase column-family-descriptor TTL instead of the
table's compiled conditional TTL expression:
```
boolean isSystemTable = pTable.getType() == PTableType.SYSTEM;
...
if (isSystemTable
||
pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
ColumnFamilyDescriptor cfd = store.getColumnFamilyDescriptor();
ttlExpr = TTLExpressionFactory.create(cfd.getTimeToLive()); // descriptor TTL,
NOT the conditional expr
} else {
ttlExpr = !pTable.getTTLExpression().equals(TTL_EXPRESSION_NOT_DEFINED)
? pTable.getCompiledTTLExpression(pConn) // conditional path — skipped for
SYSTEM
: TTL_EXPRESSION_FOREVER;
}
```
For a table whose TTL is a conditional expression, the CF-descriptor TTL is
FOREVER (Phoenix stores a numeric TTL in the descriptor only for literal
expressions; conditional TTLs are stored in SYSTEM.CATALOG and the descriptor
is left FOREVER). Consequently, for a SYSTEM table:
- ttlExprForRow resolves to a literal FOREVER, so
RowContext.hasConditionalTTL() is always false;
- therefore postProcessForConditionalTTL(...) — the only place the conditional
expression is evaluated and an expired row is dropped (result.clear()) — is
never reached;
- so expiry is enforced only as read-time masking (TTLRegionScanner), never at
compaction.
Minor compaction / flush also never purge (they use TTL_EXPRESSION_FOREVER by
design), so major compaction is the only place this should happen — and it
doesn't for SYSTEM tables.
Steps to reproduce
1. Create/use a SYSTEM table with a conditional TTL (e.g. SYSTEM.CDC_STREAM,
whose TTL is PARTITION_END_TIME IS NOT NULL AND TO_NUMBER(CURRENT_TIME()) -
TO_NUMBER(PHOENIX_ROW_TIMESTAMP()) >= <min-age>, default
PHOENIX_CDC_STREAM_PARTITION_EXPIRY_MIN_AGE_MS ≈ 30h).
2. Insert rows that satisfy the expiry condition and advance the clock past the
min age.
3. Trigger a major compaction on the table.
4. Observed: the rows are still physically present in the HFiles (a raw/setRaw
scan still returns them); they are only hidden from normal Phoenix reads.
5. Expected: the expired rows are physically removed by the major compaction.
Expected behavior
SYSTEM tables should honor their conditional TTL at (major) compaction —
evaluating the compiled conditional expression and physically purging matching
rows — the same way non-SYSTEM tables do. The isSystemTable short-circuit
should not force a conditional-TTL table onto the (FOREVER) descriptor TTL.
Proposed fix
In NonPartitionedTableTTLTracker, do not route a SYSTEM table to the descriptor
TTL when it has a conditional TTL expression; use the table's compiled
conditional TTL (pTable.getCompiledTTLExpression(...)) so hasConditionalTTL()
becomes true and postProcessForConditionalTTL(...) runs at major compaction.
(Keep the descriptor-TTL path only for tables that genuinely define TTL in the
descriptor / literal TTLs.)
Test
Add an integration test that:
- uses a SYSTEM table with a conditional TTL (SYSTEM.CDC_STREAM),
- writes rows satisfying the condition and ages them past the configured
min-age (~30h),
- runs a major compaction, and
- asserts the expired rows are physically removed (e.g. a raw scan /
all-versions scan no longer returns them), not merely masked.
Impact
Any SYSTEM table with a conditional TTL retains expired rows on disk forever,
causing unbounded growth of the system table and defeating the intent of the
conditional TTL (rows are hidden but never reclaimed).
was:
Summary
A SYSTEM table configured with a conditional TTL never has that TTL applied
during compaction. Expired rows are hidden at read time (masking) but are
never physically removed from the HFiles, so they accumulate indefinitely.
SYSTEM.CDC_STREAM — which is created with a conditional TTL to expire
closed-partition rows after a minimum age — is the concrete instance where this
shows up.
Root cause
In CompactionScanner.NonPartitionedTableTTLTracker, SYSTEM tables are
unconditionally routed to the HBase column-family-descriptor TTL instead of the
table's compiled conditional TTL expression:
boolean isSystemTable = pTable.getType() == PTableType.SYSTEM;
...
if (isSystemTable
||
pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR)) {
ColumnFamilyDescriptor cfd = store.getColumnFamilyDescriptor();
ttlExpr = TTLExpressionFactory.create(cfd.getTimeToLive()); // descriptor TTL,
NOT the conditional expr
} else {
ttlExpr = !pTable.getTTLExpression().equals(TTL_EXPRESSION_NOT_DEFINED)
? pTable.getCompiledTTLExpression(pConn) // conditional path — skipped for
SYSTEM
: TTL_EXPRESSION_FOREVER;
}
For a table whose TTL is a conditional expression, the CF-descriptor TTL is
FOREVER (Phoenix stores a numeric TTL in the descriptor only for literal
expressions; conditional TTLs are stored in SYSTEM.CATALOG and the descriptor
is left FOREVER). Consequently, for a SYSTEM table:
- ttlExprForRow resolves to a literal FOREVER, so
RowContext.hasConditionalTTL() is always false;
- therefore postProcessForConditionalTTL(...) — the only place the conditional
expression is evaluated and an expired row is dropped (result.clear()) — is
never reached;
- so expiry is enforced only as read-time masking (TTLRegionScanner), never at
compaction.
Minor compaction / flush also never purge (they use TTL_EXPRESSION_FOREVER by
design), so major compaction is the only place this should happen — and it
doesn't for SYSTEM tables.
Steps to reproduce
1. Create/use a SYSTEM table with a conditional TTL (e.g. SYSTEM.CDC_STREAM,
whose TTL is PARTITION_END_TIME IS NOT NULL AND TO_NUMBER(CURRENT_TIME()) -
TO_NUMBER(PHOENIX_ROW_TIMESTAMP()) >= <min-age>, default
PHOENIX_CDC_STREAM_PARTITION_EXPIRY_MIN_AGE_MS ≈ 30h).
2. Insert rows that satisfy the expiry condition and advance the clock past the
min age.
3. Trigger a major compaction on the table.
4. Observed: the rows are still physically present in the HFiles (a raw/setRaw
scan still returns them); they are only hidden from normal Phoenix reads.
5. Expected: the expired rows are physically removed by the major compaction.
Expected behavior
SYSTEM tables should honor their conditional TTL at (major) compaction —
evaluating the compiled conditional expression and physically purging matching
rows — the same way non-SYSTEM tables do. The isSystemTable short-circuit
should not force a conditional-TTL table onto the (FOREVER) descriptor TTL.
Proposed fix
In NonPartitionedTableTTLTracker, do not route a SYSTEM table to the descriptor
TTL when it has a conditional TTL expression; use the table's compiled
conditional TTL (pTable.getCompiledTTLExpression(...)) so hasConditionalTTL()
becomes true and postProcessForConditionalTTL(...) runs at major compaction.
(Keep the descriptor-TTL path only for tables that genuinely define TTL in the
descriptor / literal TTLs.)
Test
Add an integration test that:
- uses a SYSTEM table with a conditional TTL (SYSTEM.CDC_STREAM),
- writes rows satisfying the condition and ages them past the configured
min-age (~30h),
- runs a major compaction, and
- asserts the expired rows are physically removed (e.g. a raw scan /
all-versions scan no longer returns them), not merely masked.
Impact
Any SYSTEM table with a conditional TTL retains expired rows on disk forever,
causing unbounded growth of the system table and defeating the intent of the
conditional TTL (rows are hidden but never reclaimed).
> Conditional TTL is not enforced at compaction for SYSTEM tables — expired
> rows are never physically purged (e.g. SYSTEM.CDC_STREAM)
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-8000
> URL: https://issues.apache.org/jira/browse/PHOENIX-8000
> Project: Phoenix
> Issue Type: Bug
> Components: phoenix
> Affects Versions: 5.3.0, 5.3.1, 5.3.2
> Reporter: Shubham Roy
> Assignee: Shubham Roy
> Priority: Minor
> Fix For: 5.3.3
>
>
> Description
> Summary
> A SYSTEM table configured with a conditional TTL never has that TTL applied
> during compaction. Expired rows are hidden at read time (masking) but are
> never physically removed from the HFiles, so they accumulate indefinitely.
> SYSTEM.CDC_STREAM — which is created with a conditional TTL to expire
> closed-partition rows after a minimum age — is the concrete instance where
> this shows up.
> Root cause
> In CompactionScanner.NonPartitionedTableTTLTracker, SYSTEM tables are
> unconditionally routed to the HBase column-family-descriptor TTL instead of
> the table's compiled conditional TTL expression:
> ```
> boolean isSystemTable = pTable.getType() == PTableType.SYSTEM;
> ...
> if (isSystemTable
> ||
> pTable.getTTLExpression().equals(TTL_EXPRESSION_DEFINED_IN_TABLE_DESCRIPTOR))
> {
> ColumnFamilyDescriptor cfd = store.getColumnFamilyDescriptor();
> ttlExpr = TTLExpressionFactory.create(cfd.getTimeToLive()); // descriptor
> TTL, NOT the conditional expr
> } else {
> ttlExpr = !pTable.getTTLExpression().equals(TTL_EXPRESSION_NOT_DEFINED)
> ? pTable.getCompiledTTLExpression(pConn) // conditional path — skipped for
> SYSTEM
> : TTL_EXPRESSION_FOREVER;
> }
> ```
> For a table whose TTL is a conditional expression, the CF-descriptor TTL is
> FOREVER (Phoenix stores a numeric TTL in the descriptor only for literal
> expressions; conditional TTLs are stored in SYSTEM.CATALOG and the descriptor
> is left FOREVER). Consequently, for a SYSTEM table:
> - ttlExprForRow resolves to a literal FOREVER, so
> RowContext.hasConditionalTTL() is always false;
> - therefore postProcessForConditionalTTL(...) — the only place the
> conditional expression is evaluated and an expired row is dropped
> (result.clear()) — is never reached;
> - so expiry is enforced only as read-time masking (TTLRegionScanner), never
> at compaction.
> Minor compaction / flush also never purge (they use TTL_EXPRESSION_FOREVER by
> design), so major compaction is the only place this should happen — and it
> doesn't for SYSTEM tables.
> Steps to reproduce
> 1. Create/use a SYSTEM table with a conditional TTL (e.g. SYSTEM.CDC_STREAM,
> whose TTL is PARTITION_END_TIME IS NOT NULL AND TO_NUMBER(CURRENT_TIME()) -
> TO_NUMBER(PHOENIX_ROW_TIMESTAMP()) >= <min-age>, default
> PHOENIX_CDC_STREAM_PARTITION_EXPIRY_MIN_AGE_MS ≈ 30h).
> 2. Insert rows that satisfy the expiry condition and advance the clock past
> the min age.
> 3. Trigger a major compaction on the table.
> 4. Observed: the rows are still physically present in the HFiles (a
> raw/setRaw scan still returns them); they are only hidden from normal Phoenix
> reads.
> 5. Expected: the expired rows are physically removed by the major compaction.
> Expected behavior
> SYSTEM tables should honor their conditional TTL at (major) compaction —
> evaluating the compiled conditional expression and physically purging
> matching rows — the same way non-SYSTEM tables do. The isSystemTable
> short-circuit should not force a conditional-TTL table onto the (FOREVER)
> descriptor TTL.
> Proposed fix
> In NonPartitionedTableTTLTracker, do not route a SYSTEM table to the
> descriptor TTL when it has a conditional TTL expression; use the table's
> compiled conditional TTL (pTable.getCompiledTTLExpression(...)) so
> hasConditionalTTL() becomes true and postProcessForConditionalTTL(...) runs
> at major compaction.
> (Keep the descriptor-TTL path only for tables that genuinely define TTL in
> the descriptor / literal TTLs.)
> Test
> Add an integration test that:
> - uses a SYSTEM table with a conditional TTL (SYSTEM.CDC_STREAM),
> - writes rows satisfying the condition and ages them past the configured
> min-age (~30h),
> - runs a major compaction, and
> - asserts the expired rows are physically removed (e.g. a raw scan /
> all-versions scan no longer returns them), not merely masked.
> Impact
> Any SYSTEM table with a conditional TTL retains expired rows on disk forever,
> causing unbounded growth of the system table and defeating the intent of the
> conditional TTL (rows are hidden but never reclaimed).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
