L-Gryps opened a new pull request, #8804:
URL: https://github.com/apache/paimon/pull/8804
Closes #8803
### Purpose
When a primary-key table has `'table-read.sequence-number.enabled' =
'true'`, querying its `$audit_log` (or `$binlog`) system table and **then**
querying the base table causes:
- Column data to be shifted by one position (the internal
`_SEQUENCE_NUMBER` value leaks into the first column, all real columns shift
right), or
- A hard `ClassCastException` when a base column is non-numeric, e.g.
`HeapLongVector cannot be cast to BytesColumnVector`, surfaced by Flink as
`Recovery is suppressed by NoRestartBackoffTimeStrategy`.
**Root cause**
`AuditLogTable`'s constructor mutates the wrapped `FileStoreTable`'s
options map in place when sequence-number reading is enabled:
```java
this.wrapped.options().put(CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(),
"true");
FileStoreTable.options() returns schema().options(), which is the same
Map<String, String> reference held by the catalog-cached FileStoreTable. After
the first SELECT * FROM t$audit_log, the cached base table permanently
carries key-value.sequence_number.enabled=true. Subsequent reads of the
base table go through KeyValueTableRead.unwrap, which then makes
ValueContentRowDataRecordIterator prepend a long _SEQUENCE_NUMBER column to
every row.
The physical row now has one extra column ahead of the logical row type,
causing column mis-alignment or vector type mismatch.
Fix
Instead of mutating the shared options map, create a private copy of the
wrapped FileStoreTable that carries the internal
KEY_VALUE_SEQUENCE_NUMBER_ENABLED=true option only for the audit-log's own
reads. The catalog-cached
base table is left untouched.
wrapped = wrapped.copyWithoutTimeTravel(
Collections.singletonMap(
CoreOptions.KEY_VALUE_SEQUENCE_NUMBER_ENABLED.key(),
"true"));
### Test
Added regression coverage at two layers:
-
paimon-core/src/test/java/org/apache/paimon/table/system/AuditLogTableTest.java
— testReadBaseTableAfterAuditLogDoesNotIncludeSequenceNumber: constructs an
AuditLogTable and asserts the wrapped base table's row type /
options are not polluted with _SEQUENCE_NUMBER /
KEY_VALUE_SEQUENCE_NUMBER_ENABLED.
-
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/BatchFileStoreITCase.java
— testReadBaseTableAfterAuditLogWithSequenceNumberEnabled: end-to-end
reproduction of the reported user scenario (query
t$audit_log, then re-query t); base-table result must remain unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
