mxdzs0612 opened a new pull request, #10807:
URL: https://github.com/apache/paimon/pull/10807
### Purpose
Debug logging of every written key value in `KeyValueDataFileWriter#write`
crashes when the row contains complex fields (ARRAY / MAP / MULTISET / ROW).
The values of such fields are
internal data structures like `ColumnarRow` or `ColumnarMap`, and
`Object#toString()` builds the identity string via `hashCode()`, which is not
supported for columnar rows:
```
java.lang.UnsupportedOperationException: ColumnarRowData do not support
hashCode, please hash fields one by one!
at
org.apache.paimon.data.columnar.ColumnarRow.hashCode(ColumnarRow.java:258)
at java.lang.Object.toString(Object.java:236)
at org.apache.paimon.KeyValue.rowDataToString(KeyValue.java:226)
at
org.apache.paimon.io.KeyValueDataFileWriter.write(KeyValueDataFileWriter.java:123)
```
It always reproduces when the log level is DEBUG and the table has complex
fields: the exception is thrown from a compaction task, `prepareCommit` fails
and then the checkpoints fail.
This PR fixes the problem in two layers:
1. `[common]` `ColumnarRow#toString()` no longer delegates to the
unsupported `hashCode()`, mirroring the existing `ColumnarMap#toString()`. Any
string conversion of a columnar row
becomes safe.
2. `[core]` `KeyValue#rowDataToString()` now converts values according to
the field types and recurses into complex values (arrays as `[...]`, maps and
multisets as `{k=v}`, rows as
`(...)`, binary values as byte lists) instead of calling
`Object#toString()`. The output of scalar fields is unchanged. The debug call
in `KeyValueDataFileWriter` also uses SLF4J placeholders now.
### Tests
- `ColumnarRowTest#testToStringDoesNotUseHashCode`: converting a columnar
row to a string must not throw.
- `KeyValueTest#testRowDataToStringScalars`,
`KeyValueTest#testRowDataToStringComplexTypes`: scalar output is unchanged,
complex and nested fields are rendered field by field, and binary
rows format identically to generic rows.
- `KeyValueTest#testRowDataToStringNestedColumnarRow` is the regression
case: without this patch it throws exactly the `UnsupportedOperationException`
shown above, because nested rows
read from columnar formats are `ColumnarRow`s.
- `KeyValueTest#testKeyValueToString` covers the full `KeyValue` debug
format.
All new tests fail without the patch and pass with it. `ColumnarRowTest`,
`KeyValueTest`, `KeyValueSerializerTest` and `MergeSorterTest` pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
