MehulBatra opened a new issue, #11980:
URL: https://github.com/apache/gravitino/issues/11980
### Version
main branch
### Describe what's wrong
Version: 1.3.0 (also present in 1.2.0 and current main)
Component: core / relational entity store, generic (lakehouse-generic)
catalog
Describe the bug:
When a table managed by the relational entity store is loaded, its columns
can come back in an order that does not match the order they were declared in.
The order is also unstable: it can differ between loads (for example after a
cache reload), because the backing store returns columns in an arbitrary order.
The column ordinal is stored correctly (the column_position column in
table_column_version_info, "starting from 0").
The problem is only on the read path:
ManagedTableOperations.toGenericTable() builds the column array directly
from tableEntity.columns() without sorting by position.
Expected behavior:
loadTable / describe should always return columns in their declared position
order, regardless of the order the store returns rows in.
Proposed fix:
Sort by position in toGenericTable():
```
.withColumns(
tableEntity.columns().stream()
.sorted(Comparator.comparingInt(ColumnEntity::position))
.map(this::toGenericColumn)
.toArray(Column[]::new))
```
column_position is already persisted correctly, so this fixes existing
tables on read with no migration or re-registration. Comparator and
ColumnEntity are already imported in the file.
### Error message and/or stacktrace
Root cause:
core/src/main/java/org/apache/gravitino/catalog/ManagedTableOperations.java,
method toGenericTable():
`.withColumns(tableEntity.columns().stream().map(this::toGenericColumn).toArray(Column[]::new))
`
There is no sort by position. The alter path in the same class already
accounts for this and sorts
`(.sorted(Comparator.comparingInt(ColumnEntity::position)))` ,with the comment
"columns maybe unordered when retrieved from the store". The load path was
never given the same sort.
bash
/eb8a58db-74a0-4753-9420-c38bfdc4ca1a/scratchpad/lance_positional_bug_repro.sh
```
== 1. CATALOG order (Lance Rest Catalog DescribeTable; creds filtered out) ==
catalog order : ['image_uri', 'embedding', 'product_id', 'caption']
external flag : true
== 2. PHYSICAL order (pylance reads the .lance dataset directly) ==
physical order: ['product_id', 'image_uri', 'embedding', 'caption']
== 3. THE CRASH: attached-catalog SELECT * (positional read) ==
select * from catalog.schema.table
-- Loading resources from /Users/mbatra/.duckdbrc
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
Invalid Input Error:
Vector::Reference used on vector of different type (source FLOAT[512]
referenced VARCHAR)
Stack Trace:
/opt/ext/lance.duckdb_extension(+0x135c6ed) [0x7f7ecc3b06ed]
/opt/ext/lance.duckdb_extension(+0x135c758) [0x7f7ecc3b0758]
/opt/ext/lance.duckdb_extension(+0x135eb75) [0x7f7ecc3b2b75]
/opt/ext/lance.duckdb_extension(+0x126ba27) [0x7f7ecc2bfa27]
/opt/ext/lance.duckdb_extension(+0x90bc1a) [0x7f7ecb95fc1a]
/opt/ext/lance.duckdb_extension(+0x127f97c) [0x7f7ecc2d397c]
/opt/ext/lance.duckdb_extension(+0xc2fc94) [0x7f7ecbc83c94]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0x13feb6f)
[0x7f7e4b798b6f]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb45a2b)
[0x7f7e4aedfa2b]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb4ef6d)
[0x7f7e4aee8f6d]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb4f342)
[0x7f7e4aee9342]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb47266)
[0x7f7e4aee1266]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb514e4)
[0x7f7e4aeeb4e4]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb35470)
[0x7f7e4aecf470]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb3559c)
[0x7f7e4aecf59c]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb356fc)
[0x7f7e4aecf6fc]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb37b91)
[0x7f7e4aed1b91]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0xb3817c)
[0x7f7e4aed217c]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZN6duckdb11QuackServer21HandleMessageInternalERNS_16DatabaseInstanceERNS_12QuackMessageENS_12optional_ptrINS_15QuackConnectionELb1EEE+0x376)
[0x7f7e4a73f706]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZN6duckdb11QuackServer13HandleMessageERNS_12MemoryStreamE+0xbe7)
[0x7f7e4a741ee7]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(+0x39302a)
[0x7f7e4a72d02a]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZNK14duckdb_httplib6Server35dispatch_request_for_content_readerERNS_7RequestERNS_8ResponseENS_13ContentReaderERKSt6vectorISt4pairIN10duckdb_re25RegexESt8functionIFvRKS1_S4_RKS5_EEESaISH_EE+0xb5)
[0x7f7e4a706375]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZN14duckdb_httplib6Server7routingERNS_7RequestERNS_8ResponseERNS_6StreamE+0x403)
[0x7f7e4a71a263]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZN14duckdb_httplib6Server15process_requestERNS_6StreamERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiSA_ibRbRKSt8functionIFvRNS_7RequestEEE+0x144e)
[0x7f7e4a71bbfe]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZN14duckdb_httplib6Server24process_and_close_socketEi+0x4e0)
[0x7f7e4a71d3d0]
/root/.duckdb/extensions/v1.5.4/linux_amd64/quack.duckdb_extension(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJN14duckdb_httplib10ThreadPool6workerEEEEEE6_M_runEv+0x14e)
[0x7f7e4a72920e]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xe1224) [0x7f7ee6c9e224]
/lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x7f7eeadf6b7b]
/lib/x86_64-linux-gnu/libc.so.6(+0x1107f8) [0x7f7eeae747f8]
```
### How to reproduce
To reproduce:
1. Create a table with several columns through a generic/managed lance
rest catalog backed by the relational (JDBC/Postgres) entity store.
2. Load or describe the table (ideally after the entity cache has
reloaded, so the store re-reads rows).
3. Observe that the returned column order does not match the declared
order, and can vary between loads.
A deterministic unit-test reproduction: store a TableEntity whose columns()
list order differs from the columns' position values, call loadTable, and
assert the returned columns are in position order. Without the fix the
assertion fails.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
