JeonDaehong opened a new issue, #1482:
URL: https://github.com/apache/paimon-rust/issues/1482

   Compacting a primary-key table with about 64 BIGINT value columns fails 
under default options with
   
   ```
   ResourceExhausted { message: "Decoded Lookup IPC exceeds the memory budget" }
   ```
   
   both with `changelog-producer = lookup` and with deletion vectors. The same 
table with 48 columns compacts fine.
   
   After decoding a payload, `LookupFileCache::read` checks 
`batch.get_array_memory_size()` against the budget. The decoded columns are all 
views into the one IPC body, and `get_array_memory_size` counts the whole 
backing buffer for every column, so the body is charged once per column. For a 
1,024-row payload with 69 columns the body is 575 KB, while the check sees 39.5 
MB, over the 32 MB limit (`lookup.cache-max-memory-size` / 8).
   
   Repro: `bucket = 1`, `id` primary key plus 66 BIGINT columns, write 500k 
keys and compact, then overwrite a third of the keys and compact again. The 
second compaction fails.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to