Fokko commented on issue #3260:
URL: 
https://github.com/apache/iceberg-python/issues/3260#issuecomment-6834850767

   I was just reviewing #4342. The problem there is that it will always convert 
from dictionary to plain encoding. Dictionary encoding is there for a reason, 
to reduce memory. My LLM friend did the following benchmark:
   
   ```
   
┌────────────────────────────────────────────────────────────────┬───────────────────────┬──────────────────────────┐
   │                            Scenario                            │         
main          │            PR            │
   
├────────────────────────────────────────────────────────────────┼───────────────────────┼──────────────────────────┤
   │ Wide plain table: 300 cols, 200 files × 1k rows                │ 1017 ms   
            │ 1008 ms (no difference)  │
   
├────────────────────────────────────────────────────────────────┼───────────────────────┼──────────────────────────┤
   │ Plain strings: 20 files × 1M rows                              │ 17.2 ms / 
400 MB      │ 17.0 ms / 400 MB         │
   
├────────────────────────────────────────────────────────────────┼───────────────────────┼──────────────────────────┤
   │ All-dictionary strings: 20 files × 1M rows, 10 distinct values │ 11.3 ms / 
80 MB, dict │ 61.8 ms / 402 MB, string │
   
├────────────────────────────────────────────────────────────────┼───────────────────────┼──────────────────────────┤
   │ Same table, to_arrow(dictionary_columns=("s",))                │ —         
            │ 19.4 ms / 80 MB, dict    │
   
└────────────────────────────────────────────────────────────────┴───────────────────────┴──────────────────────────┘
   ```
   
   The `dictionary_columns` is also a bit of a workaround already. Did we check 
if there is anything upstream in Arrow that we can fix? That would allow 
concatenating different encodings? In the end, it is just an implemention 
detail. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to