pepijnve commented on issue #18411: URL: https://github.com/apache/datafusion/issues/18411#issuecomment-3497666584
Is it currently possible to dynamically switch between `Utf8` and `Utf8View`? I was thinking that for the very short strings like the ones in `l_returnflag` and `l_linestatus` it's actually counterproductive to use `Utf8View`. Those short string will all get inlined and end up using 16 bytes per entry instead of the ~5 (offset + 1 data byte) you would use per entry in `Utf8`. Additionally there's the conditional for each entry to check if it's inline or not. The only problem I see is that it's hard to know up front which of the two you options (view or not) to choose based on the parquet metadata. On a per row group basis you can decide this since the dictionary is entirely available in one page so you can figure out the maximum string length for the row group. But if that choice needs to be made for the entire file it's not so simple. Here too `ENUM` could be a possible heuristic here, but I'm not sure how often that actually gets used in practice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
