paleolimbot commented on issue #1005: URL: https://github.com/apache/arrow-adbc/issues/1005#issuecomment-1702713064
The problem of int64 -> R vector comes up a lot, so even though it shows up when roundtripping through SQLite it almost certainly will come up elsewhere. The ideal behaviour you described is basically what the arrow package does, because it has the fancy SIMD-dispatching logic that makes computing the value range cheap. I don't think nanoarrow will be able to do that particularly soon (almost entirely because I don't have time to do it). > Do you any ideas for (cheaply) getting information on value ranges (only when needed) before materializing the data as R object? In the near future, you could use `arrow` (e.g., `arrow::as_arrow_table(stream)` + `min()`/`max()` on `table$column`), or you could *always* materialize to `bit64::integer64()` and then do your range checking in R. In a more far off future nanoarrow might be able to extend its scope to deal with this particular issue. As short-term action items I will: - Add `integer64` support to nanoarrow - Ensure that converting double values where `abs(value) > 2^51` incurs a warning -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
