Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2992671689
`label_issue.py` automatically added labels {'arrow'} from #7403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb closed issue #7423: Variant: Rust API to Read Variant Values URL: https://github.com/apache/arrow-rs/issues/7423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb closed issue #7423: Variant: Rust API to Read Variant Values URL: https://github.com/apache/arrow-rs/issues/7423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2897586489 BTW I think @mapleFU has made a PR with a proposed C/C++ API for accessing variants which might have some additional inspiration: - https://github.com/apache/arrow/pull/46372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
mkarbo commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2894743866 Yeah I agree, thanks for pointing it out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2894610251 > I personally favor the latter approach (safe and easy to use, even if not always the absolutely max efficient), but the topic probably needs a wider discussion. I likewise think the "safe / return error by default" is the right model. I also agree it should be an API goal that there should be no panics on invalid variant data (it should return an error instead) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2894597621 @scovich had some great comments on https://github.com/apache/arrow-rs/pull/7452#discussion_r2095809708 that I wanted to copy/paste into this ticket perhaps for wider discussion: > I have not reviewed the code carefully at all yet, and what follows is a general observation based on the inherent nature of variant data and rust notions of safety: > > It will be really tempting to have "efficient" code that e.g. uses [from_utf8_unchecked](https://doc.rust-lang.org/std/primitive.str.html#method.from_utf8_unchecked) to extract a `&str` from a `&[u8]`, or to use indexing operations like `v[10]` to extract bytes. But variant data is generally untrusted user input and whatever `Variant` struct/enum we define will become the first -- and often only -- line of defense against malicious or malformed input. > > Hopefully we can code carefully, with the goal that sizes and/or contents of metadata and value slices will never cause a panic? > > Additinoally, it seems like we have a few choices for values such as strings and decimals even a right-sized byte slice can contain invalid values: > > 1. Return obviously unvalidated values, e.g. `&[u8]` instead of `&str` for strings, and `&[u8]` instead of whatever `VariantDecimal` struct we might otherwise define -- leaving the user responsible to finish the conversion as (un)safely as they deem prudent. > 2. Return ostensibly validated values, with (safe) checked and (unsafe) unchecked constructors and/or getters that let the user choose the one they deem appropriate. > > I personally favor the latter approach (safe and easy to use, even if not always the absolutely max efficient), but the topic probably needs a wider discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
mkarbo commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890516466 We'd like to pick this up, can you assign this one to me @alamb? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890606191 BTW there are a few known issues with the example variant values https://github.com/apache/parquet-testing/issues/75 in parquet-testing: Specifically - https://github.com/apache/parquet-testing/issues/81 - https://github.com/apache/parquet-testing/issues/82 I think they will be relatively easy to solve / workaround for the time being, but I wanted to bring them to your attention -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890601195 FYI @PinkCrow007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]
alamb commented on issue #7423: URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890600766 Thanks @mkarbo ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
