jorisvandenbossche commented on PR #50951: URL: https://github.com/apache/arrow/pull/50951#issuecomment-5428343339
Inline comment of @ngoldbaum (moving it here it make it more prominent and not hidden in a potentially outdated/collapsed review comment): > Why is the output missing sentinel `None` for all input `na_object` choices, in particular for strings? > > Note that this also disagrees with NumPy: > > ``` > >>> np.array(["hello", "__placeholder__", "world"], dtype="T").tolist() > ['hello', '__placeholder__', 'world'] > ``` > > Is there a reason why you can't more faithfully translate NumPy's string missing data semantics? I think this is the main API design question to discuss: the numpy StringDtype has a configurable missing value sentinel model, while Arrow has no such option and just has nulls (through the bitmask). So that means it is not really possible to fully "faithfully" translate those missing semantics in the numpy->arrow conversion. Either: - we use the actual placeholder value (or at least if it is a string) as the resulting string value in the Arrow string array. That preserves the value, but looses the fact that it is missing - we translate the placeholder values to nulls (what this PR currently does). That preserves the missingness, but looses the information about the original placeholder value used. So we always loose something (unless we would create an extension type ..). Personally, my feeling is that preserving "missingness" is the most relevant. FWIW, this also means that a fully faithful roundtrip (once the conversion from arrow -> numpy exists) is also not possible out of the box, only if the user specifies the resulting numpy dtype (which has the information about which placeholder to use) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
