jorisvandenbossche commented on PR #35036: URL: https://github.com/apache/arrow/pull/35036#issuecomment-1517599393
> Do we have dictionary scalars? It's actually a good question what we see as "logical" nulls for dictionary encoded arrays. The difference with REE and union is that dictionary arrays _do_ have a top-level validity bitmap (for the indices), but you can also have nulls in the dictionary, and so in that sense it becomes similar as run_end encoded arrays. The spec explicitly mentions that the `null_count` only considers the top-level validity bitmap: > The null count of such arrays is dictated only by the validity bitmap of its indices, irrespective of any null values in the dictionary. But we could have a similar discussion about whether "logical" nulls would deviate from this physical null count, just as we now do for REE and union. (this is probably food for a separate issue, though) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
