tustvold opened a new pull request #10073: URL: https://github.com/apache/arrow/pull/10073
Currently calling concat on two dictionary encoded arrays blinding concatenates the keys arrays, and keeps only the dictionary of the first. This can lead to invalid data, including keys referring to value indexes beyond the bounds of the values array. This PR alters MutableArrayData to concatenate the child data of any dictionary arrays passed to it, and offset the source keys to correctly map to the relevant "slice" of the resulting child data. This does mean that the resulting dictionary array may contain duplicates, and will not be sorted, but I'm not sure if this is a problem? Signed-off-by: Raphael Taylor-Davies <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
