I will leave some comments about this below. I believe we should try to tokenize into the Arrow varbinary layout to avoid this extra conversion step
[ Full content available at: https://github.com/apache/arrow/pull/2576 ] This message was relayed via gitbox.apache.org for [email protected]
