tustvold commented on PR #3399: URL: https://github.com/apache/arrow-rs/pull/3399#issuecomment-1383276937
Sorry for the delay with this, I'm currently reimplementing the JSON reader to not be row-based, and instead process batches at a time to drastically improve performance. Whilst initially this will be a separate opt-in RawDecoder the eventual goal would be to replace the current implementation. I'm therefore a bit apprehensive about this feature, as it will likely be impossible to port across... On a related note, such a behaviour is not possible for CSV for much the same reason. This leads me to the position of not really knowing how best to proceed... Could you perhaps expand a bit on the use-case for this? Perhaps we could parse errors as nulls or something instead? Sorry to contradict my earlier statements, but I'm wary of painting ourselves into a corner here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
