Reranko05 commented on PR #50945: URL: https://github.com/apache/arrow/pull/50945#issuecomment-5436325509
@cyb70289 I don't think the issue is JSON compatibility between RapidJSON and simdjson. The main issue I ran into is the boundary/streaming semantics. The previous RapidJSON implementation uses `kParseStopWhenDoneFlag`, so it stops as soon as the first complete top-level value is parsed, even if the remaining input contains a partial or malformed value. With `iterate_many()`, I wasn't able to reproduce that behavior: errors in subsequent/incomplete data can affect whether we successfully identify the preceding complete value. I agree that writing our own optimized JSON parser would not be ideal. The manual approach I used, and which @rok also implemented in rok#47, only scans for the boundary of the first complete object or array while respecting strings and escapes, and then lets simdjson perform the actual JSON validation. But I agree this still introduces complexity and needs careful testing. If there is a way to use simdjson directly while preserving the old stop-after-one-value semantics, that would definitely be preferable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
