pauldix opened a new pull request #7064: URL: https://github.com/apache/arrow/pull/7064
This isn't even close to being mergable, but I'm hoping to get some additional eyes on this at this point. I have an initial skeleton of the integration test. I started out only implementing part of the `arrow-json-integration-test` executor. I started by trying to get Rust to consume JSON and produce Arrow for the `generated_primitive` test/JSON. I was only able to build the Go version of things. For some reason none of the docker-compose builds worked for C++ or Java and I was completely unsuccessful getting them to build on OSX Catalina. I figured if Go works then I'd use that as my comparison for now and try to get C++ to build later when I need to hook up the Flight integration test. On running this with Archery and having Rust producing and Go consuming, it comes up with the following error output from the Go executable when trying to validate the Arrow file produced by Rust: ``` arrow-json: could not read record 0 from ARROW file: arrow/ipc: invalid file metadata=1580 position for record 0 ``` I checked what's being passed to the Rust FileWriter and I think it looks correct (2 record batches with 17 and 20 rows respectively). I can continue digging to see if it's something in the FileWriter, but I wanted to get some feedback on all of this before I go even deeper into this rabbit hole. So here are a few specific questions: * Does the structure of this look roughly correct? * How should I go about troubleshooting and fixing the generated Arrow file that is failing to be read by the Go reader? Apologies if there is a more preferred way to open up this kind of discussion, any pointers are appreciated. I'd love to contribute to the Rust implementation and get it over the line to be included in the integration tests for release. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
