lwhite1 commented on issue #13405: URL: https://github.com/apache/arrow/issues/13405#issuecomment-1182004773
Hi @adzcai, I looked into this a bit more. The method ScanTask#batchIterator that you're using was removed in Arrow 8, when the Arrow Datasets Parquet reader was re-implemented to use the c-data-interface. While the Parquet reader has no tests to demonstrate that complex type supports work, the c-data-interface code does test for all types. With the new implementation complex types should be supported. If you're able to update to the latest Arrow version you should be ok, but some code modifications will be required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
