Tom-Newton commented on PR #12914: URL: https://github.com/apache/arrow/pull/12914#issuecomment-1553278082
A skeleton PR for this weekend sounds very promising 🙂 . I'll leave that with you for now @srilman but I'll keep a fairly close eye in case I can help. I might try the HDFS filesystem. If I can work out how to do the auth maybe I can tell you how the memory usage compares. I should probably also point out that my 11X number comes largely from how badly the fsspec filesystem suffers with high latency (I'm connecting from the UK to a US East ADLS gen2 account). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
