houqp commented on issue #1923: URL: https://github.com/apache/arrow-datafusion/issues/1923#issuecomment-1066041247
We ran into the same issue in delta-rs. I think the ideal solution would be to normalize table path within the objecstore implementation when it's being created. The issue with using URI in fileMeta is all the object stores' list calls do return full URIs, so we will have to perform a lot of string creations in the heap to construct the URIs, this could be expensive when we need to deal with millions of files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
