GitHub user keen85 added a comment to the discussion: Azure Storage Gen2 (hierarchical namespaces) - use DFS endpoint to improve performance
Hi @tustvold > > recursive directory listings > > Why would it provide better performance in this case? Because the DFS endpoint exposes a filesystem-like API that supports recursive listing in one call. The Blob endpoint is object-store oriented: it only lists a flat set of objects under a given prefix. To traverse a hierarchy recursively with the Blob API, the client must repeatedly enumerate each "folder level" and stitch results together. That means many network round-trips and higher latency. The DFS endpoint, by contrast, is built for hierarchical namespaces. Its `List Path` API accepts a `recursive=true` flag, so the storage service itself performs the traversal server-side and streams all matching entries back in a single operation. This avoids multiple client-side iterations and accelerates recursive listings, especially in deep or large directory trees. GitHub link: https://github.com/apache/arrow-rs-object-store/discussions/481#discussioncomment-14591899 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
