I definitely think it would be a good thing to have a C++ ADLS filesystem interface that is on par in quality with our S3 and GCS C++ interfaces — these should also provide material performance benefits to Python users over a pure-Python interface (I'm not sure if pyarrow's S3 interface via C++ has been studied relative to s3fs but there should be performance benefits in many datasets interactions).
On Wed, Oct 26, 2022 at 1:09 PM Lee, David <david....@blackrock.com.invalid> wrote: > > Or this.. > > https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems-with-arrow > > import adlfs > # ... load your credentials and configure the filesystem > fs = adlfs.AzureBlobFileSystem(account_name=account_name, > account_key=account_key) > > > -----Original Message----- > From: Lee, David > Sent: Wednesday, October 26, 2022 11:04 AM > To: dev@arrow.apache.org > Subject: RE: Apache Arrow filesystem question > > There are fsspec implementations to support other filesystems. > > https://pypi.org/project/pyarrowfs-adlgen2/ > > -----Original Message----- > From: Jerry Adair <jerry.ad...@sas.com.INVALID> > Sent: Wednesday, October 26, 2022 10:59 AM > To: dev@arrow.apache.org > Subject: RE: Apache Arrow filesystem question > > External Email: Use caution with links and attachments > > > Hi- > > I am curious about the Apache Arrow filesystem support. I noticed that there > are classes for S3 and GCS. I am currently using those classes to read/write > from/to those cloud storage options. However I noticed that there is no > class for Azure (ADLS). And so I am curious why that is? Does anyone know? > > Thanks! > Jerry > > > This message may contain information that is confidential or privileged. If > you are not the intended recipient, please advise the sender immediately and > delete this message. See > http://www.blackrock.com/corporate/compliance/email-disclaimers for further > information. Please refer to > http://www.blackrock.com/corporate/compliance/privacy-policy for more > information about BlackRock’s Privacy Policy. > > > For a list of BlackRock's office addresses worldwide, see > http://www.blackrock.com/corporate/about-us/contacts-locations. > > © 2022 BlackRock, Inc. All rights reserved.
