I agree. I already utilize the AWS and GCS classes. We have a customer need for ADLS & Parquet. I've recommended that we wait for the new implementation to become available in the next Arrow/Parquet C++ library release. In a way I am making a "bet" that the feature makes it into that release (11.0.0), so hopefully the contributors will make that happen.
-----Original Message----- From: Wes McKinney <wesmck...@gmail.com> Sent: Thursday, October 27, 2022 4:42 PM To: dev@arrow.apache.org Subject: Re: Apache Arrow filesystem question [You don't often get email from wesmck...@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL I definitely think it would be a good thing to have a C++ ADLS filesystem interface that is on par in quality with our S3 and GCS C++ interfaces - these should also provide material performance benefits to Python users over a pure-Python interface (I'm not sure if pyarrow's S3 interface via C++ has been studied relative to s3fs but there should be performance benefits in many datasets interactions). On Wed, Oct 26, 2022 at 1:09 PM Lee, David <david....@blackrock.com.invalid> wrote: > > Or this.. > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Farro > w.apache.org%2Fdocs%2Fpython%2Ffilesystems.html%23using-fsspec-compati > ble-filesystems-with-arrow&data=05%7C01%7CJerry.Adair%40sas.com%7C > b0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C > 0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd > ata=gnpmJKK3%2FcuYbnNlhEr%2F4notr7lCf0F9dJFTH8IeqYA%3D&reserved=0 > > import adlfs > # ... load your credentials and configure the filesystem fs = > adlfs.AzureBlobFileSystem(account_name=account_name, > account_key=account_key) > > > -----Original Message----- > From: Lee, David > Sent: Wednesday, October 26, 2022 11:04 AM > To: dev@arrow.apache.org > Subject: RE: Apache Arrow filesystem question > > There are fsspec implementations to support other filesystems. > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpypi > .org%2Fproject%2Fpyarrowfs-adlgen2%2F&data=05%7C01%7CJerry.Adair%4 > 0sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552 > 373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi > MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 > C%7C&sdata=Zhzd2XQa1pAkKYolRG4%2Bvgpr%2Fk7tmKSv6fL11Kg5xIg%3D& > reserved=0 > > -----Original Message----- > From: Jerry Adair <jerry.ad...@sas.com.INVALID> > Sent: Wednesday, October 26, 2022 10:59 AM > To: dev@arrow.apache.org > Subject: RE: Apache Arrow filesystem question > > External Email: Use caution with links and attachments > > > Hi- > > I am curious about the Apache Arrow filesystem support. I noticed that there > are classes for S3 and GCS. I am currently using those classes to read/write > from/to those cloud storage options. However I noticed that there is no > class for Azure (ADLS). And so I am curious why that is? Does anyone know? > > Thanks! > Jerry > > > This message may contain information that is confidential or privileged. If > you are not the intended recipient, please advise the sender immediately and > delete this message. See > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fcompliance%2Femail-disclaimers&data=05%7C01%7CJerry.Adair%40sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=a7lIrYIWXA3WNA%2Bc%2BRFvY952NWyjb3f03WLxUeOfOY4%3D&reserved=0 > for further information. Please refer to > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fcompliance%2Fprivacy-policy&data=05%7C01%7CJerry.Adair%40sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=X93wWDdHJ6oPQT1S8sZqURTZl1borPzC8VCRisblioI%3D&reserved=0 > for more information about BlackRock's Privacy Policy. > > > For a list of BlackRock's office addresses worldwide, see > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fabout-us%2Fcontacts-locations&data=05%7C01%7CJerry.Adair%40sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=okDt4J9JZ78nIOx913vltrblW%2B9XFTJpUpNOv%2BthNwA%3D&reserved=0. > > (c) 2022 BlackRock, Inc. All rights reserved.
