djouallah opened a new issue, #38726: URL: https://github.com/apache/arrow/issues/38726
### Describe the enhancement requested Delta_rs added support for Fabric OneLake recently, it will be nice to add the support for pyarrow dataset to read parquet and csv etc from OneLake Fabric https://github.com/delta-io/delta-rs/pull/1642 currently I am using this code to read from dataset and save ad delta table but it works only with local path ``` import pyarrow.dataset as ds from deltalake.writer import write_deltalake aadToken = mssparkutils.credentials.getToken('storage') storage_options={"bearer_token": aadToken, "use_fabric_endpoint": "true"} sf=100 rowgroup = 2000000 nbr_rowgroup_File = 8 * rowgroup for tbl in ['lineitem','nation','region','customer','supplier','orders','part','partsupp'] : print(tbl) dataset = ds.dataset(f'/lakehouse/default/Files/{sf}/{tbl}',format="parquet") write_deltalake(f"abfss://[email protected]/test.Lakehouse/Tables/{tbl}"\ ,dataset\ ,mode='overwrite',overwrite_schema=True,max_rows_per_file =nbr_rowgroup_File,min_rows_per_group=rowgroup,max_rows_per_group=rowgroup\ ,storage_options=storage_options) ``` ### Component(s) Format, Integration, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
