alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1125007908
##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
def _adlfs(properties: Properties) -> AbstractFileSystem:
from adlfs import AzureBlobFileSystem
- fs = AzureBlobFileSystem(**properties)
- return fs
+ return AzureBlobFileSystem(
+ connection_string=properties.get("adlfs.connection-string"),
+ account_name=properties.get("adlfs.account-name"),
+ account_key=properties.get("adlfs.account-key"),
+ sas_token=properties.get("adlfs.sas-token"),
+ tenant_id=properties.get("adlfs.tenant-id"),
+ client_id=properties.get("adlfs.client-id"),
+ client_secret=properties.get("adlfs.client-secret"),
+ )
SCHEME_TO_FS = {
Review Comment:
I just tested wasb on pandas. Pandas is using adlfs as well. It is not
supported:
code:
```
import pandas as pd
storage_options={'account_key': "<account_key>"}
df =
pd.read_parquet('wasb://<container_name>@<account_name>.blob.core.windows.net/path/to/file.parquet',
storage_options=storage_options)
df.head()
```
output:
```
File
[c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:493](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:493),
in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes,
**kwargs)
446 """
447 Load a parquet object from the file path, returning a DataFrame.
448
(...)
489 DataFrame
490 """
491 impl = get_engine(engine)
--> 493 return impl.read(
494 path,
495 columns=columns,
496 storage_options=storage_options,
497 use_nullable_dtypes=use_nullable_dtypes,
498 **kwargs,
499 )
File
[c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:233](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:233),
in PyArrowImpl.read(self, path, columns, use_nullable_dtypes, storage_options,
**kwargs)
...
--> 208 raise ValueError("Protocol not known: %s" % protocol)
209 bit = known_implementations[protocol]
210 try:
ValueError: Protocol not known: wasb
```
The same works with abfs protocol.
Using following url:
`abfs://<container_name>@<account_name>.dfs.core.windows.net/path/to/file.parquet`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
