[
https://issues.apache.org/jira/browse/ARROW-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prakhar Sandhu updated ARROW-17672:
-----------------------------------
Description:
I am running the below commands in databricks.
When I am trying to read a file which is stored in adls using pandas:
{code:java}
pip install adlfs
import pandas as pd
data = pd.read_parquet("abfss://data.parquet", storage_options= {}){code}
Then I got the below error:
{code:java}
File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py",
line 310, in read_parquet
return impl.read(path, columns=columns, **kwargs)
File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py",
line 125, in read
path, columns=columns, **kwargs
File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line
1573, in read_table
ignore_prefixes=ignore_prefixes,
File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line
1434, in __init__
ignore_prefixes=ignore_prefixes)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line
667, in dataset
return _filesystem_dataset(source, **kwargs)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line
424, in _filesystem_dataset
fs, paths_or_selector = _ensure_single_source(source, filesystem)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line
371, in _ensure_single_source
filesystem, path = FileSystem.from_uri(path)
File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI:
abfss:///data.parquet {code}
was:
When I am trying to read a file which is stored in adls using pandas:
{code:java}
pip install adlfs
import pandas as pd
data = pd.read_parquet("abfss://data.parquet", storage_options= {}){code}
Then I got the below error:
{code:java}
File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py",
line 310, in read_parquet
return impl.read(path, columns=columns, **kwargs)
File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py",
line 125, in read
path, columns=columns, **kwargs
File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line
1573, in read_table
ignore_prefixes=ignore_prefixes,
File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line
1434, in __init__
ignore_prefixes=ignore_prefixes)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line
667, in dataset
return _filesystem_dataset(source, **kwargs)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line
424, in _filesystem_dataset
fs, paths_or_selector = _ensure_single_source(source, filesystem)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line
371, in _ensure_single_source
filesystem, path = FileSystem.from_uri(path)
File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI:
abfss:///data.parquet {code}
> Unrecognized filesystem type in URI: abfss://
> ---------------------------------------------
>
> Key: ARROW-17672
> URL: https://issues.apache.org/jira/browse/ARROW-17672
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet, Python
> Reporter: Prakhar Sandhu
> Priority: Major
>
> I am running the below commands in databricks.
> When I am trying to read a file which is stored in adls using pandas:
> {code:java}
> pip install adlfs
> import pandas as pd
> data = pd.read_parquet("abfss://data.parquet", storage_options= {}){code}
> Then I got the below error:
> {code:java}
> File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py",
> line 310, in read_parquet
> return impl.read(path, columns=columns, **kwargs)
> File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py",
> line 125, in read
> path, columns=columns, **kwargs
> File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py",
> line 1573, in read_table
> ignore_prefixes=ignore_prefixes,
> File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py",
> line 1434, in __init__
> ignore_prefixes=ignore_prefixes)
> File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py",
> line 667, in dataset
> return _filesystem_dataset(source, **kwargs)
> File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py",
> line 424, in _filesystem_dataset
> fs, paths_or_selector = _ensure_single_source(source, filesystem)
> File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py",
> line 371, in _ensure_single_source
> filesystem, path = FileSystem.from_uri(path)
> File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri
> File "pyarrow/error.pxi", line 122, in
> pyarrow.lib.pyarrow_internal_check_status
> File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI:
> abfss:///data.parquet {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
