sl2902 opened a new issue, #34370:
URL: https://github.com/apache/arrow/issues/34370
### Describe the bug, including details regarding any error messages,
version, and platform.
I am building a custom dataset using HuggingFace Datasets. However, I run
into an issue with the schema.
My schema has a few nested fields, and one of the fields, where I encounter
the error, is defined as a list of dictionaries.
Hugging Face requires you to define the schema, and this particular field is
defined like so:
"names" : datasets.Sequence(
datasets.Features(
{
"url": datasets.Value("string"),
"score": datasets.Value("float16"),
}
)
I read the file using `pd.read_json(filename, compression="gzip",
lines=True)`
When I call the script, it failed with the following error
```
Downloading and preparing dataset aliases/default to
/Users/home/.cache/huggingface/datasets/aliases/default/1.0.0/a6ef1d39bec6e9117853e395cfa3061bd17e23da73677bf1acba8e50cebe3196...
Generating train split: 0%|
| 0/98149 [00:04<?, ? examples/s]Traceback (most recent call last):
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/builder.py",
line 1833, in _prepare_split_single
writer.write_table(table)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/arrow_writer.py",
line 567, in write_table
pa_table = table_cast(pa_table, self._schema)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2312, in table_cast
return cast_table_to_schema(table, schema)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2271, in cast_table_to_schema
arrays = [cast_array_to_feature(table[name], feature) for name, feature
in features.items()]
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2271, in <listcomp>
arrays = [cast_array_to_feature(table[name], feature) for name, feature
in features.items()]
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1837, in wrapper
return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in
array.chunks])
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1837, in <listcomp>
return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in
array.chunks])
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2087, in cast_array_to_feature
casted_values = _c(array.values, feature.feature)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1839, in wrapper
return func(array, *args, **kwargs)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2065, in cast_array_to_feature
arrays = [_c(array.field(name), subfeature) for name, subfeature in
feature.items()]
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2065, in <listcomp>
arrays = [_c(array.field(name), subfeature) for name, subfeature in
feature.items()]
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1839, in wrapper
return func(array, *args, **kwargs)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2131, in cast_array_to_feature
return array_cast(array, feature(),
allow_number_to_str=allow_number_to_str)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1839, in wrapper
return func(array, *args, **kwargs)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2021, in array_cast
return array.cast(pa_type)
File "pyarrow/array.pxi", line 926, in pyarrow.lib.Array.cast
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/pyarrow/compute.py",
line 391, in cast
return call_function("cast", [arr], options)
File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 144, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to
halffloat using function cast_half_float
```
To try to resolve this, I explicitly defined the schema for the arrow table
while converting from pandas to pyarrow
`pa.Table.from_pandas(df, schema=schema)`
But this also throws the same error
```
Downloading and preparing dataset aliases/default to
/Users/home/.cache/huggingface/datasets/aliases/default/1.0.0/5d933aa1538259f753a65ea696f0e78a15480e53c1852b167f94d41433e6a1d7...
Generating train split: 0%|
| 0/98149 [00:04<?, ? examples/s]Traceback (most recent call last):
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/builder.py",
line 1833, in _prepare_split_single
writer.write_table(table)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/arrow_writer.py",
line 567, in write_table
pa_table = table_cast(pa_table, self._schema)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2312, in table_cast
return cast_table_to_schema(table, schema)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2272, in cast_table_to_schema
return pa.Table.from_arrays(arrays, schema=schema)
File "pyarrow/table.pxi", line 3657, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 1421, in pyarrow.lib._sanitize_arrays
File "pyarrow/array.pxi", line 347, in pyarrow.lib.asarray
File "pyarrow/table.pxi", line 523, in pyarrow.lib.ChunkedArray.cast
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/pyarrow/compute.py",
line 391, in cast
return call_function("cast", [arr], options)
File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 144, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from list<item:
struct<url: string, score: double>> to struct using function cast_struct
```
I casted the field while reading the file but that didn't help. The only
way, I could resolve it is by converting the field to `str` and also making
sure that the `dtype` parameter in pandas casts the field to str, otherwise it
fails with the following error
```
Downloading and preparing dataset aliases/default to
/Users/home/.cache/huggingface/datasets/aliases/default/1.0.0/cd5506c5eb8c660aee4304538efdc03778baee11e78f15d409f59ecba349f81a...
Generating train split: 0%|
| 0/98149 [00:04<?, ? examples/s]Traceback (most recent call last):
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/builder.py",
line 1833, in _prepare_split_single
writer.write_table(table)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/arrow_writer.py",
line 567, in write_table
pa_table = table_cast(pa_table, self._schema)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2312, in table_cast
return cast_table_to_schema(table, schema)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2271, in cast_table_to_schema
arrays = [cast_array_to_feature(table[name], feature) for name, feature
in features.items()]
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2271, in <listcomp>
arrays = [cast_array_to_feature(table[name], feature) for name, feature
in features.items()]
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1837, in wrapper
return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in
array.chunks])
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1837, in <listcomp>
return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in
array.chunks])
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2131, in cast_array_to_feature
return array_cast(array, feature(),
allow_number_to_str=allow_number_to_str)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 1839, in wrapper
return func(array, *args, **kwargs)
File
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
line 2022, in array_cast
raise TypeError(f"Couldn't cast array of
type\n{array.type}\nto\n{pa_type}")
TypeError: Couldn't cast array of type
list<item: struct<score: double, url: string>>
to
string
```
numpy 1.24.2
pandas 1.5.3
pyarrow 11.0.0
MacoS Catalina 10.15.7
I came across a similar issue,
https://issues.apache.org/jira/browse/ARROW-1888 which was resolved a couple of
weeks ago. However, in my case, I don't think I am changing the type.
Nevertheless it fails. Do you know why this is happening? Is converting it to a
`str` the only workaround for now? Any help is appreciated
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]