sl2902 opened a new issue, #34370:
URL: https://github.com/apache/arrow/issues/34370

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   I am building a custom dataset using HuggingFace Datasets. However, I run 
into an issue with the schema.
   My schema has a few nested fields, and one of the fields, where I encounter 
the error, is defined as a list of dictionaries.
   Hugging Face requires you to define the schema, and this particular field is 
defined like so:
   "names" : datasets.Sequence(
                   datasets.Features(
                           {
                               "url": datasets.Value("string"),
                               "score": datasets.Value("float16"),
                           }
                       )
   
   I read the file using `pd.read_json(filename, compression="gzip", 
lines=True)`
   When I call the script, it failed with the following error
   ```
   Downloading and preparing dataset aliases/default to 
/Users/home/.cache/huggingface/datasets/aliases/default/1.0.0/a6ef1d39bec6e9117853e395cfa3061bd17e23da73677bf1acba8e50cebe3196...
   Generating train split:   0%|                                                
                                                                                
                                                                                
  | 0/98149 [00:04<?, ? examples/s]Traceback (most recent call last):
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/builder.py",
 line 1833, in _prepare_split_single
       writer.write_table(table)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/arrow_writer.py",
 line 567, in write_table
       pa_table = table_cast(pa_table, self._schema)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2312, in table_cast
       return cast_table_to_schema(table, schema)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2271, in cast_table_to_schema
       arrays = [cast_array_to_feature(table[name], feature) for name, feature 
in features.items()]
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2271, in <listcomp>
       arrays = [cast_array_to_feature(table[name], feature) for name, feature 
in features.items()]
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1837, in wrapper
       return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in 
array.chunks])
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1837, in <listcomp>
       return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in 
array.chunks])
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2087, in cast_array_to_feature
       casted_values = _c(array.values, feature.feature)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1839, in wrapper
       return func(array, *args, **kwargs)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2065, in cast_array_to_feature
       arrays = [_c(array.field(name), subfeature) for name, subfeature in 
feature.items()]
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2065, in <listcomp>
       arrays = [_c(array.field(name), subfeature) for name, subfeature in 
feature.items()]
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1839, in wrapper
       return func(array, *args, **kwargs)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2131, in cast_array_to_feature
       return array_cast(array, feature(), 
allow_number_to_str=allow_number_to_str)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1839, in wrapper
       return func(array, *args, **kwargs)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2021, in array_cast
       return array.cast(pa_type)
     File "pyarrow/array.pxi", line 926, in pyarrow.lib.Array.cast
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/pyarrow/compute.py",
 line 391, in cast
       return call_function("cast", [arr], options)
     File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
     File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
     File "pyarrow/error.pxi", line 144, in 
pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to 
halffloat using function cast_half_float
   ```
   
   To try to resolve this, I explicitly defined the schema for the arrow table 
while converting from pandas to pyarrow
   `pa.Table.from_pandas(df, schema=schema)`
   But this also throws the same error
   ```
   Downloading and preparing dataset aliases/default to 
/Users/home/.cache/huggingface/datasets/aliases/default/1.0.0/5d933aa1538259f753a65ea696f0e78a15480e53c1852b167f94d41433e6a1d7...
   Generating train split:   0%|                                                
                                                                                
                                                                                
  | 0/98149 [00:04<?, ? examples/s]Traceback (most recent call last):
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/builder.py",
 line 1833, in _prepare_split_single
       writer.write_table(table)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/arrow_writer.py",
 line 567, in write_table
       pa_table = table_cast(pa_table, self._schema)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2312, in table_cast
       return cast_table_to_schema(table, schema)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2272, in cast_table_to_schema
       return pa.Table.from_arrays(arrays, schema=schema)
     File "pyarrow/table.pxi", line 3657, in pyarrow.lib.Table.from_arrays
     File "pyarrow/table.pxi", line 1421, in pyarrow.lib._sanitize_arrays
     File "pyarrow/array.pxi", line 347, in pyarrow.lib.asarray
     File "pyarrow/table.pxi", line 523, in pyarrow.lib.ChunkedArray.cast
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/pyarrow/compute.py",
 line 391, in cast
       return call_function("cast", [arr], options)
     File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
     File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
     File "pyarrow/error.pxi", line 144, in 
pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: Unsupported cast from list<item: 
struct<url: string, score: double>> to struct using function cast_struct
   ```
   I casted the field while reading the file but that didn't help. The only 
way, I could resolve it is by converting the field to `str` and also making 
sure that the `dtype` parameter in pandas casts the field to str, otherwise it 
fails with the following error
   ```
   Downloading and preparing dataset aliases/default to 
/Users/home/.cache/huggingface/datasets/aliases/default/1.0.0/cd5506c5eb8c660aee4304538efdc03778baee11e78f15d409f59ecba349f81a...
   Generating train split:   0%|                                                
                                                                                
                                                                                
  | 0/98149 [00:04<?, ? examples/s]Traceback (most recent call last):
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/builder.py",
 line 1833, in _prepare_split_single
       writer.write_table(table)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/arrow_writer.py",
 line 567, in write_table
       pa_table = table_cast(pa_table, self._schema)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2312, in table_cast
       return cast_table_to_schema(table, schema)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2271, in cast_table_to_schema
       arrays = [cast_array_to_feature(table[name], feature) for name, feature 
in features.items()]
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2271, in <listcomp>
       arrays = [cast_array_to_feature(table[name], feature) for name, feature 
in features.items()]
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1837, in wrapper
       return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in 
array.chunks])
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1837, in <listcomp>
       return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in 
array.chunks])
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2131, in cast_array_to_feature
       return array_cast(array, feature(), 
allow_number_to_str=allow_number_to_str)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 1839, in wrapper
       return func(array, *args, **kwargs)
     File 
"/Users/home/.local/share/virtualenvs/env-sIFPHfLo/lib/python3.8/site-packages/datasets/table.py",
 line 2022, in array_cast
       raise TypeError(f"Couldn't cast array of 
type\n{array.type}\nto\n{pa_type}")
   TypeError: Couldn't cast array of type
   list<item: struct<score: double, url: string>>
   to
   string
   ```
   numpy              1.24.2
   pandas             1.5.3
   pyarrow            11.0.0
   MacoS Catalina 10.15.7
   
   I came across a similar issue, 
https://issues.apache.org/jira/browse/ARROW-1888 which was resolved a couple of 
weeks ago. However, in my case, I don't think I am changing the type. 
Nevertheless it fails. Do you know why this is happening? Is converting it to a 
`str` the only workaround for now? Any help is appreciated
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to