plamb-viso opened a new issue, #34455:
URL: https://github.com/apache/arrow/issues/34455

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   I'm using Huggingface Datasets to encode a dataset which uses pyarrow under 
the covers. The mapper function allows you to split encoding over multiple 
processors each which get chunks of the dataset. At the end of encoding, the 
results are flattened and written to disk using pyarrow.
   
   When run over multiple processors, the dataset correctly encodes, but then 
hangs indefinitely once it gets to the flattening portion. When run over a 
single processor, it passes mapping and once `save_to_disk` is called, another 
flattening process occurs in pyarrow. The following exception is thrown:
   
   ```
   File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 1348, in save_to_disk
       dataset = self.flatten_indices(num_proc=num_proc) if self._indices is 
not None else self
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 528, in wrapper
       out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
     File "/opt/conda/lib/python3.8/site-packages/datasets/fingerprint.py", 
line 511, in wrapper
       out = func(dataset, *args, **kwargs)
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 3541, in flatten_indices
       return self.map(
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 563, in wrapper
       out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 528, in wrapper
       out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 2953, in map
       for rank, done, content in Dataset._map_single(**dataset_kwargs):
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", 
line 3346, in _map_single
       writer.write_batch(batch)
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py", 
line 555, in write_batch
       self.write_table(pa_table, writer_batch_size)
     File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py", 
line 567, in write_table
       pa_table = pa_table.combine_chunks()
     File "pyarrow/table.pxi", line 3241, in pyarrow.lib.Table.combine_chunks
     File "pyarrow/error.pxi", line 144, in 
pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: concatenation of 
extension<arrow.py_extension_type<Array2DExtensionType>>
   ```
   My immediate interpretation of the error is that the `Array2D` simply hasn't 
been implemented yet in the way I'm attempting to use it. The problem is that 
I've used it the same way many times in the past.
   
   To get a feel for how I'm using it, the only two columns I have with the 
Array2D datatype are:
   ```python
           'seg_data': Array2D(dtype="float32", shape=(1024,4)),
           'visual_seg_data': Array2D(dtype="int64", shape=(196, 4)),
   ```
   
   I posted this on the Datasets forum as well, but thought I might try here.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to