[
https://issues.apache.org/jira/browse/ARROW-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Athanassios Hatzis updated ARROW-9505:
--------------------------------------
Description:
Hi, in this PyArrow structured array
{code:java}
struct_array.slice(0,3)
Out[52]:
<pyarrow.lib.StructArray object at 0x7f92061e9dc0>
-- is_valid: all not null
-- child 0 type: int16
[
991,
992,
993
]
-- child 1 type: decimal(6, 3)
[
36.100,
42.300,
15.300
]
{code}
I have tried to apply dictionary_encode() method and I got back this error
{code:java}
struct_array.dictionary_encode()
File "<ipython-input-51-440741990dd7>", line 1, in <module>
struct_array.dictionary_encode()
File "pyarrow/array.pxi", line 750, in pyarrow.lib.Array.dictionary_encode
File "pyarrow/error.pxi", line 106, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: dictionary-encode not implemented for
struct<catpid: int16, catcost: decimal(6, 3)>
{code}
I also tried to define a struct_array type with dictionary encode data types
and you get the same error.
I know that it is possible to apply dictionary_encode() to each field of the
struct_array and you can create a RecordBatch from the dictionary encoded
fields of the array. So I am not sure why this functionality is not implemented.
I also noticed that there is a transformation RecordBatch.from_struct_array()
but I want the columns to be dictionary encoded and the only way to do this in
the current PyArrow version is to process each field, column separately.
BTW: In my project I am addressing a basic problem which is how to transform
tuples from any database table to dictionary encoded columns of a PyArrow
RecordBatch (Table).
was:
Hi, in this PyArrow structured array
{code:java}
struct_array.slice(0,3)
Out[52]:
<pyarrow.lib.StructArray object at 0x7f92061e9dc0>
-- is_valid: all not null
-- child 0 type: int16
[
991,
992,
993
]
-- child 1 type: decimal(6, 3)
[
36.100,
42.300,
15.300
]
{code}
I have tried to apply dictionary_encode() method and I got back this error
{code:java}
struct_array.dictionary_encode()
File "<ipython-input-51-440741990dd7>", line 1, in <module>
struct_array.dictionary_encode()
File "pyarrow/array.pxi", line 750, in pyarrow.lib.Array.dictionary_encode
File "pyarrow/error.pxi", line 106, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: dictionary-encode not implemented for
struct<catpid: int16, catcost: decimal(6, 3)>
{code}
I also tried to define a struct_array type with dictionary encode data types
and you get the same error.
I know that it is possible to apply dictionary_encode() to each field of the
struct_array and you can create a RecordBatch from the dictionary encoded
fields of the array. So I am not sure why this functionality is not implemented.
I also noticed that there is a transformation RecordBatch.from_struct_array()
but I want the columns to be dictionary encoded and the only way to do this in
the current version is to process each field, column separately.
BTW: In my project I am addressing a basic problem which is how to transform
tuples from any database table to dictionary encoded columns of a PyArrow
RecordBatch (Table).
> [Python] pa.struct() dictionary-encode() not implemented
> --------------------------------------------------------
>
> Key: ARROW-9505
> URL: https://issues.apache.org/jira/browse/ARROW-9505
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Affects Versions: 0.17.1
> Reporter: Athanassios Hatzis
> Priority: Major
>
> Hi, in this PyArrow structured array
> {code:java}
> struct_array.slice(0,3)
> Out[52]:
> <pyarrow.lib.StructArray object at 0x7f92061e9dc0>
> -- is_valid: all not null
> -- child 0 type: int16
> [
> 991,
> 992,
> 993
> ]
> -- child 1 type: decimal(6, 3)
> [
> 36.100,
> 42.300,
> 15.300
> ]
> {code}
> I have tried to apply dictionary_encode() method and I got back this error
> {code:java}
> struct_array.dictionary_encode()
> File "<ipython-input-51-440741990dd7>", line 1, in <module>
> struct_array.dictionary_encode()
> File "pyarrow/array.pxi", line 750, in pyarrow.lib.Array.dictionary_encode
> File "pyarrow/error.pxi", line 106, in pyarrow.lib.check_status
> pyarrow.lib.ArrowNotImplementedError: dictionary-encode not implemented for
> struct<catpid: int16, catcost: decimal(6, 3)>
> {code}
> I also tried to define a struct_array type with dictionary encode data types
> and you get the same error.
> I know that it is possible to apply dictionary_encode() to each field of the
> struct_array and you can create a RecordBatch from the dictionary encoded
> fields of the array. So I am not sure why this functionality is not
> implemented.
> I also noticed that there is a transformation RecordBatch.from_struct_array()
> but I want the columns to be dictionary encoded and the only way to do this
> in the current PyArrow version is to process each field, column separately.
> BTW: In my project I am addressing a basic problem which is how to transform
> tuples from any database table to dictionary encoded columns of a PyArrow
> RecordBatch (Table).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
