Re: ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)

[Дима Фомин]

Hello, Micah!

 

Gotcha! Seems the problem in in pyarrow.array() handling of decimal values.

 

data1 = [decimal.Decimal('234234234234.43')]

 

#pa.array somehow broke precision and scale?

data2 = pa.array(data1)

 

sss1 = pa.schema({'decimal_col': pa.decimal128(22,2)})

 

pa.Table.from_arrays([data1], schema=sss1)

 

is OK

 

bur this one

pa.Table.from_arrays([data2], schema=sss1)

 

raise ArrowNotImplementedError: No cast implemented from decimal(14, 2) to decimal(22, 2)

 

see it here

https://colab.research.google.com/drive/1ejOyFBVs8GPFTosuRMDyTwPvt4145rgV

 

Should I post in to JIRA as well? Or this is the same reason posted above?

 

 

25.10.2019, 08:00, "Micah Kornfield" <emkornfi...@gmail.com>:

Hi Dima,

Thanks for finding this bug would you mind filing a JIRA (https://issues.apache.org/jira).  

 

Note that this code works fine if  you put quotes around the decimal value (e.g. Decimal('123.234')), but as you state it crashes python because the Decimal created without quotes requires more then 38 decimal places to represent.

 

Thanks,

Micah

 

 

 

On Thu, Oct 24, 2019 at 7:53 AM Дима Фомин <dat...@yandex.ru> wrote:

Digging this I just found that this code breaks my Python Kernel :(

 

import sqlalchemy as sq

import pyarrow as pa

 

aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234), sq.sql.sqltypes.decimal.Decimal(123.234)

ddd = [aaa, bbb]

pa.array(ddd)

 

So seems pyarrow.array is limited with types of array elements, doesnt'it?

 

WBR,

Dima Fomin

 

24.10.2019, 15:47, "Dima Fomin" <dat...@yandex.ru>:

Hello!

First of all thank you very much for your efforts on Apache Arrow! This looks very useful and prominent tech to be able use it in self-BI area.

 

I faced such issues - get Subj Error

ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)

when trying to pyarrow.Table.from_batches array of values defined in schema as pyarrow.decimal128(12,2).

 

Which values are in fact column of sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)" values, which I map in schema like this pyarrow.decimal128(12,2).

 

What could be a reason of issue?

Could it be that somehow declared decimal type of data is reduced to minimum value which met in data? :/

 

With best regards,

Dima Fomin