[
https://issues.apache.org/jira/browse/TRAFODION-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006906#comment-16006906
]
Hans Zeller commented on TRAFODION-2615:
----------------------------------------
The problem is that the new TINYINT type sometimes is stored in 1 byte, other
times in 2 bytes. The TMUDF code makes the assumption that we can compute the
storage size from the precision. I need to fix this in the binder and make sure
the storage size matches the storage size that the TMUDF code will compute.
> TMUDF returns wrong results with small numeric inputs
> -----------------------------------------------------
>
> Key: TRAFODION-2615
> URL: https://issues.apache.org/jira/browse/TRAFODION-2615
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
> Affects Versions: 2.1-incubating
> Reporter: Hans Zeller
> Assignee: Hans Zeller
> Fix For: 2.2-incubating
>
> Attachments: EchoUdf.cpp
>
>
> I got an incorrect result with this query:
> {noformat}
> >>select * from udf(echo(table(select * from (values (1), (22), (3), (4))
> >>t(x))));
> X
> ------
> 1
> -234
> 3
> 772
> --- 4 row(s) selected.
> >>
> {noformat}
> Note that -234 if FF16 (and 16 is hex for 22) and that 772 is 304, so it's a
> junk byte followed by the true values 22 and 4 that is returned.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
