[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852493#comment-16852493 ] Wes McKinney commented on ARROW-1837: - [~emkornfi...@gmail.com] if you are interested in unsigned integers this would benefit from some attention > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Wes McKinney >Priority: Blocker > Labels: columnar-format-1.0 > Fix For: 0.14.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765281#comment-16765281 ] Wes McKinney commented on ARROW-1837: - [~pravindra] any chance someone on your end might be able to look at this? We'd also have to make changes in C++, so we probably would want to create a branch where we can collaborate on this change > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Wes McKinney >Priority: Blocker > Labels: columnar-format-1.0 > Fix For: 0.13.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510130#comment-16510130 ] Wes McKinney commented on ARROW-1837: - [~jnadeau] [~siddteotia] any thoughts on what would be involved with better supporting unsigned integers in Java? I think they need to be supported. I'm moving this issue to 0.11 so it does not block 0.10 > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors >Reporter: Wes McKinney >Priority: Blocker > Fix For: 0.11.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356266#comment-16356266 ] Wes McKinney commented on ARROW-1837: - Moving this to 0.10.0. Maybe we can deal with unsigned integers in Java in the next release cycle > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors >Reporter: Wes McKinney >Priority: Blocker > Fix For: 0.10.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345174#comment-16345174 ] Li Jin commented on ARROW-1837: --- We can check with Dremio folks whether those are being used. If not we are free to change the them. cc [~siddteotia] > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors >Reporter: Wes McKinney >Priority: Blocker > Fix For: 0.9.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344472#comment-16344472 ] Jingyuan Wang commented on ARROW-1837: -- [~icexelloss], any opinion on whether UInt2Vector should return char/Character? > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors >Reporter: Wes McKinney >Priority: Blocker > Fix For: 0.9.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341439#comment-16341439 ] Jingyuan Wang commented on ARROW-1837: -- For Java 8 & 9, Integer and Long can partially handle unsigned values, but looks highly inconvenient. The example I found is to assign the value by parsing a string. I agree with you on your proposal of promoting unsigned values to to a wider type when retrieved. But I also think this might require other Java developers their opinions as currently UInt2Vector return char/Character. I am not sure about if this vector has any necessary use cases. If so, it is at least confusing if not wrong. > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors >Reporter: Wes McKinney >Priority: Blocker > Fix For: 0.9.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[
https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340464#comment-16340464
]
Wes McKinney commented on ARROW-1837:
-
Since Java doesn't have unsigned integer values, I presume that we'll want to
return them in {{getObject}} as a larger signed integer object for now – as
long as unsigned values sent from other applications can be understood
correctly. So a {{UInt2Vector}} would return {{int}} values, {{UInt4Vector}}
return {{long}} values, and so forth (would {{UInt8}} have to return
{{BigInteger}}?)
> [Java] Unable to read unsigned integers outside signed range for bit width in
> integration tests
> ---
>
> Key: ARROW-1837
> URL: https://issues.apache.org/jira/browse/ARROW-1837
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Vectors
>Reporter: Wes McKinney
>Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: generated_primitive.json
>
>
> I believe this was introduced recently (perhaps in the refactors), but there
> was a problem where the integration tests weren't being properly run that hid
> the error from us
> see https://github.com/apache/arrow/pull/1294#issuecomment-345553066
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[ https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340275#comment-16340275 ] Jingyuan Wang commented on ARROW-1837: -- In the defined Java Vector types, there are two 16-bit types: SmallInt and UInt2. The first one is associated with signed short (Java does not support unsigned short) and the latter is associated with char/Character. I need to look at the code a bit more to understand how these two types are determined to use. > [Java] Unable to read unsigned integers outside signed range for bit width in > integration tests > --- > > Key: ARROW-1837 > URL: https://issues.apache.org/jira/browse/ARROW-1837 > Project: Apache Arrow > Issue Type: Bug > Components: Java - Vectors >Reporter: Wes McKinney >Priority: Blocker > Fix For: 0.9.0 > > Attachments: generated_primitive.json > > > I believe this was introduced recently (perhaps in the refactors), but there > was a problem where the integration tests weren't being properly run that hid > the error from us > see https://github.com/apache/arrow/pull/1294#issuecomment-345553066 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[
https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340263#comment-16340263
]
Jingyuan Wang commented on ARROW-1837:
--
a more complete stack trace
{code:none}
Error accessing files
Numeric value (65350) out of range of Java short
at [Source: /tmp/tmp7z9ijgqc/generated_primitive.json; line: 783, column: 18]
17:46:40.350 [main] ERROR org.apache.arrow.tools.Integration - Error accessing
files
com.fasterxml.jackson.core.JsonParseException: Numeric value (65350) out of
range of Java short
at [Source: /tmp/tmp7z9ijgqc/generated_primitive.json; line: 783, column: 18]
at
com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
at
com.fasterxml.jackson.core.JsonParser.getShortValue(JsonParser.java:1075)
at
org.apache.arrow.vector.ipc.JsonFileReader$BufferHelper$3.read(JsonFileReader.java:248)
at
org.apache.arrow.vector.ipc.JsonFileReader$BufferReader.readBuffer(JsonFileReader.java:199)
at
org.apache.arrow.vector.ipc.JsonFileReader.readIntoBuffer(JsonFileReader.java:491)
at
org.apache.arrow.vector.ipc.JsonFileReader.readFromJsonIntoVector(JsonFileReader.java:539)
at
org.apache.arrow.vector.ipc.JsonFileReader.read(JsonFileReader.java:180)
at
org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:196)
at org.apache.arrow.tools.Integration.run(Integration.java:114)
at org.apache.arrow.tools.Integration.main(Integration.java:65)
Suppressed: java.lang.IllegalStateException: Memory was leaked by
query. Memory leaked: (1148)
Allocator(ROOT) 0/1148/1148/2147483647 (res/actual/peak/limit)
at
org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:406)
at
org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:220)
... 2 common frames omitted
{code}
> [Java] Unable to read unsigned integers outside signed range for bit width in
> integration tests
> ---
>
> Key: ARROW-1837
> URL: https://issues.apache.org/jira/browse/ARROW-1837
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Vectors
>Reporter: Wes McKinney
>Priority: Blocker
> Fix For: 0.9.0
>
>
> I believe this was introduced recently (perhaps in the refactors), but there
> was a problem where the integration tests weren't being properly run that hid
> the error from us
> see https://github.com/apache/arrow/pull/1294#issuecomment-345553066
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1837) [Java] Unable to read unsigned integers outside signed range for bit width in integration tests
[
https://issues.apache.org/jira/browse/ARROW-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340222#comment-16340222
]
Jingyuan Wang commented on ARROW-1837:
--
Wes mentioned in the GitHub issue page:
--
I found the problem -- one of the primitive integration test files was being
clobbered and not run, which was suppressing a failure that should have been
raised a long time ago.
In the meantime, there was also a regression from the Java refactor, and we are
no longer able to fully read unsigned integer types anymore. I will hack the
integration tests for now and open a JIRA about fixing,
here's an example of trying to read a {{uint16}} vector:
{{16:49:51.051 [main] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio:
8 Error accessing files Numeric value (65350) out of range of Java short at
[Source: /tmp/tmpwgopllpl/generated_primitive.json; line: , column: 18]
16:49:51.065 [main] ERROR org.apache.arrow.tools.Integration - Error accessing
files com.fasterxml.jackson.core.JsonParseException: Numeric value (65350) out
of range of Java short}}
> [Java] Unable to read unsigned integers outside signed range for bit width in
> integration tests
> ---
>
> Key: ARROW-1837
> URL: https://issues.apache.org/jira/browse/ARROW-1837
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Vectors
>Reporter: Wes McKinney
>Priority: Blocker
> Fix For: 0.9.0
>
>
> I believe this was introduced recently (perhaps in the refactors), but there
> was a problem where the integration tests weren't being properly run that hid
> the error from us
> see https://github.com/apache/arrow/pull/1294#issuecomment-345553066
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
