[
https://issues.apache.org/jira/browse/HIVE-26840?focusedWorklogId=835967&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-835967
]
ASF GitHub Bot logged work on HIVE-26840:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Dec/22 16:01
Start Date: 28/Dec/22 16:01
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3859:
URL: https://github.com/apache/hive/pull/3859#issuecomment-1366756444
@abstractdog There is a test
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe which is failing
with this error :
`java.lang.NullPointerException: src
at io.netty.util.internal.ObjectUtil.checkNotNull(ObjectUtil.java:39)
at
io.netty.buffer.UnsafeByteBufUtil.setBytes(UnsafeByteBufUtil.java:537)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:199)
at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:486)
at
io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:34)
at io.netty.buffer.ArrowBuf.setBytes(ArrowBuf.java:937)
at
org.apache.arrow.vector.BaseVariableWidthVector.setBytes(BaseVariableWidthVector.java:1155)
at
org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:990)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:418)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:213)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:279)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:219)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:142)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.serialize(Serializer.java:528)
at
org.apache.hadoop.hive.ql.io.arrow.ArrowColumnarBatchSerDe.serialize(ArrowColumnarBatchSerDe.java:260)
at
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe.serializeAndDeserialize(TestArrowColumnarBatchSerDe.java:240)
at
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe.initAndSerializeAndDeserialize(TestArrowColumnarBatchSerDe.java:204)
at
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe.testComprehensive(TestArrowColumnarBatchSerDe.java:381)`
This is because arrow 0.x had io.netty version of 4.1.22.Final which is the
old version. When we try to upgrade it to 4.1.69.Final netty and exclude the
netty jar from arrow dependency as I have done in this PR, then we get the
above exception in the test. The solutions are as follows :
1. Fix this test with the current netty version. I am still working on it
but till now not found a fix. If you know of something please let me know.
2. Revert the netty version. This test works perfectly fine if I change the
netty version to 4.1.48.Final.
3. Upgrade to arrow version 2.0.0 as done in master which has its own
implementation of netty classes but not sure how many cherry picks will be
required for a green build.
Issue Time Tracking
-------------------
Worklog Id: (was: 835967)
Time Spent: 3h 20m (was: 3h 10m)
> Backport of HIVE-23073 and HIVE-24138
> -------------------------------------
>
> Key: HIVE-26840
> URL: https://issues.apache.org/jira/browse/HIVE-26840
> Project: Hive
> Issue Type: Sub-task
> Reporter: Aman Raj
> Assignee: Aman Raj
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
