amanraj2520 commented on PR #3859:
URL: https://github.com/apache/hive/pull/3859#issuecomment-1366756444
@abstractdog There is a test
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe which is failing
with this error :
`java.lang.NullPointerException: src
at io.netty.util.internal.ObjectUtil.checkNotNull(ObjectUtil.java:39)
at
io.netty.buffer.UnsafeByteBufUtil.setBytes(UnsafeByteBufUtil.java:537)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:199)
at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:486)
at
io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:34)
at io.netty.buffer.ArrowBuf.setBytes(ArrowBuf.java:937)
at
org.apache.arrow.vector.BaseVariableWidthVector.setBytes(BaseVariableWidthVector.java:1155)
at
org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:990)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:418)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:213)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:279)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:219)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:142)
at
org.apache.hadoop.hive.ql.io.arrow.Serializer.serialize(Serializer.java:528)
at
org.apache.hadoop.hive.ql.io.arrow.ArrowColumnarBatchSerDe.serialize(ArrowColumnarBatchSerDe.java:260)
at
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe.serializeAndDeserialize(TestArrowColumnarBatchSerDe.java:240)
at
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe.initAndSerializeAndDeserialize(TestArrowColumnarBatchSerDe.java:204)
at
org.apache.hadoop.hive.ql.io.arrow.TestArrowColumnarBatchSerDe.testComprehensive(TestArrowColumnarBatchSerDe.java:381)`
This is because arrow 0.x had io.netty version of 4.1.22.Final which is the
old version. When we try to upgrade it to 4.1.69.Final netty and exclude the
netty jar from arrow dependency as I have done in this PR, then we get the
above exception in the test. The solutions are as follows :
1. Fix this test with the current netty version. I am still working on it
but till now not found a fix. If you know of something please let me know.
2. Revert the netty version. This test works perfectly fine if I change the
netty version to 4.1.48.Final.
3. Upgrade to arrow version 2.0.0 as done in master which has its own
implementation of netty classes but not sure how many cherry picks will be
required for a green build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
