[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913381#comment-16913381 ] Arina Ielchiieva commented on DRILL-7355: - Awesome :) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. During the writing of these vectors, the offset value overflow is not > processed. Therefore, upon subsequent reading of such a vector to the > corresponding holder, the value in the "end" field may be less than the value > of the "start" field. In some cases, the "start" and "end" values are used to > get the size of the buffer (for example,ParquetOutputRecordWriter). This > causes the target buffer size to be negative. > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913379#comment-16913379 ] Oleg Zinoviev commented on DRILL-7355: -- [~arina], looks like it's already fixed, since I cannot reproduce it on drill 1.17.0. > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. During the writing of these vectors, the offset value overflow is not > processed. Therefore, upon subsequent reading of such a vector to the > corresponding holder, the value in the "end" field may be less than the value > of the "start" field. In some cases, the "start" and "end" values are used to > get the size of the buffer (for example,ParquetOutputRecordWriter). This > causes the target buffer size to be negative. > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913284#comment-16913284 ] Oleg Zinoviev commented on DRILL-7355: -- Sorry, I missed that UInt4 is a Java 'int' vector. Description need to be fixed > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > * to be fixed * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913283#comment-16913283 ] Arina Ielchiieva commented on DRILL-7355: - [~le.louch] please add data file and example of the select statement so if any volunteers to fix this could be able to reproduce the issue. Regarding the fix availability in 1.17, contributions always welcome :) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > * to be fixed * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913276#comment-16913276 ] Oleg Zinoviev commented on DRILL-7355: -- Any chance this can be fixed in 1.17.0? > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. When reading values into the corresponding holder, the values of > their offset vectors are written into the "start" and "end" fields as > ordinary integers. In the subsequent operations is not considered that > originally this values were unsigned integers and (for example in the > calculation of the buffer size at ParquetOutputRecordWriter) usual > subtraction "end" - "start" used. This causes the target buffer size to be > negative. -- This message was sent by Atlassian Jira (v8.3.2#803003)
