Usama Kaleem created HIVE-23431: ----------------------------------- Summary: unable to wrtie a few records to hive in orc format gives Column has wrong number of index entries found: 0 expected: 1 Key: HIVE-23431 URL: https://issues.apache.org/jira/browse/HIVE-23431 Project: Hive Issue Type: Bug Environment: Hive version 2.3.3 Reporter: Usama Kaleem
I'm writing to hive from NiFi. the table is created and it works fine except for a few records after a few weeks. It gives this error. I have a bunch of failed records piled up. ERROR [put-hive-streaming-0] o.a.h.h.streaming.AbstractRecordWriter Unable to close org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater[hdfs://hdfscluster/user/hive/metastore/bi_sureyield_db.db/bi_sureyield_events/year=2020/month=1/day=10/delta_127318799_127318808/bucket_00000] due to: Column has wrong number of index entries found: 0 expected: 1ERROR [put-hive-streaming-0] o.a.h.h.streaming.AbstractRecordWriter Unable to close org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater[hdfs://hdfscluster/user/hive/metastore/bi_sureyield_db.db/bi_sureyield_events/year=2020/month=1/day=10/delta_127318799_127318808/bucket_00000] due to: Column has wrong number of index entries found: 0 expected: 1java.lang.IllegalArgumentException: Column has wrong number of index entries found: 0 expected: 1 at org.apache.orc.impl.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:695) at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:2147) at org.apache.orc.impl.WriterImpl.flushStripe(WriterImpl.java:2661) at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2834) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:321) at org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.close(OrcRecordUpdater.java:502) at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.closeBatch(AbstractRecordWriter.java:218) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$6.run(HiveEndPoint.java:998) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$6.run(HiveEndPoint.java:995) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.closeImpl(HiveEndPoint.java:994) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:760) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commit(HiveEndPoint.java:854) at org.apache.nifi.util.hive.HiveWriter$4.call(HiveWriter.java:237) at org.apache.nifi.util.hive.HiveWriter$4.call(HiveWriter.java:234) at org.apache.nifi.util.hive.HiveWriter.lambda$null$3(HiveWriter.java:373) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:373) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-29 14:55:39,542 ERROR [put-hive-streaming-0] o.a.h.h.streaming.AbstractRecordWriter Unable to close org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater[hdfs://hdfscluster/user/hive/metastore/bi_sureyield_db.db/bi_sureyield_events/year=2020/month=1/day=10/delta_127318799_127318808/bucket_00001] due to: nulljava.nio.channels.ClosedChannelException: null at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1521) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.orc.impl.PhysicalFsWriter$BufferedStream.spillToDiskAndClear(PhysicalFsWriter.java:286) at org.apache.orc.impl.PhysicalFsWriter.finalizeStripe(PhysicalFsWriter.java:337) at org.apache.orc.impl.WriterImpl.flushStripe(WriterImpl.java:2665) at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2834) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:321) at org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.close(OrcRecordUpdater.java:502) at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.closeBatch(AbstractRecordWriter.java:218) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$6.run(HiveEndPoint.java:998) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$6.run(HiveEndPoint.java:995) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.closeImpl(HiveEndPoint.java:994) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:760) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commit(HiveEndPoint.java:854) at org.apache.nifi.util.hive.HiveWriter$4.call(HiveWriter.java:237) at org.apache.nifi.util.hive.HiveWriter$4.call(HiveWriter.java:234) at org.apache.nifi.util.hive.HiveWriter.lambda$null$3(HiveWriter.java:373) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:373) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)