Issue DRILL-4184 is created to track this. A small sample parquet file is attached that can be used to reproduce it.
Dave Oshinsky -----Original Message----- From: Dave Oshinsky [mailto:[email protected]] Sent: Wednesday, December 09, 2015 1:35 PM To: [email protected] Subject: query failure on DECIMAL field of parquet file Has anyone seen a failure as shown below my signature? I have seen it with Drill 1.3.0 and 1.2.0. The DECIMAL values are encoded in big-endian byte order, as converted from java.math.BigInteger. Dave Oshinsky Commvault Systems Here is a successful query on a UTF8 field in the same parquet file: 0: jdbc:drill:zk=local> select count(*) from dfs.`c:/dao/DBArchivePredictor/acct_mstr.parquet` where opr_mode='JO'; +-----------+ | EXPR$0 | +-----------+ | 37050847 | +-----------+ 1 row selected (9.934 seconds) 0: jdbc:drill:zk=local> Here is a query that fails, on a DECIMAL field acct_no: 0: jdbc:drill:zk=local> select count(*) from dfs.`c:/dao/DBArchivePredictor/acct_mstr.parquet` where acct_no=70000020; ........ Error: SYSTEM ERROR: ClassCastException: org.apache.drill.exec.vector.Decimal28S parseVector cannot be cast to org.apache.drill.exec.vector.VariableWidthVector Fragment 1:0 [Error Id: bfde2a91-2c86-48b3-b188-7a7bf3d8e6ac on DaveOshinsky-PC.gp.cv.commvau lt.com:31010] (state=,code=0) 0: jdbc:drill:zk=local> The output is quite lengthy from the failure, including the following: org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet recor d reader. Message: Failure in setting up reader Parquet Metadata: ParquetMetaData{FileMetaData{schema: message my_table { required binary ACCT_NO (DECIMAL(20,0)); optional binary SF_NO (UTF8); optional binary LF_NO (UTF8); optional binary BRANCH_NO (DECIMAL(20,0)); optional binary INTRO_CUST_NO (DECIMAL(20,0)); optional binary INTRO_ACCT_NO (DECIMAL(20,0)); optional binary INTRO_SIGN (UTF8); optional binary TYPE (UTF8); optional binary OPR_MODE (UTF8); optional binary CUR_ACCT_TYPE (UTF8); optional binary TITLE (UTF8); optional binary CORP_CUST_NO (DECIMAL(20,0)); optional binary APLNDT (UTF8); optional binary OPNDT (UTF8); optional binary VERI_EMP_NO (DECIMAL(20,0)); optional binary VERI_SIGN (UTF8); optional binary MANAGER_SIGN (UTF8); optional binary CURBAL (DECIMAL(8,2)); optional binary STATUS (UTF8); } ...... blocks: [BlockMetaData{12710100, 1108111133 [ColumnMetaData{SNAPPY [ACCT_NO] BINA RY [BIT_PACKED, PLAIN], 4}, ColumnMetaData{SNAPPY [SF_NO] BINARY [RLE, BIT_PAC KED, PLAIN_DICTIONARY], 56990517}, ColumnMetaData{SNAPPY [LF_NO] BINARY [RLE, B IT_PACKED, PLAIN_DICTIONARY], 57125602}, ColumnMetaData{SNAPPY [BRANCH_NO] BINAR Y [RLE, BIT_PACKED, PLAIN_DICTIONARY], 57233332}, ColumnMetaData{SNAPPY [INTRO_ CUST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 57281915}, ColumnMe taData{SNAPPY [INTRO_ACCT_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 89933 829}, ColumnMetaData{SNAPPY [INTRO_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTION ARY], 110250177}, ColumnMetaData{SNAPPY [TYPE] BINARY [RLE, BIT_PACKED, PLAIN_D ICTIONARY], 110291904}, ColumnMetaData{SNAPPY [OPR_MODE] BINARY [RLE, BIT_PACKE D, PLAIN_DICTIONARY], 110346542}, ColumnMetaData{SNAPPY [CUR_ACCT_TYPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 110406593}, ColumnMetaData{SNAPPY [TITLE] B INARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 110467110}, ColumnMetaData{SNAPPY [C ORP_CUST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 110628878}, Col umnMetaData{SNAPPY [APLNDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 143290986}, ColumnMetaData{SNAPPY [OPNDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIO NARY, PLAIN], 158585878}, ColumnMetaData{SNAPPY [VERI_EMP_NO] BINARY [RLE, BIT_ PACKED, PLAIN_DICTIONARY, PLAIN], 173880774}, ColumnMetaData{SNAPPY [VERI_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 205251396}, ColumnMetaData{SNAPPY [ MANAGER_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 205294552}, ColumnMet aData{SNAPPY [CURBAL] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 205337 708}, ColumnMetaData{SNAPPY [STATUS] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY] , 268041364}]}, BlockMetaData{16740100, 877627388 [ColumnMetaData{SNAPPY [ACCT_N O] BINARY [BIT_PACKED, PLAIN], 268084308}, ColumnMetaData{SNAPPY [SF_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 339879714}, ColumnMetaData{SNAPPY [LF_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 340048465}, ColumnMetaData{SNAPPY [BRANCH_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 340178908}, ColumnMetaD ata{SNAPPY [INTRO_CUST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 34023528 0}, ColumnMetaData{SNAPPY [INTRO_ACCT_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIO NARY], 361297427}, ColumnMetaData{SNAPPY [INTRO_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 382437967}, ColumnMetaData{SNAPPY [TYPE] BINARY [RLE, BIT_PA CKED, PLAIN_DICTIONARY], 382492898}, ColumnMetaData{SNAPPY [OPR_MODE] BINARY [R LE, BIT_PACKED, PLAIN_DICTIONARY], 382563576}, ColumnMetaData{SNAPPY [CUR_ACCT_T YPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 382632643}, ColumnMetaData{SNA PPY [TITLE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 382703353}, ColumnMetaD ata{SNAPPY [CORP_CUST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 383347416 }, ColumnMetaData{SNAPPY [APLNDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PL AIN], 404409563}, ColumnMetaData{SNAPPY [OPNDT] BINARY [RLE, BIT_PACKED, PLAIN_ DICTIONARY, PLAIN], 418195496}, ColumnMetaData{SNAPPY [VERI_EMP_NO] BINARY [RLE , BIT_PACKED, PLAIN_DICTIONARY], 431981127}, ColumnMetaData{SNAPPY [VERI_SIGN] B INARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 451668116}, ColumnMetaData{SNAPPY [M ANAGER_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 451723163}, ColumnMeta Data{SNAPPY [CURBAL] BINARY [RLE, BIT_PACKED, PLAIN], 451779553}, ColumnMetaDat a{SNAPPY [STATUS] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 535811598}]}, Blo ckMetaData{15390100, 1010393203 [ColumnMetaData{SNAPPY [ACCT_NO] BINARY [BIT_PA CKED, PLAIN], 535866529}, ColumnMetaData{SNAPPY [SF_NO] BINARY [RLE, BIT_PACKED , PLAIN_DICTIONARY], 603471354}, ColumnMetaData{SNAPPY [LF_NO] BINARY [RLE, BIT _PACKED, PLAIN_DICTIONARY], 603636501}, ColumnMetaData{SNAPPY [BRANCH_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 603756535}, ColumnMetaData{SNAPPY [INTRO_ CUST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 603807077}, ColumnMetaData {SNAPPY [INTRO_ACCT_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 623171740}, ColumnMetaData{SNAPPY [INTRO_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 642536380}, ColumnMetaData{SNAPPY [TYPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIO NARY], 642586922}, ColumnMetaData{SNAPPY [OPR_MODE] BINARY [RLE, BIT_PACKED, PL AIN_DICTIONARY], 642652002}, ColumnMetaData{SNAPPY [CUR_ACCT_TYPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 642717093}, ColumnMetaData{SNAPPY [TITLE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 642782179}, ColumnMetaData{SNAPPY [CORP_C UST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 651988784}, ColumnMetaData{ SNAPPY [APLNDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 671353447}, ColumnMetaData{SNAPPY [OPNDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN] , 689248030}, ColumnMetaData{SNAPPY [VERI_EMP_NO] BINARY [RLE, BIT_PACKED, PLAI N_DICTIONARY], 707142273}, ColumnMetaData{SNAPPY [VERI_SIGN] BINARY [RLE, BIT_P ACKED, PLAIN_DICTIONARY], 726506776}, ColumnMetaData{SNAPPY [MANAGER_SIGN] BINAR Y [RLE, BIT_PACKED, PLAIN_DICTIONARY], 726558408}, ColumnMetaData{SNAPPY [CURBA L] BINARY [RLE, BIT_PACKED, PLAIN], 726610040}, ColumnMetaData{SNAPPY [STATUS] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 803868149}]}, BlockMetaData{1487010 0, 997915332 [ColumnMetaData{SNAPPY [ACCT_NO] BINARY [BIT_PACKED, PLAIN], 80391 9875}, ColumnMetaData{SNAPPY [SF_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY] , 869264876}, ColumnMetaData{SNAPPY [LF_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICT IONARY], 869417004}, ColumnMetaData{SNAPPY [BRANCH_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 869532661}, ColumnMetaData{SNAPPY [INTRO_CUST_NO] BINARY [R LE, BIT_PACKED, PLAIN_DICTIONARY], 876919008}, ColumnMetaData{SNAPPY [INTRO_ACCT _NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 895628500}, ColumnMetaData{SNA PPY [INTRO_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 915616293}, Column MetaData{SNAPPY [TYPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 915665431}, ColumnMetaData{SNAPPY [OPR_MODE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 91 5728109}, ColumnMetaData{SNAPPY [CUR_ACCT_TYPE] BINARY [RLE, BIT_PACKED, PLAIN_ DICTIONARY], 915790793}, ColumnMetaData{SNAPPY [TITLE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 915853477}, ColumnMetaData{SNAPPY [CORP_CUST_NO] BINARY [RL E, BIT_PACKED, PLAIN_DICTIONARY], 924595366}, ColumnMetaData{SNAPPY [APLNDT] BIN ARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 943304858}, ColumnMetaData{SNAP PY [OPNDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 960868676}, Colum nMetaData{SNAPPY [VERI_EMP_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 9784 32494}, ColumnMetaData{SNAPPY [VERI_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIO NARY], 997142542}, ColumnMetaData{SNAPPY [MANAGER_SIGN] BINARY [RLE, BIT_PACKED , PLAIN_DICTIONARY], 997192537}, ColumnMetaData{SNAPPY [CURBAL] BINARY [RLE, BI T_PACKED, PLAIN], 997242440}, ColumnMetaData{SNAPPY [STATUS] BINARY [RLE, BIT_P ACKED, PLAIN_DICTIONARY], 1071891224}]}, BlockMetaData{7154452, 447096168 [Colum nMetaData{SNAPPY [ACCT_NO] BINARY [BIT_PACKED, PLAIN], 1071941219}, ColumnMetaD ata{SNAPPY [SF_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 1103384569}, Col umnMetaData{SNAPPY [LF_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 11034553 36}, ColumnMetaData{SNAPPY [BRANCH_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONAR Y], 1103509961}, ColumnMetaData{SNAPPY [INTRO_CUST_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 1107598352}, ColumnMetaData{SNAPPY [INTRO_ACCT_NO] BINARY [ RLE, BIT_PACKED, PLAIN_DICTIONARY], 1116601592}, ColumnMetaData{SNAPPY [INTRO_SI GN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 1125605021}, ColumnMetaData{SNA PPY [TYPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 1125628519}, ColumnMetaD ata{SNAPPY [OPR_MODE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 1125658015}, ColumnMetaData{SNAPPY [CUR_ACCT_TYPE] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY ], 1125687511}, ColumnMetaData{SNAPPY [TITLE] BINARY [RLE, BIT_PACKED, PLAIN_DI CTIONARY], 1125717007}, ColumnMetaData{SNAPPY [CORP_CUST_NO] BINARY [RLE, BIT_P ACKED, PLAIN_DICTIONARY], 1130010136}, ColumnMetaData{SNAPPY [APLNDT] BINARY [R LE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 1139013376}, ColumnMetaData{SNAPPY [OP NDT] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY, PLAIN], 1146672810}, ColumnMeta Data{SNAPPY [VERI_EMP_NO] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY], 115433224 4}, ColumnMetaData{SNAPPY [VERI_SIGN] BINARY [RLE, BIT_PACKED, PLAIN_DICTIONARY ], 1163335628}, ColumnMetaData{SNAPPY [MANAGER_SIGN] BINARY [RLE, BIT_PACKED, P LAIN_DICTIONARY], 1163359126}, ColumnMetaData{SNAPPY [CURBAL] BINARY [RLE, BIT_ PACKED, PLAIN], 1163382624}, ColumnMetaData{SNAPPY [STATUS] BINARY [RLE, BIT_PA CKED, PLAIN_DICTIONARY], 1199297307}]}]} at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader .handleAndRaise(ParquetRecordReader.java:346) at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader .setup(ParquetRecordReader.java:339) at org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:1 01) at org.apache.drill.exec.store.parquet.ParquetScanBatchCreator.getBatch( ParquetScanBatchCreator.java:168) at org.apache.drill.exec.store.parquet.ParquetScanBatchCreator.getBatch( ParquetScanBatchCreator.java:56) at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCr eator.java:151) at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreat or.java:174) at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCr eator.java:131) at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreat or.java:174) at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCr eator.java:131) at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreat or.java:174) at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCr eator.java:131) at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreat or.java:174) at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCr eator.java:131) at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreat or.java:174) at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreat or.java:105) at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.j ava:79) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExec utor.java:230) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable .java:38) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassCastException: org.apache.drill.exec.vector.Decimal28S parseVector cannot be cast to org.apache.drill.exec.vector.VariableWidthVector at org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColu mn.<init>(VarLengthValuesColumn.java:44) at org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnRead ers$Decimal28Column.<init>(VarLengthColumnReaders.java:52) at org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory .getReader(ColumnReaderFactory.java:178) at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader .setup(ParquetRecordReader.java:319) ... 20 more Error: SYSTEM ERROR: ClassCastException: org.apache.drill.exec.vector.Decimal28S parseVector cannot be cast to org.apache.drill.exec.vector.VariableWidthVector Fragment 1:0 [Error Id: bfde2a91-2c86-48b3-b188-7a7bf3d8e6ac on DaveOshinsky-PC.gp.cv.commvau lt.com:31010] (state=,code=0) 0: jdbc:drill:zk=local> ***************************Legal Disclaimer*************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. Thank you." ********************************************************************** ***************************Legal Disclaimer*************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. Thank you." **********************************************************************
