[
https://issues.apache.org/jira/browse/DRILL-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka reopened DRILL-5495:
------------------------------------
Looks like the issue is gone for
{{NullableFixedByteAlignedReaders.NullableFixedBinaryAsTimeStampReader}}
{code}set `store.parquet.reader.int96_as_timestamp` = true;{code}
but still exists for
{{NullableFixedByteAlignedReaders.NullableFixedBinaryReader}}
{code}set `store.parquet.reader.int96_as_timestamp` = false;{code}
{code}
0: jdbc:drill:zk=local> select convert_from(d, 'TIMESTAMP_IMPALA') from
dfs.`/home/vitalii/Downloads/d4`;
........
........
| 2011-06-19 19:10:02.0 |
| 2011-06-19 19:10:02.0 |
| 2011-06-19 19:10:02.0 |
| 2011-06-19 19:10:02.0 |
Error: INTERNAL_ERROR ERROR: Error in parquet record reader.
Message:
Hadoop path:
/home/vitalii/Downloads/d4/part-r-00003-08c5c621-62ea-4fee-b690-11576eddc39c.snappy.parquet
Total records read: 0
Row group index: 0
Records in row group: 10000
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message spark_schema {
optional int32 a;
optional binary b (UTF8);
optional int32 c (DATE);
optional int96 d;
}
, metadata:
\{org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"a","type":"integer","nullable":true,"metadata":{}},\{"name":"b","type":"string","nullable":true,"metadata":{}},\{"name":"c","type":"date","nullable":true,"metadata":{}},\{"name":"d","type":"timestamp","nullable":true,"metadata":{}}]}}},
blocks: [BlockMetaData\{10000, 8627 [ColumnMetaData{SNAPPY [a] optional int32
a [BIT_PACKED, PLAIN, RLE], 4}, ColumnMetaData\{SNAPPY [b] optional binary b
(UTF8) [BIT_PACKED, PLAIN_DICTIONARY, RLE], 2348}, ColumnMetaData\{SNAPPY [c]
optional int32 c (DATE) [BIT_PACKED, PLAIN, RLE], 4578}, ColumnMetaData\{SNAPPY
[d] optional int96 d [BIT_PACKED, PLAIN_DICTIONARY, RLE], 5822}]}]}
Fragment 1:0
[Error Id: 7cae35f2-eade-4e12-aaca-25aaf11e63d5 on vitalii-pc:31010]
(state=,code=0)
{code}
> convert_from function on top of int96 data results in
> ArrayIndexOutOfBoundsException
> ------------------------------------------------------------------------------------
>
> Key: DRILL-5495
> URL: https://issues.apache.org/jira/browse/DRILL-5495
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.10.0
> Reporter: Rahul Challapalli
> Assignee: Vitalii Diravka
> Priority: Major
> Fix For: 1.14.0
>
> Attachments: 26edf56f-6bc6-1e1f-5aa4-d98aec858a4a.sys.drill,
> d4.tar.gz, drillbit.log
>
>
> git.commit.id.abbrev=1e0a14c
> The data set used is generated from spark and contains a timestamp stored as
> int96
> {code}
> [root@qa-node190 framework]# /home/parquet-tools-1.5.1-SNAPSHOT/parquet-meta
> /home/framework/framework/resources/Datasources/parquet_date/spark_generated/d4/part-r-00000-08c5c621-62ea-4fee-b690-11576eddc39c.snappy.parquet
>
> creator: parquet-mr (build 32c46643845ea8a705c35d4ec8fc654cc8ff816d)
> extra: org.apache.spark.sql.parquet.row.metadata =
> {"type":"struct","fields":[{"name":"a","type":"integer","nullable":true,"metadata":{}},{"name":"b","type":"strin
> [more]...
> file schema: spark_schema
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> a: OPTIONAL INT32 R:0 D:1
> b: OPTIONAL BINARY O:UTF8 R:0 D:1
> c: OPTIONAL INT32 O:DATE R:0 D:1
> d: OPTIONAL INT96 R:0 D:1
> row group 1: RC:10000 TS:8661
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> a: INT32 SNAPPY DO:0 FPO:4 SZ:2367/2571/1.09 VC:10000
> ENC:RLE,PLAIN,BIT_PACKED
> b: BINARY SNAPPY DO:0 FPO:2371 SZ:2329/2843/1.22 VC:10000
> ENC:RLE,PLAIN_DICTIONARY,BIT_PACKED
> c: INT32 SNAPPY DO:0 FPO:4700 SZ:1374/1507/1.10 VC:10000
> ENC:RLE,PLAIN,BIT_PACKED
> d: INT96 SNAPPY DO:0 FPO:6074 SZ:1597/1740/1.09 VC:10000
> ENC:RLE,PLAIN_DICTIONARY,BIT_PACKED
> {code}
> The below query fails with an ArrayIndexOutOfBoundsException
> {code}
> select convert_from(d, 'TIMESTAMP_IMPALA') from
> dfs.`/drill/testdata/resource-manager/d4`;
> Fails with below error after displaying a bunch of records
> Error: SYSTEM ERROR: ArrayIndexOutOfBoundsException: 0
> Fragment 1:0
> [Error Id: f963f6c0-3306-49a6-9d98-a193c5e7cfee on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Attached the logs, profiles and data files
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
