[
https://issues.apache.org/jira/browse/DRILL-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874263#comment-16874263
]
Stuart Teasdale commented on DRILL-5983:
----------------------------------------
I'm seeing basically the same issue with Drill 1.16.0. I've attached the file
that does this
> select * from test;
Error: INTERNAL_ERROR ERROR: Error in parquet record reader.
Message: Failure in setting up reader
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message schema {
optional int32 chrom_int (INT_8);
optional int64 __index_level_0__;
}
, metadata: \{pandas={"index_columns": ["__index_level_0__"], "column_indexes":
[{"name": null, "field_name": null, "pandas_type": "unicode", "numpy_type":
"object", "metadata": {"encoding": "UTF-8"}}], "columns": [\{"name":
"chrom_int", "field_name": "chrom_int", "pandas_type": "int8", "numpy_type":
"int8", "metadata": null}, \{"name": null, "field_name": "__index_level_0__",
"pandas_type": "int64", "numpy_type": "int64", "metadata": null}],
"pandas_version": "0.24.2"}}}, blocks: [BlockMetaData\{1, 140
[ColumnMetaData{SNAPPY [chrom_int] optional int32 chrom_int (INT_8)
[PLAIN_DICTIONARY, RLE, PLAIN], 24}, ColumnMetaData\{SNAPPY [__index_level_0__]
optional int64 __index_level_0__ [PLAIN_DICTIONARY, RLE, PLAIN], 158}]}]}
Fragment 0:0
Please, refer to logs for more information.
[Error Id: c3e0e2ea-0e51-4732-96af-88ffe669b22c on
mephistopheles.londc.genomicsplc.com:31010] (state=,code=0)
and from the logs:
org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet
record reader.
Message: Failure in setting up reader
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message schema {
optional int32 chrom_int (INT_8);
optional int64 __index_level_0__;
}
, metadata: \{pandas={"index_columns": ["__index_level_0__"], "column_indexes":
[{"name": null, "field_name": null, "pandas_type": "unicode", "numpy_type":
"object", "metadata": {"encoding": "UTF-8"}}], "columns": [\{"name":
"chrom_int", "field_name": "chrom_int", "pandas_type": "int8", "numpy_type":
"int8", "metadata": null}, \{"name": null, "field_name": "__index_level_0__",
"pandas_type": "int64", "numpy_type": "int64", "metadata": null}],
"pandas_version": "0.24.2"}}}, blocks: [BlockMetaData\{1, 140
[ColumnMetaData{SNAPPY [chrom_int] optional int32 chrom_int (INT_8)
[PLAIN_DICTIONARY, RLE, PLAIN], 24}, ColumnMetaData\{SNAPPY [__index_level_0__]
optional int64 __index_level_0__ [PLAIN_DICTIONARY, RLE, PLAIN], 158}]}]}
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException(ParquetRecordReader.java:269)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.setup(ParquetRecordReader.java:253)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas(ScanBatch.java:321)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:216)
~[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:271)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
~[drill-java-exec-1.16.0.jar:1.16.0]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)
~[drill-java-exec-1.16.0.jar:1.16.0]
at .......(:0) ~[na:na]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
~[hadoop-common-2.7.4.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:283)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
~[drill-common-1.16.0.jar:1.16.0]
at .......(:0) ~[na:na]
Caused by: org.apache.drill.common.exceptions.ExecutionSetupException:
Unsupported nullable converted type INT_8 for primitive type INT32
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory.getNullableColumnReader(ColumnReaderFactory.java:288)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReaderFactory.createFixedColumnReader(ColumnReaderFactory.java:203)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetColumnMetadata.makeFixedWidthReader(ParquetColumnMetadata.java:141)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.parquet.columnreaders.ReadState.buildReader(ReadState.java:123)
~[drill-java-exec-1.16.0.jar:1.16.0]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.setup(ParquetRecordReader.java:251)
~[drill-java-exec-1.16.0.jar:1.16.0]
... 18 common frames omitted
2019-06-27 16:34:19,439 [22eb1e03-b923-c996-d78e-90759f23981a:frag:0:0] WARN
o.a.d.e.w.f.QueryStateProcessor - Dropping request to move to COMPLETED state
as query is already at FAILED state (which is terminal).
> Unsupported nullable converted type INT_8 for primitive type INT32 error
> ------------------------------------------------------------------------
>
> Key: DRILL-5983
> URL: https://issues.apache.org/jira/browse/DRILL-5983
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
> Affects Versions: 1.10.0, 1.11.0
> Environment: NAME="Ubuntu"
> VERSION="16.04.2 LTS (Xenial Xerus)"
> Reporter: Hakan Sarıbıyık
> Priority: Major
> Labels: parquet, read, types
> Attachments: test.parquet
>
>
> When I query a table with byte in it, then it gives an error;
> _Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> ExecutionSetupException: Unsupported nullable converted type INT_8 for
> primitive type INT32 Fragment 1:6 [Error Id:
> 46636b05-cff5-455b-ba25-527217346b3e on bigdata7:31010]_
> Actualy, it has been solved with
> [DRILL-4764] - Parquet file with INT_16, etc. logical types not supported by
> simple SELECT
> according to https://drill.apache.org/docs/apache-drill-1-10-0-release-notes/
> But i tried it with even 1-11-0 it didnt worked.
> I am querying parquet formatted file with pySpark
> tablo1
> sourceid: byte (nullable = true)
> select sourceid from tablo1
> works as expected with pySpark. But not with Drill v1.11.0
> Thanx.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
