[
https://issues.apache.org/jira/browse/DRILL-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6990:
------------------------------------
Fix Version/s: 1.17.0
> IllegalStateException: The current reader doesn't support getting next
> information
> ----------------------------------------------------------------------------------
>
> Key: DRILL-6990
> URL: https://issues.apache.org/jira/browse/DRILL-6990
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.14.0
> Reporter: Khurram Faraaz
> Assignee: Igor Guzenko
> Priority: Major
> Fix For: 1.17.0
>
> Attachments: parqt_nestedArray.parquet.tar
>
>
> Reading a parquet file created from Spark, returns IllegalStateException: The
> current reader doesn't support getting next information
> Drill 1.14.0, parquet file created from Spark is attached here.
> //Steps to create parquet file from Spark 2.3.1
> [root@ba102-495 ~]# cd /opt/mapr/spark/spark-2.3.1
> [root@ba102-495 spark-2.3.1]# cd bin
> [root@ba102-495 bin]# ./spark-shell
> 19/01/21 22:57:05 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Spark context Web UI available at http://qa102-45.qa.lab:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1548111430809).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
> /_/
> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import spark.implicits._
> import spark.implicits._
> scala> val df = spark.read.json("/apps/nestedDataJson.json")
> df: org.apache.spark.sql.DataFrame = [id: bigint, nested_array:
> array<array<bigint>>]
> scala> df.write.parquet("/apps/parqt_nestedArray.parquet")
> Data used in test
> {noformat}
> [root@ba102-495 ~]# cat nestedDataJson.json
> {"id":19,"nested_array":[[1,2,3,4],[5,6,7,8],[9,10,12]]}
> {"id":14121,"nested_array":[[1,3,4],[5,6,8],[9,11,12]]}
> {"id":18894,"nested_array":[[1,3,4],[5,6,7,8],[9,10,11,12]]}
> {"id":12499,"nested_array":[[1,4],[5,7,8],[9,11,12]]}
> {"id":120,"nested_array":[[1,4],[5,7,8],[9,10,11,12]]}
> {"id":12,"nested_array":[[1,2,3,4],[5,6,7,8],[11,12]]}
> {"id":13,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
> {"id":14,"nested_array":[[1,2,3,4],[5,68],[9,10,11,12]]}
> {"id":123,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
> {"id":124,"nested_array":[[1,2,4],[5,6,7,8],[9,10,11,12]]}
> {"id":134,"nested_array":[[1,4],[5,8],[9,12]]}
> {noformat}
> From drillbit.log
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> IllegalStateException: The current reader doesn't support getting next
> information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on
> ba102-495.qa.lab:31010]
> ...
> ...
> 2019-01-21 23:08:11,268 [23b9af24-10b9-ad11-5583-ecc3e0c562e6:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
> The current reader doesn't support getting next information.
> Fragment 0:0
> [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: The current reader doesn't support getting next
> information.
> Fragment 0:0
> [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
> [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
> [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
> [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.14.0-mapr.jar:1.14.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
> Caused by: java.lang.IllegalStateException: The current reader doesn't
> support getting next information.
> at
> org.apache.drill.exec.vector.complex.impl.AbstractBaseReader.next(AbstractBaseReader.java:64)
> ~[vector-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.next(SingleMapReaderImpl.java:31)
> ~[vector-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.test.generated.ProjectorGen971.doEval(ProjectorTemplate.java:35)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.ProjectorGen971.projectRecords(ProjectorTemplate.java:67)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:231)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:294)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:281)
> ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_181]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_181]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> ~[hadoop-common-2.7.0-mapr-1808.jar:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:281)
> [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
> ... 4 common frames omitted
> {noformat}
> Doing a project of nested_array column from the parquet file returns
> incorrect data, the representation of the nested array is wrong.
> {noformat}
> apache drill 1.14.0-mapr
> "start your sql engine"
> 0: jdbc:drill:schema=dfs.tmp> select nested_array from
> dfs.`/apps/parqt_nestedArray.parquet`;
> +--------------+
> | nested_array |
> +--------------+
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":2},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":6},\{"element":7},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":6},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":6},\{"element":7},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":7},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":7},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":2},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":6},\{"element":7},\{"element":8}]}},\{"element":{"list":[{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":2},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":2},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":68}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":2},\{"element":3},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":2},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":6},\{"element":7},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":10},\{"element":11},\{"element":12}]}}]}
> |
> |
> \{"list":[{"element":{"list":[{"element":1},\{"element":4}]}},\{"element":{"list":[{"element":5},\{"element":8}]}},\{"element":{"list":[{"element":9},\{"element":12}]}}]}
> |
> +--------------+
> 11 rows selected (0.293 seconds)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
