[jira] [Updated] (SPARK-23340) Empty float/double array columns in ORC file should not raise EOFException
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23340: -- Description: This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 more patches including bug fixes (https://s.apache.org/Fll8). Especially, the following ORC-285 is fixed at 1.4.3. {code} scala> val df = Seq(Array.empty[Float]).toDF() scala> df.write.format("orc").save("/tmp/floatarray") scala> spark.read.orc("/tmp/floatarray") res1: org.apache.spark.sql.DataFrame = [value: array] scala> spark.read.orc("/tmp/floatarray").show() 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.io.IOException: Error reading file: file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) ... Caused by: java.io.EOFException: Read past EOF for compressed stream Stream for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 {code} was: This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 more patches (https://s.apache.org/Fll8). Especially, the following ORC-285 is fixed at 1.4.3. {code} scala> val df = Seq(Array.empty[Float]).toDF() scala> df.write.format("orc").save("/tmp/floatarray") scala> spark.read.orc("/tmp/floatarray") res1: org.apache.spark.sql.DataFrame = [value: array] scala> spark.read.orc("/tmp/floatarray").show() 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.io.IOException: Error reading file: file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) ... Caused by: java.io.EOFException: Read past EOF for compressed stream Stream for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 {code} > Empty float/double array columns in ORC file should not raise EOFException > -- > > Key: SPARK-23340 > URL: https://issues.apache.org/jira/browse/SPARK-23340 > Project: Spark > Issue Type: Bug > Components: Build, SQL >Affects Versions: 2.3.0 >Reporter: Dongjoon Hyun >Priority: Critical > > This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. > Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 > more patches including bug fixes (https://s.apache.org/Fll8). > Especially, the following ORC-285 is fixed at 1.4.3. > {code} > scala> val df = Seq(Array.empty[Float]).toDF() > scala> df.write.format("orc").save("/tmp/floatarray") > scala> spark.read.orc("/tmp/floatarray") > res1: org.apache.spark.sql.DataFrame = [value: array] > scala> spark.read.orc("/tmp/floatarray").show() > 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) > java.io.IOException: Error reading file: > file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc > at > org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) > at > org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) > ... > Caused by: java.io.EOFException: Read past EOF for compressed stream Stream > for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23340) Empty float/double array columns in ORC file should not raise EOFException
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23340: -- Description: This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 more patches (https://s.apache.org/Fll8). Especially, the following ORC-285 is fixed at 1.4.3. {code} scala> val df = Seq(Array.empty[Float]).toDF() scala> df.write.format("orc").save("/tmp/floatarray") scala> spark.read.orc("/tmp/floatarray") res1: org.apache.spark.sql.DataFrame = [value: array] scala> spark.read.orc("/tmp/floatarray").show() 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.io.IOException: Error reading file: file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) ... Caused by: java.io.EOFException: Read past EOF for compressed stream Stream for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 {code} was: This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 more patches (https://s.apache.org/Fll8). Especially, the following ORC-285 is fixed at 1.4.3. {code} scala> df.write.format("orc").save("/tmp/floatarray") scala> spark.read.orc("/tmp/floatarray") res1: org.apache.spark.sql.DataFrame = [value: array] scala> spark.read.orc("/tmp/floatarray").show() 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.io.IOException: Error reading file: file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) ... Caused by: java.io.EOFException: Read past EOF for compressed stream Stream for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 {code} > Empty float/double array columns in ORC file should not raise EOFException > -- > > Key: SPARK-23340 > URL: https://issues.apache.org/jira/browse/SPARK-23340 > Project: Spark > Issue Type: Bug > Components: Build, SQL >Affects Versions: 2.3.0 >Reporter: Dongjoon Hyun >Priority: Critical > > This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. > Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 > more patches (https://s.apache.org/Fll8). > Especially, the following ORC-285 is fixed at 1.4.3. > {code} > scala> val df = Seq(Array.empty[Float]).toDF() > scala> df.write.format("orc").save("/tmp/floatarray") > scala> spark.read.orc("/tmp/floatarray") > res1: org.apache.spark.sql.DataFrame = [value: array] > scala> spark.read.orc("/tmp/floatarray").show() > 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) > java.io.IOException: Error reading file: > file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc > at > org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) > at > org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) > ... > Caused by: java.io.EOFException: Read past EOF for compressed stream Stream > for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23340) Empty float/double array columns in ORC file should not raise EOFException
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23340: -- Summary: Empty float/double array columns in ORC file should not raise EOFException (was: Empty float/double array columns in ORC file raise EOFException) > Empty float/double array columns in ORC file should not raise EOFException > -- > > Key: SPARK-23340 > URL: https://issues.apache.org/jira/browse/SPARK-23340 > Project: Spark > Issue Type: Bug > Components: Build, SQL >Affects Versions: 2.3.0 >Reporter: Dongjoon Hyun >Priority: Critical > > This issue updates Apache ORC dependencies to 1.4.3 released on February 9th. > Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 > more patches (https://s.apache.org/Fll8). > Especially, the following ORC-285 is fixed at 1.4.3. > {code} > scala> df.write.format("orc").save("/tmp/floatarray") > scala> spark.read.orc("/tmp/floatarray") > res1: org.apache.spark.sql.DataFrame = [value: array] > scala> spark.read.orc("/tmp/floatarray").show() > 18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) > java.io.IOException: Error reading file: > file:/tmp/floatarray/part-0-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc > at > org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191) > at > org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) > ... > Caused by: java.io.EOFException: Read past EOF for compressed stream Stream > for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org