[
https://issues.apache.org/jira/browse/SPARK-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15113641#comment-15113641
]
Sam Stoelinga edited comment on SPARK-12947 at 1/23/16 9:16 AM:
----------------------------------------------------------------
I'm using Ceph with Swift compatible front-end. It's deployed on 2 VMs + 1 phys
server which run OpenStack controllers and RadosGW (Lab env non-production
env). I will check if I can find anything in the RadosGW logs.
I have not done any benchmarking of the Ceph cluster so not sure about ability
to handle Swift requests.
I'm using Spark-1.6.0 snapshot built from source non-official 1.6.0 release
with Hadoop 2.6 baked in I believe.
Edit: There are indeed some errors in RadosGW so it seems to be a Swift/Ceph
issue:
2016-01-23 09:14:15.376036 7fb68efed700 1 ====== starting new request
req=0x7fb6980516c0 =====
2016-01-23 09:14:15.414023 7fb68efed700 0 validated token: spark:spark
expires: 1453543940
2016-01-23 09:14:15.432711 7fb68efed700 0 RGWObjManifest::operator++():
result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
2016-01-23 09:14:15.432814 7fb68efed700 0 RGWObjManifest::operator++():
result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0
2016-01-23 09:14:15.432871 7fb68efed700 0 RGWObjManifest::operator++():
result: ofs=13107200 stripe_ofs=13107200 part_ofs=0 rule->part_size=0
2016-01-23 09:14:15.432921 7fb68efed700 0 RGWObjManifest::operator++():
result: ofs=17301504 stripe_ofs=17301504 part_ofs=0 rule->part_size=0
2016-01-23 09:14:15.592533 7fb68efed700 0 RGWObjManifest::operator++():
result: ofs=21495808 stripe_ofs=21495808 part_ofs=0 rule->part_size=0
2016-01-23 09:14:15.650454 7fb68efed700 0 RGWObjManifest::operator++():
result: ofs=21778686 stripe_ofs=21778686 part_ofs=0 rule->part_size=0
2016-01-23 09:14:15.653057 7fb68efed700 0 ERROR: flush_read_list():
d->client_c->handle_data() returned -1
was (Author: samos123):
I'm using Ceph with Swift compatible front-end. It's deployed on 2 VMs + 1 phys
server which run OpenStack controllers and RadosGW (Lab env non-production
env). I will check if I can find anything in the RadosGW logs.
I have not done any benchmarking of the Ceph cluster so not sure about ability
to handle Swift requests.
I'm using Spark-1.6.0 snapshot built from source non-official 1.6.0 release
with Hadoop 2.6 baked in I believe.
> Spark with Swift throws EOFException when reading parquet file
> --------------------------------------------------------------
>
> Key: SPARK-12947
> URL: https://issues.apache.org/jira/browse/SPARK-12947
> Project: Spark
> Issue Type: Bug
> Components: ML, SQL
> Affects Versions: 1.6.0
> Environment: Spark 1.6.0-SNAPSHOT
> Reporter: Sam Stoelinga
>
> I'm using Swift as underlying storage for my spark jobs but it sometimes
> throws EOFExceptions for some parts of the data.
> Another user has hit the same issue:
> http://stackoverflow.com/questions/32400137/spark-swift-integration-parquet
> Code to reproduce:
> ```
> val features = sqlContext.read.parquet(featurePath)
> // Flatten the features into the array exploded
> val exploded =
> features.select(explode(features("features"))).toDF("features")
> val kmeans = new KMeans()
> .setK(k)
> .setFeaturesCol("features")
> .setPredictionCol("prediction")
> val model = kmeans.fit(exploded)
> ```
> val features is a dataframe with 2 columns:
> image: String, features: Array[Vector]
> val exploded is a dataframe with a single column:
> features: Vector
> The following exception is shown when running takeSample on a large dataset
> saved as parquet file (~1+GB):
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at
> org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:756)
> at
> org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:494)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127)
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
> at
> org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:168)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:350)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$30$$anon$1.hasNext(RDD.scala:827)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1563)
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1119)
> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1119)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1840)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1840)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
