[ https://issues.apache.org/jira/browse/KUDU-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245380#comment-16245380 ]
Andrew Ya edited comment on KUDU-2210 at 11/9/17 8:41 AM: ---------------------------------------------------------- We are using kudu-spark_2.10-1.2.0.jar Logs look like {code} 17/11/03 10:15:55 INFO executor.Executor: Running task 93.0 in stage 1.0 (TID 92) 17/11/03 10:17:27 INFO executor.Executor: Finished task 93.0 in stage 1.0 (TID 92). 1145 bytes result sent to driver 17/11/03 10:17:27 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 104 17/11/03 10:17:27 INFO executor.Executor: Running task 103.0 in stage 1.0 (TID 104) 17/11/03 10:19:00 INFO executor.Executor: Finished task 103.0 in stage 1.0 (TID 104). 1145 bytes result sent to driver 17/11/03 10:19:00 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 113 17/11/03 10:19:00 INFO executor.Executor: Running task 118.0 in stage 1.0 (TID 113) 17/11/03 10:21:43 INFO executor.Executor: Finished task 118.0 in stage 1.0 (TID 113). 1145 bytes result sent to driver 17/11/03 10:21:43 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 129 17/11/03 10:21:43 INFO executor.Executor: Running task 131.0 in stage 1.0 (TID 129) 17/11/03 10:25:03 INFO executor.Executor: Finished task 131.0 in stage 1.0 (TID 129). 1145 bytes result sent to driver 17/11/03 10:25:03 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 139 17/11/03 10:25:03 INFO executor.Executor: Running task 142.0 in stage 1.0 (TID 139) 17/11/07 09:54:59 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown 17/11/07 09:54:59 INFO storage.MemoryStore: MemoryStore cleared {code} was (Author: andrew_ya): We are using kudu-spark_2.10-1.2.0.jar > Apache Spark stucks while reading Kudu table. > --------------------------------------------- > > Key: KUDU-2210 > URL: https://issues.apache.org/jira/browse/KUDU-2210 > Project: Kudu > Issue Type: Bug > Components: client, perf, spark > Reporter: Andrew Ya > > When I try reading Kudu table with Apache Spark using following code > {code} > import org.apache.kudu.spark.kudu._ > import sqlContext.implicits._ > val kuduOptions: Map[String, String] = Map( > "kudu.table" -> "test_table", > "kudu.master" -> "host1:7051,host2:7051,host3:7051") > val kuduDF = sqlContext.read.options(kuduOptions).kudu > kuduDF.registerTempTable("t") > sqlContext.sql(" SELECT * FROM t where id in (1111,2222) ").show(50, false) > {code} > after completing 95% of tasks the job stucks for more than three days. The > table is partitioned by date and partitions have uneven size. Table have one > partition 12 Gb size, about 20 partitions with size between 1 Gb and 3 Gb and > some partitions with Mb's and kb's of data. -- This message was sent by Atlassian JIRA (v6.4.14#64029)