[ 
https://issues.apache.org/jira/browse/KUDU-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245380#comment-16245380
 ] 

Andrew Ya edited comment on KUDU-2210 at 11/9/17 8:50 AM:
----------------------------------------------------------

We are using spark 1.6.0 and kudu-spark_2.10-1.2.0.jar
Logs look like
{code}
17/11/03 10:15:55 INFO executor.Executor: Running task 93.0 in stage 1.0 (TID 
92)
17/11/03 10:17:27 INFO executor.Executor: Finished task 93.0 in stage 1.0 (TID 
92). 1145 bytes result sent to driver
17/11/03 10:17:27 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
104
17/11/03 10:17:27 INFO executor.Executor: Running task 103.0 in stage 1.0 (TID 
104)
17/11/03 10:19:00 INFO executor.Executor: Finished task 103.0 in stage 1.0 (TID 
104). 1145 bytes result sent to driver
17/11/03 10:19:00 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
113
17/11/03 10:19:00 INFO executor.Executor: Running task 118.0 in stage 1.0 (TID 
113)
17/11/03 10:21:43 INFO executor.Executor: Finished task 118.0 in stage 1.0 (TID 
113). 1145 bytes result sent to driver
17/11/03 10:21:43 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
129
17/11/03 10:21:43 INFO executor.Executor: Running task 131.0 in stage 1.0 (TID 
129)
17/11/03 10:25:03 INFO executor.Executor: Finished task 131.0 in stage 1.0 (TID 
129). 1145 bytes result sent to driver
17/11/03 10:25:03 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
139
17/11/03 10:25:03 INFO executor.Executor: Running task 142.0 in stage 1.0 (TID 
139)
17/11/07 09:54:59 INFO executor.CoarseGrainedExecutorBackend: Driver commanded 
a shutdown
17/11/07 09:54:59 INFO storage.MemoryStore: MemoryStore cleared
{code}

During the execution of spark job I got some exceptions:
{code}
org.apache.kudu.client.NonRecoverableException: Invalid call sequence ID in 
scan request
        at 
org.apache.kudu.client.TabletClient.dispatchTSErrorOrReturnException(TabletClient.java:557)
        at org.apache.kudu.client.TabletClient.decode(TabletClient.java:488)
        at org.apache.kudu.client.TabletClient.decode(TabletClient.java:82)
...
{code}

but the job didn't fail. Rather the failed tasks were re-executed and 
succesfully completed.


was (Author: andrew_ya):
We are using kudu-spark_2.10-1.2.0.jar
Logs look like
{code}
17/11/03 10:15:55 INFO executor.Executor: Running task 93.0 in stage 1.0 (TID 
92)
17/11/03 10:17:27 INFO executor.Executor: Finished task 93.0 in stage 1.0 (TID 
92). 1145 bytes result sent to driver
17/11/03 10:17:27 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
104
17/11/03 10:17:27 INFO executor.Executor: Running task 103.0 in stage 1.0 (TID 
104)
17/11/03 10:19:00 INFO executor.Executor: Finished task 103.0 in stage 1.0 (TID 
104). 1145 bytes result sent to driver
17/11/03 10:19:00 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
113
17/11/03 10:19:00 INFO executor.Executor: Running task 118.0 in stage 1.0 (TID 
113)
17/11/03 10:21:43 INFO executor.Executor: Finished task 118.0 in stage 1.0 (TID 
113). 1145 bytes result sent to driver
17/11/03 10:21:43 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
129
17/11/03 10:21:43 INFO executor.Executor: Running task 131.0 in stage 1.0 (TID 
129)
17/11/03 10:25:03 INFO executor.Executor: Finished task 131.0 in stage 1.0 (TID 
129). 1145 bytes result sent to driver
17/11/03 10:25:03 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 
139
17/11/03 10:25:03 INFO executor.Executor: Running task 142.0 in stage 1.0 (TID 
139)
17/11/07 09:54:59 INFO executor.CoarseGrainedExecutorBackend: Driver commanded 
a shutdown
17/11/07 09:54:59 INFO storage.MemoryStore: MemoryStore cleared
{code}

During the execution of spark job I got some exceptions:
{code}
org.apache.kudu.client.NonRecoverableException: Invalid call sequence ID in 
scan request
        at 
org.apache.kudu.client.TabletClient.dispatchTSErrorOrReturnException(TabletClient.java:557)
        at org.apache.kudu.client.TabletClient.decode(TabletClient.java:488)
        at org.apache.kudu.client.TabletClient.decode(TabletClient.java:82)
...
{code}

but the job didn't fail. Rather the failed tasks were re-executed and 
succesfully completed.

> Apache Spark stucks while reading Kudu table.
> ---------------------------------------------
>
>                 Key: KUDU-2210
>                 URL: https://issues.apache.org/jira/browse/KUDU-2210
>             Project: Kudu
>          Issue Type: Bug
>          Components: client, perf, spark
>            Reporter: Andrew Ya
>
> When I try reading Kudu table with Apache Spark using following code
> {code}
> import org.apache.kudu.spark.kudu._
> import sqlContext.implicits._
> val kuduOptions: Map[String, String] = Map(
> "kudu.table"  -> "test_table", 
> "kudu.master" -> "host1:7051,host2:7051,host3:7051")
> val kuduDF = sqlContext.read.options(kuduOptions).kudu
> kuduDF.registerTempTable("t")
> sqlContext.sql(" SELECT * FROM t  where id in (1111,2222) ").show(50, false)
> {code}
> after completing 95% of tasks the job stucks for more than three days.  The 
> table is partitioned by date and partitions have uneven size. Table have one 
> partition 12 Gb size, about 20 partitions with size between 1 Gb and 3 Gb and 
> some partitions with Mb's and kb's of data.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to