[
https://issues.apache.org/jira/browse/SPARK-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14653440#comment-14653440
]
Poorvi Lashkary edited comment on SPARK-9594 at 8/4/15 10:42 AM:
-----------------------------------------------------------------
Use case: I need to create auto increment sequence column for a data frame. I
have created a UDF which updates value of accumulator and returns that value.
Used accumulators so that executors can share the same.
PFB sample code snippet:
static Accumulator<Integer> start = sc.accumulator(0);
SQLContext.udf().register("seq",new UDF1<Integer,Integer>(){
public Integer call(Integer l) throws
Exception{
l = startCtr.value() + 1;
start.setValue(l);
return l;
}
},DataTypes.IntegerType);
Query---- "Select seq("+start.value()+") as ID from df";
was (Author: poorvi_767):
Use case: I need to create auto increment sequence column for a data frame. I
have created a UDF which updates value of accumulator and returns that value.
Used accumulators so that executors can share the same.
PFB sample code snippet:
static Accumulator<Integer> start = sc.accumulator(0);
SQLContext.udf().register("seq",new UDF1<Integer,Integer>(){
public Integer call(Integer l) throws
Exception{
l = startCtr.value() + 1;
startCtr.setValue(l);
return l;
}
},DataTypes.IntegerType);
Query---- "Select seq("+start.value()+") as ID from df";
> Failed to get broadcast_33_piece0 while using Accumulators in UDF
> ------------------------------------------------------------------
>
> Key: SPARK-9594
> URL: https://issues.apache.org/jira/browse/SPARK-9594
> Project: Spark
> Issue Type: Test
> Environment: Amazon Linux AMI release 2014.09
> Reporter: Poorvi Lashkary
>
> Getting Below Exception while using accumulator in a UDF.
> java.io.IOException: org.apache.spark.SparkException: Failed to get
> broadcast_33_piece0 of broadcast_33
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1156)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
> at
> org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
> at
> org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
> at
> org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
> at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.SparkException: Failed to get broadcast_33_piece0
> of broadcast_33
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:137)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:137)
> at scala.Option.getOrElse(Option.scala:120)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:136)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:119)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:119)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:119)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:174)
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1153)
> ... 11 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
