Better line number hints for logging?

I have created some extension methods for RDDs in RichRecordRDD and these are working exceptionally well for me. However, when looking at the logs, its impossible to tell what's going on because all the line number hints point to RichRecordRDD.scala rather than the code that uses it. For example:

Re: Better line number hints for logging?

like this, please open a JIRA > issue for it. > > Matei > > On Jun 3, 2014, at 5:22 PM, John Salvatier wrote: > > I have created some extension methods for RDDs in RichRecordRDD and these > are working exceptionally well for me. > > However, when looking at the logs,

setCallSite for API backtraces not showing up in logs?

What's the correct way to use setCallSite to get the change to show up in the spark logs? I have something like class RichRDD (rdd : RDD[MyThing]) { def mySpecialOperation() { rdd.context.setCallSite("bubbles and candy!") rdd.map() val result = rdd.groupBy()

"apply at Option.scala:120" callback in Spark 1.1, but no user code involved?

In Spark 1.1, I'm seeing tasks with callbacks that don't involve my code at all! I'd seen something like this before in 1.0.0, but the behavior seems to be back apply at Option.scala:120 org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.

How are exceptions in map functions handled in Spark?

I'm trying to get a clear idea about how exceptions are handled in Spark? Is there somewhere where I can read about this? I'm on spark .7 For some reason I was under the impression that such exceptions are swallowed and the value that produced them ignored but the exception is logged. However, rig

Re: How are exceptions in map functions handled in Spark?

> > Matei > > On Apr 4, 2014, at 10:40 AM, John Salvatier wrote: > > > I'm trying to get a clear idea about how exceptions are handled in > Spark? Is there somewhere where I can read about this? I'm on spark .7 > > > > For some reason I was under t

Re: How are exceptions in map functions handled in Spark?

Btw, thank you for your help. On Fri, Apr 4, 2014 at 11:49 AM, John Salvatier wrote: > Is there a way to log exceptions inside a mapping function? logError and > logInfo seem to freeze things. > > > On Fri, Apr 4, 2014 at 11:02 AM, Matei Zaharia wrote: > >> Exceptions sh