Is there a way to log exceptions inside a mapping function? logError and logInfo seem to freeze things.
On Fri, Apr 4, 2014 at 11:02 AM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > Exceptions should be sent back to the driver program and logged there > (with a SparkException thrown if a task fails more than 4 times), but there > were some bugs before where this did not happen for non-Serializable > exceptions. We changed it to pass back the stack traces only (as text), > which should always work. I'd recommend trying a newer Spark version, 0.8 > should be easy to upgrade to from 0.7. > > Matei > > On Apr 4, 2014, at 10:40 AM, John Salvatier <jsalvat...@gmail.com> wrote: > > > I'm trying to get a clear idea about how exceptions are handled in > Spark? Is there somewhere where I can read about this? I'm on spark .7 > > > > For some reason I was under the impression that such exceptions are > swallowed and the value that produced them ignored but the exception is > logged. However, right now we're seeing the task just re-tried over and > over again in an infinite loop because there's a value that always > generates an exception. > > > > John > >
