Re: pyspark exception catch

2014-12-19 Thread imazor
Hi, Thanks for the answer. regarding 2,3, its indeed the solution, but as I mentioned in my question, I can as well do input checks (using .map) before applying any other rdd operations. I still think that its overhead. Regarding 1, this will make all the other rdd operations more complex, as I

Re: pyspark exception catch

2014-12-16 Thread cfregly
hey igor! a few ways to work around this depending on the level of exception-handling granularity you're willing to accept: 1) use mapPartitions() to wrap the entire partition handling code in a try/catch -- this is fairly coarse-grained, however, and will fail the entire partition. 2) modify your