I am currently using Spark 2.0.1 with Scala 2.11.8. However same code works
with Scala 2.10.6. Please advise if I am missing something
import org.apache.spark.sql.functions.udf
val getFileName = udf{z:String => z.takeRight(z.length
-z.lastIndexOf("/")-1)}
and this gives me following error
I am seeing this in my logs. It appears to reopen connection to Shuffle
Service. Whenever this happens, I am seeing the partition to take longer
time to complete. I was running a job with 1000 partitions. About 600
partitions, it was completing in less than 20 mins a partition and after
that I
