Hello,
I’m at a loss trying to diagnose why my spark job is failing. (works fine on
small data)
It is failing during the repartition, or on the subsequent steps.. which then
seem to fail and fall back to repartitioning..
I’ve tried adjusting every parameter I can find, but have had no success.
I
It seems like the problem is related to —executor-cores. Is there possibly some
sort of race condition when using multiple cores per executor?
On Nov 22, 2015, at 12:38 PM, Jeremy Davis
mailto:jda...@marketshare.com>> wrote:
Hello,
I’m at a loss trying to diagnose why my spark job is f
0)))
.withColumn("min"+(-win),min("value").over(baseWin.rowsBetween(-win,0)))
.withColumn("max"+(-win),max("value").over(baseWin.rowsBetween(-win,0)))
}
resultFrame.show()
}
}
> On Sep 20, 2016, at 10:26 PM, Jeremy Dav
I have an application that does many thousand univariate GLM regressions that
seems to break down after completing around 25K jobs.
Plenty of resources: disk, network, memory, CPU are free, but eventually it is
only scheduling on a few threads (out of 400+ possible on the cluster)
No task failu
