How much memory have you allocated to the driver? Driver stores some state
for tracking the task, stage and job history that you can see in the spark
console, it does take up a significant portion of the heap, anywhere from
200MB - 1G, depending no your map reduce steps.
Either way that is a good
This isn't related to the progress bar, it just happened while in that
section of code. Something else is taking memory in the driver, usually a
broadcast table or something else that requires a lot of memory and happens
on the driver.
You should check your driver memory settings and the query
the spark version is 2.1.0
--发件人:方孝健(玄弟)
发送时间:2017年2月10日(星期五) 12:35收件人:spark-dev
; spark-user 主 题:Driver hung and
happend out of memory while writing to
[Stage 172:==> (10328 + 93) / 16144]
[Stage 172:==> (10329 + 93) / 16144]
[Stage 172:==> (10330 + 93) / 16144]
[Stage 172:==>