In our case, we are dealing with 20TB text data which is separated to about 200k map tasks and 200k reduce tasks, and our driver's memory is 15G,.
-- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/MapStatus-too-large-for-drvier-tp14704p14707.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
