Increase or Decrease the number of data partitions: Since a data partition represents the quantum of data to be processed together by a single Spark Task, there could be situations: (a) Where existing number of data partitions are not sufficient enough in order to maximize the usage of available resources (b) Where existing number of data partitions are too heavy to be computed reliably without memory overruns. (c) Where existing number of data partitions are too high in number such that task scheduling overhead becomes the bottleneck in the overall processing time.
----- ᐅ Targeted Web Traffic AFFORDABLE web traffic package is the best ideal for small businesses 👉 Website Traffic Packages : Turn Traffic Increase Into Revenue - -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
