Shuffle performance tuning. How to tune netty?
I am facing a very tricky issue here. I have a treeReduce task. The reduce-function returns a very large object. In fact it is a Map[Int, Array[Double]]. Each reduce task inserts and/or updates values into the map or updates the array. My problem is, that this Map can become very large. Currently, I am looking at about 500 MB (serialized size). The performance of the entire reduce task is incredibly slow. While my reduce function takes only about 10 seconds to execute, the shuffle-subsystem of Spark takes very long. My task returns after about 100-300 seconds. This even happens if I just have 2 nodes with 2 worker cores. So the only thing spark would have to do is to send the 500 MB over the network (both machines are connected via Gigabit Ethernet) which should take a couple of seconds. It is also interesting to note that if I choose "nio" as block transport manager, the speed is very good. Only a couple of seconds as expected. But just discovered that "nio" support is discontinued. So, how can I get good performance for such a usage scenario. Large objects, treeReduce, not very many nodes with netty? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-performance-tuning-How-to-tune-netty-tp25433.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Prevent partitions from moving
I have a cluster with 2 nodes (32 CPU cores each). My data is distributed evenly, but the processing times for each partition can vary greatly. Now, sometimes Spark seems to conclude from the current workload on both nodes that it might be better to shift one partition from node1 to node2 (because that guy has cores waiting for work). Am i hallucinating or is that really the happening? Is there any way I prevent this from happening? Greetings, T3L -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Prevent-partitions-from-moving-tp25216.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
(SOLVED) Ahhhh... Spark creates >30000 partitions... What can I do?
I was able to solve this by myself. What I did is changing the way spark computes the partitioning for binary files. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/A-Spark-creates-3-partitions-What-can-I-do-tp25140p25170.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Partition for each executor
If I have a cluster with 7 nodes, each having an equal amount of cores and create an RDD with sc.parallelize() it looks as if the Spark will always tries to distribute the partitions. Question: (1) Is that something I can rely on? (2) Can I rely that sc.parallelize() will assign partitions to as many executers as possible. Meaning: Let's say I request 7 partitions, is each node guaranteed to get 1 of these partitions? If I select 14 partitions, is each node guaranteed to grab 2 partitions? P.S.: I am aware that for other cases like sc.hadoopFile, this might depend in the actual storage location of the data. I am merely asking for the sc.parallelize() case. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Partition-for-each-executor-tp25141.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Ahhhh... Spark creates >30000 partitions... What can I do?
I have dataset consisting of 5 binary files (each between 500kb and
2MB). They are stored in HDFS on a Hadoop cluster. The datanodes of the
cluster are also the workers for Spark. I open the files as a RDD using
sc.binaryFiles("hdfs:///path_to_directory").When I run the first action that
involves this RDD, Spark spawns a RDD with more than 3 partitions. And
this takes ages to process these partitions even if you simply run "count".
Performing a "repartition" directly after loading does not help, because
Spark seems to insist on materializing the RDD created by binaryFiles first.
How I can get around this?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/A-Spark-creates-3-partitions-What-can-I-do-tp25140.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
