Hello all. I have been running a Spark Job that eventually needs to do a large join.
24 million x 150 million A broadcast join is infeasible in this instance clearly, so I am instead attempting to do it with Hash Partitioning by defining a custom partitioner as: class RDD2Partitioner(partitions: Int) extends HashPartitioner(partitions) { override def getPartition(key: Any): Int = key match { case k: Tuple2[Int, String] => super.getPartition(k._1) case _ => super.getPartition(key) } } I then partition both arrays using this partitioner. However, the job eventually fails with the following exception which if I had to guess indicated that a network connection was interrupted during the shuffle stage, causing things to get lost and ultimately resulting in a fetch failure: 14/11/14 12:56:21 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(innovationdatanode08.cof.ds.capitalone.com,37590) 14/11/14 12:56:21 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@7369b398 14/11/14 12:56:21 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@7369b398 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) In the spark UI, I still see a substantial amount of shuffling going on at this stage, I am wondering if I’m perhaps using the partitioner incorrectly? ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.