Hello,
In the app below, when I run it with local[1] or local [3], I get the
expected result - a list of the square roots of the numbers from 1 to 20.
When I try the same app as standalone with one or two workers on the same
machine, it will only print 1.0.
Adding print statements into the reduce function reveals that three times it
calculated Set(1.0) ++ Set(1.0) to yield Set(1.0).
Any ideas? Thanks!
Best, Oliver
Oliver Ruebenacker | Solutions Architect
Altisource(tm)
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
oliver.ruebenac...@altisource.com<mailto:oliver.ruebenac...@altisource.com> |
www.Altisource.com
package sandbox
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object SquareRoots extends App {
def sqrt(x: Double, nIters: Long) = {
var iIter: Long = 0
var root = 1.0
while (iIter < nIters) {
iIter += 1
root = 0.5 * (root + x / root)
}
root
}
def format(x:Double) :String = {
val string = "" + x
if(string.length > 5) { string.substring(0,5) } else { string }
}
val nNums = 20
val nIters = 1000000000 // for 1e9, runs about 50-55 secs per stage on my
laptop with --master local[4]
val nStages = 10
System.setProperty("hadoop.home.dir", "c:\\Users\\ruebenac\\winutil\\")
val conf = new SparkConf().setAppName("Square roots")
val sc = new SparkContext(conf)
val logPrefix = " [###] "
def log(line: String) = { println(logPrefix + line) }
log("Let's go!")
for (iStage <- 0 to nStages) {
log("Starting stage " + iStage)
val nums = sc.parallelize((1 to nNums).map(_.toDouble))
val roots = nums.map(sqrt(_, nIters)).map(Set(_)).reduce((roots1, roots2)
=> roots1 ++ roots2).toList.sorted
log("Square roots from 1 to " + nNums + " in " + nIters + " iterations:")
log(roots.map(format(_)).mkString(" "))
log("Completed stage " + iStage)
}
log("Done!")
}
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system. This message and any attachments may contain information that
is confidential, privileged or exempt from disclosure. Delivery of this message
to any person other than the intended recipient is not intended to waive any
right or privilege. Message transmission is not guaranteed to be secure or free
of software viruses.
***********************************************************************************************************************
