currently we migrate from 1.1 to 1.2, and found our program 3x slower, maybe due to the singleton hack?
could you explain in detail why or how "The singleton hack works very different in spark 1.2.0 " thanks! 2015-01-18 20:56 GMT+08:00 octavian.ganea <[email protected]>: > The singleton hack works very different in spark 1.2.0 (it does not work if > the program has multiple map-reduce jobs in the same program). I guess > there > should be an official documentation on how to have each machine/node do an > init step locally before executing any other instructions (e.g. loading > locally a very big object once at the begining that can be used in all > further map jobs that will be assigned to that worker). > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-share-a-NonSerializable-variable-among-tasks-in-the-same-worker-node-tp11048p21219.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
