Deenar, the singleton pattern I'm suggesting would look something like this:
public class TaskNonce {
private transient boolean mIsAlreadyDone;
private static transient TaskNonce mSingleton = new TaskNonce();
private transient Object mSyncObject = new Object();
public TaskNonce getSingleton() { return mSingleton; }
public void doThisOnce() {
if (mIsAlreadyDone) return;
lock (mSyncObject) {
mIsAlreadyDone = true;
...
}
}
which you would invoke as TaskNonce.getSingleton().doThisOnce() from within
the map closure. If you're using the Spark Java API, you can put all this
code in the mapper class itself.
There is no need to require one-row RDD partitions to achieve what you
want, if I understand your problem statement correctly.
--
Christopher T. Nguyen
Co-founder & CEO, Adatao <http://adatao.com>
linkedin.com/in/ctnguyen
On Tue, Mar 25, 2014 at 11:07 AM, deenar.toraskar <deenar.toras...@db.com>wrote:
> Christopher
>
> It is once per JVM. TaskNonce would meet my needs. I guess if I want it
> once
> per thread, then a ThreadLocal would do the same.
>
> But how do I invoke TaskNonce, what is the best way to generate a RDD to
> ensure that there is one element per executor.
>
> Deenar
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Running-a-task-once-on-each-executor-tp3203p3208.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
