Hi Akhil, thanks for your reply. I am using the latest version of Spark 1.2.1 (also tried 1.3 developer branch). If I am not mistaken the TorrentBroadcast is the default there, isn't it?
Thanks, Tassilo On Sun, Feb 22, 2015 at 10:59 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Did you try with torrent broadcast factory? > > Thanks > Best Regards > > On Sun, Feb 22, 2015 at 3:29 PM, TJ Klein <tjkl...@gmail.com> wrote: > >> Hi, >> >> I am trying to broadcast large objects (order of a couple of 100 MBs). >> However, I keep getting errors when trying to do so: >> >> Traceback (most recent call last): >> File "/LORM_experiment.py", line 510, in <module> >> broadcast_gradient_function = sc.broadcast(gradient_function) >> File "/scratch/users/213444/spark/python/pyspark/context.py", line 643, >> in >> broadcast >> return Broadcast(self, value, self._pickled_broadcast_vars) >> File "/scratch/users/213444/spark/python/pyspark/broadcast.py", line 65, >> in __init__ >> self._path = self.dump(value, f) >> File "/scratch/users/213444/spark/python/pyspark/broadcast.py", line 82, >> in dump >> cPickle.dump(value, f, 2) >> SystemError: error return without exception set >> 15/02/22 04:52:14 ERROR Utils: Uncaught exception in thread delete Spark >> local dirs >> java.lang.IllegalStateException: Shutdown in progress >> >> Any idea how to prevent that? I got plenty of RAM, so there shouldn't be >> any >> problem with that. >> >> Thanks, >> Tassilo >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Broadcasting-Large-Objects-Fails-tp21752.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
