Why would you ever need to do this? I'm genuinely curious. I view collects as being solely for interactive work.
On Thursday, April 28, 2016, <ioannis.deligian...@nomura.com> wrote: > Hi, > > > > It is a common pattern to process an RDD, collect (typically a subset) to > the driver and then broadcast back. > > > > Adding an RDD method that can do that using the torrent broadcast > mechanics would be much more efficient. In addition, it would not require > the Driver to also utilize its Heap holding this broadcast. > > > > I guess this can become complicated if the resulting broadcast is required > to keep lineage information, but assuming a torrent distribution, once the > broadcast is synced then lineage would not be required. I’d also expect the > call to rdd.brodcast to be an action that eagerly distributes the broadcast > and returns when the operation has succeeded. > > > > Is this something that could be implemented or are there any reasons that > prohibits this? > > > > Thanks > > Ioannis > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from > taking action on the basis of information in this e-mail and must contact > the sender immediately, delete this e-mail (and all attachments) and > destroy any hard copies. Nomura will not accept responsibility or liability > for the accuracy or completeness of, or the presence of any virus or > disabling code in, this e-mail. If verification is sought please request a > hard copy. Any reference to the terms of executed transactions should be > treated as preliminary only and subject to formal written confirmation by > Nomura. Nomura reserves the right to retain, monitor and intercept e-mail > communications through its networks (subject to and in accordance with > applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm > -- Want to work at Handy? Check out our culture deck and open roles <http://www.handy.com/careers> Latest news <http://www.handy.com/press> at Handy Handy just raised $50m <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led by Fidelity
