RE: Broadcast destroy

2017-01-02 Thread bryan.jeffrey
All, Anyone have a thought? Thank you, Bryan Jeffrey From: bryan.jeff...@gmail.com Sent: Friday, December 30, 2016 1:20 PM To: user Subject: Broadcast destroy All, If we are updating broadcast variables do we need to manually destroy the replaced broadcast, or will they be automatically

Broadcast destroy

2016-12-30 Thread bryan.jeffrey
All, If we are updating broadcast variables do we need to manually destroy the replaced broadcast, or will they be automatically pruned? Thank you, Bryan Jeffrey Sent from my Windows 10 phone

RE: How to Check Dstream is empty or not?

2016-12-01 Thread bryan.jeffrey
The stream is just a wrapper over batch operations. You can check if a batch is empty by simply doing: val isEmpty = stream.transform(rdd => rdd.isEmpty) This will give you a stream of Boolean indicating if given batches are empty. Bryan Jeffrey From: rockinf...@gmail.com Sent: Thursday,