Re: Spark 1.6.1: Unexpected partition behavior?

2016-06-26 Thread Randy Gelhausen
Sorry, please ignore the above. I now see I called coalesce on a different reference, than I used to register the table. On Sun, Jun 26, 2016 at 6:34 PM, Randy Gelhausen <rgel...@gmail.com> wrote: > > val enriched_web_logs = sqlContext.sql(""" > select we

Spark 1.6.1: Unexpected partition behavior?

2016-06-26 Thread Randy Gelhausen
val enriched_web_logs = sqlContext.sql(""" select web_logs.datetime, web_logs.node as app_host, source_ip, b.node as source_host, log from web_logs left outer join (select distinct node, address from nodes) b on source_ip = address """)