Re: SparkSQL and multiple roots in 1.6

2016-03-25 Thread Spencer Uresk
you can set > "spark.sql.sources.partitionDiscovery.enabled=false". Otherwise, I think > you need to use the unioning approach. > > On Fri, Mar 25, 2016 at 1:35 PM, Spencer Uresk <sur...@gmail.com> wrote: > >> Thanks for the suggestion - I didn't try it at fir

Re: SparkSQL and multiple roots in 1.6

2016-03-25 Thread Spencer Uresk
Thanks for the suggestion - I didn't try it at first because it seems like I have multiple roots and not necessarily partitioned data. Is this the correct way to do that? sqlContext.read.option("basePath", "hdfs://user/hdfs/analytics/").json("hdfs://user/hdfs/analytics/*/PAGEVIEW/*/*") If so, it