Re: Spark SQL Hive - merge small files
So there is no good way to merge spark files in a manage hive table right now? On Wed, Aug 5, 2015 at 10:02 AM, Michael Armbrust wrote: > This feature isn't currently supported. > > On Wed, Aug 5, 2015 at 8:43 AM, Brandon White > wrote: > >> Hello, >> >> I would love to have hive merge the small files in my managed hive >> context after every query. Right now, I am setting the hive configuration >> in my Spark Job configuration but hive is not managing the files. Do I need >> to set the hive fields in around place? How do you set Hive configurations >> in Spark? >> >> Here is what I'd like to set >> >> hive.merge.mapfilestrue >> hive.merge.mapredfilestrue >> hive.merge.size.per.task25600 >> hive.merge.smallfiles.avgsize1600 >> > >
Re: Spark SQL Hive - merge small files
This feature isn't currently supported. On Wed, Aug 5, 2015 at 8:43 AM, Brandon White wrote: > Hello, > > I would love to have hive merge the small files in my managed hive context > after every query. Right now, I am setting the hive configuration in my > Spark Job configuration but hive is not managing the files. Do I need to > set the hive fields in around place? How do you set Hive configurations in > Spark? > > Here is what I'd like to set > > hive.merge.mapfilestrue > hive.merge.mapredfilestrue > hive.merge.size.per.task25600 > hive.merge.smallfiles.avgsize1600 >
Spark SQL Hive - merge small files
Hello, I would love to have hive merge the small files in my managed hive context after every query. Right now, I am setting the hive configuration in my Spark Job configuration but hive is not managing the files. Do I need to set the hive fields in around place? How do you set Hive configurations in Spark? Here is what I'd like to set hive.merge.mapfilestrue hive.merge.mapredfilestrue hive.merge.size.per.task25600 hive.merge.smallfiles.avgsize1600
