Re: Issue of Hive parquet partitioned table schema mismatch

spark.sql.hive.convertMetastoreParquet is true. I can't repro the issue of scanning all partitions now.. : P Anyway, I found another email thread "Re: Spark Sql behaves strangely with tables with a lot of partitions" I observe the same issue as Jerrick, spark driver will call listStatus for the

Re: Issue of Hive parquet partitioned table schema mismatch

Is there any chance that " spark.sql.hive.convertMetastoreParquet" is turned off? Cheng On 11/4/15 5:15 PM, Rex Xiong wrote: Thanks Cheng Lian. I found in 1.5, if I use spark to create this table with partition discovery, the partition pruning can be performed, but for my old table

Re: Issue of Hive parquet partitioned table schema mismatch

SPARK-11153 should be irrelevant because you are filtering on a partition key while SPARK-11153 is about Parquet filter push-down and doesn't affect partition pruning. Cheng On 11/3/15 7:14 PM, Rex Xiong wrote: We found the query performance is very poor due to this issue

Re: Issue of Hive parquet partitioned table schema mismatch

We found the query performance is very poor due to this issue https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-11153 We usually use filter on partition key, the date, it's in string type in 1.3.1 and works great. But in 1.5, it needs to do parquet scan for all partitions.

Re: Issue of Hive parquet partitioned table schema mismatch

Add back this thread to email list, forgot to reply all. 2015年10月31日 下午7:23，"Michael Armbrust" 写道： > Not that I know of. > > On Sat, Oct 31, 2015 at 12:22 PM, Rex Xiong wrote: > >> Good to know that, will have a try. >> So there is no easy way to

Re: Issue of Hive parquet partitioned table schema mismatch

What Storage Format? > On 30 Oct 2015, at 12:05, Rex Xiong wrote: > > Hi folks, > > I have a Hive external table with partitions. > Every day, an App will generate a new partition day=-MM-dd stored by > parquet and run add-partition Hive command. > In some cases, we

Issue of Hive parquet partitioned table schema mismatch

Hi folks, I have a Hive external table with partitions. Every day, an App will generate a new partition day=-MM-dd stored by parquet and run add-partition Hive command. In some cases, we will add additional column to new partitions and update Hive table schema, then a query across new and old

Re: Issue of Hive parquet partitioned table schema mismatch

> > We have tried schema merging feature, but it's too slow, there're hundreds > of partitions. > Which version of Spark?