Re: Doubt on Hive Partitioning.

Partition pruning works also with older Hive version, but you have to put the filter in the join statement and not only in the where statement > On 02 Aug 2016, at 09:53, Furcy Pin wrote: > > I'm using Hive 1.1 on MR and dynamic partition pruning does not seem to work.

Re: Doubt on Hive Partitioning.

I do not think so, but never tested it. > On 02 Aug 2016, at 03:45, Qiuzhuang Lian wrote: > > Is this partition pruning fixed in MR too except for TEZ in newer hive > version? > > Regards, > Q > >> On Mon, Aug 1, 2016 at 8:48 PM, Jörn Franke

Re: Doubt on Hive Partitioning.

I'm using Hive 1.1 on MR and dynamic partition pruning does not seem to work. Since MR is deprecated in 2.0, I assume we should not expect any future perf optimisation on this side. It has been implemented for Hive on Spark, though. https://issues.apache.org/jira/browse/HIVE-9152 On Tue, Aug

Re: Doubt on Hive Partitioning.

Is this partition pruning fixed in MR too except for TEZ in newer hive version? Regards, Q On Mon, Aug 1, 2016 at 8:48 PM, Jörn Franke wrote: > It happens in old hive version of the filter is only in the where clause > and NOT in the join clause. This should not happen in

Re: Doubt on Hive Partitioning.

> WHERE p IN (SELECT p FROM t2) > here we could argue that Hive could optimize this by computing the sub >query first, > and then do the partition pruning, but sadly I don't think this >optimisation has been implemented yet It is implemented already -

Re: Doubt on Hive Partitioning.

It happens in old hive version of the filter is only in the where clause and NOT in the join clause. This should not happen in newer hive version. You can check it by executing explain dependency query. > On 01 Aug 2016, at 11:07, Abhishek Dubey wrote: > > Hi All,

Re: Doubt on Hive Partitioning.

Hi Abhishek, Yes, it can happen. The only such scenarios I can think of are when you use a WHERE clause with a non-constant clause. As far as I know, partition only work on constant clauses, because it has to evaluate them *before* starting the query in order to prune the partitions. For

Doubt on Hive Partitioning.

Hi All, I have a very big table t with billions of rows and it is partitioned on a column p. Column p has datatype text and values like '201601', '201602'...upto '201612'. And, I am running a query like : Select columns from t where p='201604'. My question is : Can there be a