You need to enable nonstrict mode to be able to use dynamic partitioning. Dynamic partitioning is more automatic, but might take more time because Hive has to write to multiple outputs at once. With MapReduce, this uses MultipleOutputs, and can fail with OOM when writing too many partitions at once (especially ORC), there is a fix for that (hive.optimize.sort.dynamic.partition, true by default) which causes Hive to sort the output before writing it, when dynamic partitioning is used, this might make some queries much slower (especially when copying a partitioned table).
Also, I think dynamic partitioning requires to put an exclusive write lock on the table, meaning it can't be read while you write in it. When you use static partitioning, you should be able to write a new partition and still be able to read from the other partitions at the same time. In short: if you know where your partitions are going and can generate your Hive queries programatically, use static partitioning, otherwise, use dynamic partitioning. 2018-01-08 20:21 GMT+01:00 Alan Gates <alanfga...@gmail.com>: > When doing dynamic partitioning, Hive needs to look at each record and > determine which partition to place it in. In the case where all records go > to the same partition, it is more efficient to tell Hive up front, that is, > to use static partitioning. So you can use dynamic partition for large > files, it will just take more processing power and time. > > Alan. > > On Tue, Jan 2, 2018 at 5:53 PM, Sachit Murarka <connectsac...@gmail.com> > wrote: > >> Hi, >> >> I am unclear about Dynamic and static partition. >> >> 1. Why static partition is good for loading large files and why can’t >> we use dynamic partition for the same? >> 2. Why does dynamic partition take more time in loading data than >> static partitions? >> >> Also please explain when to use strict and nonstrict mode. >> >> Kind Regards, >> Sachit Murarka >> > >
