So we're seeing the following error during some of our hive loads:
2011-07-05 12:26:52,927 Stage-2 map = 100%, reduce = 100%
Ended Job = job_201106302113_3864
Loading data to table default.merged_weblogs partition (day=null)
Failed with exception Number of dynamic partitions created is 1013,
which is more than 1000. To solve this try to set
hive.exec.max.dynamic.partitions to at least 1013.
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
Here is a sample script we're running:
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions.pernode=10000;
SET hive.exec.max.dynamic.partitions=10000;
SET hive.exec.max.created.files=150000;
SET hive.exec.compress.intermediate=true;
SET hive.intermediate.compression.codec=com.hadoop.compression.lzo.LzoCodec;
SET hive.intermediate.compression.type=BLOCK;
SET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;
SET hive.exec.compress.output=true;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
SET mapred.output.compression.type=BLOCK;
FROM (
SELECT hostname, name, ip, day
FROM logsStaging
UNION ALL
SELECT hostname, name, ip, day
FROM logs
) a
INSERT OVERWRITE TABLE logs PARTITION(day)
SELECT DISTINCT hostname, name, ip, day
DISTRIBUTE BY day;
QUIT;
Has anyone run into this problem before? And I've noticed that
increasing the number of partitions hasn't been working. Been looking
for the config.xml setting already configured with 'final' in the
properties but no go so far. I believe the default is 100 partitions
and the job (when running) does show 10000 partitions (from the above
script)
thoughts on what else to look at?
Thanks!
