Hi Saurabh, The right configuration parameter is: set mapred.output.compression.type=BLOCK;
Sorry about pointing you to the wrong configuration parameter. Zheng On Mon, Jul 27, 2009 at 10:02 PM, Saurabh Nanda<[email protected]> wrote: > >> The 1600MB number looks like record-level compression. Are you sure >> you've turned on block compression? > > Here's the exact snippet from my shell script. Do I have to set these > configuration parameters directly in the hadoop configuration file: > > ${HIVE_COMMAND} -e "set hive.exec.compress.output=true; set > io.seqfile.compression.type=BLOCK; set > mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec; set > io.seqfile.compress.blocksize=50000000; insert overwrite table > raw_compressed partition(dt='${D}') select line from raw where dt='${D}'" > > Saurabh. > -- > http://nandz.blogspot.com > http://foodieforlife.blogspot.com > -- Yours, Zheng
