Hi Prabakaran,
Multiple small part files in the output directory is because each reducer
task output is coming as one part file.
1. Do each part file will take 64MB block size?
Based on the output size of the reducer one part file is
created. Filesize can be smaller size than the hdfs block size, i.e. it not be
mandatorily be of 64MB
2. How to merge these multiple RC format part files into one RC file?
One way (may be longer way ) is to get the part files in to
local diretory and write a tool to merge all the RC files.
But anyway i feel in the first place we need to ensure we have
single reducer so that there is no need for merging
3. What is the pros-cons of having multiple part files?
Depends on the next operation what you want to do,
Like if you are planning to load into Hive then based on Hive
paritions better to configure the MR to be partitioned as per Hive partiions
and loading would be easier? etc ...
4. Do merging part files will improve performance?
Performance of the Map reduce or later operation ? I think
if the overall scenario is known then we will be able to support better
Regards,
Naga
Huawei Technologies Co., Ltd.
Phone:
Fax:
Mobile: +91 9980040283
Email: [email protected]<mailto:[email protected]>
Huawei Technologies Co., Ltd.
Bantian, Longgang District,Shenzhen 518129, P.R.China
http://www.huawei.com
¡This e-mail and its attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed above.
Any use of the information contained herein in any way (including, but not
limited to, total or partial disclosure, reproduction, or dissemination) by
persons other than the intended recipient(s) is prohibited. If you receive this
e-mail in error, please notify the sender by phone or email immediately and
delete it!
________________________________
From: Natarajan, Prabakaran 1. (NSN - IN/Bangalore)
[[email protected]]
Sent: Thursday, July 17, 2014 15:52
To: [email protected]
Subject: Multiple Part files
Hi
After Map Reduce job, we are seeing multiple small part files in the output
directory. We are using RC file format (snappy codec)
1. Do each part file will take 64MB block size?
2. How to merge these multiple RC format part files into one RC file?
3. What is the pros-cons of having multiple part files?
4. Do merging part files will improve performance?
Thanks and Regards
Prabakaran.N aka NP
nsn, Bangalore
When "I" is replaced by "We" - even Illness becomes "Wellness"
