Lei, Are you still using hive 4.1 or have you upgraded, the merge options mentioned above were probable not present until 5.0
Edward On Mon, Aug 9, 2010 at 9:59 PM, Todd Lee <[email protected]> wrote: > as long as the files are inside the same directory, hive will treat them as a > table. > > > Todd > > On Aug 9, 2010, at 6:07 PM, "Bakshi, Ankita" <[email protected]> > wrote: > >> >> Hi, >> >> Sorry to hijack this thread. But I am curious if there any other in-built >> option to merge files in the directory before loading data into the table. >> >> I have a directory in the local file system which contains many small files. >> I want to load it to a single hive table. I am wondering what would be the >> best approach to this problem. >> >> Thanks, >> Ankita >> >> >> -----Original Message----- >> From: Namit Jain [mailto:[email protected]] >> Sent: Monday, August 09, 2010 9:32 AM >> To: [email protected] >> Subject: RE: How to merge small files >> >> Yes, it will try to run another map-reduce job to merge the files >> ________________________________________ >> From: lei liu [[email protected]] >> Sent: Monday, August 09, 2010 8:57 AM >> To: [email protected] >> Subject: Re: How to merge small files >> >> Could you tell me whether the query is slower if I two parameters both are >> true? >> >> 2010/8/9 Namit Jain <[email protected]><mailto:[email protected]>> >> That's right >> >> ________________________________________ >> From: lei liu [[email protected]<mailto:[email protected]>] >> Sent: Sunday, August 08, 2010 7:18 PM >> To: [email protected]<mailto:[email protected]> >> Subject: Re: How to merge small files >> >> Thank you for your reply. >> >> Your mean is I will execute below statement: >> >> statement.execute("set hive.merge.mapfiles=true"); >> statement.execute("set hive.merge.mapredfiles=true"); >> >> The two parementers are both true, right? >> >> 2010/8/6 Namit Jain >> <[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>>> >> HIVEMERGEMAPFILES("hive.merge.mapfiles", true), >> HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false), >> >> >> Set the above parameters to true before your query. >> >> >> >> ________________________________________ >> From: lei liu >> [[email protected]<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>>] >> Sent: Thursday, August 05, 2010 8:47 PM >> To: >> [email protected]<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> >> Subject: How to merge small files >> >> When I run below sql: INSERT OVERWRITE TABLE tablename1 select_statement1 >> FROM from_statement, there are many files which size is zero are stored to >> hadoop, >> >> How can I merge these small files? >> >> Thanks, >> >> >> >> LiuLei >> >> >> >> The information contained in this email message and its attachments is >> intended only for the private and confidential use of the recipient(s) named >> above, unless the sender expressly agrees otherwise. Transmission of email >> over the Internet is not a secure communications medium. If you are >> requesting or have requested the transmittal of personal data, as defined in >> applicable privacy laws by means of email or in an attachment to email, you >> must select a more secure alternate means of transmittal that supports your >> obligations to protect such personal data. If the reader of this message is >> not the intended recipient and/or you have received this email in error, you >> must take no action based on the information in this email and you are >> hereby notified that any dissemination, misuse or copying or disclosure of >> this communication is strictly prohibited. If you have received this >> communication in error, please notify us immediately by email and delete the >> original message. >> >
