as long as the files are inside the same directory, hive will treat them as a table.
Todd On Aug 9, 2010, at 6:07 PM, "Bakshi, Ankita" <[email protected]> wrote: > > Hi, > > Sorry to hijack this thread. But I am curious if there any other in-built > option to merge files in the directory before loading data into the table. > > I have a directory in the local file system which contains many small files. > I want to load it to a single hive table. I am wondering what would be the > best approach to this problem. > > Thanks, > Ankita > > > -----Original Message----- > From: Namit Jain [mailto:[email protected]] > Sent: Monday, August 09, 2010 9:32 AM > To: [email protected] > Subject: RE: How to merge small files > > Yes, it will try to run another map-reduce job to merge the files > ________________________________________ > From: lei liu [[email protected]] > Sent: Monday, August 09, 2010 8:57 AM > To: [email protected] > Subject: Re: How to merge small files > > Could you tell me whether the query is slower if I two parameters both are > true? > > 2010/8/9 Namit Jain <[email protected]><mailto:[email protected]>> > That's right > > ________________________________________ > From: lei liu [[email protected]<mailto:[email protected]>] > Sent: Sunday, August 08, 2010 7:18 PM > To: [email protected]<mailto:[email protected]> > Subject: Re: How to merge small files > > Thank you for your reply. > > Your mean is I will execute below statement: > > statement.execute("set hive.merge.mapfiles=true"); > statement.execute("set hive.merge.mapredfiles=true"); > > The two parementers are both true, right? > > 2010/8/6 Namit Jain > <[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>>> > HIVEMERGEMAPFILES("hive.merge.mapfiles", true), > HIVEMERGEMAPREDFILES("hive.merge.mapredfiles", false), > > > Set the above parameters to true before your query. > > > > ________________________________________ > From: lei liu > [[email protected]<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>>] > Sent: Thursday, August 05, 2010 8:47 PM > To: > [email protected]<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> > Subject: How to merge small files > > When I run below sql: INSERT OVERWRITE TABLE tablename1 select_statement1 > FROM from_statement, there are many files which size is zero are stored to > hadoop, > > How can I merge these small files? > > Thanks, > > > > LiuLei > > > > The information contained in this email message and its attachments is > intended only for the private and confidential use of the recipient(s) named > above, unless the sender expressly agrees otherwise. Transmission of email > over the Internet is not a secure communications medium. If you are > requesting or have requested the transmittal of personal data, as defined in > applicable privacy laws by means of email or in an attachment to email, you > must select a more secure alternate means of transmittal that supports your > obligations to protect such personal data. If the reader of this message is > not the intended recipient and/or you have received this email in error, you > must take no action based on the information in this email and you are hereby > notified that any dissemination, misuse or copying or disclosure of this > communication is strictly prohibited. If you have received this communication > in error, please notify us immediately by email and delete the original > message. >
