Thanx Chinna, I have another question. Now I know that InputForamt is an interface in apache hadoop, so what I get (and correct me if I m wrong) that it is Hive implementation who makes all that splits , is it ? or it is because of using of Cassandra? In both cases, can I do anything to manage the size of splits?
PS. I ve started writing new aplication mapred to read data from cassandra after that insert the results to hive, I do not klnow if it is a good choice , *I m kind of newbie in Hadoop mapred and Hive :)* Thanx in advance for your reply Amjad On Fri, Mar 28, 2014 at 2:09 PM, Chinna Rao Lalam < [email protected]> wrote: > Hi, > > Number of mappers will depend on some factors like, > > 1.The number of mappers is defined based on our data size (By default set > to one per HDFS block). > > 2.The number of mappers depends on amount of InputSplit generated by > InputFormat#getInputSplits method. In particular FileInputSplit splits > input directory in respect to blocks and files. > > Ex: > > Two files: > f1 [ block1, block2], > f2 [block3, block4] > becomes 4 mappers > f1(offset of block1), > f1(offset of block2), > f2(offest of block3), > f2(offset of block4) > Other InputFormat has its own methods for files splitting (for example > Hbase splits input on region boundaries). > > > Hope It Helps, > Chinna > > > On Fri, Mar 28, 2014 at 4:47 PM, Amjad ALSHABANI <[email protected]>wrote: > >> Hello All, >> I have tow tables created in Hive, one read the data immidiately from >> Cassandra DB and the other takes the data from stored files (data already >> exported in somewhow from the same Cassandra DB) >> >> Both tables are identical (in data) >> but when running the same request it gives me very different number of >> mapper for each of them (P.S I m using the same hive config for both >> requests) >> >> *hive -e " select count(1) from keyring.cred"* >> >> will take 2338 mapper and 1 reducer >> >> while: >> >> *hive -e "select count(1) from keyring.cred_seq"* >> >> will take just 151 mapper and 1 reducer >> >> Any idea how to minimize this number, and where does this explosion in >> mapper number come from?? >> >> any response will be appreciated :) >> >> Cheers!! >> >> Amjad >> >> >
