Thanx Chinna,

I have another question. Now I know that InputForamt is an interface in
apache hadoop, so what I get  (and correct me if I m wrong) that it is Hive
implementation who makes all that splits , is it ? or it is because of
using of Cassandra?
In both cases, can I do anything to manage the size of splits?

PS. I ve started writing new aplication mapred to read data from cassandra
after that insert the results to hive, I do not klnow if it is a good
choice , *I m kind of newbie in Hadoop mapred and Hive :)*

Thanx in advance for your reply

Amjad


On Fri, Mar 28, 2014 at 2:09 PM, Chinna Rao Lalam <
[email protected]> wrote:

> Hi,
>
> Number of mappers will depend on some factors like,
>
> 1.The number of mappers is defined based on our data size (By default set
> to one per HDFS block).
>
> 2.The number of mappers depends on amount of InputSplit generated by
> InputFormat#getInputSplits method. In particular FileInputSplit splits
> input directory in respect to blocks and files.
>
> Ex:
>
> Two files:
> f1 [ block1, block2],
> f2 [block3, block4]
> becomes 4 mappers
> f1(offset of block1),
> f1(offset of block2),
> f2(offest of block3),
> f2(offset of block4)
> Other InputFormat has its own methods for files splitting (for example
> Hbase splits input on region boundaries).
>
>
> Hope It Helps,
> Chinna
>
>
> On Fri, Mar 28, 2014 at 4:47 PM, Amjad ALSHABANI <[email protected]>wrote:
>
>> Hello All,
>> I have tow tables created in Hive, one  read the data immidiately from
>> Cassandra DB and the other takes the data from stored files (data already
>> exported in somewhow from the same Cassandra DB)
>>
>> Both tables are identical (in data)
>> but when running the same request it gives me very different number of
>> mapper for each of them (P.S I m using the same hive config for both
>> requests)
>>
>> *hive -e " select count(1) from keyring.cred"*
>>
>> will take 2338 mapper and 1 reducer
>>
>> while:
>>
>> *hive -e "select count(1) from keyring.cred_seq"*
>>
>> will take just 151 mapper and 1 reducer
>>
>> Any idea how to minimize this number, and where does this explosion in
>> mapper number come from??
>>
>> any response will be appreciated :)
>>
>> Cheers!!
>>
>> Amjad
>>
>>
>

Reply via email to