I agree. I'm not an expert though. I do more pig jobs than anything. Any one else on the thread have more experience creating MR jobs on HBase data?
On Wednesday, July 13, 2016, Frank Luo <[email protected]> wrote: > It will work, but it is pretty awkward way to create more mappers. > > > > *From:* Billy Watson [mailto:[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>] > *Sent:* Wednesday, July 13, 2016 3:57 PM > *To:* Frank Luo <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> > *Cc:* [email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');> > *Subject:* Re: Re:is possible to create multiple TableSplit per region? > > > > It seems like it might be faster then to consider a map job followed by > another map job. Or, depending on the web service calls, maybe a combine > step? > > > William Watson > Lead Software Engineer > > > > On Wed, Jul 13, 2016 at 4:40 PM, Frank Luo <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > > It makes a number of web-service calls. > > > > *From:* Billy Watson [mailto:[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>] > *Sent:* Wednesday, July 13, 2016 3:27 PM > *To:* [email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');> > *Cc:* Frank Luo <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> > *Subject:* Re: Re:is possible to create multiple TableSplit per region? > > > > What do you mean by "heavy work downstream"? > > > > I think the mailing list might need a *few* more details to help out > better. > > > William Watson > > > > On Wed, Jul 13, 2016 at 12:32 PM, Frank Luo <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > > Thanks for the prompt reply, Lu. > > It is true that having a smaller region file size can solve the problem. > But it also have side effects. For example, total number of regions can be > easily doubled/tripled, and I am already facing a challenge of having too > many regions per server. So I cannot go to that route. > > From: 陆巍 [mailto:[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>] > Sent: Wednesday, July 13, 2016 11:24 AM > To: [email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>; Frank Luo < > [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');>> > Subject: Re:is possible to create multiple TableSplit per region? > > here is an archived mail: > http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%[email protected]%3E > > At 2016-07-13 23:20:28, "Frank Luo" <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');><mailto: > [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');>>> > wrote: > > >We have mapper only jobs operating on a result of a Scan. Because of > heavy work downstream, the mapper runs fairly slowly. So I am wondering if > there is a way to create multiple TableSplit on one region hence multiple > mappers can be created to work on different piece of date on the region. > > > > > >I am aware of MultithreadedTableMapper class, which could be my solution, > but I hesitate to use it as my code is not thread safe. > > > > > >So any suggestions, or code to share? > > > > > > > > > > > >Download the latest installment of our annual Marketing Imperatives, > “Winning with People-Based Marketing”< > http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1> > > > > > >This email and any attachments transmitted with it are intended for use > by the intended recipient(s) only. If you have received this email in > error, please notify the sender immediately and then delete it. If you are > not the intended recipient, you must not keep, use, disclose, copy or > distribute this email without the author’s prior permission. We take > precautions to minimize the risk of transmitting software viruses, but we > advise you to perform your own virus checks on any attachment to this > message. We cannot accept liability for any loss or damage caused by > software viruses. The information contained in this communication may be > confidential and may be subject to the attorney-client privilege. > > > > > Download the latest installment of our annual Marketing Imperatives, > “Winning with People-Based Marketing”< > http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1> > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > > > > *Download the latest installment of our annual Marketing Imperatives, > “Winning with People-Based Marketing”* > <http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1> > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > > > > *Download the latest installment of our annual Marketing Imperatives, > “Winning with People-Based Marketing”* > <http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1> > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > -- William Watson Lead Software Engineer
