patch for review...
>
>
> From: Ted Yu
> Sent: Tuesday, October 04, 2016 6:28 PM
> To: dev@hbase.apache.org
> Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started
> by Master or RS)
>
> Refactoring work over in HBASE-16727 is read
(WAS => Re: [DISCUSSION] MR jobs started by
Master or RS)
Refactoring work over in HBASE-16727 is ready for review.
Kindly provide your feedback.
Thanks
On Mon, Oct 3, 2016 at 3:05 PM, Andrew Purtell wrote:
> This sounds good to me.
> I'd be at least +0 as to merging the branc
take a backup/restore his/her tables), we can discuss the
> "backup
> > service" or something else.
> > Folks - Stack / Andrew / Matteo / others, please speak up if you disagree
> > with the above. Would like to get over this merge-to-master hump
> obviously
ionov
> Sent: Monday, September 26, 2016 11:48 AM
> To: dev@hbase.apache.org
> Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs
> started by Master or RS)
>
> Ok, we had internal discussion and this is what we are suggesting now:
>
> 1. We will create separ
; Sent: Monday, September 26, 2016 11:48 AM
> To: dev@hbase.apache.org
> Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs
> started by Master or RS)
>
> Ok, we had internal discussion and this is what we are suggesting now:
>
> 1. We will create separate module
apache.org
Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by
Master or RS)
Ok, we had internal discussion and this is what we are suggesting now:
1. We will create separate module (hbase-backup) and move server-side code
there.
2. Master and RS will be MR and backup free
would
> > >> cause
> > >>>>>>>>> probabilistic
> > >>>>>>>>>>> dead
> > >>>>>>>>>>>>> lock or some strange NPEs...
> > >>>>>>>>&g
e service other than master
> >>>>>>>>>>>>>>>>> 2. Shell out from the master
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>&g
gt;>>>>> ever being able to launch MR jobs.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> + MR is dead. We should be busy working hard to undo
> it
> > > > >
gt;>>> + Master is a rats nest of state. Matteo, Stephen, and
> > Appy
> > > > >>>>> are
> > > > >>>>>>>> busy
> > > > >>>>>>>>>>>> working hard on moving it up on to a new foundation.
> Lets
> > > > >>>> not
> > > > >>>&g
t;>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <
> > > >>>>> yuzhih...@gmail.com
> > > >>>>>>>
> > > >>>>
e-server
>>>>>>>>>>>>>>> moving it out to be an optional module (Spark would be its
>>>>>>>>> peer).
>>>>>>>>>>>>>>> + Master is a rats nest of state. Matteo, Stephen, and Appy
>>>>>>>> are
>>>>>&
gt;>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If in the future, we find better ways of
>>>>>> doing
>>>>>>>>> this
>>>&g
>>>>>>>>> is
> > >>>>>>>>>> to
> > >>>>>>>>>>>>> make
> > >>>>>>>>>>>>>> Master more stable.
> > >>>>>>>>>>>>>
;>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I suggest you look at Matteo's work for
>> >>>> AssignmentManager
>> >>>>>>> which
>> >>>>>>
gt;>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> No, not your fault, at lease, not this time:)
> >>>>>>>>>>>>>>>
> >>>
p 23, 2016 at 5:32 AM, 张铎 <
> >>>>> palomino...@gmail.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> No, not your fault, at lease, not this time:)
> &
ster is also a
>>>> regionserver
>>>>>> so
>>>>>>> it
>>>>>>>>>>> extends
>>>>>>>>>>>>>>> HRegionServer, and the initialization of
>>>> HRegionServer
>>&
t;>>>>>>>>>>> add
> >>>>>>>>>>>>> external dependencies to HMaster, especially add more
> >>>> works
> >>>>>> for
> >>>>>>>> the
> >>>>>>>
; :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I read through HADOOP-13433
>>>>>>>>>>>>>> <https://issues.apache.org/
>> jira/browse/HADOOP-13433>
>>> -
>>>&g
gt; > > > > > > > > > > > > If in the future, we find better ways of
> doing
> > > this
> > > > > > > without
> > > > > > > > > > using
> > > > > > > > > > > > MR,
&
t; taking a simple look at it? This is what I mean,
> ugly
> > > > > code...
> > > > > > > > > logout
> > > > > > > > > > > and
> > > > > > > > > > > > > destroy the credentia
t; > > > and
> > > > > > the
> > > > > > > > only
> > > > > > > > > > way
> > > > > > > > > > > > to fix it is to write another piece of ugly code...
> > > > > > > > > > > >
> > > > > > >
t; > > > using
> > > > > > > > > > MR,
> > > > > > > > > > > we
> > > > > > > > > > > > can certainly consider that
> > > > > > > > > > > >
> > >
t; > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> > > > > > d...@hortonworks.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
&
R-based
> > > > > > > > > > > compactions.. But I was thinking more about the
> > > SpliceMachine
> > > > > > > > approach
> > > > > > > > > of
> > > > > > > > > > > managing compactions in Spark where apparently they
> saw a
> > > lot
> >
gt; > > > > > > > of
> > > > > > > > > > > managing compactions in Spark where apparently they
> saw a
> > > lot
> > > > > of
> > > > > > > > > > benefits.
> > > > > > >
> > we
> > > > > > > > > > > >>>> have not used at all), it introduced another cost
> > for
> > > > > > > maintain.
> > > > > > > > > I
> > > > > > > > > > > >>>> don't think it is a good idea.
> > > > > &
> > > > >> if you won't be doing backups.
> > > > > > > > > > >>
> > > > > > > > > > >> This discussion (we do not want see M/R dependency)
> g
sed by HBase already for some operations.
> > > > > > > >
> > > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > > server
> > > > > not
> > > > > > > > being the least of them all. Security (kerberos
&
m
> > > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> still
> > > > > > functions
> > > > > > > >
ember 23, 2016 8:40 AM
To: dev
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
That is the point, Matteo.
Put it another way, there is nothing that stops a user from deploying
custom procedure, custom co-processor that calls out MR job.
The optional feature should satisfy some basic require
etc. etc.). IMO, that approach is DOA.
> > > Instead
> > > > > > let's
> > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > good
> > > > > reason
> > > > >
tab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > > Instead
> > > > > > let's
> > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > good
> > > > > reason
> > > > > > > why the HBase master
> > > > > agreed.
> > > > > >
> > > > > > Now before going to (2), let's see what are the benefits of
> running
> > > the
> > > > > > backup/restore jobs from the master. I think Ted has summarized
> > some
&
> some
> > of
> > > > the
> > > > > issues that we need to take care of - basically, the master can
> keep
> > > > track
> > > > > of running jobs, and should it fail, the backup master can continue
> > > > keeping
> > &
r can also do cleanup, etc. of failed backup/restore processes.
> > > > Security is another issue - the job needs to run as 'hbase' since it
> > owns
> > > > the data. Having the master launch the job makes it get that
> privilege.
> > > In
> &g
gt; > > the data. Having the master launch the job makes it get that privilege.
> > In
> > > the (2) approach, it's hard to do some of the above management.
> > >
> > > Guys, just to reiterate, the patch as such is ready from the overall
> > >
ome of the above management.
> >
> > Guys, just to reiterate, the patch as such is ready from the overall
> > design/arch point of view (maybe code review is still pending from
> Matteo).
> > If in the future, we find better ways of doing this without using MR, we
>
n the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merged.
>
>
> From: 张铎
> Sent: Thursday, September 22, 2016 8:32 PM
> To: dev@hbase.apache.org
>
code review is still pending from Matteo).
>> If in the future, we find better ways of doing this without using MR, we
>> can certainly consider that. But IMO don't think we should block this patch
>> from getting merged.
>>
>> ___
s such is ready from the overall
> design/arch point of view (maybe code review is still pending from Matteo).
> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merge
f in the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merged.
>
>
> From: 张铎
> Sent: Thursday, September 22, 2016 8:32 PM
> To: dev@hbase.apache.o
From: 张铎
Sent: Thursday, September 22, 2016 8:32 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
So what about a standalone service other than master? You can use your own
procedure store in that service?
2016-09-23 11:28
;>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > andrew.purt...@gmail.com
> > > >:
> > > >>>>>>
> > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a
> fair
> > > >>>>>
fair
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> Can this be driven by a utility derived from Tool like our other
> MR
> > >>>> apps?
> > >>>>>>> The issue is ne
cessController to decide if allowed? But
> >>>> nothing
> >>>>>>> prevents the user from running the job manually/independently,
> right?
> >>>>>>>
> >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi &
tozzi <
>>>> theo.berto...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> just a remark. my query was not about tools using MR (everyone i
>>>> think
>>>>>>> is
>>>>>>>
eding the AccessController to decide if allowed? But
>>>>> nothing
>>>>>>>> prevents the user from running the job manually/independently,
>>> right?
>>>>>>>>
>>>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>>>>
>>>>
>>>>>>> just a remark. my query was not about tools using MR (everyone i
>>> think
>>>>>> is
>>>>>>> ok with those).
>>>>>>> the topic was about: "are we ok with running MR jobs from Mast
>>> prevents the user from running the job manually/independently,
>>> right?
>>> >> >>>
>>> >> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>> >> theo.berto...@gmail.com>
>
; >> >>> > just a remark. my query was not about tools using MR (everyone i
>> >> think
>> >> >>> is
>> >> >>> > ok with those).
>> >> >>> > the topic was about: "are we ok with run
>> >>> > just a remark. my query was not about tools using MR (everyone i
> >> think
> >> >>> is
> >> >>> > ok with those).
> >> >>> > the topic was about: "are we ok with running MR jobs from Master
> and
>
gt; ok with those).
> > >>> > the topic was about: "are we ok with running MR jobs from Master
> and
> > RSs
> > >>> > code?" since this will be the first time we do this
> > >>> >
> > >>> > Matteo
> > >>> >
> &g
h those).
>> >>> > the topic was about: "are we ok with running MR jobs from Master and
>> RSs
>> >>> > code?" since this will be the first time we do this
>> >>> >
>> >>> > Matteo
>> >>> >
&g
de?" since this will be the first time we do this
> >>> >
> >>> > Matteo
> >>> >
> >>> >
> >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das
> >>> wrote:
> >>> >>
> >>> >> Very much agree; for tools like ExportSnapshot /
ch agree; for tools like ExportSnapshot / Backup / Restore, it's
>>> >> fine to be dependent on MR. MR is the right framework for such. We
>>> should
>>> >> also do compactions using MR (just saying :) )
>>> >> __
t;> >
>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das
>> wrote:
>> >>
>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
>> >> fine to be dependent on MR. MR is the right framework for such. We
>> sh
right framework for such. We
> should
> >> also do compactions using MR (just saying :) )
> >> ________________________
> >> From: Ted Yu
> >> Sent: Thursday, September 22, 2016 2:00 PM
> >> To: dev@hbase.apache.org
> >
tore, it's fine
>> to be dependent on MR. MR is the right framework for such. We should also do
>> compactions using MR (just saying :) )
>>
>> From: Ted Yu
>> Sent: Thursday, September 22, 2016 2:00 PM
>> To: dev@hb
rk for such. We should also do
> compactions using MR (just saying :) )
>
> From: Ted Yu
> Sent: Thursday, September 22, 2016 2:00 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> I agre
ight framework for such. We should
>> also do compactions using MR (just saying :) )
>>
>> From: Ted Yu
>> Sent: Thursday, September 22, 2016 2:00 PM
>> To: dev@hbase.apache.org
>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>
>>
t; Sent: Thursday, September 22, 2016 4:29 PM
> To: dev
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> Well, I'm just not using those features ;) But was hopping for the MOBs ;)
> My point is, if we can do it without MR, then, why not? )
>
> 2016-09-22 1
R? Curious.
From: Jean-Marc Spaggiari
Sent: Thursday, September 22, 2016 4:29 PM
To: dev
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
Well, I'm just not using those features ;) But was hopping for the MOBs ;)
My point is, if we can do it without MR, then, why
From: Matteo Bertozzi
Sent: Thursday, September 22, 2016 3:44 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
just a remark. my query was not about tools using MR (everyone i think is
ok with those).
the topic was about: "are we ok with runni
R jobs from Master and
> RSs
> >> > code?" since this will be the first time we do this
> >> >
> >> > Matteo
> >> >
> >> >
> >> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das
> >> wrote:
> >> >
> >> > > V
> >
>> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das
>> wrote:
>> >
>> > > Very much agree; for tools like ExportSnapshot / Backup / Restore,
>> it's
>> > > fine to be dependent on MR. MR is the right framework for such. We
>> should
>> > > also do compactions
e:
> >
> > > Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> > > fine to be dependent on MR. MR is the right framework for such. We
> should
> > > also do compactions using MR (just saying :) )
> > > ____________
t; From: Ted Yu
> > Sent: Thursday, September 22, 2016 2:00 PM
> > To: dev@hbase.apache.org
> > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >
> > I agree - backup / restore is in the same category as import / export.
> >
> > On Thu, Sep 22, 2016 at
ht framework for such. We should
> > also do compactions using MR (just saying :) )
> >
> > From: Ted Yu
> > Sent: Thursday, September 22, 2016 2:00 PM
> > To: dev@hbase.apache.org
> > Subject: Re: [DISCUSSION] MR jobs sta
tember 22, 2016 2:00 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> I agree - backup / restore is in the same category as import / export.
>
> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell
> wrote:
>
> > Backup is extra tooli
PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
I agree - backup / restore is in the same category as import / export.
On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell
wrote:
> Backup is extra tooling around core in my opinion. Like import or export.
I agree - backup / restore is in the same category as import / export.
On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell
wrote:
> Backup is extra tooling around core in my opinion. Like import or export.
> Or the optional MOB tool. It's fine.
>
> > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi
> w
Backup is extra tooling around core in my opinion. Like import or export. Or
the optional MOB tool. It's fine.
> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi wrote:
>
> What's the latest opinion around running MR jobs from hbase (Master or RS)?
>
> I remember in the past that there was discus
I would be -1 a requirement for MR for something core to HBase.
> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi wrote:
>
> What's the latest opinion around running MR jobs from hbase (Master or RS)?
>
> I remember in the past that there was discussion about not having MR has
> direct dependency
What's the latest opinion around running MR jobs from hbase (Master or RS)?
I remember in the past that there was discussion about not having MR has
direct dependency of hbase.
I think some of discussion where around MOB that had a MR job to compact,
that later was transformed in a non-MR job to
76 matches
Mail list logo
