Re: [DISCUSS] EOM branch-1.3

[Francis Liu]

Hi Andy,

I see, I chimed in because the previous conversation sounded like 1.3 was
just missing an RM and non-dev 1.3 users. I understand, I wouldn't want to
be the only reason 1.3 is kept alive.

We have a few patches we need to push upstream before we can get off of
1.3. We are working on it but it'll take some time.

Thanks,
Francis





On Thu, Dec 5, 2019 at 7:09 PM Andrew Purtell  wrote:

> Hey Francis,
>
> What is preventing an upgrade to 1.4? Are there specific concerns
> remaining? It's been out there for a long time now, and bug fixed through
> 12 releases. Feel free to contact me offlist if you prefer, no problem.
>
> Committers are voting to EOM 1.3 with their git clients already. It is
> spotty if changes make it that far back even when they should. See extra
> work the RM had to do for the last 1.3 release to compare history and port
> back stuff. Nothing prevents you or someone else from continuing to make
> 1.3 releases but the writing is on the wall here.
>
>
>
> On Thu, Dec 5, 2019 at 6:52 PM Francis Christopher Liu <
> toffer@gmail.com>
> wrote:
>
> > Hi Guys,
> >
> > We are still on 1.3 so it would be in our interest if I can continue to
> > rollout 1.3.z releases. Having said that it is the oldest release branch
> > and I understand the effort it takes to maintain another branch hence I
> > didn't push for it unless there are other reasons than our own for
> keeping
> > it going. If it works for you guys I can send an email to the user list
> to
> > see if that criteria is met?
> >
> > Also I was wondering if retired does that prevent us from rolling out
> > releases with critical/needed fixes?
> >
> > Thanks,
> > Francis
> >
> >
> >
> > On Tue, Dec 3, 2019 at 3:27 AM Sean Busbey  wrote:
> >
> > > If it would change anyone's willingness to maintain the branch, then I
> > > encourage them to go ask about the need on user@hbase.
> > >
> > > AFAIK in the year since we started talking about shutting down
> branch-1.3
> > > no committer or PMC has expressed that their interest would change if
> > > someone on user@hbase felt stuck on 1.3.z.
> > >
> > > Also worth noting that in the month since the 1.3.6 announcement went
> out
> > > noone has showed up to say they can't move off of the release line.
> > >
> > >
> > > On Mon, Dec 2, 2019, 22:53 Andrew Purtell 
> > > wrote:
> > >
> > > > And if a non dev says they won’t move off 1.3? Will it change any
> > > > committer or PMC minds on actually continuing to do 1.3 releases? If
> > not
> > > I
> > > > think we have to call it for lack of interest and bandwidth.
> > > >
> > > > 1.4 is a functional superset of 1.3 and the current stable line
> anyway.
> > > > Seems little reason not to upgrade save inertia or risk aversion.
> > > >
> > > >
> > > > > On Dec 2, 2019, at 5:43 PM, Sean Busbey  wrote:
> > > > >
> > > > > Anyone who wants branch-1.3 to keep having releases has to be
> > willing
> > > > > to volunteer to maintain it. If the note in the 1.3.6 release
> wasn't
> > > > > sufficient motivation to get them to show up on dev@hbase to do
> so,
> > I
> > > > > could put a more explicit mention of it in the EOM message. We'd
> need
> > > > > to come up with some phrasing that didn't leave the status of the
> > > > > release line ambiguous though.
> > > > >
> > > > > For reference, these are the last two EOM announcements we did:
> > > > >
> > > > > * 2.0.z in Sep 2019: https://s.apache.org/slgsa
> > > > > * 1.2.z in Jun 2019:  https://s.apache.org/g8lnu
> > > > >
> > > > > 2.0 and 1.3 were never a release line with the "stable" marker on
> it.
> > > > > 1.2 was the stable release line prior to 1.4.
> > > > >
> > > > >> On Mon, Dec 2, 2019 at 1:58 PM Misty Linville 
> > > wrote:
> > > > >>
> > > > >> Whether any non-dev users are unable to move off 1.3, I suppose.
> > > > >>
> > > > >>> On Mon, Dec 2, 2019 at 11:04 AM Sean Busbey 
> > > wrote:
> > > > >>>
> > > > >>> On what, specifically?
> > > > >>>
> > > >  On Mon, Dec 2, 2019, 11:24 Misty Linville 
> > wrote:
> > > > >>>
> > > >  Should the user list be allowed to weigh in?
> > > > 
> > > >  On Mon, Dec 2, 2019 at 7:33 AM Andrew Purtell <
> > > > andrew.purt...@gmail.com>
> > > >  wrote:
> > > > 
> > > > > I think there is a consensus on moving the stable pointer,
> based
> > on
> > > > > earlier discussion. What I would suggest is a separate thread
> to
> > > > >>> propose
> > > > > it, and if nobody objects, do it.
> > > > >
> > > > >> On Dec 2, 2019, at 5:14 AM, 张铎(Duo Zhang) <
> > palomino...@gmail.com>
> > > >  wrote:
> > > > >>
> > > > >> +1.
> > > > >>
> > > > >> And I think it is time to move the stable pointer to 2.2.x? I
> > know
> > > > >>> that
> > > > >> 2.2.x still has some bugs, especially on the procedure store,
> > but
> > > >  anyway,
> > > > >> we have HBCK2 to fix them.
> > > > >>
> > > > >> And for the current stable release line, 1.4.x, the assignment
> > > > >>> manager
> > > > 

[jira] [Reopened] (HBASE-11288) Splittable Meta

[Francis Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu reopened HBASE-11288:
-

> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Francis Liu
>        Assignee: Francis Liu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Need one more binding vote for 1.3.4 release candidate (RC0)

[Francis Liu]

Hi,

So far this RC has only 2 binding votes, needs one more. I've extended it
until friday (4/26), hopefully we can close this by then.

Thanks!
Francis

Need one more binding vote for 1.3.4 release candidate (RC0)

[Francis Liu]

Hi,

Just a reminder if any of the PMCs has time to help vote on this release.
We need only one more binding vote so far it's just mine and Andy's.

Thanks,
Francis

[VOTE] The first HBase 1.3.4 release candidate (RC0) is available

[Francis Liu]

Hi Folks,

The first HBase 1.3.4 release candidate (RC0) is available for download at
https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.4RC0/ and Maven
artifacts are available in the temporary repository
*https://repository.apache.org/content/repositories/orgapachehbase-1302/
*

The git tag corresponding to the candidate is '1.3.4RC0' (5d44375).

A detailed source and binary compatibility report for this release is
available at
https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.4RC0/compat-check-report.html
.
Please review and raise any concerns if you have them.

A list of the 25 issues resolved in this release can be found at
 https://s.apache.org/uyEI .

Please try out the candidate and vote +1/0/-1.

Prior to making this announcement I made the following checks:

RAT check passes (7u211)
Unit test suite passes (7u211)
Loaded the UI in a browser, poked around (8u144)
A few shell commands (8u144)
ITBLL 25M rows (8u144)

This vote will be open for one week and close April 22, 2018.

Thanks,
Francis

[jira] [Created] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3

[Francis Liu (JIRA)]

Francis Liu created HBASE-22058:
---

 Summary: backport HBASE-HBASE-21791 (Upgrade thrift dependency to 
0.12.0) to 1.4 and 1.3
 Key: HBASE-22058
 URL: https://issues.apache.org/jira/browse/HBASE-22058
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Reporter: Francis Liu
 Fix For: 1.4.10, 1.3.4


Creating a separate Jira to do the backport since the .thrift files differ 
between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from 
branch-1 and regenerated the thrift configs. 

cc [~apurtell]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover

[Francis Liu (JIRA)]

Francis Liu created HBASE-20724:
---

 Summary: Sometimes some compacted storefiles are still opened 
after region failover
 Key: HBASE-20724
 URL: https://issues.apache.org/jira/browse/HBASE-20724
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0
Reporter: Francis Liu


It is important that compacted storefiles of a given compaction execution are 
wholly opened or archived to insure data consistency. ie a storefile containing 
delete tombstones can be archived while older storefiles containing cells that 
were supposed to be deleted are left unarchived thereby undeleting those cells.

When a server fails compaction markers (in the wal edit) are used to determine 
which storefiles are compacted and should be excluded during region open 
(during failover). But the WALs containing compaction markers can be 
prematurely archived even though there are still compacted storefiles for that 
particular compaction event that hasn't been archived yet. Thus losing 
compaction information that needs to be replayed. This is because hlog 
archiving logic only keeps track of flushed storefiles and not archived ones.

https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20704) Sometimes compacted storefiles are archived on region close

[Francis Liu (JIRA)]

Francis Liu created HBASE-20704:
---

 Summary: Sometimes compacted storefiles are archived on region 
close
 Key: HBASE-20704
 URL: https://issues.apache.org/jira/browse/HBASE-20704
 Project: HBase
  Issue Type: Bug
  Components: Compaction
Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0
Reporter: Francis Liu


During region close compacted files which have not yet been archived by the 
discharger are archived as part of the region closing process. It is important 
that these files are wholly archived to insure data consistency. ie a storefile 
containing delete tombstones can be archived while older storefiles containing 
cells that were supposed to be deleted are left unarchived. 

On region close a compacted storefile is skipped from archiving if it has read 
references (ie open scanners). This behavior is correct for when the discharger 
chore runs but on region close consistency is of course more important so we 
should add a special case to ignore any references on the storefile and go 
ahead and archive it. 

Attached patch contains a unit test that reproduces the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Please welcome Francis Liu to the HBase PMC

[Francis Liu]

Thanks a lot guys! Looking forward to more 1.3 releases and beyond.

Francis


On Thu, Apr 12, 2018 at 4:45 PM Stack <st...@duboce.net> wrote:

> Hot dog!
>
> On Wed, Apr 11, 2018 at 1:03 PM, Andrew Purtell <apurt...@apache.org>
> wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Francis
> > Liu has accepted our invitation to become a PMC member on the Apache
> > HBase project. We appreciate Francis stepping up to take more
> > responsibility in the HBase project. He has been an active contributor to
> > HBase for many years and recently took over responsibilities as branch RM
> > for branch-1.3.
> >
> > Please join me in welcoming Francis to the HBase PMC!
> >
> > --
> > Best regards,
> > Andrew
> >
>

[ANNOUNCE] Apache HBase 1.3.2 is now available for download

[Francis Liu]

The HBase team is happy to announce the immediate availability of Apach​e
HBase 1.3.2!

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

Download through an ASF mirror:

https://www.apache.org/dyn/closer.lua/hbase/1.3.2

HBase 1.3.2 is the third release of the new HBase 1.3 line, continuing on
the theme of bringing a stable, reliable database to the Apache Big Data
ecosystem and beyond.

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

There are 256 issues resolved which includes 19 blockers, 47 critical
and 121 major issues. The full list can be found at
*https://s.apache.org/aMSp *

Thanks to all the contributors who made this release possible!

Cheers,
The HBase Dev Team

[VOTE] HBase 1.3.2 (RC1) - Needs one more vote

[Francis Liu]

Hi Guys,

HBase 1.3.2 (RC1) needs one more binding vote. I'm extending the vote by 72
hours which ends Thursday 22:00 PST. If any PMC member has the time to take
a look would appreciate the help.

Thanks,
Francis

[VOTE] First release candidate for HBase 1.3.2 (RC1) is available

[Francis Liu]

Hi,

The second release candidate for Apache HBase 1.3.2 is available to
download and testing.

Artifacts are available here:

*https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.2RC1/
*

Maven artifacts are available in the staging repository:

*https://repository.apache.org/content/repositories/orgapachehbase-1205
*

All artifacts are signed with my code signing key D646D9F3, which is also
in the project KEYS file at

http://www.apache.org/dist/hbase/KEYS

these artifacts correspond to commit hash

1bedb5bfbb5a99067e7bc54718c3124f632b6e17 tagged as 1.3.2RC1.

HBase 1.3.2 is the second maintenance release in the HBase 1.3.z release
line, continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities. This release includes 256 resolved issues
since 1.3.1 released 11 months ago.

The full list of issues addressed is available at

*https://s.apache.org/aMSp *

and also in the CHANGES.txt file in the root directory of the source
tarball.

Please take a few minutes to verify the release and vote on
releasing it:

[ ] +1 Release this package as Apache HBase 1.3.2
[ ] +0 no opinion
[ ] -1 Do not release this package because...

This Vote will run for one week and close Mon Mar 26, 2018 22:00 PST.

Thanks

[jira] [Created] (HBASE-20174) Fix TestZKLessMergeOnCluster flakiness

[Francis Liu (JIRA)]

Francis Liu created HBASE-20174:
---

 Summary: Fix TestZKLessMergeOnCluster flakiness
 Key: HBASE-20174
 URL: https://issues.apache.org/jira/browse/HBASE-20174
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu
 Fix For: 1.5.0, 1.4.3, 1.3.2


TestZKLessMergeOnCluster.testCleanMergeReference() sometimes fails because the 
condition used to wait for region compaction to complete is not comprehensive 
enough which causes a race sometimes leaving the new compacted storefile not 
fully committed preventing the discharger from cleaning up the storefile 
references.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] First release candidate for HBase 1.3.2 (RC0) is available

[Francis Liu]

Hi,

I'm happy to announce the first release candidate for Apache HBase 1.3.2 is
available to download and testing.

Artifacts are available here:

*https://dist.apache.org/repos/dist/dev/hbase/hbase-1.3.2RC0/
*

Maven artifacts are available in the staging repository:

*https://repository.apache.org/content/repositories/orgapachehbase-1202
*

All artifacts are signed with my code signing key D646D9F3, which is also
in the project KEYS file at

http://www.apache.org/dist/hbase/KEYS

these artifacts correspond to commit hash

4fc36c6e4ee84091d02a7e26fc45bae3fcb0e30f tagged as 1.3.2RC0.

HBase 1.3.2 is the second maintenance release in the HBase 1.3.z release
line, continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities. This release includes 241 resolved issues
since 1.3.1 released 11 months ago.

The full list of issues addressed is available at

*https://s.apache.org/aMSp *

and also in the CHANGES.txt file in the root directory of the source
tarball.

Please take a few minutes to verify the release and vote on
releasing it:

[ ] +1 Release this package as Apache HBase 1.3.2
[ ] +0 no opinion
[ ] -1 Do not release this package because...

This Vote will run for one week and close Wed Mar 14, 2018 22:00 PST.

Thanks

Also extra thanks to Andy for helping me with this release.

Francis

[jira] [Created] (HBASE-20001) cleanIfNoMetaEntry() uses encoded instead of region name to lookup region

[Francis Liu (JIRA)]

Francis Liu created HBASE-20001:
---

 Summary: cleanIfNoMetaEntry() uses encoded instead of region name 
to lookup region
 Key: HBASE-20001
 URL: https://issues.apache.org/jira/browse/HBASE-20001
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.1.7, 1.4.0, 1.3.0, 1.2.0
Reporter: Francis Liu
Assignee: Thiruvel Thirumoolan


In RegionStates.cleanIfNoMetaEntry()

{{if (MetaTableAccessor.getRegion(server.getConnection(), 
hri.getEncodedNameAsBytes()) == null) {}}

{{regionOffline(hri);}}{{ }}

{{FSUtils.deleteRegionDir(server.getConfiguration(), hri);}}
{{ }}}

But api expects regionname

{{ public static Pair<HRegionInfo, ServerName> getRegion(Connection connection, 
byte [] regionName)}}

So we might cleaning good regions.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Considering branching for 1.5 and other branch-1 release planning

[Francis Liu]

> If someone else is using 1.3 your feedback would be very
valuable.

We are running 1.3 in production, full rollout ongoing. Ran into some
issues but it's generally been stable. We'll prolly gonna be on 1.3 for a
while.

Cheers,
Francis


On Sun, Feb 4, 2018 at 10:59 AM Andrew Purtell  wrote:

> Hi Ted,
>
> If Hadoop 3 support is in place for an (eventual) 1.5.0 release, I think
> that would be great.
>
>
> On Sun, Feb 4, 2018 at 10:55 AM, Ted Yu  wrote:
>
> > Andrew:
> > Do you think making 1.5 release support hadoop 3 is among the goals ?
> >
> > Cheers
> >
> > On Fri, Feb 2, 2018 at 3:28 PM, Andrew Purtell 
> > wrote:
> >
> > > The backport of RSGroups to branch-1 triggered the opening of the 1.4
> > code
> > > line as branch-1.4 and releases 1.4.0 and 1.4.1.
> > >
> > > After the commit of HBASE-19858 (Backport HBASE-14061 (Support CF-level
> > > Storage Policy) to branch-1), storage policy aware file placement might
> > be
> > > useful enough to trigger a new minor release from branch-1. This would
> be
> > > branch-1.5, and at least release 1.5.0. I am not sure about this yet.
> It
> > > needs testing. I'd like to mock up a couple of use cases and determine
> if
> > > what we have is sufficient on its own or more changes will be needed. I
> > > want to get the idea of a 1.5 on your radar. though.
> > >
> > > Also, I would like to make one more release of branch-1.3 before we
> > retire
> > > it. Mikhail passed the reins. We might have a volunteer to RM 1.3.2. If
> > > not, I will do it. I'm expecting 1.4 will supersede 1.3 but this will
> be
> > > decided organically depending on uptake.
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[jira] [Created] (HBASE-18235) LoadBalancer.BOGUS_SERVER_NAME should not have a bogus hostname

[Francis Liu (JIRA)]

Francis Liu created HBASE-18235:
---

 Summary: LoadBalancer.BOGUS_SERVER_NAME should not have a bogus 
hostname
 Key: HBASE-18235
 URL: https://issues.apache.org/jira/browse/HBASE-18235
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu


The original patch used localhost to have assignment fail fast. Avoiding 
misleading DNS exceptions, delays due to dns lookup, etc. 

Was wondering what the reason was for changing it?




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] No regions on Master node in 2.0

[Francis Liu]

> That doesn't solve the same problem.Agreed as mentioned regionserver groups 
>only provides user-system region isolation. 
> That still means that the most important operations are competing for rpc 
>queue time.Given the previous setup. For meta access contention this should be 
>addressed by higher priority rpc access no?

On Tuesday, June 6, 2017 9:17 AM, Elliott Clark <ecl...@apache.org> wrote:
 

 That doesn't solve the same problem. Dedicating a remote server for the
system tables still means that all the master to system tables mutations
and reads are done over rpc. That still means that the most important
operations are competing for rpc queue time.

On Fri, Nov 18, 2016 at 11:37 AM, Francis Liu <tof...@ymail.com.invalid>
wrote:

> Just some extra bits of information:
>
> Another way to isolate user regions from meta is you can create a
> regionserver group (HBASE-6721) dedicated to the system tables. This is
> what we do at Y!. If the load on meta gets too high (and it does), we split
> meta so the load gets spread across more regionservers (HBASE-11165) this
> way availability for any client is not affected. Tho agreeing with Stack
> that something is really broken if high priority rpcs cannot get through to
> meta.
> Does single writer to meta refer to the zkless assignment feature? If
> isn't that feature has been available since 0.98.6 (meta _not_ on master)?
> and we've been running with it on all our clusters for quite sometime now
> (with some enhancements ie split meta etc).
> Cheers,Francis
>
>    On Wednesday, November 16, 2016 10:47 PM, Stack <st...@duboce.net>
> wrote:
>
>
>  On Wed, Nov 16, 2016 at 10:44 PM, Stack <st...@duboce.net> wrote:
>
> > On Wed, Nov 16, 2016 at 10:57 AM, Gary Helmling <ghelml...@gmail.com>
> > wrote:
> >
> >>
> >> Do you folks run the meta-carrying-master form G?
> >
> > Pardon me. I missed a paragraph. I see you folks do deploy this form.
> St.Ack
>
>
>
>
>
> > St.Ack
> >
> >
> >
> >
> >
> >>
> >>
> >> > > >
> >> > > Is this just because meta had a dedicated server?
> >> > >
> >> > >
> >> > I'm sure that having dedicated resources for meta helps.  But I don't
> >> think
> >> > that's sufficient.  The key is that master writes to meta are local,
> and
> >> do
> >> > not have to contend with the user requests to meta.
> >> >
> >> > It seems premature to be discussing dropping a working implementation
> >> which
> >> > eliminates painful parts of distributed consensus, until we have a
> >> complete
> >> > working alternative to evaluate.  Until then, why are we looking at
> >> > features that are in use and work well?
> >> >
> >> >
> >> >
> >> How to move forward here? The Pv2 master is almost done. An ITBLL
> bakeoff
> >> of new Pv2 based assign vs a Master that exclusively hosts hbase:meta?
> >>
> >>
> >> I think that's a necessary test for proving out the new AM
> implementation.
> >> But remember that we are comparing a feature which is actively
> supporting
> >> production workloads with a line of active development.  I think there
> >> should also be additional testing around situations of high meta load
> and
> >> end-to-end assignment latency.
> >>
> >
> >
>
>
>
>


   

Re: DISCUSS: Merge new Assignment Manager (AMv2), HBASE-14614

[Francis Liu]

 > The list of items for 2.0 are here [1]. I think you are talking about
> something that is NOT in this doc. If so, yeah, we are way behind for 2.0.

I see good to know, I was mainly asking for split meta and favored nodes both 
of which are in the doc. FN has most of the stuff in and split meta I can start 
rebasing on AMv2. 
> Perhaps we could add a few place-holders for your incoming in a 2.0 so it
> could come in at a later time, in a 2.1 (or 3.0 if it was close behind)?
This is good to know as well. 

Thanks,Francis

On Wednesday, May 24, 2017 5:24 PM, Stack <st...@duboce.net> wrote:
 

 On Wed, May 24, 2017 at 1:56 PM, Francis Liu <tof...@apache.org> wrote:

> I'd be up for taking this for a spin. As I need to get split meta rebased
> on top of this anyway.


Great.



> Also I'd generally like to know more about AMv2 (transition from zkless
> etc).



See the linked doc. If you need more, call it out and I'll fill it in.



> BTW once 2.0 is branched does that mean we can't merge in any big features
> anymore?
> Thanks,Francis
>
>
The list of items for 2.0 are here [1]. I think you are talking about
something that is NOT in this doc. If so, yeah, we are way behind for 2.0.
Perhaps we could add a few place-holders for your incoming in a 2.0 so it
could come in at a later time, in a 2.1 (or 3.0 if it was close behind)?

Thanks,
S

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.v21r9nz8g01j



>
>
>
>    On Wednesday, May 24, 2017 8:37 AM, Stack <st...@duboce.net> wrote:
>
>
>  On Wed, May 24, 2017 at 8:11 AM, Sean Busbey <bus...@apache.org> wrote:
>
> > Presuming merge happens, looking at the TODOs I see 1 called out
> > expressly as a blocker and 2 called out as minor. Is it safe to assume
> > the remainder have been evaluated and found some where in between, or
> > will we need a triaging step once they're in JIRA to determine number
> > of blockers before branch-2 would be ready for release?
> >
> > The former.
>
> The item called out as a BLOCKER I have not been unable to reproduce in a
> few weeks now. I should probably knock down its priority given this fact
> (and that should it happen -- we are talking about dropping Procedures
> because of Procedure WAL Corruption -- it will be for hbck to cover until
> source of corruption is figured).
>
> I do not know of any other BLOCKERs, not at this point. We want hbase2
> rolling upgradeable from hbase1 but that I see as a separate task, one we
> might as a community want to punt on, and besides, AMv2, as I see it, would
> be a prerequisite.
>
> The others are tasks of some substance, worthy of call out, but for the
> most part improvement and finishing.
>
> No harm in me filing JIRAs for the non-nebulous of [1]. Let me work on this
> today.
>
> Thanks Sean,
> St.Ack
>
> 1.
> https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.xp9zndoycwj
>
>
>
> > On Wed, May 24, 2017 at 2:22 AM, Stack <st...@duboce.net> wrote:
> > > I would like to discuss merging branch HBASE-14614 to master.
> > >
> > > The new AssignmentManager on HBASE-14614 branch promises new levels of
> > > scaling, insight, resilience and assignment performance. A skeleton dev
> > > overview is available here [1].
> > >
> > > It has been in development a good while now. Currently it is working at
> > > least as well as our current assign. Cluster testing has me into a new
> > > realm of scale where the failures are for reasons other than AM; e.g.
> > > massive WALs that take tens of minutes to split holding up restore post
> > > crash (TODO).
> > >
> > > The AMv2 patch is large, unfortunately. Most of the bulk comes of
> > generated
> > > protobuf changes but the patch is also large because the commit could
> not
> > > deliver an half-baked machine.
> > >
> > > The patch is in need of review. Independent confirmation that it is
> > > basically working would be sweet.
> > >
> > > There are outstanding items to complete but they can come in
> post-merge.
> > > Critical is reenabling a set of tests disabled because they depend on
> > > workings no longer present or they depended on a broken behavior in
> AMv1
> > > (Disabled tests and the list of TODOs are in [1]).
> > >
> > > Currently I am working on polish making it pass the outstanding failing
> > > unit tests, fixing white space, and findbugs but it is ready for
> commit.
> > > I've put up an RB patch that is absent protobufs so th

Re: DISCUSS: Merge new Assignment Manager (AMv2), HBASE-14614

[Francis Liu]

I'd be up for taking this for a spin. As I need to get split meta rebased on 
top of this anyway. Also I'd generally like to know more about AMv2 (transition 
from zkless etc). BTW once 2.0 is branched does that mean we can't merge in any 
big features anymore?
Thanks,Francis


 

On Wednesday, May 24, 2017 8:37 AM, Stack  wrote:
 

 On Wed, May 24, 2017 at 8:11 AM, Sean Busbey  wrote:

> Presuming merge happens, looking at the TODOs I see 1 called out
> expressly as a blocker and 2 called out as minor. Is it safe to assume
> the remainder have been evaluated and found some where in between, or
> will we need a triaging step once they're in JIRA to determine number
> of blockers before branch-2 would be ready for release?
>
> The former.

The item called out as a BLOCKER I have not been unable to reproduce in a
few weeks now. I should probably knock down its priority given this fact
(and that should it happen -- we are talking about dropping Procedures
because of Procedure WAL Corruption -- it will be for hbck to cover until
source of corruption is figured).

I do not know of any other BLOCKERs, not at this point. We want hbase2
rolling upgradeable from hbase1 but that I see as a separate task, one we
might as a community want to punt on, and besides, AMv2, as I see it, would
be a prerequisite.

The others are tasks of some substance, worthy of call out, but for the
most part improvement and finishing.

No harm in me filing JIRAs for the non-nebulous of [1]. Let me work on this
today.

Thanks Sean,
St.Ack

1.
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.xp9zndoycwj



> On Wed, May 24, 2017 at 2:22 AM, Stack  wrote:
> > I would like to discuss merging branch HBASE-14614 to master.
> >
> > The new AssignmentManager on HBASE-14614 branch promises new levels of
> > scaling, insight, resilience and assignment performance. A skeleton dev
> > overview is available here [1].
> >
> > It has been in development a good while now. Currently it is working at
> > least as well as our current assign. Cluster testing has me into a new
> > realm of scale where the failures are for reasons other than AM; e.g.
> > massive WALs that take tens of minutes to split holding up restore post
> > crash (TODO).
> >
> > The AMv2 patch is large, unfortunately. Most of the bulk comes of
> generated
> > protobuf changes but the patch is also large because the commit could not
> > deliver an half-baked machine.
> >
> > The patch is in need of review. Independent confirmation that it is
> > basically working would be sweet.
> >
> > There are outstanding items to complete but they can come in post-merge.
> > Critical is reenabling a set of tests disabled because they depend on
> > workings no longer present or they depended on a broken behavior in AMv1
> > (Disabled tests and the list of TODOs are in [1]).
> >
> > Currently I am working on polish making it pass the outstanding failing
> > unit tests, fixing white space, and findbugs but it is ready for commit.
> > I've put up an RB patch that is absent protobufs so the bulk is less
> > intimidating [3].
> >
> > This feature is the last hurdle to our branching for hbase2.
> >
> > Thanks,
> > St.Ack
> >
> > P.S. This patch is mostly the work of Matteo Bertozzi (inception, bulk of
> > implementation). Others have contributed including Stephen Yuan Jiang.
> > Umesh Agashe and Appy also have contributed fixes that got added to the
> > HBASE-14614 branch.
> >
> > 1.
> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > 2.
> > https://docs.google.com/document/d/1QLXlVERKt5EMbx_
> EL3Y2u0j64FN-_TrVoM5WWxIXh6o/edit#heading=h.df9krsl9k16
> > 3. https://reviews.apache.org/r/55117/
>


   

Re: Moving 2.0 forward

[Francis Liu]

Hi Stack,
I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a 2.x 
draft up next week. Was working on the 1.x version internally.
Also if you'd like I can be the owner for rsgroups as well. 
Thanks,Francis






On Wednesday, January 18, 2017 11:29 AM, Stack  wrote:
 

 Done Thiruvel (And thanks Guanghao for adding hbase-replication).
St.Ack

On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
thiru...@yahoo-inc.com.invalid> wrote:

> Hi Stack,
> I would like to add Favored Nodes to the ancillary section.
> HBASE-15531: Favored Nodes EnhancementsStatus: Active development.Owner:
> Thiruvel Thanks!-Thiruvel
>
>    On Monday, January 16, 2017 2:10 PM, Stack  wrote:
>
>
>  On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang 
> wrote:
>
> > For 6. Significant contirbs in master only, there are some issues about
> > replication operations routed through master. They are sub-task
> > of HBASE-10504. And there are other umbrella issue for replication, like
> > HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> > tracking from Zookeeper to HBase. So i thought we can add a new section
> > named
> > hbase-replication to possible 2.0.0s. This will help us to track the
> state.
> > Thanks.
> >
>
> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to have a
> go at a first cut, be my guest. If nothing done in the next day or so, I'll
> add this section Sir.
> Thanks,
> M
>
>
>
>


   

Re: HBASE 2.0 release progress

[Francis Liu]

> I suspect this feature will be considered risky but I'd be willing to try it 
> out and give it some production-like load (and with chaos). By the time we 
> would be iterating our platform on 2.0 we'd have the type of production loads 
> fielded that could overpressure meta on a single server. 

Thanks Andy. Am working on a 1.x (for internal use) and will try to put both a 
1.x (for reference) and 2.x version.

 

On Tuesday, November 22, 2016 10:05 AM, Andrew Purtell 
 wrote:
 

 Thanks. Are your experiences with FNv2 in production documented? Perhaps on 
the JIRA? That would go a long way.  


> On Nov 22, 2016, at 12:44 PM, Thiruvel Thirumoolan  
> wrote:
> 
> Hi Andrew,
> 
> Thanks for your comments.
> 
> FN v2 happened due to the same reasons you mention. We believe v2 is complete 
> based on our experience using and operating it in production since early 
> 2016. We have a design doc @ umbrella jira HBASE-15531 and have responded to 
> all questions we had so far. More eyes will always help.
> 
> -Thiruvel
> 
> 
> On Tuesday, November 22, 2016, 7:13:08 AM PST, Andrew Purtell 
>  wrote:
> Favored nodes version 1 did not see much adoption and was hard to use or 
> unusable, depending on who you ask, so please expect a focus on usability and 
> feature completeness when/if v2 is reviewed for possible inclusion in 2.0. 
> While I am not suggesting this is the case, if the v2 candidate shares the 
> feature incompleteness or usability problems of its predecessor it should not 
> go in. 
> 
> 
> > On Nov 21, 2016, at 6:01 PM, Thiruvel Thirumoolan 
> >  wrote:
> > 
> > We would like to add the favored node enhancements as part of 
> > https://issues.apache.org/jira/browse/HBASE-15531 to 2.0. Most of the code 
> > should be new, there will be very less code changes to Master/AM etc.
> > 
> > -Thiruvel
> > 
> > On Monday, November 14, 2016, 10:06:49 PM PST, Anoop John 
> >  wrote:>-  Anyone (Ram, Anoop?) wants to post a 
> > high-level writeup on the current
> > up-to-date state of offheaping?
> > 
> > Off heaping the read path (HBASE-11425) this is completed some time
> > back.  As per my knowledge at least 2 users back ported this work to
> > 1.x based version and even running in production. Am leaving it them
> > for giving details..
> > 
> > Write path off heaping HBASE-15179 is the umbrella issue.  As u can
> > see most of the sub tasks are done by now..  Mainly 2 more sub tasks
> > are yet to get committed.  Both are some what bigger sized also..
> > Patch is already available for them.  We hope it can be completed by
> > Nov end.
> > 
> > Some more off heaping work, (in compaction path etc) might be taken up
> > after write path work. Alibaba guys might be joining with that.  Had
> > some discuss with them.. We will come up with jira and doc for that
> > before actual work begin.  But as Enis said ya all that can go in 2.x
> > (x>0) :-)
> > 
> > Just giving some high level status update. Thanks ..
> > 
> > -Anoop-
> > 
> > 
> >> On Tue, Nov 15, 2016 at 7:31 AM, Nick Dimiduk  wrote:
> >> Enis, by your criteria it seems the log4j2 upgrade ticket should be
> >> considered for 2.0 as well: HBASE-12341.
> >> 
> >>> On Monday, November 14, 2016, Enis Söztutar  wrote:
> >>> 
> >>> Another way to look at whether a feature is a "blocker" or not for 2.0 is 
> >>> a
> >>> simple test:
> >>>  - Can this feature be committed for 2.1, 2.2, etc or not assuming it does
> >>> not make into 2.0.
> >>> 
> >>> It is a simple test, but affectively validates whether the feature is
> >>> client-visible and have backwards compatibility implications. If it is ok
> >>> to bring the feature in for 2.1 or later, it means that the release should
> >>> never block on it. Otherwise, we should at least have the API / Client
> >>> visible parts of it in 2.0 if not the full thing.
> >>> 
> >>> Looking at the features under discussion through this lens reveals easier
> >>> decision points I think. A couple of examples:
> >>>  - Java async client / C++ Client: It is independent work, can come in 2.1
> >>> or 2.2, etc. Whenever it is completed.
> >>>  - Offheaping : transparent ti clients, so can come in incrementally in
> >>> 2.0, 2.1, 2.2, etc.
> >>> 
> >>> On the other hand, 1.x releases have been going on for >1.5 years, and 
> >>> will
> >>> likely go on for another year at least. So the choices (dependency, APIs,
> >>> etc) that we make now for 2.0 will likely live for >2 years. In that
> >>> respect I would rather be a bit more aggressive in terms of dropping
> >>> support for older stuff and updating wherever we can. For example require
> >>> Hadoop-2.6+, Java-8, etc.
> >>> 
> >>> Enis
> >>> 
> >>> On Mon, Nov 14, 2016 at 12:15 PM, Mikhail Antonov  >>> >
> >>> wrote:
> >>> 
>  Great to see progress here.
>  
> 

Re: HBASE 2.0 release progress

[Francis Liu]

Late to respond to this but I'd like to get "Split Meta" HBASE-11165 into HBase 
2.0 as well. We've been running with a version of it internally. I'll start 
putting up patches soon. 

On Friday, November 11, 2016 5:15 PM, Stephen Jiang 
 wrote:
 

 Hello, fellow HBASE developers,

We are making progress towards HBASE 2.0 releases.  I am using the
following queries to search for on-going HBASE 2.0 feature work items
(project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND
resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND
issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have
247!  That is a lot.  At some time in near future, we definitely need to
trim down the list; otherwise, 2.0 will never be released.

For now, Matteo and I are tracking some big projects that are on-going:

(1). HBASE-14350 the stable Assignment Manager (using Procedure V2)
- This is a blocker to have branch-2 cut.  In the past few weeks, we made
good progress and majority of implementation is done.  The patches are
under review and testing.  Matto is drafting a document for review.

(2). HBASE-14414 Backup/Restore Phase 3
- Currently it is blocked by HBASE-14123.  The giant HBASE-14123 patches
was discussed and reviewed from the community (see
http://www.mail-archive.com/dev@hbase.apache.org/msg41090.html for the long
discussion); and all feedback were taken care of in the latest patch.
Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further
develpment of backup/restore is blocked - the backup/restore is a key
enterprise feature for 2.0 release.  I think HBASE-14123 is ready for
another round of vote.

(3). HBASE-15179 Offheap
- This is another important feature that I think it is MUST for 2.0.  Since
stack works closely with Intel developers, he has some insight on this
project: "Intel are betting on this. Alibaba are using the offheap read
path and interested in write path too. This work is still very much up in
the air and being worked out as we go (especially the Y! Israel inmemory
compaction component).  It is a little shakey dependent on mslab pooling,
blockcache being on by default, async wal being default, and then dependent
on lots of perf and ITBLL testing."

(4). HBASE-16952 Protobuf3
- Good news, stack got the majority of work done already.  This is a MUST
for 2.0 release.  Now we only have a small sub-task HBASE-16967 (findbugs)
left.

(5). HBASE-14070 Logical Clock
- This needs to be done before 2.0 release.  At this time, seems not making
much progress.

(6). HBASE-16833 JAVA Async Client
- A lot of progress was made by Duo and his associates.  We should be in a
good shape in JAVA client when 2.0 release.

(7). HBASE-14850 C++ Async Client
- Another project that is making good progress.  I know some customer wants
this.  This is long overdue project.  Hopefully we will make those customer
happy in 2.0 release with a good performed C++ client.

Please let me know other on-going projects that needs to be in HBASE 2.0
release (stack mentioned "logical file system tier", but I am not sure
whether it would make enough progress to have a realistic chance making
2.0).  As I said at the beginning of this email, at some point soon, we
will be more picky and trim down the unfinished features in 2.0.

Happy Weekend!
Stephen


   

Re: [DISCUSS] No regions on Master node in 2.0

[Francis Liu]

Just some extra bits of information:

Another way to isolate user regions from meta is you can create a regionserver 
group (HBASE-6721) dedicated to the system tables. This is what we do at Y!. If 
the load on meta gets too high (and it does), we split meta so the load gets 
spread across more regionservers (HBASE-11165) this way availability for any 
client is not affected. Tho agreeing with Stack that something is really broken 
if high priority rpcs cannot get through to meta. 
Does single writer to meta refer to the zkless assignment feature? If isn't 
that feature has been available since 0.98.6 (meta _not_ on master)? and we've 
been running with it on all our clusters for quite sometime now (with some 
enhancements ie split meta etc). 
Cheers,Francis 

On Wednesday, November 16, 2016 10:47 PM, Stack  wrote:
 

 On Wed, Nov 16, 2016 at 10:44 PM, Stack  wrote:

> On Wed, Nov 16, 2016 at 10:57 AM, Gary Helmling 
> wrote:
>
>>
>> Do you folks run the meta-carrying-master form G?
>
> Pardon me. I missed a paragraph. I see you folks do deploy this form.
St.Ack





> St.Ack
>
>
>
>
>
>>
>>
>> > > >
>> > > Is this just because meta had a dedicated server?
>> > >
>> > >
>> > I'm sure that having dedicated resources for meta helps.  But I don't
>> think
>> > that's sufficient.  The key is that master writes to meta are local, and
>> do
>> > not have to contend with the user requests to meta.
>> >
>> > It seems premature to be discussing dropping a working implementation
>> which
>> > eliminates painful parts of distributed consensus, until we have a
>> complete
>> > working alternative to evaluate.  Until then, why are we looking at
>> > features that are in use and work well?
>> >
>> >
>> >
>> How to move forward here? The Pv2 master is almost done. An ITBLL bakeoff
>> of new Pv2 based assign vs a Master that exclusively hosts hbase:meta?
>>
>>
>> I think that's a necessary test for proving out the new AM implementation.
>> But remember that we are comparing a feature which is actively supporting
>> production workloads with a line of active development.  I think there
>> should also be additional testing around situations of high meta load and
>> end-to-end assignment latency.
>>
>
>


   

[jira] [Created] (HBASE-15733) Forward port some fixes/tweaks fined in branch-1 backport

[Francis Liu (JIRA)]

Francis Liu created HBASE-15733:
---

 Summary: Forward port some fixes/tweaks fined in branch-1 backport
 Key: HBASE-15733
 URL: https://issues.apache.org/jira/browse/HBASE-15733
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu


Main thing that was found broken is a small typo in get_rsgroup cli command. 
Till it's fixed the cli command won't work.

Other small fixes are: 
- maven dependency cleanup in hbase-rsgroup module
- Removing a try{} catch block in AM since the exception is handled properly by 
callers.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] No regions on Master node in 2.0

[Francis Liu]

Very late to the party
IMHO having the master doing only gardening and not become a part of the user 
access path is a good design and something we should stick to. It's good SoC 
(ie makes gardening tasks more isolated from user workload).
> we double assign and lose data.
Given that meta only has a single writer/manager (aka master) this IMHO is more 
about having a clean state machine than because of remotely writing to a 
region. We should be able to remain in a good state in the event of write 
failures. After all even the writes to the filesystem involve remote writes. 
> Running ITBLL on a loop that creates a new table every time, and without meta 
>on master everything will fail pretty reliably in ~2 days.
This is interesting. I'll give it a try. Just run generator for 2 days? 
Creating a new table everytime? Do I drop the old one? 
> Short circuit local reads, Caching blocks in jvm, etc. Bringing data closer 
>to the interested party has a long history of making things faster and better.
AFAIK All the metadata that the master needs is already cached in memory during 
startup. It does not require meta to be on master.
> Master is in charge of just about all mutations of all systems tables.
Locality is not as useful here writes still end up being remote by virtue of 
hdfs.
> At FB we've seen read throughput to meta doubled or more by swapping it to 
>master. Writes to meta are also much faster since there's no rpc hop, no 
>queueing, to fighting with reads. So far it has been the single biggest thing 
>to make meta faster.
This can be addressed with region server groups. :-) As in this case that's 
pretty much what you're doing here having a special region server, serve system 
tables, isolating it from user tables. The upside is you can have more than one 
"system regionserver" in this case. This is how we do things internally so 
we've never experienced user region access interfering with meta. 
> For example, it makes it possible to cold-start HBase under load, where a 
>non-meta-serving master is never able to successfully complete initialization.
Is this problem here because meta is affected by user region workloads? If so 
region server groups should help in this case as well.
> If we really think that normal master housekeeping functions are work enough 
>that we shouldn't combine with region serving, then why do we think that those 
>will _not_ have to be scaled by splitting the metadata space across multiple 
>servers when we encounter meta-scaling issues that require splitting meta to 
>distribute it across multiple servers?  
Based on our tests a single master (without meta) is fine with handling a few 
million regions the bottlenecks are elsewhere (ie updating meta). 







 



 

On Tuesday, April 12, 2016 11:55 AM, Gary Helmling  
wrote:
 

 >
> # Without meta on master, we double assign and lose data.
>
> I doubt meta on master solve this problem.
> This has more to do on the fact that balancer, assignment, split, merge
> are disjoint operations that are not aware of each other.
> also those operation in general consist of multiple steps and if the master
> crashes you may end up in an inconsistent state.
>
>
Meta-on-master does dramatically improve things.  For example, it makes it
possible to cold-start HBase under load, where a non-meta-serving master is
never able to successfully complete initialization.  This is the difference
between a cluster being able to come to a healthy state vs. one that is
never able to complete assignments, communicate those assignments to
clients and come to a steady state.


> this is what proc-v2 should solve. since we are aware of each operation
> there is no chance of double assignment and similar by design.
>
>
Again, I think it is difficult to compare an existing feature that is
working in production use vs. one that is actively being developed in
master.

Preventing double assignment sounds great.  When happens when the update of
meta to communicate this to clients fails?  So long as meta is served
elsewhere you still have distributed state.

Until we have an alternative that is feature complete and has demonstrated
success and stability in production use, I don't see how we can even
propose removing a feature that is solving real problems.

I also think that this proposed direction will amplify our release problems
and get us further away from regular, incremental releases.  Master will
remain unreleaseable indefinitely until proc v2 development is finished,
and even initial releases will have problems that need to be ironed out.
Ironing out issues in initial releases is not unexpected, but by removing
existing solution we would be forcing a big-bang approach where everything
has to work before anyone can move over to 2.0, which increases pressure
for users to stay on 1.x releases, which increases pressure to backport
features and brings us closer to the Hadoop way.  I would much rather see
us working on 

Re: Please welcome new HBase Committer Francis Liu

[Francis Liu]

Thanks guys! Looking forward to working on more great stuff with everyone.  

On Monday, April 11, 2016 9:17 AM, Stephen Jiang <syuanjiang...@gmail.com> 
wrote:
 

 Congratulations and welcome!

Stephen

On Sun, Apr 10, 2016 at 10:23 PM, rajeshb...@apache.org <
chrajeshbab...@gmail.com> wrote:

> Congratulations Francis!!
>
> On Mon, Apr 11, 2016 at 10:30 AM, Anoop John <anoop.hb...@gmail.com>
> wrote:
>
> > Congrats.
> >
> > On Mon, Apr 11, 2016 at 9:19 AM, Yu Li <car...@gmail.com> wrote:
> > > Congratulations!
> > >
> > > Best Regards,
> > > Yu
> > >
> > > On 11 April 2016 at 10:12, ramkrishna vasudevan <
> > > ramkrishna.s.vasude...@gmail.com> wrote:
> > >
> > >> Congrats Francis !!!
> > >>
> > >> On Mon, Apr 11, 2016 at 7:34 AM, Pankaj kr <pankaj...@huawei.com>
> > wrote:
> > >>
> > >> > Congratulations Francis..!!
> > >> >
> > >> > -Original Message-
> > >> > From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of
> > >> Stack
> > >> > Sent: Friday, April 08, 2016 1:19 PM
> > >> > To: HBase Dev List
> > >> > Subject: Please welcome new HBase Committer Francis Liu
> > >> >
> > >> > Francis has been around forever looking after one of the biggest
> HBase
> > >> > deploys. He has contributed a bunch of big features during this time
> > --
> > >> > namespacing and grouping to mention a few -- and has more coming
> down
> > the
> > >> > pipe.
> > >> >
> > >> > Please welcome Francis to the committer fold.
> > >> >
> > >> > Thanks for all the great work Francis,
> > >> > St.Ack
> > >> >
> > >>
> >
>


  

[jira] [Created] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1

[Francis Liu (JIRA)]

Francis Liu created HBASE-15631:
---

 Summary: Backport Regionserver Groups (HBASE-6721) to branch-1 
 Key: HBASE-15631
 URL: https://issues.apache.org/jira/browse/HBASE-15631
 Project: HBase
  Issue Type: New Feature
Affects Versions: 1.4.0
Reporter: Francis Liu
Assignee: Francis Liu


Based on dev list discussion backporting region server group should not be an 
issue as it does not: 1. destabilize the code. 2. cause backward 
incompatibility. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15533) RSGroup related favored nodes enhancements

[Francis Liu (JIRA)]

Francis Liu created HBASE-15533:
---

 Summary: RSGroup related favored nodes enhancements
 Key: HBASE-15533
 URL: https://issues.apache.org/jira/browse/HBASE-15533
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Thiruvel Thirumoolan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15532) core favored nodes enhancements

[Francis Liu (JIRA)]

Francis Liu created HBASE-15532:
---

 Summary: core favored nodes enhancements
 Key: HBASE-15532
 URL: https://issues.apache.org/jira/browse/HBASE-15532
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Thiruvel Thirumoolan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15531) Favored Nodes Enhancements

[Francis Liu (JIRA)]

Francis Liu created HBASE-15531:
---

 Summary: Favored Nodes Enhancements
 Key: HBASE-15531
 URL: https://issues.apache.org/jira/browse/HBASE-15531
 Project: HBase
  Issue Type: Umbrella
Reporter: Francis Liu
Assignee: Francis Liu


We been working on enhancing favored nodes at Yahoo!  See draft document. Feel 
free to comment. I'll add more info.

https://docs.google.com/document/d/1948RKX_-kGUNOHjiYFiKPZnmKWybJkahJYsNibVhcCk/edit?usp=sharing

These enhancements have recently started running in production.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98

[Francis Liu]

I see, that is indeed undesirable. 
Tho based on this discussion there's a bunch of good features that users want 
that won't fit the criteria for #1 and #2. So allowing a backport of the few 
that can fit into the criteria shouldn't significantly affect future release 
from trunk. This way we can have some progress on some features. 


 

On Monday, March 21, 2016 10:23 AM, Sean Busbey <bus...@cloudera.com> wrote:
 

 Hadoop's trunk branch hasn't had a release in a very very long time. So it 
continues to accumulate changes that aren't in a release while folks drive 
their particular desired features back into branch-2.
On Mon, Mar 21, 2016 at 12:01 PM, Francis Liu <tof...@apache.org> wrote:

To summarize so far it seems the concerns for backporting are:
1. compatibility - api, wire, rolling upgradeabability2. stability - 
destabilizing code and deploys for those that don't want the new feature
Is there anything else? 
Elliot what happened to hadoop? Is it neither of the two?
Francis




    On Thursday, March 17, 2016 12:01 PM, Enis Söztutar <e...@apache.org> wrote:


 As long as there is interest in doing the backport work, maintaining and
experimenting with the feature on an actual release (which seems to be the
case for RSGroups and Spark at least) and keeping the code base for
branch-1 stable, there is no reason not to backport. RsGroups (and I
believe Spark support as well) is marked experimental in release notes
which means that there is some room for our client-visible guarantees.

2.0 has it's own timeline and feature set and will come in time. Of course
anybody can propose to cut a release and push for getting it sooner rather
than later. We should all help the effort, but realistically, even if after
2.0 is released, there will be people running 1.x for years after that.

Enis

On Thu, Mar 17, 2016 at 11:28 AM, Mikhail Antonov <olorinb...@gmail.com>
wrote:

> >>>"Why MOB and RegionServer Groups should be in a new major version and
> stuff
> like the new RPC queue (HBASE-15136), date based tiered compactions
> (HBASE-15181), special handling for system tables WALs (HBASE-13557), keep
> table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103)
> are considered for or already in 1.x?"
>
> To me the differentiator would be "how much does it change the codebase
> around". If all/most of code change is the code which is new/ignored when
> feature is turned off, and by default it's off until well-tested by various
> users - then it should be fine to include. In the list above MOBs probably
> don't satisfy that, Spark and RS Groups probably do.
>
> If we make 2.0 release with just Mobs and RS Groups, that would mean new AM
> would have to be postponed to 3.0? What about procsV2? Although we would
> want rolling upgrade to 2.0, still it's our chance to release something
> big, invasive and new (since as was mentioned, the user expectation anyway
> would be that in new major version some incompatibilities would be present
> and some migration may be required)?
>
> -Mikhail
>
> On Thu, Mar 17, 2016 at 7:48 AM, Matteo Bertozzi <theo.berto...@gmail.com>
> wrote:
>
> > Why MOB and RegionServer Groups should be in a new major version and
> stuff
> > like the new RPC queue (HBASE-15136), date based tiered compactions
> > (HBASE-15181), special handling for system tables WALs (HBASE-13557),
> keep
> > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103)
> > are considered for or already in 1.x?
> >
> > to me, and probably most of the users, a new Major version means that
> APIs
> > will break, wire may break, there may be an upgrade of some sort and so
> on.
> > which is not true for MOB and RS groups.
> >
> > In case we do a major for RS groups and Mob will that still based on the
> > 1.x branch?
> >
> > On Wed, Mar 16, 2016 at 11:23 PM, Andrew Purtell <
> andrew.purt...@gmail.com
> > >
> > wrote:
> >
> > > I remember explicitly saying I was not against a backport of the MOB
> > > feature. You are misrepresenting my position a bit. Sure, I'm a
> skeptic.
> > > Not a big deal because I don't think we can or should seek a blanket
> > rule.
> > > Sometimes a feature will have sufficient interest and applicability
> that
> > a
> > > backport is worth considering, and then there's a question of what kind
> > of
> > > risk the changes envisioned carry. RS groups as implemented are low
> risk.
> > > Meanwhile MOB is highly invasive. I think RS groups would have two
> large
> > > production users soon after introduction into branch-1. I'm not sure
> > about
> > > MOB. They are worth consideration on their own m

Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98

[Francis Liu]

To summarize so far it seems the concerns for backporting are:
1. compatibility - api, wire, rolling upgradeabability2. stability - 
destabilizing code and deploys for those that don't want the new feature
Is there anything else? 
Elliot what happened to hadoop? Is it neither of the two?
Francis


 

On Thursday, March 17, 2016 12:01 PM, Enis Söztutar  wrote:
 

 As long as there is interest in doing the backport work, maintaining and
experimenting with the feature on an actual release (which seems to be the
case for RSGroups and Spark at least) and keeping the code base for
branch-1 stable, there is no reason not to backport. RsGroups (and I
believe Spark support as well) is marked experimental in release notes
which means that there is some room for our client-visible guarantees.

2.0 has it's own timeline and feature set and will come in time. Of course
anybody can propose to cut a release and push for getting it sooner rather
than later. We should all help the effort, but realistically, even if after
2.0 is released, there will be people running 1.x for years after that.

Enis

On Thu, Mar 17, 2016 at 11:28 AM, Mikhail Antonov 
wrote:

> >>>"Why MOB and RegionServer Groups should be in a new major version and
> stuff
> like the new RPC queue (HBASE-15136), date based tiered compactions
> (HBASE-15181), special handling for system tables WALs (HBASE-13557), keep
> table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103)
> are considered for or already in 1.x?"
>
> To me the differentiator would be "how much does it change the codebase
> around". If all/most of code change is the code which is new/ignored when
> feature is turned off, and by default it's off until well-tested by various
> users - then it should be fine to include. In the list above MOBs probably
> don't satisfy that, Spark and RS Groups probably do.
>
> If we make 2.0 release with just Mobs and RS Groups, that would mean new AM
> would have to be postponed to 3.0? What about procsV2? Although we would
> want rolling upgrade to 2.0, still it's our chance to release something
> big, invasive and new (since as was mentioned, the user expectation anyway
> would be that in new major version some incompatibilities would be present
> and some migration may be required)?
>
> -Mikhail
>
> On Thu, Mar 17, 2016 at 7:48 AM, Matteo Bertozzi 
> wrote:
>
> > Why MOB and RegionServer Groups should be in a new major version and
> stuff
> > like the new RPC queue (HBASE-15136), date based tiered compactions
> > (HBASE-15181), special handling for system tables WALs (HBASE-13557),
> keep
> > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103)
> > are considered for or already in 1.x?
> >
> > to me, and probably most of the users, a new Major version means that
> APIs
> > will break, wire may break, there may be an upgrade of some sort and so
> on.
> > which is not true for MOB and RS groups.
> >
> > In case we do a major for RS groups and Mob will that still based on the
> > 1.x branch?
> >
> > On Wed, Mar 16, 2016 at 11:23 PM, Andrew Purtell <
> andrew.purt...@gmail.com
> > >
> > wrote:
> >
> > > I remember explicitly saying I was not against a backport of the MOB
> > > feature. You are misrepresenting my position a bit. Sure, I'm a
> skeptic.
> > > Not a big deal because I don't think we can or should seek a blanket
> > rule.
> > > Sometimes a feature will have sufficient interest and applicability
> that
> > a
> > > backport is worth considering, and then there's a question of what kind
> > of
> > > risk the changes envisioned carry. RS groups as implemented are low
> risk.
> > > Meanwhile MOB is highly invasive. I think RS groups would have two
> large
> > > production users soon after introduction into branch-1. I'm not sure
> > about
> > > MOB. They are worth consideration on their own merits and on user
> demand
> > > for them.
> > >
> > > Another thing we could do is get 2.0 started right now. Whatever major
> > > that doesn't make the cut could go into 3.0. I think the requests for
> > these
> > > backports are coming because there is no near time horizon for a 2.0
> > > release. So set it soon?
> > >
> > >
> > > > On Mar 16, 2016, at 9:27 PM, Elliott Clark 
> wrote:
> > > >
> > > > On Wed, Mar 16, 2016 at 6:26 PM, Andrew Purtell <
> > > andrew.purt...@gmail.com>
> > > > wrote:
> > > >
> > > >> Because I for one might well want to run RS groups in production
> with
> > > >> branch-1 code without waiting for or dealing with the other stuff
> > > coming in
> > > >> any 2.0.
> > > >
> > > >
> > > > This is the same email that I sent for MOB. Which you agreed with
> then.
> > > But
> > > > not now. There's nothing substantively different about this feature
> > that
> > > > makes it different from any other feature. It's a large change that
> > > wasn't
> > > > there in 1.X line.
> > > >
> > > > I would like backups, and 

Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98

[Francis Liu]

RS groups is also pretty non-invasive. It's a cp and the core code is in a 
separate module. 

On Thursday, March 17, 2016 9:06 AM, Andrew Purtell 
 wrote:
 

 I also don't see why we can't have the spark connector module in a 1.x 
release. I know our internal users would be delighted to have it available. 

> On Mar 17, 2016, at 9:00 AM, Matteo Bertozzi  wrote:
> 
> spark is no different than RS group for this discussion.
> We forced francis to move everything in a module and use coprocessors to be
> external as possible.
> so, if RS group is not going to a 1.x we should have spark not going in for
> the same reason.
> 
> MOB is deep into HRegion and all the other stuff even if everything is
> under if isMobCF.
> so, I see why we may not want it just by looking at the code.
> API, and usefulness is another thing to consider but it doesn't look like
> we are discussing this in this thread.
> 
> 
>> On Thu, Mar 17, 2016 at 8:53 AM, Ted Yu  wrote:
>> 
>> For Spark, backporting to branch-1 makes sense since
>> 
>> it is in hbase-spark module - only adds one line to root pom.xml.
>> there has been frequent user query for when hbase-spark would be in an
>> hbase release
>> 
>> On Thu, Mar 17, 2016 at 8:26 AM, Matteo Bertozzi 
>> wrote:
>> 
>>> If we cut master now we have
>>> - no rolling upgrade (am switched to zkless only and the other code was
>>> removed)
>>> - no api compatibility (we removed the deprecated)
>>> - Offheaping on the read path
>>> - Spark
>>> - MOB
>>> - RS Groups
>>> - some other stuff...
>>> 
>>> is that worth a major?
>>> The offheaping work is probably the one that cannot be backported and it
>>> may be nice to have.
>>> but stuff like spark, mob and RS Groups can be in a 1.x and avoid
>> migration
>>> pain, and those are probably the ones that some people wants to try now.
>>> 
>>> If we have a 2.0 for features like Spark, Mob and RS Groups I'd like to
>>> have it based on branch-1. At least users can move there without having
>> to
>>> worry about compatibility, even if the version number itself will
>> probably
>>> force the users to stick with the 1.x because the assumption is that
>>> something will break or there is a migration required.
>>> 
>>> On Thu, Mar 17, 2016 at 8:05 AM, Andrew Purtell <
>> andrew.purt...@gmail.com>
>>> wrote:
>>> 
 I don't think we need to do a major for RS groups.
 
 I do think Elliott's points can be addressed by getting a 2.0 out the
>>> door
 soon containing whatever we have on deck now to go in.
 
 Probably not going to satisfy everyone here - but maybe?
 
 
> On Mar 17, 2016, at 7:48 AM, Matteo Bertozzi <
>> theo.berto...@gmail.com>
 wrote:
> 
> Why MOB and RegionServer Groups should be in a new major version and
 stuff
> like the new RPC queue (HBASE-15136), date based tiered compactions
> (HBASE-15181), special handling for system tables WALs (HBASE-13557),
 keep
> table state in meta (HBASE-13017) or the Region Normalizer
>>> (HBASE-13103)
> are considered for or already in 1.x?
> 
> to me, and probably most of the users, a new Major version means that
 APIs
> will break, wire may break, there may be an upgrade of some sort and
>> so
 on.
> which is not true for MOB and RS groups.
> 
> In case we do a major for RS groups and Mob will that still based on
>>> the
> 1.x branch?
> 
> On Wed, Mar 16, 2016 at 11:23 PM, Andrew Purtell <
 andrew.purt...@gmail.com>
> wrote:
> 
>> I remember explicitly saying I was not against a backport of the MOB
>> feature. You are misrepresenting my position a bit. Sure, I'm a
>>> skeptic.
>> Not a big deal because I don't think we can or should seek a blanket
 rule.
>> Sometimes a feature will have sufficient interest and applicability
 that a
>> backport is worth considering, and then there's a question of what
>>> kind
 of
>> risk the changes envisioned carry. RS groups as implemented are low
 risk.
>> Meanwhile MOB is highly invasive. I think RS groups would have two
>>> large
>> production users soon after introduction into branch-1. I'm not sure
 about
>> MOB. They are worth consideration on their own merits and on user
>>> demand
>> for them.
>> 
>> Another thing we could do is get 2.0 started right now. Whatever
>> major
>> that doesn't make the cut could go into 3.0. I think the requests
>> for
 these
>> backports are coming because there is no near time horizon for a 2.0
>> release. So set it soon?
>> 
>> 
>>> On Mar 16, 2016, at 9:27 PM, Elliott Clark 
>>> wrote:
>>> 
>>> On Wed, Mar 16, 2016 at 6:26 PM, Andrew Purtell <
>> andrew.purt...@gmail.com>
>>> wrote:
>>> 
 Because I for one might well want to run RS groups in production
>>> with

[DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98

[Francis Liu]

Hi,
HBASE-6721 is now committed to trunk. It'd be great if it can be backported to 
1.x and 0.98 so that we can use it internally as well as push up features and 
fixes. We have been running an internal version for around 4 years. There's 
seems to be interest (HW, Bloomberg, Salesforce, etc). Also given how modular 
the code is. There's barely any effect in existing code paths. 
Seeding the criteria with Andy's suggestions in jira:
1. Stability - Unit tests and ?2. functional3. Performance - Read/write path 
was not affected. Some small changes related to assignment. 
Thanks,
Francis

[jira] [Resolved] (HBASE-7043) Region Server Group CLI commands

[Francis Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu resolved HBASE-7043.

Resolution: Fixed

was finished as part of parent patch.

> Region Server Group CLI commands
> 
>
> Key: HBASE-7043
> URL: https://issues.apache.org/jira/browse/HBASE-7043
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Francis Liu
>        Assignee: Francis Liu
>  Labels: hbase-6721
> Attachments: HBASE-6721_94_2.patch, HBASE-7043_94.patch, 
> HBASE-7043_94_2.patch, HBASE-7043_94_3.patch, HBASE-7043_94_4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15382) Expose regionserver metadata (ie groups, tables, servers) via JMX

[Francis Liu (JIRA)]

Francis Liu created HBASE-15382:
---

 Summary: Expose regionserver metadata (ie groups, tables, servers) 
via JMX
 Key: HBASE-15382
 URL: https://issues.apache.org/jira/browse/HBASE-15382
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu


This feature was removed from the base patch. So we can come up with a proper 
interface for other components to use as well, as directly accessing jmx is not 
an option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15156) Support first assignment of split daughters to non-parent RS

[Francis Liu (JIRA)]

Francis Liu created HBASE-15156:
---

 Summary: Support first assignment of split daughters to non-parent 
RS
 Key: HBASE-15156
 URL: https://issues.apache.org/jira/browse/HBASE-15156
 Project: HBase
  Issue Type: New Feature
Reporter: Francis Liu
Assignee: Francis Liu


On region split, the region's daughter is always opened by the same region 
server hosting the parent region. In some cases this is not ideal:

This feature was mainly needed for favored nodes to allow for more freedom when 
selecting favored nodes for daughter regions. ie The daughter doesn't have to 
always select the regionserver hosting the split as a favored node which should 
allow for better favored node distribution.

Though this feature is actually useful in cases where region splits occur much 
more often than the balancer is run. It also is a bit more efficient as the 
major compaction that occurs after daughter assignment does not go to waste (ie 
cancelled half-way, loss of locality due to move, etc). We actually run it this 
way in some of our clusters even without favored nodes enabled. Hence I am 
supplying a patch which is independent of favored nodes.







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14981) Shell unit tests are not running in trunk?

[Francis Liu (JIRA)]

Francis Liu created HBASE-14981:
---

 Summary: Shell unit tests are not running in trunk?
 Key: HBASE-14981
 URL: https://issues.apache.org/jira/browse/HBASE-14981
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu


TestShell seems to be replaced by an AbstractTestShell class. Thus it does not 
seem that any of the shell unit tests are running? If so, is the intent to 
create a TestShell which subclasses the abstract class? 

On a related note I've written shell unit tests for HBASE-6721 and it requires 
a different Shell subclass as I need to have a different balancer and 
coprocessor installed. I'm hoping we can address that use case in this jira as 
well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14981) Shell unit tests are not running in trunk?

[Francis Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu resolved HBASE-14981.
-
Resolution: Invalid

> Shell unit tests are not running in trunk?
> --
>
> Key: HBASE-14981
> URL: https://issues.apache.org/jira/browse/HBASE-14981
> Project: HBase
>  Issue Type: Bug
>        Reporter: Francis Liu
>        Assignee: Francis Liu
>
> TestShell seems to be replaced by an AbstractTestShell class. Thus it does 
> not seem that any of the shell unit tests are running? If so, is the intent 
> to create a TestShell which subclasses the abstract class? 
> On a related note I've written shell unit tests for HBASE-6721 and it 
> requires a different Shell subclass as I need to have a different balancer 
> and coprocessor installed. I'm hoping we can address that use case in this 
> jira as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14857) add region server groups in master UI

[Francis Liu (JIRA)]

Francis Liu created HBASE-14857:
---

 Summary: add region server groups in master UI
 Key: HBASE-14857
 URL: https://issues.apache.org/jira/browse/HBASE-14857
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14856) document region servers groups in the book

[Francis Liu (JIRA)]

Francis Liu created HBASE-14856:
---

 Summary: document region servers groups in the book
 Key: HBASE-14856
 URL: https://issues.apache.org/jira/browse/HBASE-14856
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14312) Forward port some fixes from hbase-6721-0.98 to hbase-6721

[Francis Liu (JIRA)]

Francis Liu created HBASE-14312:
---

 Summary: Forward port some fixes from hbase-6721-0.98 to hbase-6721
 Key: HBASE-14312
 URL: https://issues.apache.org/jira/browse/HBASE-14312
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu


Some fixes where checked into hbase-6721-0.98 to address some testing failures 
and resync'ing with internal Y! implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14169) API to refreshSuperUserGroupsConfiguration

[Francis Liu (JIRA)]

Francis Liu created HBASE-14169:
---

 Summary: API to refreshSuperUserGroupsConfiguration
 Key: HBASE-14169
 URL: https://issues.apache.org/jira/browse/HBASE-14169
 Project: HBase
  Issue Type: New Feature
Reporter: Francis Liu
Assignee: Francis Liu


For deployments that use security. User impersonation (AKA doAs()) is needed 
for some services (ie Stargate, thriftserver, Oozie, etc). Impersonation 
definitions are defined in a xml config file and read and cached by the 
ProxyUsers class. Calling this api will refresh cached information, eliminating 
the need to restart the master/regionserver whenever the configuration is 
changed. 

Implementation just adds another method to AccessControlService.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-13134) mutateRow and checkAndMutate apis don't throw region level exceptions

[Francis Liu (JIRA)]

Francis Liu created HBASE-13134:
---

 Summary: mutateRow and checkAndMutate apis don't throw region 
level exceptions
 Key: HBASE-13134
 URL: https://issues.apache.org/jira/browse/HBASE-13134
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu


The HTable.mutateRow() and checkAndMutate api internally uses multi(). Tho 
after getting a response it does not go through returned region action result 
list to check for exceptions. 

Adding a patch for trunk tho we've observed this issue in 0.98 as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'

[Francis Liu (JIRA)]

Francis Liu created HBASE-12511:
---

 Summary: namespace permissions - add support from table creation 
privilege in a namespace 'C'
 Key: HBASE-12511
 URL: https://issues.apache.org/jira/browse/HBASE-12511
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu


As discussed in namespace permission Jira. A user granted a 'C' on a namespace 
enables a user to create tables within the namespace. 'C' on a namespace does 
not enable a user to create/drop the namespace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-12512) namespace permissions - add support for namespace list privilege 'L'

[Francis Liu (JIRA)]

Francis Liu created HBASE-12512:
---

 Summary: namespace permissions - add support for namespace list 
privilege 'L'
 Key: HBASE-12512
 URL: https://issues.apache.org/jira/browse/HBASE-12512
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu


As discussed in namespace permisssion jira. A user granted an 'L' on a 
namespace is able to list the tables associated with the namespace else an 
AccessDeniedException is throw. The listTables command will list only tables a 
user has 'L' privilege on it's associated namespace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-11291) Parallelize meta scan during startup

[Francis Liu (JIRA)]

Francis Liu created HBASE-11291:
---

 Summary: Parallelize meta scan during startup
 Key: HBASE-11291
 URL: https://issues.apache.org/jira/browse/HBASE-11291
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu


During startup meta gets scanned twice. This takes a few mins each with large 
number of regions. Let's parallelize one scan. And have a switch to disable the 
second one or perhaps store data that meta has been migrated.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11286) BulkDisabler should use a bulk RPC call for opening regions (just like BulkAssigner)

[Francis Liu (JIRA)]

Francis Liu created HBASE-11286:
---

 Summary: BulkDisabler should use a bulk RPC call for opening 
regions (just like BulkAssigner)
 Key: HBASE-11286
 URL: https://issues.apache.org/jira/browse/HBASE-11286
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Virag Kothari






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11287) Flush table cmd should make RPC calls to region server in parallel

[Francis Liu (JIRA)]

Francis Liu created HBASE-11287:
---

 Summary: Flush table cmd should make RPC calls to region server in 
parallel 
 Key: HBASE-11287
 URL: https://issues.apache.org/jira/browse/HBASE-11287
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11288) Splittable Meta

[Francis Liu (JIRA)]

Francis Liu created HBASE-11288:
---

 Summary: Splittable Meta
 Key: HBASE-11288
 URL: https://issues.apache.org/jira/browse/HBASE-11288
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11289) Speedup balance

[Francis Liu (JIRA)]

Francis Liu created HBASE-11289:
---

 Summary: Speedup balance
 Key: HBASE-11289
 URL: https://issues.apache.org/jira/browse/HBASE-11289
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-9573) provide namespaceExists api

[Francis Liu (JIRA)]

Francis Liu created HBASE-9573:
--

 Summary: provide namespaceExists api
 Key: HBASE-9573
 URL: https://issues.apache.org/jira/browse/HBASE-9573
 Project: HBase
  Issue Type: Improvement
Reporter: Francis Liu
Assignee: Francis Liu


Presently in order to test existence of a namespace one has to use the get api 
and catch the NamespaceNotFound exception. This is undersirable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9540) Fix race condition in namespace janitor

[Francis Liu (JIRA)]

Francis Liu created HBASE-9540:
--

 Summary: Fix race condition in namespace janitor
 Key: HBASE-9540
 URL: https://issues.apache.org/jira/browse/HBASE-9540
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu


Namespace janitor cleans up orphaned namespace directories on hdfs and znodes 
on zk. As it stand it could potentially cleanup namespaces that are in the 
process of being created.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9435) Fix jersey serialization/deserialization of json objects

[Francis Liu (JIRA)]

Francis Liu created HBASE-9435:
--

 Summary: Fix jersey serialization/deserialization of json objects
 Key: HBASE-9435
 URL: https://issues.apache.org/jira/browse/HBASE-9435
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
 Attachments: HBASE-9435.patch

Stargate uses the default json marshaller/unmarshaller in natural mode. In this 
mode the unmarshaller has trouble unmarshalling json instances. 

This patch fixes this issue by using jackson as the marshaller/unmarshaller 
instead. 

I've also updated all the model unit tests to test json 
serialization/deserialization. Backwards compatibilty can be verified by modify 
the test base class to use the original marshaller/unmarshaller and see that 
model tests pass.

The patch is backward compatible except for StorageClusterStatusModel, which is 
broken anyway. It only shows one node in the liveNodes field.





--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9206) namespace permissions

[Francis Liu (JIRA)]

Francis Liu created HBASE-9206:
--

 Summary: namespace permissions
 Key: HBASE-9206
 URL: https://issues.apache.org/jira/browse/HBASE-9206
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu


Now that we have namespaces let's address how we can give admins more 
flexibility.

Let's list out the privileges we'd like. Then we can map it to existing 
privileges and see if we need more. 

So far we have:

1. Modify namespace descriptor (ie quota, other values)
2. create namespace
3. delete namespace
4. list tables in namespace
5. create/drop tables in a namespace
6. All namespace's tables create
7. All namespace's tables write
8. All namespace's tables execute
9. All namespace's tables delete
10. All namespace's tables admin

1-3, is currently set to global admin only. Which seems acceptable to me.




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9191) Update Loadbalancer method to throw HBaseIOException

[Francis Liu (JIRA)]

Francis Liu created HBASE-9191:
--

 Summary: Update Loadbalancer method to throw HBaseIOException
 Key: HBASE-9191
 URL: https://issues.apache.org/jira/browse/HBASE-9191
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu


Some load balancers need a way to communicate failure scenarios so the AM has 
the opportunity to better handle them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9182) In secure mode CLI list command throws a security exception for non-admin users

[Francis Liu (JIRA)]

Francis Liu created HBASE-9182:
--

 Summary: In secure mode CLI list command throws a security 
exception for non-admin users
 Key: HBASE-9182
 URL: https://issues.apache.org/jira/browse/HBASE-9182
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.9
Reporter: Francis Liu


HBASE-8692, prevents users retrieving table descriptors they don't have admin 
access to. As a side-effect it also prevents non-admin users from being able to 
list tables in the CLI. One proposal to fix this is to provide an API in 
HMaster which returns a list of table names only. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9183) Secure Admin Page when in secure mode

[Francis Liu (JIRA)]

Francis Liu created HBASE-9183:
--

 Summary: Secure Admin Page when in secure mode
 Key: HBASE-9183
 URL: https://issues.apache.org/jira/browse/HBASE-9183
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu


Currently admin page is recommended to be protected and only accessible by 
administrators as it exposes potentially sensitive information as well as 
actions (ie compact, split, etc).

Other hadoop components (ie HDFS, YARN), secures their admin pages. We should 
do the same. We should be able to leverage some of the pieces they used.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9147) Mechanism to limit system table deletion and creation in non-secure mode

[Francis Liu (JIRA)]

Francis Liu created HBASE-9147:
--

 Summary: Mechanism to limit system table deletion and creation in 
non-secure mode
 Key: HBASE-9147
 URL: https://issues.apache.org/jira/browse/HBASE-9147
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9148) Add wal and OpenRegionhandler tier for system namespace

[Francis Liu (JIRA)]

Francis Liu created HBASE-9148:
--

 Summary: Add wal and OpenRegionhandler tier for system namespace
 Key: HBASE-9148
 URL: https://issues.apache.org/jira/browse/HBASE-9148
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9150) Add priority for system tables

[Francis Liu (JIRA)]

Francis Liu created HBASE-9150:
--

 Summary: Add priority for system tables
 Key: HBASE-9150
 URL: https://issues.apache.org/jira/browse/HBASE-9150
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-9149) javadoc cleanup of to reflect .META. rename to hbase:meta

[Francis Liu (JIRA)]

Francis Liu created HBASE-9149:
--

 Summary: javadoc cleanup of to reflect .META. rename to hbase:meta
 Key: HBASE-9149
 URL: https://issues.apache.org/jira/browse/HBASE-9149
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8743) upgrade hadoop-23 version to 0.23.7

[Francis Liu (JIRA)]

Francis Liu created HBASE-8743:
--

 Summary: upgrade hadoop-23 version to 0.23.7
 Key: HBASE-8743
 URL: https://issues.apache.org/jira/browse/HBASE-8743
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.8
Reporter: Francis Liu
 Fix For: 0.94.9


There's a missing class that's causing compile time issues when building with 
security. This got fixed in 0.23.7. No patch needed just need to bump up the 
version in pom

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

Sounds like I should give the auto-generate approach a try. 

In essence it'll do the following:

- Tables without '.' will be moved into the default namespace. 
- Tables with '.' will be move into new namespaces 
- namespaces will be delimited by the last '.' in the table name 
ie org.apache.hbase.MyTable, namespace = org.apache.hbase
- In both cases the oldTableName is the same as the fullTableName
- all existing apis and cli commands will be expecting full table names unless 
explicitly stated
- a RenameTable tool will be provided to rename offline tables

Let me know if any clarification is needed.

-Francis


On May 8, 2013, at 8:40 PM, Stack wrote:

 On Wed, May 8, 2013 at 7:03 PM, Ted Yu yuzhih...@gmail.com wrote:
 
 w.r.t. auto-generate option, we need some heuristics.
 
 e.g. would namespace.schemaname.tablename be supported ?
 
 The auto-generate option may create namespace name out of existing schema
 name.
 
 
 
 There is no schema name in hbase.  James's convention up in phoenix at the
 hbase-level is a table name w/ a dot in it.
 
 This is already a difficult enough issue.  No need to add complications.
 
 St.Ack

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

I can probably incorporate the migration into the main patch will see how big 
it gets. And the rename tool as a subtask.

On May 9, 2013, at 4:21 PM, Ted Yu wrote:

 The plan is feasible.
 This would be done in sub-task of HBASE-8015, right ?
 
 Thanks
 
 On Thu, May 9, 2013 at 4:03 PM, Francis Liu tof...@apache.org wrote:
 
 Sounds like I should give the auto-generate approach a try.
 
 In essence it'll do the following:
 
 - Tables without '.' will be moved into the default namespace.
 - Tables with '.' will be move into new namespaces
- namespaces will be delimited by the last '.' in the table name
ie org.apache.hbase.MyTable, namespace = org.apache.hbase
 - In both cases the oldTableName is the same as the fullTableName
 - all existing apis and cli commands will be expecting full table names
 unless explicitly stated
 - a RenameTable tool will be provided to rename offline tables
 
 Let me know if any clarification is needed.
 
 -Francis
 
 
 On May 8, 2013, at 8:40 PM, Stack wrote:
 
 On Wed, May 8, 2013 at 7:03 PM, Ted Yu yuzhih...@gmail.com wrote:
 
 w.r.t. auto-generate option, we need some heuristics.
 
 e.g. would namespace.schemaname.tablename be supported ?
 
 The auto-generate option may create namespace name out of existing
 schema
 name.
 
 
 
 There is no schema name in hbase.  James's convention up in phoenix at
 the
 hbase-level is a table name w/ a dot in it.
 
 This is already a difficult enough issue.  No need to add complications.
 
 St.Ack
 
 

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

 Francis, your proposal for auto-generating the namespaces looks good, but I
 am wondering whether the user might be surprised later when they find about
 the all the namespaces.


The ones that don't read the release notes will probably be surprised though I 
don't think it'll be a big issue.
As it doesn't affect existing behavior, apart from not being able to create new 
tables with dots. 
And they only have to do extra work if they choose to start using namespaces 
the way it should be.

 BTW, renaming a table will break snapshots, unless it is done as a clone /
 restore.

I see, clone/restore was one approach I had in mind to implement renaming.

On May 9, 2013, at 6:00 PM, Enis Söztutar wrote:

 Elliot's suggestion sounds good, but unfortunately there are places where
 we use the table name as a string. We cannot expect all those places to
 also have namespaces. Some of the examples are metrics names per table,
 hfile links encoding the table name in it, and maybe znodes, and of course
 the shell. I would agree though that we should handle the schama and table
 names as separate entities within inside hbase.
 
 Francis, your proposal for auto-generating the namespaces looks good, but I
 am wondering whether the user might be surprised later when they find about
 the all the namespaces.
 
 BTW, renaming a table will break snapshots, unless it is done as a clone /
 restore.
 
 
 
 
 
 
 On Thu, May 9, 2013 at 4:43 PM, Francis Liu tof...@apache.org wrote:
 
 I can probably incorporate the migration into the main patch will see how
 big it gets. And the rename tool as a subtask.
 
 On May 9, 2013, at 4:21 PM, Ted Yu wrote:
 
 The plan is feasible.
 This would be done in sub-task of HBASE-8015, right ?
 
 Thanks
 
 On Thu, May 9, 2013 at 4:03 PM, Francis Liu tof...@apache.org wrote:
 
 Sounds like I should give the auto-generate approach a try.
 
 In essence it'll do the following:
 
 - Tables without '.' will be moved into the default namespace.
 - Tables with '.' will be move into new namespaces
   - namespaces will be delimited by the last '.' in the table name
   ie org.apache.hbase.MyTable, namespace = org.apache.hbase
 - In both cases the oldTableName is the same as the fullTableName
 - all existing apis and cli commands will be expecting full table names
 unless explicitly stated
 - a RenameTable tool will be provided to rename offline tables
 
 Let me know if any clarification is needed.
 
 -Francis
 
 
 On May 8, 2013, at 8:40 PM, Stack wrote:
 
 On Wed, May 8, 2013 at 7:03 PM, Ted Yu yuzhih...@gmail.com wrote:
 
 w.r.t. auto-generate option, we need some heuristics.
 
 e.g. would namespace.schemaname.tablename be supported ?
 
 The auto-generate option may create namespace name out of existing
 schema
 name.
 
 
 
 There is no schema name in hbase.  James's convention up in phoenix at
 the
 hbase-level is a table name w/ a dot in it.
 
 This is already a difficult enough issue.  No need to add
 complications.
 
 St.Ack
 
 
 
 

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

In the current implementation it will be only allowed as a namespace delimiter. 
We can relax that to support applications which support more than one '.' 

ie org.apache.hbase.MyTable

Namespace = org.apache.hbase
Table = MyTable



On May 7, 2013, at 11:36 PM, Stack wrote:

 With namespaces in place, will '.' be illegal in a table name?
 
 With namespaces, is there a no-namespace/default location?  If so, what
 will it be called or how will you refer to tables in the
 no-namespace/default namespace?
 
 I just took a user's production website where there are hundreds of tables.
 For no good reason that I can see, they happened to have choosen '_' and
 '-' as table name partitioner: i.e. application_feature, etc.  My sense is
 they could just as easily have gone with '.' but maybe the '.META.' name
 frightens people away from '.'?
 
 Anyone using '.' in their table names?
 
 St.Ack

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

There shouldn't be any ambiguity. There's fully-qualified table names and 
there's table names.  Table name constant changes were to make the names less 
funky. 

I like your suggestion since it simplifies migration. Though it seems we're 
kicking the can down the road here. In a way we're avoiding the problem by 
specifying an internal delimiter and adding extra complexity to prevent the 
user from using it. Having a way of specifying a table fully qualified seems to 
be something fundamental and convenient, if we don't support one now we'll have 
even more trouble in the future. Looking at the suggestions we can potentially 
make migration painless by recognizing existing tables with . as part of the 
default namespace or automatically create namespaces for tables with dots in 
them. Neither requires renaming tables. They only need to rename tables if they 
want to start organizing things into namespaces which they will have to do in 
any scenario.

-Francis

On May 8, 2013, at 1:27 PM, Elliott Clark wrote:

 With this solution there's no naming ambiguity.  There's no
 overloading table name to actually be two different things.  There's
 no need for users to rename their tables. Most code that is already
 written will still be source compatible.  No need to change table name
 constants or anything like that.
 

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

 I think putting existing tables with . in table name as part of default
 namespace is better choice among the two.

Correct me if I missed something here. That is probably the more difficult of 
the two as you can't derive membership information by just parsing the table 
name. Which is what is passed around internally most of the time. Because of 
that I prefer the auto namespace generation approach. 

On May 8, 2013, at 5:11 PM, Ted Yu wrote:

 bq. by recognizing existing tables with . as part of the default
 namespace or automatically create namespaces for tables with dots in them.
 
 I think putting existing tables with . in table name as part of default
 namespace is better choice among the two.
 
 Cheers
 
 On Wed, May 8, 2013 at 5:02 PM, Francis Liu tof...@apache.org wrote:
 
 There shouldn't be any ambiguity. There's fully-qualified table names and
 there's table names.  Table name constant changes were to make the names
 less funky.
 
 I like your suggestion since it simplifies migration. Though it seems
 we're kicking the can down the road here. In a way we're avoiding the
 problem by specifying an internal delimiter and adding extra complexity to
 prevent the user from using it. Having a way of specifying a table fully
 qualified seems to be something fundamental and convenient, if we don't
 support one now we'll have even more trouble in the future. Looking at the
 suggestions we can potentially make migration painless by recognizing
 existing tables with . as part of the default namespace or automatically
 create namespaces for tables with dots in them. Neither requires renaming
 tables. They only need to rename tables if they want to start organizing
 things into namespaces which they will have to do in any scenario.
 
 -Francis
 
 On May 8, 2013, at 1:27 PM, Elliott Clark wrote:
 
 With this solution there's no naming ambiguity.  There's no
 overloading table name to actually be two different things.  There's
 no need for users to rename their tables. Most code that is already
 written will still be source compatible.  No need to change table name
 constants or anything like that.
 
 
 

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

 I think if we only have a string for fully qualified table name internally
 and rely on parsing the dot we are going to have migration issues somewhere.

What migration issues do you see?

 parameters, but internally IMHO we need separate namespace argument, or if

We can get there incrementally as the change is pervasive. As a first cut well 
have to use a delimiter.
Also we'll have to use delimiters for zk, hdfs and catalog.

 it is too painful, use different internal separator that will not cause
 backward compat and change it for display only.

Wouldn't it be confusing to have an internal and external delimiter?
We would have to know what's external and internal and then
would have to remember to do the translation when making calls between the two.

On May 8, 2013, at 6:24 PM, Sergey Shelukhin wrote:

 I think if we only have a string for fully qualified table name internally
 and rely on parsing the dot we are going to have migration issues somewhere.
 Externally I agree with Elliot, we can do all kinds of things like add
 parameters, but internally IMHO we need separate namespace argument, or if
 it is too painful, use different internal separator that will not cause
 backward compat and change it for display only.
 
 On Wed, May 8, 2013 at 6:00 PM, Francis Liu tof...@apache.org wrote:
 
 I think putting existing tables with . in table name as part of default
 namespace is better choice among the two.
 
 Correct me if I missed something here. That is probably the more difficult
 of the two as you can't derive membership information by just parsing the
 table name. Which is what is passed around internally most of the time.
 Because of that I prefer the auto namespace generation approach.
 
 On May 8, 2013, at 5:11 PM, Ted Yu wrote:
 
 bq. by recognizing existing tables with . as part of the default
 namespace or automatically create namespaces for tables with dots in
 them.
 
 I think putting existing tables with . in table name as part of default
 namespace is better choice among the two.
 
 Cheers
 
 On Wed, May 8, 2013 at 5:02 PM, Francis Liu tof...@apache.org wrote:
 
 There shouldn't be any ambiguity. There's fully-qualified table names
 and
 there's table names.  Table name constant changes were to make the names
 less funky.
 
 I like your suggestion since it simplifies migration. Though it seems
 we're kicking the can down the road here. In a way we're avoiding the
 problem by specifying an internal delimiter and adding extra complexity
 to
 prevent the user from using it. Having a way of specifying a table fully
 qualified seems to be something fundamental and convenient, if we don't
 support one now we'll have even more trouble in the future. Looking at
 the
 suggestions we can potentially make migration painless by recognizing
 existing tables with . as part of the default namespace or
 automatically
 create namespaces for tables with dots in them. Neither requires
 renaming
 tables. They only need to rename tables if they want to start organizing
 things into namespaces which they will have to do in any scenario.
 
 -Francis
 
 On May 8, 2013, at 1:27 PM, Elliott Clark wrote:
 
 With this solution there's no naming ambiguity.  There's no
 overloading table name to actually be two different things.  There's
 no need for users to rename their tables. Most code that is already
 written will still be source compatible.  No need to change table name
 constants or anything like that.
 
 
 
 
 

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

 Looks like using '#' as delimiter becomes an attractive option.

Doesn't the auto-generate option sound reasonable?

On May 8, 2013, at 6:27 PM, Ted Yu wrote:

 bq. use different internal separator that will not cause backward compat
 and change it for display only.
 
 User would copy / paste table name from UI and expect it to be accepted by
 shell, etc.
 
 Looks like using '#' as delimiter becomes an attractive option.
 
 On Wed, May 8, 2013 at 6:24 PM, Sergey Shelukhin 
 ser...@hortonworks.comwrote:
 
 I think if we only have a string for fully qualified table name internally
 and rely on parsing the dot we are going to have migration issues
 somewhere.
 Externally I agree with Elliot, we can do all kinds of things like add
 parameters, but internally IMHO we need separate namespace argument, or if
 it is too painful, use different internal separator that will not cause
 backward compat and change it for display only.
 
 On Wed, May 8, 2013 at 6:00 PM, Francis Liu tof...@apache.org wrote:
 
 I think putting existing tables with . in table name as part of
 default
 namespace is better choice among the two.
 
 Correct me if I missed something here. That is probably the more
 difficult
 of the two as you can't derive membership information by just parsing the
 table name. Which is what is passed around internally most of the time.
 Because of that I prefer the auto namespace generation approach.
 
 On May 8, 2013, at 5:11 PM, Ted Yu wrote:
 
 bq. by recognizing existing tables with . as part of the default
 namespace or automatically create namespaces for tables with dots in
 them.
 
 I think putting existing tables with . in table name as part of
 default
 namespace is better choice among the two.
 
 Cheers
 
 On Wed, May 8, 2013 at 5:02 PM, Francis Liu tof...@apache.org wrote:
 
 There shouldn't be any ambiguity. There's fully-qualified table names
 and
 there's table names.  Table name constant changes were to make the
 names
 less funky.
 
 I like your suggestion since it simplifies migration. Though it seems
 we're kicking the can down the road here. In a way we're avoiding the
 problem by specifying an internal delimiter and adding extra
 complexity
 to
 prevent the user from using it. Having a way of specifying a table
 fully
 qualified seems to be something fundamental and convenient, if we
 don't
 support one now we'll have even more trouble in the future. Looking at
 the
 suggestions we can potentially make migration painless by recognizing
 existing tables with . as part of the default namespace or
 automatically
 create namespaces for tables with dots in them. Neither requires
 renaming
 tables. They only need to rename tables if they want to start
 organizing
 things into namespaces which they will have to do in any scenario.
 
 -Francis
 
 On May 8, 2013, at 1:27 PM, Elliott Clark wrote:
 
 With this solution there's no naming ambiguity.  There's no
 overloading table name to actually be two different things.  There's
 no need for users to rename their tables. Most code that is already
 written will still be source compatible.  No need to change table
 name
 constants or anything like that.
 
 
 
 
 
 

[DISCUSS] Namespace Delimiter

[Francis Liu]

Hi,

As part of the namespace patch (HBASE-8015). We will need a delimiter to 
separate namespace name from table name. The obvious choice here would be a dot 
'.'. Since dot is presently a valid character for table names that would 
require users to migrate their tables (ie renaming tables) as part of upgrade 
to 0.96. Another option is to use a different delimiter to avoid the table 
migration altogether. Thoughts?

-Francis

Re: [DISCUSS] Namespace Delimiter

[Francis Liu]

One thing I had in mind was to automatically assume that the first dot delimits 
the namespace name. During upgrade we automatically create those namespaces and 
assign the tables accordingly. They can then eventually migrate/rename their 
tables (if needed) at a later time. In the extreme case that would be one 
namespace per table. For which we will provide a tool to rename offline tables.

I'm guessing most cases would not require a rename. What else do people use 
dots in their table name for? 

-Francis

On May 7, 2013, at 4:49 PM, Ian Varley wrote:

 I would also submit that . is a pretty universal standard (citation needed) 
 in relational databases for separating namespaces (schemas, etc.) from 
 tables. We use that now to represent the same idea, and using a different 
 delimiter would be less than ideal in the long run.
 
 But, I agree with Jon - anything in the 0.96 upgrade that causes people to 
 change client code in lock-step isn't going to fly.
 
 Is there any solution which can use . but be transparent at upgrade time? 
 I.e. you could still refer to it by its full Namespace.Table name in client 
 code, and it does a little more work to try both combinations? That'd prevent 
 cases where someone already has tables called both Y' and X.Y, but, come 
 on, who does that?
 
 Ian
 
 On May 7, 2013, at 6:43 PM, Jonathan Hsieh wrote:
 
 I prefer using a delimiter that does not require migration.  As someone who
 has to support a wide variety of users, this will cause much less confusion
 from our users (and save me grief!)
 
 From the code [1], any symbol char other than '.', '_', or '-' would be an
 ok delimiter.  howabout a ':' or '#'?
 
 [1]
 https://github.com/apache/hbase/blob/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java#L415
 
 Jon.
 
 
 On Tue, May 7, 2013 at 4:38 PM, Francis Liu tof...@apache.org wrote:
 
 Hi,
 
 As part of the namespace patch (HBASE-8015). We will need a delimiter to
 separate namespace name from table name. The obvious choice here would be a
 dot '.'. Since dot is presently a valid character for table names that
 would require users to migrate their tables (ie renaming tables) as part of
 upgrade to 0.96. Another option is to use a different delimiter to avoid
 the table migration altogether. Thoughts?
 
 -Francis
 
 
 
 
 --
 // Jonathan Hsieh (shay)
 // Software Engineer, Cloudera
 // j...@cloudera.com
 

[jira] [Created] (HBASE-8408) Implement namespace

[Francis Liu (JIRA)]

Francis Liu created HBASE-8408:
--

 Summary: Implement namespace
 Key: HBASE-8408
 URL: https://issues.apache.org/jira/browse/HBASE-8408
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8409) Security support for namespaces

[Francis Liu (JIRA)]

Francis Liu created HBASE-8409:
--

 Summary: Security support for namespaces
 Key: HBASE-8409
 URL: https://issues.apache.org/jira/browse/HBASE-8409
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8410) Basic quota support for namespaces

[Francis Liu (JIRA)]

Francis Liu created HBASE-8410:
--

 Summary: Basic quota support for namespaces
 Key: HBASE-8410
 URL: https://issues.apache.org/jira/browse/HBASE-8410
 Project: HBase
  Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-8015) Support for Namespaces

[Francis Liu (JIRA)]

Francis Liu created HBASE-8015:
--

 Summary: Support for Namespaces
 Key: HBASE-8015
 URL: https://issues.apache.org/jira/browse/HBASE-8015
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7829) zookeeper kerberos conf keytab and principal parameters interchanged

[Francis Liu (JIRA)]

Francis Liu created HBASE-7829:
--

 Summary: zookeeper kerberos conf keytab and principal parameters 
interchanged
 Key: HBASE-7829
 URL: https://issues.apache.org/jira/browse/HBASE-7829
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu
Priority: Minor
 Attachments: HBASE-7829_94.patch



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6225) create secure bulk load endpoint

[Francis Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu resolved HBASE-6225.


Resolution: Fixed

marking as resolved since parent task is done.

 create secure bulk load endpoint
 

 Key: HBASE-6225
 URL: https://issues.apache.org/jira/browse/HBASE-6225
 Project: HBase
  Issue Type: Sub-task
  Components: security
Reporter: Francis Liu
Assignee: Francis Liu



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7770) minor integration test framework fixes

[Francis Liu (JIRA)]

Francis Liu created HBASE-7770:
--

 Summary: minor integration test framework fixes
 Key: HBASE-7770
 URL: https://issues.apache.org/jira/browse/HBASE-7770
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Priority: Minor


- made FileSystem on HBaseTestingUtil.createMulti() not expect mini cluster
- added check if server is not running before deciding to restore a server

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever

[Francis Liu (JIRA)]

Francis Liu created HBASE-7771:
--

 Summary: Secure HBase Client in MR job causes tasks to wait forever
 Key: HBASE-7771
 URL: https://issues.apache.org/jira/browse/HBASE-7771
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu
 Fix For: 0.94.5


This seems to be a regression caused by HBASE-4791 wherein we check if secure 
zookeeper is enabled and if so we make use of saslLatch to verify security 
handshake is completed. But in the case of MR, we won't be negotiating a secure 
connection thus we end up waiting forever for the saslLatch.

Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is 
already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called

[Francis Liu (JIRA)]

Francis Liu created HBASE-7772:
--

 Summary: clusterId is not set in conf properly if only 
TableMapReduceUtil.initCredentials() is called
 Key: HBASE-7772
 URL: https://issues.apache.org/jira/browse/HBASE-7772
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu


clusterId is not set in the job's conf correctly if only 
TableMapReduceUtil.initCredentials() is called and thus fails to include the 
token when using an hbase client in any of the job's tasks. This is a 
regression.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: HBase on Hadoop 0.23

[Francis Liu]

Great, thanks Jon. 

-Francis

On 7/30/12 6:37 PM, Jonathan Hsieh j...@cloudera.com wrote:

Sure.  I started a poll last week and I need to close it out.

Jon.

On Mon, Jul 30, 2012 at 5:57 PM, Francis Liu tof...@apache.org wrote:
 I see, thanks for valuable info. Given that source recompilation is
 needed, can we keep the 0.23 profile for now?

 -Francis

 On 7/30/12 4:33 PM, Jonathan Hsieh j...@cloudera.com wrote:

Francis,

HBASE-6305
HBASE-6330.

I've found that I needed to recompile hbase 0.94.1 rc1 against hadoop
2.x and hadoop 0.23 -- they were source-code compatible but there are
binary incompatibilities.

I believe 0.92 doesn't have a 2.0 build profile currently, but I know
we added it to the 0.94 branch.

Jon.


On Mon, Jul 30, 2012 at 1:49 PM, Francis Liu tof...@apache.org wrote:
 Will I be able to run HBase-hadoop_2.0 built binaries on hadoop 0.23?
 We're looking into deploying HBase on hadoop_0.23 on our clusters.
Can I
 get the Jira numbers for the two issues?

 -Francis



 On 7/30/12 11:32 AM, Jonathan Hsieh j...@cloudera.com wrote:

I'm going to update these to hadoop 2.0 builds once we get the hbase +
hadoop 2.0 builds passing.  There are 2 issues (that I'm working on)
that are likely the last always failing unit tests against hadoop 2.0.

Jon.

On Mon, Jul 30, 2012 at 11:23 AM, Francis Liu tof...@apache.org
wrote:
 Hi,

 I noticed there are 0.23 builds on jenkins. I was wondering what
state
it is
 in? And who are working on it? Is there an umbrella Jira to get
HBase
 working on 0.23?

 -Francis





--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// j...@cloudera.com





--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// j...@cloudera.com





-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// j...@cloudera.com

HBase on Hadoop 0.23

[Francis Liu]

Hi,

I noticed there are 0.23 builds on jenkins. I was wondering what state it is
in? And who are working on it? Is there an umbrella Jira to get HBase
working on 0.23?

-Francis

Re: HBase on Hadoop 0.23

[Francis Liu]

Will I be able to run HBase-hadoop_2.0 built binaries on hadoop 0.23?
We're looking into deploying HBase on hadoop_0.23 on our clusters. Can I
get the Jira numbers for the two issues?

-Francis



On 7/30/12 11:32 AM, Jonathan Hsieh j...@cloudera.com wrote:

I'm going to update these to hadoop 2.0 builds once we get the hbase +
hadoop 2.0 builds passing.  There are 2 issues (that I'm working on)
that are likely the last always failing unit tests against hadoop 2.0.

Jon.

On Mon, Jul 30, 2012 at 11:23 AM, Francis Liu tof...@apache.org wrote:
 Hi,

 I noticed there are 0.23 builds on jenkins. I was wondering what state
it is
 in? And who are working on it? Is there an umbrella Jira to get HBase
 working on 0.23?

 -Francis





-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// j...@cloudera.com

Re: HBase on Hadoop 0.23

[Francis Liu]

I see, thanks for valuable info. Given that source recompilation is
needed, can we keep the 0.23 profile for now?

-Francis 

On 7/30/12 4:33 PM, Jonathan Hsieh j...@cloudera.com wrote:

Francis,

HBASE-6305
HBASE-6330.

I've found that I needed to recompile hbase 0.94.1 rc1 against hadoop
2.x and hadoop 0.23 -- they were source-code compatible but there are
binary incompatibilities.

I believe 0.92 doesn't have a 2.0 build profile currently, but I know
we added it to the 0.94 branch.

Jon.


On Mon, Jul 30, 2012 at 1:49 PM, Francis Liu tof...@apache.org wrote:
 Will I be able to run HBase-hadoop_2.0 built binaries on hadoop 0.23?
 We're looking into deploying HBase on hadoop_0.23 on our clusters. Can I
 get the Jira numbers for the two issues?

 -Francis



 On 7/30/12 11:32 AM, Jonathan Hsieh j...@cloudera.com wrote:

I'm going to update these to hadoop 2.0 builds once we get the hbase +
hadoop 2.0 builds passing.  There are 2 issues (that I'm working on)
that are likely the last always failing unit tests against hadoop 2.0.

Jon.

On Mon, Jul 30, 2012 at 11:23 AM, Francis Liu tof...@apache.org wrote:
 Hi,

 I noticed there are 0.23 builds on jenkins. I was wondering what state
it is
 in? And who are working on it? Is there an umbrella Jira to get HBase
 working on 0.23?

 -Francis





--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// j...@cloudera.com





-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// j...@cloudera.com