[jira] [Commented] (HBASE-20652) Remove internal uses of some deprecated MasterObserver hooks
[ https://issues.apache.org/jira/browse/HBASE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493089#comment-16493089 ] Appy commented on HBASE-20652: -- Thanks Ted! > Remove internal uses of some deprecated MasterObserver hooks > > > Key: HBASE-20652 > URL: https://issues.apache.org/jira/browse/HBASE-20652 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20652.master.001.patch > > > Remove internal uses of these hooks: > - preModifyNamespace > - postModifyNamespace > - preModifyTable > - postModifyTable > - preModifyTableAction > - postCompletedModifyTableAction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20592) Create a tool to verify tables do not have prefix tree encoding
[ https://issues.apache.org/jira/browse/HBASE-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493088#comment-16493088 ] Appy commented on HBASE-20592: -- bq. Why I'd prefer to have this tool in 2.0+ is to reduce the number of different HBase versions the users have to use during upgrade. Their existing 1.x cluster won't have the tool but it could be part of the new version so they wouldn't need a third one. Well if it could work this way, it'd be best. But there are few considerations: - new admin api is incompatible with 1.x. So while 2.0 uses TableDescriptor/ColumnFamilyDescriptor, 1.x uses HTableDescriptor, HColumnDescriptor. In the case (might want to check the code) that the wire bits are same (PBs), and what's written using one is readable by another, it might work. - I think that's how CM team wanted it, these tools in separate standalone jars (so shading and stuff to make then not need any external when running) > Create a tool to verify tables do not have prefix tree encoding > --- > > Key: HBASE-20592 > URL: https://issues.apache.org/jira/browse/HBASE-20592 > Project: HBase > Issue Type: New Feature > Components: Operability, tooling >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20592.master.001.patch, > HBASE-20592.master.002.patch > > > HBase 2.0.0 removed PREFIX_TREE encoding so users need to modify data block > encoding to something else before upgrading to HBase 2.0+. A tool would help > users to verify that there are no tables left with PREFIX_TREE encoding. > The tool needs to check the following: > * There are no tables where DATA_BLOCK_ENCODING => 'PREFIX_TREE' > * -Check existing hfiles that none of them have PREFIX_TREE encoding (in > case table description is changed but hfiles were not rewritten)- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20652) Remove internal uses of some deprecated MasterObserver hooks
[ https://issues.apache.org/jira/browse/HBASE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493083#comment-16493083 ] Appy commented on HBASE-20652: -- Can someone please commit this for me. Thanks :) > Remove internal uses of some deprecated MasterObserver hooks > > > Key: HBASE-20652 > URL: https://issues.apache.org/jira/browse/HBASE-20652 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-20652.master.001.patch > > > Remove internal uses of these hooks: > - preModifyNamespace > - postModifyNamespace > - preModifyTable > - postModifyTable > - preModifyTableAction > - postCompletedModifyTableAction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20592) Create a tool to verify tables do not have prefix tree encoding
[ https://issues.apache.org/jira/browse/HBASE-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491903#comment-16491903 ] Appy commented on HBASE-20592: -- So in this patch, looks like it's meant to be run against 2.0 jars? But isn't that a problem since 2.0 admin might be incompatible with 1.x deployments? I thought the initial approach was to have these tools such that they compile against 1.x branch, and give them in a separate jar so that they can be run before the upgrade is started. Has the approach changed? > Create a tool to verify tables do not have prefix tree encoding > --- > > Key: HBASE-20592 > URL: https://issues.apache.org/jira/browse/HBASE-20592 > Project: HBase > Issue Type: New Feature > Components: Operability, tooling >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20592.master.001.patch > > > HBase 2.0.0 removed PREFIX_TREE encoding so users need to modify data block > encoding to something else before upgrading to HBase 2.0+. A tool would help > users to verify that there are no tables left with PREFIX_TREE encoding. > The tool needs to check the following: > * There are no tables where DATA_BLOCK_ENCODING => 'PREFIX_TREE' > * -Check existing hfiles that none of them have PREFIX_TREE encoding (in > case table description is changed but hfiles were not rewritten)- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20592) Create a tool to verify tables do not have prefix tree encoding
[ https://issues.apache.org/jira/browse/HBASE-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491902#comment-16491902 ] Appy commented on HBASE-20592: -- Oh no, i forgot to look for existing jiras before starting to work on this. :( Anyways, here's the small tool and tests i wrote: HBASE-20652 Might want to mix the best of two patches? > Create a tool to verify tables do not have prefix tree encoding > --- > > Key: HBASE-20592 > URL: https://issues.apache.org/jira/browse/HBASE-20592 > Project: HBase > Issue Type: New Feature > Components: Operability, tooling >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20592.master.001.patch > > > HBase 2.0.0 removed PREFIX_TREE encoding so users need to modify data block > encoding to something else before upgrading to HBase 2.0+. A tool would help > users to verify that there are no tables left with PREFIX_TREE encoding. > The tool needs to check the following: > * There are no tables where DATA_BLOCK_ENCODING => 'PREFIX_TREE' > * -Check existing hfiles that none of them have PREFIX_TREE encoding (in > case table description is changed but hfiles were not rewritten)- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20594) provide utility to compare old and new descriptors
[
https://issues.apache.org/jira/browse/HBASE-20594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491901#comment-16491901
]
Appy commented on HBASE-20594:
--
LGTM. +1
Glad that i searched for jira before starting to work on this again :)
nit:
{{// CFD -> (name, CFD)}} not sure what this comment means because map is
actually declared above and is {{cf_name --> CFD}}
> provide utility to compare old and new descriptors
> --
>
> Key: HBASE-20594
> URL: https://issues.apache.org/jira/browse/HBASE-20594
> Project: HBase
> Issue Type: Improvement
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-20594.patch, HBASE-20594.v2.patch,
> HBASE-20594.v3.patch, HBASE-20594.v4.patch, HBASE-20594.v5.patch,
> HBASE-20594.v6.patch, HBASE-20594.v7.patch
>
>
> HBASE-20567 gives us hooks that give both the old and new descriptor in
> pre/postModify* events, but comparing them is still cumbersome. We should
> provide users some kind of utility for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20652) Remove internal uses of some deprecated MasterObserver hooks
[ https://issues.apache.org/jira/browse/HBASE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20652: - Status: Patch Available (was: Open) > Remove internal uses of some deprecated MasterObserver hooks > > > Key: HBASE-20652 > URL: https://issues.apache.org/jira/browse/HBASE-20652 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-20652.master.001.patch > > > Remove internal uses of these hooks: > - preModifyNamespace > - postModifyNamespace > - preModifyTable > - postModifyTable > - preModifyTableAction > - postCompletedModifyTableAction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20652) Remove internal uses of some deprecated MasterObserver hooks
[ https://issues.apache.org/jira/browse/HBASE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20652: - Attachment: HBASE-20652.master.001.patch > Remove internal uses of some deprecated MasterObserver hooks > > > Key: HBASE-20652 > URL: https://issues.apache.org/jira/browse/HBASE-20652 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-20652.master.001.patch > > > Remove internal uses of these hooks: > - preModifyNamespace > - postModifyNamespace > - preModifyTable > - postModifyTable > - preModifyTableAction > - postCompletedModifyTableAction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20652) Remove internal uses of some deprecated MasterObserver hooks
Appy created HBASE-20652: Summary: Remove internal uses of some deprecated MasterObserver hooks Key: HBASE-20652 URL: https://issues.apache.org/jira/browse/HBASE-20652 Project: HBase Issue Type: Improvement Reporter: Appy Assignee: Appy Remove internal uses of these hooks: - preModifyNamespace - postModifyNamespace - preModifyTable - postModifyTable - preModifyTableAction - postCompletedModifyTableAction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477912#comment-16477912
]
Appy commented on HBASE-20567:
--
Oh damn, my old ssh keys are in hard disk which is at home, so can't setup
apache repo to push until night. Do you mind pushing this one for me [~mdrob]?
Thanks for the review [~stack] and [~mdrob].
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch,
> HBASE-20567.master.002.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Attachment: HBASE-20567.master.002.patch
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch,
> HBASE-20567.master.002.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Attachment: (was: HBASE-20567.branch-2.001.patch)
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472874#comment-16472874
]
Appy commented on HBASE-20567:
--
Dammit, it was stuck at the password phase of my submit-patch.py script. Should
be there now :D
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.branch-2.001.patch,
> HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Attachment: HBASE-20567.branch-2.001.patch
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.branch-2.001.patch,
> HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472760#comment-16472760
]
Appy commented on HBASE-20567:
--
Makes sense.
Uploaded v2.
Added back old methods for backward compatibility. Now we can backport this
patch to 2.1 too.
I took the liberty of renaming arguments in old methods to make it clear what
they are and how they relate to new methods.
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471482#comment-16471482
]
Appy commented on HBASE-20567:
--
I don't think we can get this into 2.0.1. Does it make sense to backport it to
2.1??
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Status: Patch Available (was: Open)
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Attachment: HBASE-20567.master.001.patch
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.master.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Attachment: (was: HBASE-20567.branch-2.001.patch)
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Attachment: HBASE-20567.branch-2.001.patch
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
> Attachments: HBASE-20567.branch-2.001.patch
>
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Summary: Pass both old and new descriptors to pre/post hooks of modify
operations for table and namespace (was: Give both old and new descriptors to
pre/post hooks of modify operations for table and namespace)
> Pass both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20567) Give both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20567:
-
Summary: Give both old and new descriptors to pre/post hooks of modify
operations for table and namespace (was: Given both old and new descriptors to
pre/post hooks of modify operations for table and namespace)
> Give both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Priority: Major
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20567) Give both old and new descriptors to pre/post hooks of modify operations for table and namespace
[
https://issues.apache.org/jira/browse/HBASE-20567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy reassigned HBASE-20567:
Assignee: Appy
> Give both old and new descriptors to pre/post hooks of modify operations for
> table and namespace
>
>
> Key: HBASE-20567
> URL: https://issues.apache.org/jira/browse/HBASE-20567
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Major
>
> In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's
> no way of getting the old descriptor which was there before modification
> happened.
> Having both old and new descriptors will make the hooks more useful.
> We felt the need when we wanted to audit certain events but there was no way
> of deducing them by just seeing 'after-state' of modification.
> To keep the method signatures consistent, i have modified both pre and post
> hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20567) Given both old and new descriptors to pre/post hooks of modify operations for table and namespace
Appy created HBASE-20567:
Summary: Given both old and new descriptors to pre/post hooks of
modify operations for table and namespace
Key: HBASE-20567
URL: https://issues.apache.org/jira/browse/HBASE-20567
Project: HBase
Issue Type: Improvement
Reporter: Appy
In postModify* hooks like {{postModifyX(..., Descriptor newDesc)}}, there's no
way of getting the old descriptor which was there before modification happened.
Having both old and new descriptors will make the hooks more useful.
We felt the need when we wanted to audit certain events but there was no way of
deducing them by just seeing 'after-state' of modification.
To keep the method signatures consistent, i have modified both pre and post
hooks with new arguments which are well named and commented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435975#comment-16435975 ] Appy commented on HBASE-17449: -- please mention the jira number too, so people following discussion here can easily move to next jira. > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20330) ProcedureExecutor.start() gets stuck in recover lease on store.
[
https://issues.apache.org/jira/browse/HBASE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434684#comment-16434684
]
Appy commented on HBASE-20330:
--
Gave +1 on RB yesterday.
> ProcedureExecutor.start() gets stuck in recover lease on store.
> ---
>
> Key: HBASE-20330
> URL: https://issues.apache.org/jira/browse/HBASE-20330
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: hbase-20330.master.001.patch,
> hbase-20330.master.002.patch, hbase-20330.master.003.patch,
> hbase-20330.master.004.patch, hbase-20330.master.005.patch
>
>
> We have instance in our internal testing where master log is getting filled
> with following messages:
> {code}
> 2018-04-02 17:11:17,566 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recover lease on dfs file
> hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> 2018-04-02 17:11:17,567 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recovered lease, attempt=0 on
> file=hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log after 1ms
> 2018-04-02 17:11:17,574 WARN
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Unable to
> read tracker for hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> - Invalid Trailer version. got 111 expected 1
> 2018-04-02 17:11:17,576 ERROR
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Log file with
> id=19 already exists
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /hbase/MasterProcWALs/pv2-0019.log for client 10.17.202.11
> already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:381)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
> Debugging it further with [~appy], [~avirmani] and [~xiaochen] we found that
> when WALProcedureStore#rollWriter() fails and returns false for some reason,
> it keeps looping continuously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434677#comment-16434677 ] Appy commented on HBASE-20243: -- Beautiful. +100 > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch, > HBASE-20243.master.002.patch, HBASE-20243.master.003.patch, > HBASE-20243.master.004.patch, HBASE-20243.master.005.patch, > HBASE-20243.master.006.patch, HBASE-20243.master.007.patch, > HBASE-20243.master.008.patch, HBASE-20243.master.008.patch, > HBASE-20243.master.009.patch, HBASE-20243.master.010.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20350) NullPointerException in Scanner during close()
[
https://issues.apache.org/jira/browse/HBASE-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432882#comment-16432882
]
Appy commented on HBASE-20350:
--
That's probably because poll() can return null value when PQ heap is empty,
whereas, it won't happen if we iterate like this.
> NullPointerException in Scanner during close()
> --
>
> Key: HBASE-20350
> URL: https://issues.apache.org/jira/browse/HBASE-20350
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Appy
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-20350.master.001.patch
>
>
> From logs:
> {code}
> 2018-04-03 02:06:00,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Replaying edits from
> hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180403004104/834545a2ae1baa47082a3bc7aab2be2f/recovered.edits/1032167
> 2018-04-03 02:06:00,724 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Scanner
> 2120114333978460945 lease expired on region
> IntegrationTestBigLinkedList_20180403004104,\xF1\xFE\xCB\x98e1\xF8\xD4,1522742825561.ce0d91585a2d188123173c36d0b693a5.
> 2018-04-03 02:06:00,730 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server vd0510.halxg.cloudera.com,22101,1522626204176: Uncaught exception in
> executorService thread
> regionserver/vd0510.halxg.cloudera.com/10.17.226.13:22101.leaseChecker *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compareRows(CellComparatorImpl.java:202)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:74)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:61)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:207)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:190)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:228)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:483)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:464)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:224)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$ScannerListener.leaseExpired(RSRpcServices.java:460)
> at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-03 02:06:00,731 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController,
> org.apache.hadoop.hbase.security.token.TokenProvider,
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
> com.cloudera.navigator.audit.hbase.RegionAuditCoProcessor]
> 2018-04-03 02:06:00,737 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics as JSON
> on abort: {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20350) NullPointerException in Scanner during close()
[
https://issues.apache.org/jira/browse/HBASE-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20350:
-
Status: Patch Available (was: Open)
> NullPointerException in Scanner during close()
> --
>
> Key: HBASE-20350
> URL: https://issues.apache.org/jira/browse/HBASE-20350
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Appy
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-20350.master.001.patch
>
>
> From logs:
> {code}
> 2018-04-03 02:06:00,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Replaying edits from
> hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180403004104/834545a2ae1baa47082a3bc7aab2be2f/recovered.edits/1032167
> 2018-04-03 02:06:00,724 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Scanner
> 2120114333978460945 lease expired on region
> IntegrationTestBigLinkedList_20180403004104,\xF1\xFE\xCB\x98e1\xF8\xD4,1522742825561.ce0d91585a2d188123173c36d0b693a5.
> 2018-04-03 02:06:00,730 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server vd0510.halxg.cloudera.com,22101,1522626204176: Uncaught exception in
> executorService thread
> regionserver/vd0510.halxg.cloudera.com/10.17.226.13:22101.leaseChecker *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compareRows(CellComparatorImpl.java:202)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:74)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:61)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:207)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:190)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:228)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:483)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:464)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:224)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$ScannerListener.leaseExpired(RSRpcServices.java:460)
> at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-03 02:06:00,731 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController,
> org.apache.hadoop.hbase.security.token.TokenProvider,
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
> com.cloudera.navigator.audit.hbase.RegionAuditCoProcessor]
> 2018-04-03 02:06:00,737 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics as JSON
> on abort: {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20350) NullPointerException in Scanner during close()
[
https://issues.apache.org/jira/browse/HBASE-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20350:
-
Attachment: HBASE-20350.master.001.patch
> NullPointerException in Scanner during close()
> --
>
> Key: HBASE-20350
> URL: https://issues.apache.org/jira/browse/HBASE-20350
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Appy
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-20350.master.001.patch
>
>
> From logs:
> {code}
> 2018-04-03 02:06:00,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Replaying edits from
> hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180403004104/834545a2ae1baa47082a3bc7aab2be2f/recovered.edits/1032167
> 2018-04-03 02:06:00,724 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Scanner
> 2120114333978460945 lease expired on region
> IntegrationTestBigLinkedList_20180403004104,\xF1\xFE\xCB\x98e1\xF8\xD4,1522742825561.ce0d91585a2d188123173c36d0b693a5.
> 2018-04-03 02:06:00,730 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server vd0510.halxg.cloudera.com,22101,1522626204176: Uncaught exception in
> executorService thread
> regionserver/vd0510.halxg.cloudera.com/10.17.226.13:22101.leaseChecker *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compareRows(CellComparatorImpl.java:202)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:74)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:61)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:207)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:190)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:228)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:483)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:464)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:224)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$ScannerListener.leaseExpired(RSRpcServices.java:460)
> at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-03 02:06:00,731 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController,
> org.apache.hadoop.hbase.security.token.TokenProvider,
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
> com.cloudera.navigator.audit.hbase.RegionAuditCoProcessor]
> 2018-04-03 02:06:00,737 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics as JSON
> on abort: {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20350) NullPointerException in Scanner during close()
[
https://issues.apache.org/jira/browse/HBASE-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429213#comment-16429213
]
Appy commented on HBASE-20350:
--
bq. Can we check the return value from scanner.peek() before calling
scanner.close() ?
That's useless work done by process when all that's needed is getting rid of
the scanner.
Iterate and close seems right way, let me put up patch.
> NullPointerException in Scanner during close()
> --
>
> Key: HBASE-20350
> URL: https://issues.apache.org/jira/browse/HBASE-20350
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Appy
>Priority: Blocker
> Fix For: 2.0.0
>
>
> From logs:
> {code}
> 2018-04-03 02:06:00,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Replaying edits from
> hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180403004104/834545a2ae1baa47082a3bc7aab2be2f/recovered.edits/1032167
> 2018-04-03 02:06:00,724 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Scanner
> 2120114333978460945 lease expired on region
> IntegrationTestBigLinkedList_20180403004104,\xF1\xFE\xCB\x98e1\xF8\xD4,1522742825561.ce0d91585a2d188123173c36d0b693a5.
> 2018-04-03 02:06:00,730 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server vd0510.halxg.cloudera.com,22101,1522626204176: Uncaught exception in
> executorService thread
> regionserver/vd0510.halxg.cloudera.com/10.17.226.13:22101.leaseChecker *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compareRows(CellComparatorImpl.java:202)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:74)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:61)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:207)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:190)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:228)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:483)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:464)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:224)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$ScannerListener.leaseExpired(RSRpcServices.java:460)
> at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-03 02:06:00,731 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController,
> org.apache.hadoop.hbase.security.token.TokenProvider,
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
> com.cloudera.navigator.audit.hbase.RegionAuditCoProcessor]
> 2018-04-03 02:06:00,737 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics as JSON
> on abort: {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429173#comment-16429173 ] Appy commented on HBASE-17449: -- Update, i have tests to prove that it works. Still working on fixing few errors. > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20350) NullPointerException in Scanner during close()
[
https://issues.apache.org/jira/browse/HBASE-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-20350:
-
Summary: NullPointerException in Scanner during close() (was: RegionServer
is aborted due to NPE when scanner lease expired on a region)
> NullPointerException in Scanner during close()
> --
>
> Key: HBASE-20350
> URL: https://issues.apache.org/jira/browse/HBASE-20350
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Appy
>Priority: Blocker
> Fix For: 2.0.0
>
>
> From logs:
> {code}
> 2018-04-03 02:06:00,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Replaying edits from
> hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180403004104/834545a2ae1baa47082a3bc7aab2be2f/recovered.edits/1032167
> 2018-04-03 02:06:00,724 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Scanner
> 2120114333978460945 lease expired on region
> IntegrationTestBigLinkedList_20180403004104,\xF1\xFE\xCB\x98e1\xF8\xD4,1522742825561.ce0d91585a2d188123173c36d0b693a5.
> 2018-04-03 02:06:00,730 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server vd0510.halxg.cloudera.com,22101,1522626204176: Uncaught exception in
> executorService thread
> regionserver/vd0510.halxg.cloudera.com/10.17.226.13:22101.leaseChecker *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compareRows(CellComparatorImpl.java:202)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:74)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:61)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:207)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:190)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:228)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:483)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:464)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:224)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$ScannerListener.leaseExpired(RSRpcServices.java:460)
> at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-03 02:06:00,731 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController,
> org.apache.hadoop.hbase.security.token.TokenProvider,
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
> com.cloudera.navigator.audit.hbase.RegionAuditCoProcessor]
> 2018-04-03 02:06:00,737 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics as JSON
> on abort: {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20330) ProcedureExecutor.start() gets stuck in recover lease on store.
[
https://issues.apache.org/jira/browse/HBASE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429131#comment-16429131
]
Appy commented on HBASE-20330:
--
RB down for me.
[~uagashe] try the suggestion, you'll see what i am saying.
It's just a simple code refactor to make it easy to understand different code
flows.
> ProcedureExecutor.start() gets stuck in recover lease on store.
> ---
>
> Key: HBASE-20330
> URL: https://issues.apache.org/jira/browse/HBASE-20330
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: hbase-20330.master.001.patch,
> hbase-20330.master.002.patch, hbase-20330.master.003.patch,
> hbase-20330.master.004.patch
>
>
> We have instance in our internal testing where master log is getting filled
> with following messages:
> {code}
> 2018-04-02 17:11:17,566 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recover lease on dfs file
> hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> 2018-04-02 17:11:17,567 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recovered lease, attempt=0 on
> file=hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log after 1ms
> 2018-04-02 17:11:17,574 WARN
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Unable to
> read tracker for hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> - Invalid Trailer version. got 111 expected 1
> 2018-04-02 17:11:17,576 ERROR
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Log file with
> id=19 already exists
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /hbase/MasterProcWALs/pv2-0019.log for client 10.17.202.11
> already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:381)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
> Debugging it further with [~appy], [~avirmani] and [~xiaochen] we found that
> when WALProcedureStore#rollWriter() fails and returns false for some reason,
> it keeps looping continuously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20350) RegionServer is aborted due to NPE when scanner lease expired on a region
[
https://issues.apache.org/jira/browse/HBASE-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429127#comment-16429127
]
Appy commented on HBASE-20350:
--
This bug is breakdown of our very basic invariant - scanner.next() doesn't
return null.
Basic pattern in scanners is, after each scanner.next(), do a scanner.peek() to
make sure that next call to next() won't be null.
In this bug, when PQ.poll() is called, it calls peek() on scanners and gets
null from some scanner, which is bad.
The fact that the poll was done during close() which in turn was started by
lease expiry doesn't necessarily make the lease expiry culprit. I think it
*could* have very well happened if the scanner was in exact same state but
close was called by say, the client.
This will be a very difficult to reproduce without knowing exact scanner
details, and hence difficult to debug.
The difficulty adds up given not many changes were made to this part recently
(the once i saw seemed unrelated, like casting, etc)
So i'd say, don't make it a blocker.
But yes, a simple iteration over scanners in KeyValueHeap, instead of poll()s,
when closing seems a good change to make.
> RegionServer is aborted due to NPE when scanner lease expired on a region
> -
>
> Key: HBASE-20350
> URL: https://issues.apache.org/jira/browse/HBASE-20350
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Appy
>Priority: Blocker
> Fix For: 2.0.0
>
>
> From logs:
> {code}
> 2018-04-03 02:06:00,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Replaying edits from
> hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180403004104/834545a2ae1baa47082a3bc7aab2be2f/recovered.edits/1032167
> 2018-04-03 02:06:00,724 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Scanner
> 2120114333978460945 lease expired on region
> IntegrationTestBigLinkedList_20180403004104,\xF1\xFE\xCB\x98e1\xF8\xD4,1522742825561.ce0d91585a2d188123173c36d0b693a5.
> 2018-04-03 02:06:00,730 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server vd0510.halxg.cloudera.com,22101,1522626204176: Uncaught exception in
> executorService thread
> regionserver/vd0510.halxg.cloudera.com/10.17.226.13:22101.leaseChecker *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compareRows(CellComparatorImpl.java:202)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:74)
> at
> org.apache.hadoop.hbase.CellComparatorImpl.compare(CellComparatorImpl.java:61)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:207)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:190)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:228)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:483)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:464)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:224)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$ScannerListener.leaseExpired(RSRpcServices.java:460)
> at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-03 02:06:00,731 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController,
> org.apache.hadoop.hbase.security.token.TokenProvider,
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
> com.cloudera.navigator.audit.hbase.RegionAuditCoProcessor]
> 2018-04-03 02:06:00,737 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics as JSON
> on abort: {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20286: - Fix Version/s: 1.5.0 2.1.0 > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Csaba Skrabak >Priority: Minor > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20286-v2.patch, HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20286: - Resolution: Fixed Status: Resolved (was: Patch Available) > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Csaba Skrabak >Priority: Minor > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20286-v2.patch, HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy reassigned HBASE-20286: Assignee: Csaba Skrabak > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Csaba Skrabak >Assignee: Csaba Skrabak >Priority: Minor > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20286-v2.patch, HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429086#comment-16429086 ] Appy commented on HBASE-20286: -- Precisely... :) Thanks [~cskrabak]. +1 Pushing to repo. > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Csaba Skrabak >Priority: Minor > Attachments: HBASE-20286-v2.patch, HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20149) Purge dev javadoc from bin tarball (or make a separate tarball of javadoc)
[ https://issues.apache.org/jira/browse/HBASE-20149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429045#comment-16429045 ] Appy commented on HBASE-20149: -- What's the possibility - technically, policy-wise if any, user expectation, etc - of separating out all of site stuff from the release? > Purge dev javadoc from bin tarball (or make a separate tarball of javadoc) > -- > > Key: HBASE-20149 > URL: https://issues.apache.org/jira/browse/HBASE-20149 > Project: HBase > Issue Type: Sub-task > Components: build, community, documentation >Reporter: stack >Assignee: Artem Ervits >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20149.branch-2.0.001.patch > > > The bin tarball is too fat (Chia-Ping and Josh noticed it on the beta-2 > vote). A note to the dev list subsequently resulted in suggestion that we > just purge dev javadoc (or even all javadoc) from bin tarball (Andrew). Sean > was good w/ it and suggested perhaps we could do a javadoc only tgz. Let me > look into this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19488) Move to using Apache commons CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19488: - Resolution: Fixed Status: Resolved (was: Patch Available) > Move to using Apache commons CollectionUtils > > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.1.0 > > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch, HBASE-19488.5.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20286) Improving shell command compaction_state
[
https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426208#comment-16426208
]
Appy commented on HBASE-20286:
--
That issue is unrelated, it's about how to handle values returned by commands
in interactive mode, and not about whether commands should return values or
note in the first place.
{quote}I believe users don't really rely on this hidden command, the idea of a
compaction_state command came from a testing task.
{quote}
Can't know for sure, so let's put it back? Almost everything start with some
idea and outgrows its initial scope. It's plausible that someone can be using
it in a script as some sort of wait condition.
> Improving shell command compaction_state
>
>
> Key: HBASE-20286
> URL: https://issues.apache.org/jira/browse/HBASE-20286
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Csaba Skrabak
>Priority: Minor
> Attachments: HBASE-20286.patch
>
>
> Command does not output anything, let's add a formatter.row call.
> We should include the possible outputs in the help text.
> *Further improvement possibility.* This command can be used for checking if
> the compaction is done but very impractical if one wants to wait _until_ it
> is done. Wish there would be a flag in the shell that enforces synchronous
> compactions, that is, every time you issue a compact or major_compact in the
> shell while this flag is set, you won't get back the prompt until it finishes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19488) Move to using Apache commons CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19488: - Fix Version/s: 2.1.0 > Move to using Apache commons CollectionUtils > > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.1.0 > > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch, HBASE-19488.5.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19488) Move to using Apache commons CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19488: - Summary: Move to using Apache commons CollectionUtils (was: Remove Unused Code from CollectionUtils) > Move to using Apache commons CollectionUtils > > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch, HBASE-19488.5.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426172#comment-16426172 ] Appy commented on HBASE-20243: -- QA still not happy. Would have been fine committing it if qa was green, but since there's gonna be another revision, here are couple minor suggestions on the table which'll lead to more maintainable code. It's upto you to take/leave them * In our design, shell tests are not run when we submit a patch to just hbase-server module. So if someone changes error message, your last two tests will start failing. Probably checking for just the exception type is enough. * *Prefer multiple small tests over one-for-all test*. It's easy to maintain since unittest code is more chunked -> easier to understand -> easier to fix when tests fail. For eg, if your current test fails in "test for existent destination table", someone will have to read through everything else before itand that would be wasted effort since it's not needed and the test for 'existent destination table' could have very well been a simple <10 lines separate test. * Prefer to use constant declared variables if it's value matters in multiple places. For eg. NUM_SPLITS=2, then init column families with it and use the same thing in asserts. Makes it easy to understand tests. Another examples is, consider random test with two figures - 10 and 50. 10 initializes something and we are asserting 50. It's not obvious if we are expecting test to assert 10+40 or 10*5. * Avoid [coupling|https://en.wikipedia.org/wiki/Coupling_(computer_programming)] tests. If one breaks for weird reason, others might too. (ref: new_table = "test_clone_table_schema_table") > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch, > HBASE-20243.master.002.patch, HBASE-20243.master.003.patch, > HBASE-20243.master.004.patch, HBASE-20243.master.005.patch, > HBASE-20243.master.006.patch, HBASE-20243.master.007.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426113#comment-16426113 ] Appy commented on HBASE-17730: -- Pushed header addition to design doc and changes to upgrade section. Thanks for review guys. > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy resolved HBASE-17730. -- Resolution: Fixed > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426098#comment-16426098 ] Appy edited comment on HBASE-17730 at 4/4/18 8:04 PM: -- Oh! Thanks [~psomogyi] for pointing it out. Correcting it. And committing the docs change was (Author: appy): Oh! Thanks [~psomogyi] for pointing it out. Correcting it. > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426098#comment-16426098 ] Appy commented on HBASE-17730: -- Oh! Thanks [~psomogyi] for pointing it out. Correcting it. > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424832#comment-16424832 ] Appy commented on HBASE-20243: -- Patch looks good, few comments. * don't we need shell test too? * use a variable NUM_SPLITS=2 and use that in variable initializations and asserts * Add tests for cases when a) source table doesn't exist, b) destination table exists. Basically, we should have tests for both success and failure scenarios. * Does it compile? I don't see FAMILY_0 and 1 in declarations in TestAsyncTableAdminApi.java * Any way we can refactor out the common code in test? > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch, > HBASE-20243.master.002.patch, HBASE-20243.master.003.patch, > HBASE-20243.master.004.patch, HBASE-20243.master.005.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424764#comment-16424764 ] Appy commented on HBASE-17730: -- Uploaded patch v1 which adds section "3.1.2. Upgrading Coprocessors to 2.0" to our book. Ping [~busbey], [~mdrob] for review. > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17730: - Attachment: HBASE-17730.master.001.patch > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-17730 started by Appy. > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17730.master.001.patch > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17730) [DOC] Migration to 2.0 for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424710#comment-16424710 ] Appy commented on HBASE-17730: -- Uploaded ascii doc of [Coprocessor design improvements|https://docs.google.com/document/d/1mPkM1CRRvBMZL4dBQzrus8obyvNnHhR5it2yyhiFXTg/edit#heading=h.cu9hamx6jk24] to dev-support/design-docs [here|https://github.com/apache/hbase/blob/master/dev-support/design-docs/Coprocessor_Design_Improvements-Use_composition_instead_of_inheritance-HBASE-17732.adoc]. > [DOC] Migration to 2.0 for coprocessors > > > Key: HBASE-17730 > URL: https://issues.apache.org/jira/browse/HBASE-17730 > Project: HBase > Issue Type: Sub-task > Components: documentation, migration >Reporter: Appy >Assignee: Appy >Priority: Blocker > Fix For: 2.0.0 > > > Jiras breaking coprocessor compatibility should be marked with component ' > Coprocessor', and label 'incompatible'. > Close to releasing 2.0, we should go through all such jiras and write down > steps for migrating coprocessor easily. > The idea is, it might be very hard to fix coprocessor breakages by reverse > engineering errors, but will be easier we suggest easiest way to fix > breakages resulting from each individual incompatible change. > For eg. HBASE-17312 is incompatible change. It'll result in 100s of errors > because BaseXXXObserver classes are gone and will probably result in a lot of > confusion, but if we explicitly mention the fix which is just one line change > - replace "Foo extends BaseXXXObserver" with "Foo implements XXXObserver" - > it makes it very easy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20330) ProcedureExecutor.start() gets stuck in recover lease on store.
[
https://issues.apache.org/jira/browse/HBASE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424703#comment-16424703
]
Appy commented on HBASE-20330:
--
we can probably fix this easily by refreshing the list of log files in start of
each loopneed to be sure.
> ProcedureExecutor.start() gets stuck in recover lease on store.
> ---
>
> Key: HBASE-20330
> URL: https://issues.apache.org/jira/browse/HBASE-20330
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.0
>
>
> We have instance in our internal testing where master log is getting filled
> with following messages:
> {code}
> 2018-04-02 17:11:17,566 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recover lease on dfs file
> hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> 2018-04-02 17:11:17,567 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recovered lease, attempt=0 on
> file=hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log after 1ms
> 2018-04-02 17:11:17,574 WARN
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Unable to
> read tracker for hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> - Invalid Trailer version. got 111 expected 1
> 2018-04-02 17:11:17,576 ERROR
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Log file with
> id=19 already exists
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /hbase/MasterProcWALs/pv2-0019.log for client 10.17.202.11
> already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:381)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
> Debugging it further with [~appy] and [~avirmani], we found that when
> WALProcedureStore#rollWriter() fails and returns false for some reason, it
> keeps looping continuously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20330) ProcedureExecutor.start() gets stuck in recover lease on store.
[
https://issues.apache.org/jira/browse/HBASE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424695#comment-16424695
]
Appy commented on HBASE-20330:
--
Hard to reproduce problem.
Our hypothesis was, if a master during startup tries to create new proc-wal,
and fails while writing header (for 19.log in this case)
[here|https://github.com/apache/hbase/blob/d9e64aa6b83fb6ed5230b0fde86fdf8d8732e1a4/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L1041],
then it'll get stuck in infinite loop.
Exact seq of steps.
# rollWriter fails to write header (for 19.log), returns false.
# we 'continue' the loop
[here|https://github.com/apache/hbase/blob/d9e64aa6b83fb6ed5230b0fde86fdf8d8732e1a4/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L377]
# isRunning() will always be true, list of oldLogs is not refreshed (it's
still till 18.log)
# renew leases on all existing log files till 18, try to create 19
# since it was already created last time, rollWriter returns false and we
'continue' from step 2 again.
> ProcedureExecutor.start() gets stuck in recover lease on store.
> ---
>
> Key: HBASE-20330
> URL: https://issues.apache.org/jira/browse/HBASE-20330
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.0
>
>
> We have instance in our internal testing where master log is getting filled
> with following messages:
> {code}
> 2018-04-02 17:11:17,566 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recover lease on dfs file
> hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> 2018-04-02 17:11:17,567 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recovered lease, attempt=0 on
> file=hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log after 1ms
> 2018-04-02 17:11:17,574 WARN
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Unable to
> read tracker for hdfs://ns1/hbase/MasterProcWALs/pv2-0018.log
> - Invalid Trailer version. got 111 expected 1
> 2018-04-02 17:11:17,576 ERROR
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Log file with
> id=19 already exists
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /hbase/MasterProcWALs/pv2-0019.log for client 10.17.202.11
> already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:381)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2442)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
> Debugging it further with [~appy] and [~avirmani], we found that when
> WALProcedureStore#rollWriter() fails and returns false for some reason, it
> keeps looping continuously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19488) Remove Unused Code from CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422917#comment-16422917 ] Appy commented on HBASE-19488: -- Sorry for late reply, was on long vacation. bq. I don't want to have to fix checkstyle for every file I touched to make this happen. I didn't cause them in this patch and they should be fixed in another patch with that focus. bq. -1 checkstyle 1m 15s hbase-server: The patch generated 1 new + 362 unchanged - 1 fixed = 363 total (was 363) It's not expected either...smile. When you see "-1 checkstyle", only look for 'new' errors (caused by patch) and ignore 'unchanged'. Committing to master and branch-2 > Remove Unused Code from CollectionUtils > --- > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20276) [shell] confirm shell REPL change and document
[
https://issues.apache.org/jira/browse/HBASE-20276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422815#comment-16422815
]
Appy commented on HBASE-20276:
--
Sorry guys for the mess i started. :(
Another alternate is - Remove the --return-values flag and make that
functionality default (which is what we had earlier). And explicitly call it
out as bug in 1.4.0, 1.4.1 releases.
That will undo the damage in 2.0 and we won't have to carry the baggage in
future.
> [shell] confirm shell REPL change and document
> --
>
> Key: HBASE-20276
> URL: https://issues.apache.org/jira/browse/HBASE-20276
> Project: HBase
> Issue Type: Sub-task
> Components: documentation, shell
>Affects Versions: 1.4.0, 2.0.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.4.4, 2.0.0
>
>
> Feedback from [~mdrob] on HBASE-19158:
> {quote}
> Shell:
> HBASE-19770. There was another issue opened where this was identified as a
> problem so maybe the shape will change further, but I can't find it now.
> {quote}
> New commentary from [~busbey]:
> This was a follow on to HBASE-15965. That change effectively makes it so none
> of our ruby wrappers can be used to build expressions in an interactive REPL.
> This is a pretty severe change (most of my tips on HBASE-15611 will break, I
> think).
> I think we should
> a) Have a DISCUSS thread, spanning dev@ and user@
> b) based on the outcome of that thread, either default to the new behavior or
> the old behavior
> c) if we keep the HBASE-15965 behavior as the default, flag it as
> incompatible, call it out in the hbase 2.0 upgrade section, and update docs
> (two examples: the output in the shell_exercises sections would be wrong, and
> the _table_variables section won't work)
> d) In either case document the new flag in the ref guide
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20304) Flip default "return-values" in shell from false to true
[
https://issues.apache.org/jira/browse/HBASE-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy reassigned HBASE-20304:
Assignee: (was: Josh Elser)
> Flip default "return-values" in shell from false to true
>
>
> Key: HBASE-20304
> URL: https://issues.apache.org/jira/browse/HBASE-20304
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Priority: Blocker
> Attachments: HBASE-20304.001.branch-2.0.patch
>
>
> HBASE-15965 Was trying to be helpful in reducing spam in console output
> printed back to the user.
> However, [~psomogyi] and [~balazs.meszaros] made a good find that some shell
> commands (e.g. {{get_table}}) just don't work at all when the default
> {{return-values=false}}. This would be quite obtuse to try to explain.
> [~appy] and I had mulled over whether we should flip the default from false
> to true. The above example makes me sure that we should for 2.0.0.
> fyi [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20304) Flip default "return-values" in shell from false to true
[
https://issues.apache.org/jira/browse/HBASE-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy reassigned HBASE-20304:
Assignee: Josh Elser
> Flip default "return-values" in shell from false to true
>
>
> Key: HBASE-20304
> URL: https://issues.apache.org/jira/browse/HBASE-20304
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Attachments: HBASE-20304.001.branch-2.0.patch
>
>
> HBASE-15965 Was trying to be helpful in reducing spam in console output
> printed back to the user.
> However, [~psomogyi] and [~balazs.meszaros] made a good find that some shell
> commands (e.g. {{get_table}}) just don't work at all when the default
> {{return-values=false}}. This would be quite obtuse to try to explain.
> [~appy] and I had mulled over whether we should flip the default from false
> to true. The above example makes me sure that we should for 2.0.0.
> fyi [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20304) Flip default "return-values" in shell from false to true
[
https://issues.apache.org/jira/browse/HBASE-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422800#comment-16422800
]
Appy commented on HBASE-20304:
--
Good one.
Just saw this, was on week leave.
Sorry for mess i started.
> Flip default "return-values" in shell from false to true
>
>
> Key: HBASE-20304
> URL: https://issues.apache.org/jira/browse/HBASE-20304
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Josh Elser
>Priority: Blocker
> Attachments: HBASE-20304.001.branch-2.0.patch
>
>
> HBASE-15965 Was trying to be helpful in reducing spam in console output
> printed back to the user.
> However, [~psomogyi] and [~balazs.meszaros] made a good find that some shell
> commands (e.g. {{get_table}}) just don't work at all when the default
> {{return-values=false}}. This would be quite obtuse to try to explain.
> [~appy] and I had mulled over whether we should flip the default from false
> to true. The above example makes me sure that we should for 2.0.0.
> fyi [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20286: - Component/s: shell > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Csaba Skrabak >Priority: Minor > Attachments: HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20286: - Issue Type: Improvement (was: Bug) > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Csaba Skrabak >Priority: Minor > Attachments: HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420366#comment-16420366 ] Appy commented on HBASE-20286: -- This will change the return value of command function. Not sure if users rely on it, but we can easily retain old behavior too. > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Bug >Reporter: Csaba Skrabak >Priority: Major > Attachments: HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20286) Improving shell command compaction_state
[ https://issues.apache.org/jira/browse/HBASE-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20286: - Priority: Minor (was: Major) > Improving shell command compaction_state > > > Key: HBASE-20286 > URL: https://issues.apache.org/jira/browse/HBASE-20286 > Project: HBase > Issue Type: Bug > Components: shell >Reporter: Csaba Skrabak >Priority: Minor > Attachments: HBASE-20286.patch > > > Command does not output anything, let's add a formatter.row call. > We should include the possible outputs in the help text. > *Further improvement possibility.* This command can be used for checking if > the compaction is done but very impractical if one wants to wait _until_ it > is done. Wish there would be a flag in the shell that enforces synchronous > compactions, that is, every time you issue a compact or major_compact in the > shell while this flag is set, you won't get back the prompt until it finishes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411105#comment-16411105 ] Appy edited comment on HBASE-20243 at 3/23/18 9:56 AM: --- bq. I think it would be more convenient if we had a command to clone the table directly. i think so too. Edit: Because in one-off or evolving cases, unlike set pattern as you guys have, the only other way i can think of is snapshot + clone + truncate which is too many steps. (Or is there a another better way?) But the command name sure is confusing, given what 'clone' has meant historically in hbase world; cloning a snapshot means both schema + data gets copied. Can we please rename it to cloneTableSchema? was (Author: appy): bq. I think it would be more convenient if we had a command to clone the table directly. i think so too. But the command name sure is confusing, given what 'clone' has meant historically in hbase world; cloning a snapshot means both schema + data gets copied. Can we please rename it to cloneTableSchema? > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20243: - Priority: Minor (was: Major) > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20243: - Fix Version/s: 2.1.0 > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411105#comment-16411105 ] Appy commented on HBASE-20243: -- bq. I think it would be more convenient if we had a command to clone the table directly. i think so too. But the command name sure is confusing, given what 'clone' has meant historically in hbase world; cloning a snapshot means both schema + data gets copied. Can we please rename it to cloneTableSchema? > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: HBASE-20243.master.001.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20249) Investigate behavior of hbase.client.operation.timeout
[ https://issues.apache.org/jira/browse/HBASE-20249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20249: - Priority: Critical (was: Major) > Investigate behavior of hbase.client.operation.timeout > -- > > Key: HBASE-20249 > URL: https://issues.apache.org/jira/browse/HBASE-20249 > Project: HBase > Issue Type: Bug >Reporter: Appy >Priority: Critical > Fix For: 2.0.0 > > > hbase.client.operation.timeout should be hard limit on client operations, and > independent of number of retires. > Previous discussions here: > https://issues.apache.org/jira/browse/HBASE-17449?focusedCommentId=16390085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16390085 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20249) Investigate behavior of hbase.client.operation.timeout
[ https://issues.apache.org/jira/browse/HBASE-20249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20249: - Description: hbase.client.operation.timeout should be hard limit on client operations, and independent of number of retires. Previous discussions here: https://issues.apache.org/jira/browse/HBASE-17449?focusedCommentId=16390085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16390085 was:hbase.client.operation.timeout should be hard limit on client operations, and independent of number of retires. > Investigate behavior of hbase.client.operation.timeout > -- > > Key: HBASE-20249 > URL: https://issues.apache.org/jira/browse/HBASE-20249 > Project: HBase > Issue Type: Bug >Reporter: Appy >Priority: Major > > hbase.client.operation.timeout should be hard limit on client operations, and > independent of number of retires. > Previous discussions here: > https://issues.apache.org/jira/browse/HBASE-17449?focusedCommentId=16390085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16390085 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409122#comment-16409122 ] Appy edited comment on HBASE-17449 at 3/22/18 7:22 AM: --- So I did some testing to verify functionality. It seemed to be working as intended. Let me put up the patch with tests. In meantime, the patch in this improvement jira - tests and doc - looks fine to get in. Let me commit it and file a dedicated one for investigation/adding tests for behavior of hbase.client.operation.timeout. Edit: Created HBASE-20249 was (Author: appy): So I did some testing to verify functionality. It seemed to be working as intended. Let me put up the patch with tests. In meantime, the patch in this improvement jira - tests and doc - looks fine to get in. Let me commit it and file a dedicated one for investigation/adding tests for behavior of hbase.client.operation.timeout. > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20249) Investigate behavior of hbase.client.operation.timeout
[ https://issues.apache.org/jira/browse/HBASE-20249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20249: - Fix Version/s: 2.0.0 > Investigate behavior of hbase.client.operation.timeout > -- > > Key: HBASE-20249 > URL: https://issues.apache.org/jira/browse/HBASE-20249 > Project: HBase > Issue Type: Bug >Reporter: Appy >Priority: Major > Fix For: 2.0.0 > > > hbase.client.operation.timeout should be hard limit on client operations, and > independent of number of retires. > Previous discussions here: > https://issues.apache.org/jira/browse/HBASE-17449?focusedCommentId=16390085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16390085 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20249) Investigate behavior of hbase.client.operation.timeout
Appy created HBASE-20249: Summary: Investigate behavior of hbase.client.operation.timeout Key: HBASE-20249 URL: https://issues.apache.org/jira/browse/HBASE-20249 Project: HBase Issue Type: Bug Reporter: Appy hbase.client.operation.timeout should be hard limit on client operations, and independent of number of retires. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy reassigned HBASE-17449: Assignee: Peter Somogyi (was: Yu Li) > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy resolved HBASE-17449. -- Resolution: Fixed > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409122#comment-16409122 ] Appy commented on HBASE-17449: -- So I did some testing to verify functionality. It seemed to be working as intended. Let me put up the patch with tests. In meantime, the patch in this improvement jira - tests and doc - looks fine to get in. Let me commit it and file a dedicated one for investigation/adding tests for behavior of hbase.client.operation.timeout. > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406019#comment-16406019 ] Appy commented on HBASE-17449: -- yes. Tried understanding retrying caller. We have a tracker to keep track of overall operation time - https://github.com/apache/hbase/blob/2a3f4a0a4eb7bb67985050b0e007576c468f217e/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java#L99 And this check should be enforcing it - https://github.com/apache/hbase/blob/2a3f4a0a4eb7bb67985050b0e007576c468f217e/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java#L156 Let me dig more. > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403922#comment-16403922 ] Appy commented on HBASE-20212: -- Checked the plugin code (https://github.com/revelc/warbucks-maven-plugin), sadly it doesn't have any param to specify exclude pattern :( So i guess, regex lookahead method is the smallest and best way? Go for it. > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403853#comment-16403853
]
Appy commented on HBASE-20151:
--
That sounds coherent to kinds of issues i saw with filter last time. Post a
patch [~reidchan]? I'll take a look at code soon.
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403310#comment-16403310 ] Appy commented on HBASE-20212: -- yeah, can do few modules at a time to chunk up the work. > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402171#comment-16402171 ] Appy edited comment on HBASE-15809 at 3/16/18 4:50 PM: --- Checked UIs. They look great and will be very useful for many users. +1 [~stack] the zip is what i uploaded earlier, still waiting for new patches. was (Author: appy): Checked UIs. They look great and will be very useful for many users. +1 > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.0.0, 1.5.0 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch, rep_web_ui.zip > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15809) Basic Replication WebUI
[ https://issues.apache.org/jira/browse/HBASE-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402171#comment-16402171 ] Appy commented on HBASE-15809: -- Checked UIs. They look great and will be very useful for many users. +1 > Basic Replication WebUI > --- > > Key: HBASE-15809 > URL: https://issues.apache.org/jira/browse/HBASE-15809 > Project: HBase > Issue Type: New Feature > Components: Replication, UI >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.0.0, 1.5.0 > > Attachments: HBASE-15809-v0.patch, HBASE-15809-v0.png, > HBASE-15809-v1.patch, rep_web_ui.zip > > > At the moment the only way to have some insight on replication from the webui > is looking at zkdump and metrics. > the basic information useful to get started debugging are: peer information > and the view of WALs offsets for each peer. > https://reviews.apache.org/r/47275/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399970#comment-16399970 ] Appy edited comment on HBASE-20188 at 3/15/18 6:24 AM: --- bq. You going to have pretty graphs of GC usage, i/os, CPU too? Can't this time. We don't have any easy way to collect those in our open source version (or maybe there's one am not aware of?). And I can't use CM due to technical (and license too?) reasons - hbase2.0 + hadoop2.7. bq. At the moment I am getting abysmal perf because we are blocking writes because we are unable to flush in time. In that case, i'll not waste time in doing full perf testing. I'll reduce the scope by reducing number of rows and limiting to single workload. Can do complete one as mentioned previously when things look good. was (Author: appy): bq. You going to have pretty graphs of GC usage, i/os, CPU too? Can't this time. We don't have any easy way to collect those in our open source version (or maybe there's one am not aware of?). And I can't use CM due to technical (and license too?) reasons - hbase2.0 + hadoop2.7. bq. At the moment I am getting abysmal perf because we are blocking writes because we are unable to flush in time. In that case, i'll not waste time in doing full perf testing. I'll reduce the scope by reducing number of rows and limiting to single workload. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399970#comment-16399970 ] Appy commented on HBASE-20188: -- bq. You going to have pretty graphs of GC usage, i/os, CPU too? Can't this time. We don't have any easy way to collect those in our open source version (or maybe there's one am not aware of?). And I can't use CM due to technical (and license too?) reasons - hbase2.0 + hadoop2.7. bq. At the moment I am getting abysmal perf because we are blocking writes because we are unable to flush in time. In that case, i'll not waste time in doing full perf testing. I'll reduce the scope by reducing number of rows and limiting to single workload. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20185) Fix ACL check for MasterRpcServices#execProcedure
[ https://issues.apache.org/jira/browse/HBASE-20185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20185: - Resolution: Fixed Status: Resolved (was: Patch Available) > Fix ACL check for MasterRpcServices#execProcedure > - > > Key: HBASE-20185 > URL: https://issues.apache.org/jira/browse/HBASE-20185 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20185.master.001.patch > > > Mailing thread ref: > [http://mail-archives.apache.org/mod_mbox/hbase-dev/201803.mbox/%3CCAAjhxrriGy_UXpC4iHCSyBB18iAbjU3Y2%2BnjQ-66i9kPPCrPRQ%40mail.gmail.com%3E] > TLDR; HBASE-19400 messed up perms required for flushing a table. > > Looks like flush and snapshot procedures are already doing permissions check > as part of preTableFlush/preSnapshot hooks. However, > LogRollMasterProcedureManager is missing access checks ([~elserj], can > someone look at it?) > > With that, it makes no sense to put an ADMIN perm requirement which was added > by me in HBASE-19400. Removing it. > However, to make things better for future, i have made few design changes > which will ensure 1) perm checks don't slip by mistake, 2) a suitable > placeholder for checks for flush & snapshot when we remove AccessController > for good. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19770) Add '--return-values' option to Shell to print return values of commands in interactive mode
[
https://issues.apache.org/jira/browse/HBASE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399959#comment-16399959
]
Appy commented on HBASE-19770:
--
wdys? default return-value to true (And rename option to --no-return-value) or
remove the option altogether?
> Add '--return-values' option to Shell to print return values of commands in
> interactive mode
>
>
> Key: HBASE-19770
> URL: https://issues.apache.org/jira/browse/HBASE-19770
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19770.000.branch-2.addendum.patch,
> HBASE-19770.001.branch-2.patch, HBASE-19770.002.branch-2.patch,
> HBASE-19770.003.branch-2.patch, HBASE-19770.004.branch-2.patch,
> HBASE-19770.ADDENDUM.branch-2.patch
>
>
> Another good find by our Romil.
> {code}
> hbase(main):001:0> list
> TABLE
> a
> 1 row(s)
> Took 0.8385 seconds
> hbase(main):002:0> tables=list
> TABLE
> a
> 1 row(s)
> Took 0.0267 seconds
> hbase(main):003:0> puts tables
> hbase(main):004:0> p tables
> nil
> {code}
> The {{list}} command should be returning {{\['a'\]}} but is not.
> The command class itself appears to be doing the right thing -- maybe the
> retval is getting lost somewhere else?
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19770) Add '--return-values' option to Shell to print return values of commands in interactive mode
[
https://issues.apache.org/jira/browse/HBASE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399947#comment-16399947
]
Appy commented on HBASE-19770:
--
bq. I think flipping the default to true is something we could consider if this
"experiment" is deemed not as helpful as intended.
sounds right.
Pardon for starting down this road.
> Add '--return-values' option to Shell to print return values of commands in
> interactive mode
>
>
> Key: HBASE-19770
> URL: https://issues.apache.org/jira/browse/HBASE-19770
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2
>
> Attachments: HBASE-19770.000.branch-2.addendum.patch,
> HBASE-19770.001.branch-2.patch, HBASE-19770.002.branch-2.patch,
> HBASE-19770.003.branch-2.patch, HBASE-19770.004.branch-2.patch,
> HBASE-19770.ADDENDUM.branch-2.patch
>
>
> Another good find by our Romil.
> {code}
> hbase(main):001:0> list
> TABLE
> a
> 1 row(s)
> Took 0.8385 seconds
> hbase(main):002:0> tables=list
> TABLE
> a
> 1 row(s)
> Took 0.0267 seconds
> hbase(main):003:0> puts tables
> hbase(main):004:0> p tables
> nil
> {code}
> The {{list}} command should be returning {{\['a'\]}} but is not.
> The command class itself appears to be doing the right thing -- maybe the
> retval is getting lost somewhere else?
> FYI [~stack].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20185) Fix ACL check for MasterRpcServices#execProcedure
[ https://issues.apache.org/jira/browse/HBASE-20185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398610#comment-16398610 ] Appy commented on HBASE-20185: -- Thanks for reviews [~elserj] and [~stack]. Pushed to master, branch-2, branch-2.0 > Fix ACL check for MasterRpcServices#execProcedure > - > > Key: HBASE-20185 > URL: https://issues.apache.org/jira/browse/HBASE-20185 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20185.master.001.patch > > > Mailing thread ref: > [http://mail-archives.apache.org/mod_mbox/hbase-dev/201803.mbox/%3CCAAjhxrriGy_UXpC4iHCSyBB18iAbjU3Y2%2BnjQ-66i9kPPCrPRQ%40mail.gmail.com%3E] > TLDR; HBASE-19400 messed up perms required for flushing a table. > > Looks like flush and snapshot procedures are already doing permissions check > as part of preTableFlush/preSnapshot hooks. However, > LogRollMasterProcedureManager is missing access checks ([~elserj], can > someone look at it?) > > With that, it makes no sense to put an ADMIN perm requirement which was added > by me in HBASE-19400. Removing it. > However, to make things better for future, i have made few design changes > which will ensure 1) perm checks don't slip by mistake, 2) a suitable > placeholder for checks for flush & snapshot when we remove AccessController > for good. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20185) Fix ACL check for MasterRpcServices#execProcedure
[ https://issues.apache.org/jira/browse/HBASE-20185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398583#comment-16398583 ] Appy commented on HBASE-20185: -- bq. Leave a TODO comment around the "pre" CP hooks to mirror Makes sense. done. bq. Some unrelated whitespace/formatting changes make this appear bigger than it is IDEs. :( I thought of removing them, but them thought that it wasn't not too much and isolated to just one file. bq. Is the object visibility change on AccessChecker in RSRpcServices actually necessary? accessChecker declared in RsRpcServices but now also used in MasterRpcServices#execProcedure. There is already a test for that in TestAccessController- https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java#L809, - however, those tests are hacked to not run the operations but the AC hooks directly. for eg. instead of triggering flush, it just runs ACCESS_CONTROLLER.preTableFlush(). It's not possible/good to change just a few tests there given that will break the overall design and make things harder to maintain. The new test should go in TestAdminOnlyOperations (and the test class name should probably be changed to TestRpcAccessChecks). Let me commit what we have for now. > Fix ACL check for MasterRpcServices#execProcedure > - > > Key: HBASE-20185 > URL: https://issues.apache.org/jira/browse/HBASE-20185 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20185.master.001.patch > > > Mailing thread ref: > [http://mail-archives.apache.org/mod_mbox/hbase-dev/201803.mbox/%3CCAAjhxrriGy_UXpC4iHCSyBB18iAbjU3Y2%2BnjQ-66i9kPPCrPRQ%40mail.gmail.com%3E] > TLDR; HBASE-19400 messed up perms required for flushing a table. > > Looks like flush and snapshot procedures are already doing permissions check > as part of preTableFlush/preSnapshot hooks. However, > LogRollMasterProcedureManager is missing access checks ([~elserj], can > someone look at it?) > > With that, it makes no sense to put an ADMIN perm requirement which was added > by me in HBASE-19400. Removing it. > However, to make things better for future, i have made few design changes > which will ensure 1) perm checks don't slip by mistake, 2) a suitable > placeholder for checks for flush & snapshot when we remove AccessController > for good. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398100#comment-16398100 ] Appy commented on HBASE-20188: -- I was planning to do the same - test with YCSB. How does following plan sounds? 1Master + 6 RS + 2 client. Begin with hbase1.4, load 1B rows (yscb load phase) , flush, compact, take snapshot. run workload : write only (custom), C, E, F ([list of workloads|https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads]) Update to 2.0, run same workloads and compare. Do one off 1B load on 2.0 to compare with initial load on 1.4. What remains: * perf for increments * these will be done with basic configs (maybe I'll just increase heap size), so it won't test fancy stuff - off heap, etc. * what else? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17919) HBase 2.x over hadoop 3.x umbrella
[ https://issues.apache.org/jira/browse/HBASE-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398096#comment-16398096 ] Appy commented on HBASE-17919: -- sounds right. Let's discuss further on HBASE-20188. > HBase 2.x over hadoop 3.x umbrella > --- > > Key: HBASE-17919 > URL: https://issues.apache.org/jira/browse/HBASE-17919 > Project: HBase > Issue Type: Umbrella > Components: hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Priority: Critical > Fix For: 2.0.0 > > > We should try to get hbase 2.x branch working against the recently release > hadoop 3.0.0 alphas. These days 3.0.0-alpha2 is the latest. > HBASE-16733 and HBASE-17593 got the compile level checks in but we should > progress to getting unit tests to pass and a build against hadoop3 up. > This umbrella issue will capture issues around this project. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20181) Logging and minor logic improvements in BackupLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20181: - Fix Version/s: 3.0.0 > Logging and minor logic improvements in BackupLogCleaner > > > Key: HBASE-20181 > URL: https://issues.apache.org/jira/browse/HBASE-20181 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HBASE-20181.1.patch > > > # Simplify code > # Use re-usable libraries > # Use SLF4J parameter logging format > # Clean up some log messages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20181) Logging and minor logic improvements in BackupLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy resolved HBASE-20181. -- Resolution: Fixed > Logging and minor logic improvements in BackupLogCleaner > > > Key: HBASE-20181 > URL: https://issues.apache.org/jira/browse/HBASE-20181 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HBASE-20181.1.patch > > > # Simplify code > # Use re-usable libraries > # Use SLF4J parameter logging format > # Clean up some log messages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20181) Logging and minor logic improvements in BackupLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398091#comment-16398091 ] Appy commented on HBASE-20181: -- [~yuzhih...@gmail.com] reason i committed it was, his style is doing one file at a time. [~belugabehr] Ted's feedback here, and i support it is, at some point you need to grow - in size, skill, etc. One simple way here is, do piece-wise work, saving it in small git commits (squash them as needed, ++ git-fu skills) and scale up until you are comfortable to modify whole modules at a time :) > Logging and minor logic improvements in BackupLogCleaner > > > Key: HBASE-20181 > URL: https://issues.apache.org/jira/browse/HBASE-20181 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-20181.1.patch > > > # Simplify code > # Use re-usable libraries > # Use SLF4J parameter logging format > # Clean up some log messages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20181) Logging and minor logic improvements in BackupLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-20181: - Summary: Logging and minor logic improvements in BackupLogCleaner (was: Code Review for BackupLogCleaner) > Logging and minor logic improvements in BackupLogCleaner > > > Key: HBASE-20181 > URL: https://issues.apache.org/jira/browse/HBASE-20181 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-20181.1.patch > > > # Simplify code > # Use re-usable libraries > # Use SLF4J parameter logging format > # Clean up some log messages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20181) Code Review for BackupLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398086#comment-16398086 ] Appy edited comment on HBASE-20181 at 3/14/18 4:27 AM: --- +1 Thanks [~belugabehr] for nice fixes...keep them coming! Let me commit what's there right now was (Author: appy): +1 Thanks [~belugabehr] for nice fixes...keep them coming! Let me commit what's there right now > Code Review for BackupLogCleaner > > > Key: HBASE-20181 > URL: https://issues.apache.org/jira/browse/HBASE-20181 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-20181.1.patch > > > # Simplify code > # Use re-usable libraries > # Use SLF4J parameter logging format > # Clean up some log messages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20181) Code Review for BackupLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-20181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398086#comment-16398086 ] Appy commented on HBASE-20181: -- +1 Thanks [~belugabehr] for nice fixes...keep them coming! Let me commit what's there right now > Code Review for BackupLogCleaner > > > Key: HBASE-20181 > URL: https://issues.apache.org/jira/browse/HBASE-20181 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-20181.1.patch > > > # Simplify code > # Use re-usable libraries > # Use SLF4J parameter logging format > # Clean up some log messages -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398078#comment-16398078
]
Appy commented on HBASE-20151:
--
Thanks sir!
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397079#comment-16397079
]
Appy commented on HBASE-20151:
--
Trying to understand the nature of bug morehere's more data.
Interestingly, it only happens when the last two args for SVCF are specified.
Default value for filterIfMissing (first arg) is false and for
latestVersionOnly (second arg) is true.
{noformat}
hbase(main):006:0> test.scan({ FILTER => "(
SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
FamilyFilter(=,'binary:b') )"})
ROW COLUMN+CELL
1 column=b:2, timestamp=1520951043631, value=
1 row(s)
Took 0.0508 seconds hbase(main):007:0> test.scan({ FILTER => "(
SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
FamilyFilter(=,'binary:b') )"})
ROW COLUMN+CELL
0 row(s)
Took 0.0047 seconds hbase(main):008:0> test.scan({ FILTER => "(
SingleColumnValueFilter('a','1',=,'binary:') AND FamilyFilter(=,'binary:b') )"})
ROW COLUMN+CELL
1 column=b:2, timestamp=1520951043631, value=
1 row(s)
Took 0.0073 seconds hbase(main):009:0> test.scan({ FILTER => "(
SingleColumnValueFilter('a','10',=,'binary:') AND FamilyFilter(=,'binary:b')
)"})
ROW COLUMN+CELL
1 column=b:2, timestamp=1520951043631, value=
1 row(s)
Took 0.0057 seconds hbase(main):010:0> test.scan({ FILTER => "(
SingleColumnValueFilter('a','10',=,'binary:', false, true) AND
FamilyFilter(=,'binary:b') )"})
ROW COLUMN+CELL
1 column=b:2, timestamp=1520951043631, value=
1 row(s)
Took 0.0098 seconds hbase(main):011:0> test.scan({ FILTER => "(
SingleColumnValueFilter('a','1',=,'binary:', false, true) AND
FamilyFilter(=,'binary:b') )"})
ROW COLUMN+CELL
1 column=b:2, timestamp=1520951043631, value=
1 row(s)
Took 0.0068 seconds{noformat}
I won't have time to debug this further, if anyone else want to continue here
(incrementally, or all the way through)...next steps would be debugging using
breakpoints (unless you have great familiarity with filter code).
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Priority: Major
> Fix For: 2.0.0
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
