[jira] [Commented] (HBASE-19331) Region start-key/end-key corruption in Hbase meta table
[ https://issues.apache.org/jira/browse/HBASE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265608#comment-16265608 ] Shamith kumar commented on HBASE-19331: --- I did not face any issued accessing table record from split regions. However i face issues while converting the start/end key bytes from HRegionInfo to long value. ie. Bytes.toLong() throws exception. > Region start-key/end-key corruption in Hbase meta table > --- > > Key: HBASE-19331 > URL: https://issues.apache.org/jira/browse/HBASE-19331 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 0.98.8 > Environment: Reproduced on HBase 0.98.8 on hadoop-2 >Reporter: Shamith kumar > Attachments: TestSplit.java > > > when a region split happens on a key with trailing byte equals zero, the end > key of the first resulting region and and start key of the second resulting > region in meta table gets corrupted. > Here is the link to code to reproduce this issue > https://bitbucket.org/flytxt/hbase-meta-corruption-test > > *+Test Result+* > [INFO] --- > [INFO] T E S T S > [INFO] --- > [INFO] Running com.flytxt.HbaseRegionMetaTest > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > 18:23:54.346 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table > SAMPLE_TBL_1 > 18:23:56.094 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table > SAMPLE_TBL_2 > 18:23:58.107 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table > SAMPLE_TBL_1 > 18:23:58.658 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table > SAMPLE_TBL_1 > 18:23:59.212 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to > table SAMPLE_TBL_1 > 18:24:00.046 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete .. > lets split SAMPLE_TBL_1 > 18:24:00.500 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to > table SAMPLE_TBL_2 > 18:24:02.073 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete .. > lets split SAMPLE_TBL_2 > 18:24:02.753 [main] INFO com.flytxt.HbaseRegionMetaTest - region split > complete .. Lets verify region infos for table SAMPLE_TBL_1 > 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : > SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f. > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id > :1511355240515 > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key > :[] , Key length :0 > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : > [0, 0, 0, 0, 0, 19, -76] , Key length :7 > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - > --- > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : > SAMPLE_TBL_1,\x00\x00\x00\x00\x00\x13\xB4,1511355240515.c06afed17b2a5c4fb54bacf704dd8a9e. > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id > :1511355240515 > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key > :[0, 0, 0, 0, 0, 19, -76] , Key length :7 > 18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : [] > , Key length :0 > 18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - > --- > 18:24:03.005 [main] INFO com.flytxt.HbaseRegionMetaTest - region split > complete .. Lets verify region infos for table SAMPLE_TBL_2 > 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : > SAMPLE_TBL_2,,1511355242363.0679851100e16aad005c743af618452e. > 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id > :1511355242363 > 18:24:03.007 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key > :[] , Key length :0 > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : > [0, 0, 0, 0, 0, 0, 19, -57] , Key length :8 > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - > --- > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - >
[jira] [Comment Edited] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265595#comment-16265595 ] stack edited comment on HBASE-19323 at 11/25/17 5:41 AM: - Let me run a quick perf compare. No harm. Good idea. Oh, just to say, I will lean on the perf-compare done so far by [~aoxiang]... was (Author: stack): Let me run a quick perf compare. No harm. Good idea. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265595#comment-16265595 ] stack commented on HBASE-19323: --- Let me run a quick perf compare. No harm. Good idea. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19331) Region start-key/end-key corruption in Hbase meta table
[
https://issues.apache.org/jira/browse/HBASE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265591#comment-16265591
]
Ted Yu edited comment on HBASE-19331 at 11/25/17 5:28 AM:
--
I spent a bit time reviewing how mid key is calculated using branch-1.1
The code should be very similar between 0.98.x and 1.1
During splitting sample table, the following code in
StoreFile#getFileSplitPoint() is executed:
{code}
KeyValue mk = KeyValue.createKeyValueFromKey(midkey, 0, midkey.length);
{code}
When mk is not the same as first key and not the same as last key of the store
file, mk.getRow() is returned as split point.
The trailing zero is gone (as from e.g. \x00\x00\x00\x00\x00\x00\x07\x00). That
is why the assertion fails.
Did you observe problem accessing the split regions ?
I ask since \x00\x00\x00\x00\x00\x00\x07 can correctly separate the rows in the
parent region.
Did you observe imbalanced number of rows in the daughter regions or other
problems ?
was (Author: yuzhih...@gmail.com):
I spent a bit time reviewing how mid key is calculated using branch-1.1
The code should be very similar between 0.98.x and 1.1
During splitting sample table, the following code in
StoreFile#getFileSplitPoint() is executed:
{code}
KeyValue mk = KeyValue.createKeyValueFromKey(midkey, 0, midkey.length);
{code}
When mk is not the same as first key and not the same as last key of the store
file, mk.getRow() is returned as split point.
The trailing zero is gone (as in e.g. \x00\x00\x00\x00\x00\x00\x01\x00). That
is why the assertion fails.
Did you observe problem accessing the split regions ?
I ask since \x00\x00\x00\x00\x00\x00\x01 can correctly separate the rows in the
parent region.
Did you observe imbalanced number of rows in the daughter regions or other
problems ?
> Region start-key/end-key corruption in Hbase meta table
> ---
>
> Key: HBASE-19331
> URL: https://issues.apache.org/jira/browse/HBASE-19331
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 0.98.8
> Environment: Reproduced on HBase 0.98.8 on hadoop-2
>Reporter: Shamith kumar
> Attachments: TestSplit.java
>
>
> when a region split happens on a key with trailing byte equals zero, the end
> key of the first resulting region and and start key of the second resulting
> region in meta table gets corrupted.
> Here is the link to code to reproduce this issue
> https://bitbucket.org/flytxt/hbase-meta-corruption-test
>
> *+Test Result+*
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running com.flytxt.HbaseRegionMetaTest
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> 18:23:54.346 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table
> SAMPLE_TBL_1
> 18:23:56.094 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table
> SAMPLE_TBL_2
> 18:23:58.107 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table
> SAMPLE_TBL_1
> 18:23:58.658 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table
> SAMPLE_TBL_1
> 18:23:59.212 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to
> table SAMPLE_TBL_1
> 18:24:00.046 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete ..
> lets split SAMPLE_TBL_1
> 18:24:00.500 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to
> table SAMPLE_TBL_2
> 18:24:02.073 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete ..
> lets split SAMPLE_TBL_2
> 18:24:02.753 [main] INFO com.flytxt.HbaseRegionMetaTest - region split
> complete .. Lets verify region infos for table SAMPLE_TBL_1
> 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ===
> 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name :
> SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f.
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id
> :1511355240515
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key
> :[] , Key length :0
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key :
> [0, 0, 0, 0, 0, 19, -76] , Key length :7
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ---
> 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ===
> 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name :
>
[jira] [Commented] (HBASE-19331) Region start-key/end-key corruption in Hbase meta table
[
https://issues.apache.org/jira/browse/HBASE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265591#comment-16265591
]
Ted Yu commented on HBASE-19331:
I spent a bit time reviewing how mid key is calculated using branch-1.1
The code should be very similar between 0.98.x and 1.1
During splitting sample table, the following code in
StoreFile#getFileSplitPoint() is executed:
{code}
KeyValue mk = KeyValue.createKeyValueFromKey(midkey, 0, midkey.length);
{code}
When mk is not the same as first key and not the same as last key of the store
file, mk.getRow() is returned as split point.
The trailing zero is gone (as in e.g. \x00\x00\x00\x00\x00\x00\x01\x00). That
is why the assertion fails.
Did you observe problem accessing the split regions ?
I ask since \x00\x00\x00\x00\x00\x00\x01 can correctly separate the rows in the
parent region.
Did you observe imbalanced number of rows in the daughter regions or other
problems ?
> Region start-key/end-key corruption in Hbase meta table
> ---
>
> Key: HBASE-19331
> URL: https://issues.apache.org/jira/browse/HBASE-19331
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 0.98.8
> Environment: Reproduced on HBase 0.98.8 on hadoop-2
>Reporter: Shamith kumar
> Attachments: TestSplit.java
>
>
> when a region split happens on a key with trailing byte equals zero, the end
> key of the first resulting region and and start key of the second resulting
> region in meta table gets corrupted.
> Here is the link to code to reproduce this issue
> https://bitbucket.org/flytxt/hbase-meta-corruption-test
>
> *+Test Result+*
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running com.flytxt.HbaseRegionMetaTest
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> 18:23:54.346 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table
> SAMPLE_TBL_1
> 18:23:56.094 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table
> SAMPLE_TBL_2
> 18:23:58.107 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table
> SAMPLE_TBL_1
> 18:23:58.658 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table
> SAMPLE_TBL_1
> 18:23:59.212 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to
> table SAMPLE_TBL_1
> 18:24:00.046 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete ..
> lets split SAMPLE_TBL_1
> 18:24:00.500 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to
> table SAMPLE_TBL_2
> 18:24:02.073 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete ..
> lets split SAMPLE_TBL_2
> 18:24:02.753 [main] INFO com.flytxt.HbaseRegionMetaTest - region split
> complete .. Lets verify region infos for table SAMPLE_TBL_1
> 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ===
> 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name :
> SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f.
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id
> :1511355240515
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key
> :[] , Key length :0
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key :
> [0, 0, 0, 0, 0, 19, -76] , Key length :7
> 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ---
> 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ===
> 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name :
> SAMPLE_TBL_1,\x00\x00\x00\x00\x00\x13\xB4,1511355240515.c06afed17b2a5c4fb54bacf704dd8a9e.
> 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id
> :1511355240515
> 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key
> :[0, 0, 0, 0, 0, 19, -76] , Key length :7
> 18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : []
> , Key length :0
> 18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ---
> 18:24:03.005 [main] INFO com.flytxt.HbaseRegionMetaTest - region split
> complete .. Lets verify region infos for table SAMPLE_TBL_2
> 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest -
> ===
> 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name :
>
[jira] [Comment Edited] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265195#comment-16265195 ] Ted Yu edited comment on HBASE-19340 at 11/25/17 4:06 AM: -- There are other missing constants which can be addressed thru backport. was (Author: yuzhih...@gmail.com): See if this solves the problem for you. > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan > Fix For: 1.2.8 > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19340: --- Attachment: (was: 19340.branch-1.2.txt) > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan > Fix For: 1.2.8 > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265168#comment-16265168
]
Ted Yu edited comment on HBASE-19338 at 11/25/17 2:07 AM:
--
https://builds.apache.org/job/PreCommit-HBASE-Build/9997/console :
{code}
08:50:46 Modes: MultiJDK Jenkins Robot Docker ResetRepo UnitTests
08:50:46 Processing: HBASE-19338
08:50:47 ERROR: Unsure how to process HBASE-19338.
{code}
Looks like we may not get QA back today.
Lijin:
Next week people in US are returning to work.
You can wait till QA bot becomes functional again.
Thanks
was (Author: yuzhih...@gmail.com):
https://builds.apache.org/job/PreCommit-HBASE-Build/9997/console :
{code}
08:50:46 Modes: MultiJDK Jenkins Robot Docker ResetRepo UnitTests
08:50:46 Processing: HBASE-19338
08:50:47 ERROR: Unsure how to process HBASE-19338.
{code}
Looks like we may not get QA back today.
Lijin:
After verifying a few Visibility and AccessController related tests, you can go
with the commit.
Thanks
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19337) AsyncMetaTableAccessor may hang when call ScanController.terminate many times
[
https://issues.apache.org/jira/browse/HBASE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265558#comment-16265558
]
Ted Yu commented on HBASE-19337:
bq. It is not easy to add a unit test
I feel writing non-flaky test which exhibits the problem sometimes is harder
than figuring out the fix itself.
Okay to go in without a fix.
lgtm
> AsyncMetaTableAccessor may hang when call ScanController.terminate many times
> -
>
> Key: HBASE-19337
> URL: https://issues.apache.org/jira/browse/HBASE-19337
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19337.master.001.patch
>
>
> Code in ScanControllerImpl.
> {code}
> private void preCheck() {
> Preconditions.checkState(Thread.currentThread() == callerThread,
> "The current thread is %s, expected thread is %s, " +
> "you should not call this method outside onNext or onHeartbeat",
> Thread.currentThread(), callerThread);
> Preconditions.checkState(state.equals(ScanControllerState.INITIALIZED),
> "Invalid Stopper state %s", state);
> }
> @Override
> public void terminate() {
> preCheck();
> state = ScanControllerState.TERMINATED;
> }
> {code}
> So if call terminate on a already terminated scan, it will throw
> IllegalStateException.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19242) Add MOB compact support for AsyncAdmin
[
https://issues.apache.org/jira/browse/HBASE-19242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265557#comment-16265557
]
Guanghao Zhang commented on HBASE-19242:
+1. Let's wait the Hadoop QA result.
> Add MOB compact support for AsyncAdmin
> --
>
> Key: HBASE-19242
> URL: https://issues.apache.org/jira/browse/HBASE-19242
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, mob
>Reporter: Duo Zhang
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19242.master.001.patch,
> HBASE-19242.master.002.patch
>
>
> {code}
> private CompletableFuture compact(TableName tableName, byte[]
> columnFamily, boolean major,
> CompactType compactType) {
> if (CompactType.MOB.equals(compactType)) {
> // TODO support MOB compact.
> return failedFuture(new UnsupportedOperationException("MOB compact does
> not support"));
> }
> {code}
> We need to support it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19337) AsyncMetaTableAccessor may hang when call ScanController.terminate many times
[
https://issues.apache.org/jira/browse/HBASE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265556#comment-16265556
]
Guanghao Zhang commented on HBASE-19337:
bq. Is the following change intentional
The log change is not rleated. But it is useful when debug.
bq. Is it possible to add a test ?
It is not easy to add a unit test... Because it misused ScanController. The
ScanController was designed to only terminate once. If terminate again, then it
will throw a exception. And in AsyncMetaTableAccessor, the old impl don't catch
it, so the future can't complete...
> AsyncMetaTableAccessor may hang when call ScanController.terminate many times
> -
>
> Key: HBASE-19337
> URL: https://issues.apache.org/jira/browse/HBASE-19337
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19337.master.001.patch
>
>
> Code in ScanControllerImpl.
> {code}
> private void preCheck() {
> Preconditions.checkState(Thread.currentThread() == callerThread,
> "The current thread is %s, expected thread is %s, " +
> "you should not call this method outside onNext or onHeartbeat",
> Thread.currentThread(), callerThread);
> Preconditions.checkState(state.equals(ScanControllerState.INITIALIZED),
> "Invalid Stopper state %s", state);
> }
> @Override
> public void terminate() {
> preCheck();
> state = ScanControllerState.TERMINATED;
> }
> {code}
> So if call terminate on a already terminated scan, it will throw
> IllegalStateException.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19311) Promote TestAcidGuarantees to LargeTests and start mini cluster once to make it faster
[ https://issues.apache.org/jira/browse/HBASE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265538#comment-16265538 ] Appy commented on HBASE-19311: -- Oh, looks like it was committed (had stale comments history in my browser tab). Good with me. :) > Promote TestAcidGuarantees to LargeTests and start mini cluster once to make > it faster > -- > > Key: HBASE-19311 > URL: https://issues.apache.org/jira/browse/HBASE-19311 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19311-v1.patch, HBASE-19311.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19311) Promote TestAcidGuarantees to LargeTests and start mini cluster once to make it faster
[ https://issues.apache.org/jira/browse/HBASE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265536#comment-16265536 ] Appy commented on HBASE-19311: -- Sorry for taking time on this one. It's thanksgiving holiday season here, so not able to get 1-2 hours time slot to get this reviewed. > Promote TestAcidGuarantees to LargeTests and start mini cluster once to make > it faster > -- > > Key: HBASE-19311 > URL: https://issues.apache.org/jira/browse/HBASE-19311 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19311-v1.patch, HBASE-19311.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19207) Create Minimal HBase REST Client
[ https://issues.apache.org/jira/browse/HBASE-19207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265495#comment-16265495 ] Rick Kellogg commented on HBASE-19207: -- After considerable efforts, I have created a very minimal REST Client. This version only requires a total of 6 external JAR files. Not certain if this can be pulled in as there is considerable code duplication and modifications to the source included from Apache HBase 2.0-alpha4. External Github Project: https://github.com/rmkellogg/hbase-lite-rest-client I attempted to allow access to the create table functionality but ultimately decided against it. There are just too many classes and loss of functionality via the REST Server. As a consequence the RemoteAdmin interface is quite minimal. Comments welcome. > Create Minimal HBase REST Client > > > Key: HBASE-19207 > URL: https://issues.apache.org/jira/browse/HBASE-19207 > Project: HBase > Issue Type: New Feature > Components: Client, REST >Reporter: Rick Kellogg > > Create a minimal REST client with only contents of > org.apache.hadoop.hbase.rest.client and > org.apache.hadoop.hbase.rest.client.models packages in the hbase-rest > project. > Attempt to reduce the number of third party dependencies and allow user to > bring their own Apache HttpClient/Core. The HttpClient is frequently updated > and therefore should not be shaded to allow for upgrades. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17852) Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup)
[
https://issues.apache.org/jira/browse/HBASE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265492#comment-16265492
]
Vladimir Rodionov commented on HBASE-17852:
---
{quote}
Let me just assume this stuff is handled, but a walk through of what happens
when the backup table goes away in different scenarios would be good.
Is the above answered? (Copied from earlier in this dialog).
{quote}
When backup meta table goes away, bulk load will continue because bul load
observers do not write to main meta table. When second table (for bulk loaded
files) gets offlined - bulk loading fails.
> Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental
> backup)
>
>
> Key: HBASE-17852
> URL: https://issues.apache.org/jira/browse/HBASE-17852
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-17852-v1.patch, HBASE-17852-v2.patch,
> HBASE-17852-v3.patch, HBASE-17852-v4.patch, HBASE-17852-v5.patch,
> HBASE-17852-v6.patch, HBASE-17852-v7.patch, HBASE-17852-v8.patch
>
>
> Design approach rollback-via-snapshot implemented in this ticket:
> # Before backup create/delete/merge starts we take a snapshot of the backup
> meta-table (backup system table). This procedure is lightweight because meta
> table is small, usually should fit a single region.
> # When operation fails on a server side, we handle this failure by cleaning
> up partial data in backup destination, followed by restoring backup
> meta-table from a snapshot.
> # When operation fails on a client side (abnormal termination, for example),
> next time user will try create/merge/delete he(she) will see error message,
> that system is in inconsistent state and repair is required, he(she) will
> need to run backup repair tool.
> # To avoid multiple writers to the backup system table (backup client and
> BackupObserver's) we introduce small table ONLY to keep listing of bulk
> loaded files. All backup observers will work only with this new tables. The
> reason: in case of a failure during backup create/delete/merge/restore, when
> system performs automatic rollback, some data written by backup observers
> during failed operation may be lost. This is what we try to avoid.
> # Second table keeps only bulk load related references. We do not care about
> consistency of this table, because bulk load is idempotent operation and can
> be repeated after failure. Partially written data in second table does not
> affect on BackupHFileCleaner plugin, because this data (list of bulk loaded
> files) correspond to a files which have not been loaded yet successfully and,
> hence - are not visible to the system
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19328) Remove asked if splittable log messages
[
https://issues.apache.org/jira/browse/HBASE-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265475#comment-16265475
]
Chia-Ping Tsai commented on HBASE-19328:
{code}
01:46:11 -1 overall
01:46:11
01:46:11 _ _ __
01:46:11 | ___|_ _(_) |_ _ _ __ ___| |
01:46:11 | |_ / _` | | | | | | '__/ _ \ |
01:46:11 | _| (_| | | | |_| | | | __/_|
01:46:11 |_| \__,_|_|_|\__,_|_| \___(_)
01:46:11
01:46:11
01:46:11
01:46:11 | Vote | Subsystem | Runtime | Comment
01:46:11
01:46:11 | | || Prechecks
01:46:11 | +1 | hbaseanti | 0m 0s | Patch does not have any
anti-patterns.
01:46:11 | +1 |@author | 0m 0s | The patch does not contain any
@author
01:46:11 | | || tags.
01:46:11 | -1 | test4tests | 0m 0s | The patch doesn't appear to
include any
01:46:11 | | || new or modified tests. Please
justify
01:46:11 | | || why no new tests are needed
for this
01:46:11 | | || patch. Also please list what
manual
01:46:11 | | || steps were performed to verify
this
01:46:11 | | || patch.
01:46:11 | | || master Compile Tests
01:46:11 | +1 | mvninstall | 5m 8s | master passed
01:46:11 | +1 |compile | 0m 41s | master passed
01:46:11 | +1 | checkstyle | 1m 13s | master passed
01:46:11 | 0 | findbugs | 2m 9s | hbase-server in master has 11
extant
01:46:11 | | || Findbugs warnings.
01:46:11 | +1 |javadoc | 0m 56s | master passed
01:46:11 | | || Patch Compile Tests
01:46:11 | +1 | mvninstall | 0m 47s | the patch passed
01:46:11 | +1 |compile | 0m 42s | the patch passed
01:46:11 | +1 | javac | 0m 42s | the patch passed
01:46:11 | +1 | checkstyle | 0m 57s | the patch passed
01:46:11 | +1 | whitespace | 0m 0s | The patch has no whitespace
issues.
01:46:12 | +1 |hadoopcheck | 37m 15s | The patch does not cause any
errors
01:46:12 | | || with Hadoop 2.6.1 2.6.2 2.6.3
2.6.4
01:46:12 | | || 2.6.5 2.7.1 2.7.2 2.7.3.
01:46:12 | +1 | findbugs | 2m 14s | the patch passed
01:46:12 | +1 |javadoc | 0m 30s | the patch passed
01:46:12 | | || Other Tests
01:46:12 | -1 | unit | 140m 9s | hbase-server in the patch
failed.
01:46:12 | +1 | asflicense | 0m 47s | The patch does not generate
ASF License
01:46:12 | | || warnings.
01:46:12 | | | 193m 58s |
01:46:12
01:46:12
01:46:12 Reason | Tests
01:46:12 Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
01:46:12 |
hadoop.hbase.regionserver.TestCompactionInDeadRegionServer
01:46:12
01:46:12
01:46:12 || Subsystem || Report/Notes ||
01:46:12
01:46:12 | Optional Tests | asflicense javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
01:46:12 | uname | Linux 86e61538a674 4.10.0-35-generic #39-Ubuntu SMP Wed Sep
13 07:46:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
01:46:12 | Build tool | maven |
01:46:12 | Personality | /script/yetus/precommit/personality/hbase.sh |
01:46:12 | git revision | master / 73e3af00e9 |
01:46:12 | maven | version: Apache Maven 3.3.9 |
01:46:12 | Default Java | 1.8.0_131 |
01:46:12 | findbugs | v3.1.0-RC3 |
01:46:12 | unit | /patchprocess/patch-unit-hbase-server.txt |
01:46:12 | modules | C: hbase-server U: hbase-server |
01:46:12 | Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
{code}
The fix is not complicated, so I verify the patch locally. There are two failed
tests. Retrying the TestMasterFailoverWithProcedur is succeed.
{{TestCompactionInDeadRegionServer}} is flaky test. The patch is ready to be
committed I’d like to say. [~stack] any suggestion?
> Remove asked if splittable log messages
> ---
>
> Key: HBASE-19328
> URL: https://issues.apache.org/jira/browse/HBASE-19328
> Project: HBase
> Issue Type: Task
> Components: proc-v2
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 2.0.0-beta-1
>
>
[jira] [Updated] (HBASE-18112) Write RequestTooBigException back to client for NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-18112: - Attachment: HBASE-18112-v4.patch > Write RequestTooBigException back to client for NettyRpcServer > -- > > Key: HBASE-18112 > URL: https://issues.apache.org/jira/browse/HBASE-18112 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: Duo Zhang >Assignee: Toshihiro Suzuki > Attachments: HBASE-18112-v2.patch, HBASE-18112-v2.patch, > HBASE-18112-v2.patch, HBASE-18112-v3.patch, HBASE-18112-v3.patch, > HBASE-18112-v4.patch, HBASE-18112.patch > > > For now we just close the connection so NettyRpcServer can not pass TestIPC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18112) Write RequestTooBigException back to client for NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-18112: - Attachment: (was: HBASE-18112-v4.patch) > Write RequestTooBigException back to client for NettyRpcServer > -- > > Key: HBASE-18112 > URL: https://issues.apache.org/jira/browse/HBASE-18112 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: Duo Zhang >Assignee: Toshihiro Suzuki > Attachments: HBASE-18112-v2.patch, HBASE-18112-v2.patch, > HBASE-18112-v2.patch, HBASE-18112-v3.patch, HBASE-18112-v3.patch, > HBASE-18112.patch > > > For now we just close the connection so NettyRpcServer can not pass TestIPC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265448#comment-16265448
]
Ted Yu edited comment on HBASE-19159 at 11/24/17 5:22 PM:
--
bq. maybe it should check WRITE_EXECUTE
I searched thru hadoop tests - there is no reference to WRITE_EXECUTE.
I think WRITE should be used since no 'x' is not involved in backup operations.
was (Author: yuzhih...@gmail.com):
bq. maybe it should check WRITE_EXECUTE
I searched thru hadoop tests - there is no reference to WRITE_EXECUTE.
I think WRITE should be used since no 'x' is involved.
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159-v2.patch, HBASE-19159.patch,
> initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
> to=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/user/root/test-data/fb919a6f-3cb4-4d57-bbcf-561d6e5b3ae8/backupIT/backup_1509646884252/default/IntegrationTestBackupRestore.table2/.hbase-snapshot/.tmp/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.doWork(ExportSnapshot.java:1009)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:386)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.snapshotCopy(FullTableBackupClient.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:175)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTest(IntegrationTestBackupRestore.java:180)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:134)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestFromCommandLine(IntegrationTestBackupRestore.java:263)
> {code}
> It would be more user friendly if the permission is checked before taking the
> snapshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265448#comment-16265448
]
Ted Yu commented on HBASE-19159:
bq. maybe it should check WRITE_EXECUTE
I searched thru hadoop tests - there is no reference to WRITE_EXECUTE.
I think WRITE should be used since no 'x' is involved.
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159-v2.patch, HBASE-19159.patch,
> initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
> to=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/user/root/test-data/fb919a6f-3cb4-4d57-bbcf-561d6e5b3ae8/backupIT/backup_1509646884252/default/IntegrationTestBackupRestore.table2/.hbase-snapshot/.tmp/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.doWork(ExportSnapshot.java:1009)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:386)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.snapshotCopy(FullTableBackupClient.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:175)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTest(IntegrationTestBackupRestore.java:180)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:134)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestFromCommandLine(IntegrationTestBackupRestore.java:263)
> {code}
> It would be more user friendly if the permission is checked before taking the
> snapshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265442#comment-16265442
]
Ted Yu commented on HBASE-19343:
QA bot is currently out of order.
Expect some delay before the test suite comes back.
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
>
[jira] [Commented] (HBASE-17049) Do not issue sync request when there are still entries in ringbuffer
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265439#comment-16265439 ] ramkrishna.s.vasudevan commented on HBASE-17049: Good one. Will test this patch once next week. I think this could be the reason why when we have PE tool with autoflush we may do more syncs than FSHLog. (rather than autoflush = false). > Do not issue sync request when there are still entries in ringbuffer > > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-17049.patch, delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19318:
---
Attachment: HBASE-19318.002.branch-2.patch
.002 Assorted minor improvements:
* Test classification
* Javadoc on test class
* Test renamed
* Checkstyle import nits
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19318.001.branch-2.patch,
> HBASE-19318.002.branch-2.patch
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19092: --- Fix Version/s: 3.0.0 > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch, > HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch, > HBASE-19092_3.patch, HBASE-19092_4.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19112) Suspect methods on Cell to be deprecated
[
https://issues.apache.org/jira/browse/HBASE-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265432#comment-16265432
]
ramkrishna.s.vasudevan commented on HBASE-19112:
Before I prepare a patch few doubts,
Are we first going to add the necessary new APIs to RawCell/ExtendedCell and
deprecate the other APIs in Cell ?
Say for eg : Mark @deprecated for getTagsXXX and getSeqId in Cell and add
corresponding APIs in ExtendedCell/RawCell.
Let all the existing core code to still point to the
cell.getTagXXX/cell.getSeqId only. Later in master lets clean all when we
actually allow core to have RawCell/ExtendedCell flowing through?
Some reasons are like say if RawCell currently uses the PrivateCellUtil.getTags
to retrieve the tags rather than each cell having an impl. Now if I need to
really add the impl I need to make sure that the incoming cell is RawCell so
that what ever impl is added we can make use of it directly in the code instead
of typecasting. Currently we need to do ((RawCell)cell) every where.
I found that getTagsLength is getting used through out the code than
getTagsArray and getTagsOffset. So if getTagsLength moves to RawCell i need the
typecast.
Similary getSequnceId has got lot of references too. So what can be the
strategy for branch-2 and master now? Thoughts!!!
> Suspect methods on Cell to be deprecated
>
>
> Key: HBASE-19112
> URL: https://issues.apache.org/jira/browse/HBASE-19112
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> [~chia7712] suggested on the [mailing
> list|https://lists.apache.org/thread.html/e6de9af26d9b888a358ba48bf74655ccd893573087c032c0fcf01585@%3Cdev.hbase.apache.org%3E]
> that we have some methods on Cell which should be deprecated for removal:
> * {{#getType()}}
> * {{#getTimestamp()}}
> * {{#getTag()}}
> * {{#getSequenceId()}}
> Let's make a pass over these (and maybe the rest) to make sure that there
> aren't others which are either implementation details or methods returning
> now-private-marked classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265429#comment-16265429
]
Pankaj Kumar commented on HBASE-19343:
--
Attached patch for branch-1, please review.
Have to test this in master and other branches as well.
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
[jira] [Updated] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19343:
-
Fix Version/s: 1.5.0
Status: Patch Available (was: Open)
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537667635,
[jira] [Updated] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19343:
-
Attachment: HBASE-19343-branch-1.patch
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537667635, value=1511537511523
>
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265426#comment-16265426
]
Pankaj Kumar commented on HBASE-19343:
--
Thanks [~tedyu] for looking into this issue.
bq. Did you observe such problem in other scenario(s) ?
Yeah I checked this, found it in restore snapshot only.
Will attach patch soon.
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Attachments: Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
>
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265422#comment-16265422
]
Josh Elser commented on HBASE-19318:
bq. Ya that would be ideal path Josh. Hope u will start that work after 2.0
Will help with that when it comes.
Thanks, Anoop. I appreciate you working with me here! I'd certainly like to
help make this better. I think it will be pretty straightforward, but we'll see
:P
Let me fix the checkstyle issues, and, if I don't hear otherwise, I'll push
this come Monday. Thanks for the reviews.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19318.001.branch-2.patch
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265412#comment-16265412
]
Anoop Sam John commented on HBASE-19318:
Ya that would be ideal path Josh. Hope u will start that work after 2.0 :-)
Will help with that when it comes.
On the patch itself, as I said above, the change look very much ok. Only
questions/concerns were on the general approach by Ranger abt this.. Ya u got
the points already. Tks for the explain BTW.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19318.001.branch-2.patch
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18112) Write RequestTooBigException back to client for NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-18112: - Attachment: HBASE-18112-v4.patch > Write RequestTooBigException back to client for NettyRpcServer > -- > > Key: HBASE-18112 > URL: https://issues.apache.org/jira/browse/HBASE-18112 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: Duo Zhang >Assignee: Toshihiro Suzuki > Attachments: HBASE-18112-v2.patch, HBASE-18112-v2.patch, > HBASE-18112-v2.patch, HBASE-18112-v3.patch, HBASE-18112-v3.patch, > HBASE-18112-v4.patch, HBASE-18112.patch > > > For now we just close the connection so NettyRpcServer can not pass TestIPC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265409#comment-16265409
]
Josh Elser commented on HBASE-19318:
bq. So the intention is to see if the cluster has Accesscontrol services
installed and if so Ranger CP will do necessary actions before the ACL CP is
invoked, right?
Right. This is a smell, but something that Ranger has been doing already (I've
since learned).
bq. Is it right to do this on Ranger side?
No, but I think this is necessary-evil presently.
To Anoop's point, I think it would be better to separate AccessController from
being the coprocessor and the hbase:acl-based authz implementation. With this,
we can:
# Provide a proper interface for folks to implement custom authz logic
# Avoiding unnecessary "system internals" leaking to these impls
I would guess that we could do this in a backwards compat way (new AC
implementation instead of removing the current implementation) and handle it in
2.1.0 rather than waiting for 3.0.
We should involve Ranger folks though ([~vperiasamy]) to make sure that there
isn't more that we're missing which they could benefit from. I would guess that
they had no idea folks in HBase considered authz to be internal-only :)
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19318.001.branch-2.patch
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265400#comment-16265400
]
Ted Yu commented on HBASE-19343:
I looked at the code in RestoreSnapshotHelper of master branch where
getTableRegions() is used.
Did you observe such problem in other scenario(s) ?
If not, it seems option #1 is better since this is only triggered by snapshot
restore.
If you have bandwidth working on this issue, first step would be coming up with
unit test which shows the behavior.
Thanks
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Attachments: Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
[jira] [Commented] (HBASE-19112) Suspect methods on Cell to be deprecated
[
https://issues.apache.org/jira/browse/HBASE-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265391#comment-16265391
]
Josh Elser commented on HBASE-19112:
+1 [~ram_krish]. Didn't want to pursue when 19092 was still underway.
Admittedly, I haven't kept up with it. Will try to keep an eye on this one and
at least help with reviews :)
> Suspect methods on Cell to be deprecated
>
>
> Key: HBASE-19112
> URL: https://issues.apache.org/jira/browse/HBASE-19112
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> [~chia7712] suggested on the [mailing
> list|https://lists.apache.org/thread.html/e6de9af26d9b888a358ba48bf74655ccd893573087c032c0fcf01585@%3Cdev.hbase.apache.org%3E]
> that we have some methods on Cell which should be deprecated for removal:
> * {{#getType()}}
> * {{#getTimestamp()}}
> * {{#getTag()}}
> * {{#getSequenceId()}}
> Let's make a pass over these (and maybe the rest) to make sure that there
> aren't others which are either implementation details or methods returning
> now-private-marked classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19331) Region start-key/end-key corruption in Hbase meta table
[ https://issues.apache.org/jira/browse/HBASE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265387#comment-16265387 ] Josh Elser commented on HBASE-19331: Also, 0.98.8 is pretty old :). Have you checked to see that this problem also exists on the last release of 0.98? (0.98.24) > Region start-key/end-key corruption in Hbase meta table > --- > > Key: HBASE-19331 > URL: https://issues.apache.org/jira/browse/HBASE-19331 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 0.98.8 > Environment: Reproduced on HBase 0.98.8 on hadoop-2 >Reporter: Shamith kumar > Attachments: TestSplit.java > > > when a region split happens on a key with trailing byte equals zero, the end > key of the first resulting region and and start key of the second resulting > region in meta table gets corrupted. > Here is the link to code to reproduce this issue > https://bitbucket.org/flytxt/hbase-meta-corruption-test > > *+Test Result+* > [INFO] --- > [INFO] T E S T S > [INFO] --- > [INFO] Running com.flytxt.HbaseRegionMetaTest > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > 18:23:54.346 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table > SAMPLE_TBL_1 > 18:23:56.094 [main] INFO com.flytxt.HbaseRegionMetaTest - Dropping table > SAMPLE_TBL_2 > 18:23:58.107 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table > SAMPLE_TBL_1 > 18:23:58.658 [main] INFO com.flytxt.HbaseRegionMetaTest - Creating new table > SAMPLE_TBL_1 > 18:23:59.212 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to > table SAMPLE_TBL_1 > 18:24:00.046 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete .. > lets split SAMPLE_TBL_1 > 18:24:00.500 [main] INFO com.flytxt.HbaseRegionMetaTest - Starting puts to > table SAMPLE_TBL_2 > 18:24:02.073 [main] INFO com.flytxt.HbaseRegionMetaTest - Puts complete .. > lets split SAMPLE_TBL_2 > 18:24:02.753 [main] INFO com.flytxt.HbaseRegionMetaTest - region split > complete .. Lets verify region infos for table SAMPLE_TBL_1 > 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:02.754 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : > SAMPLE_TBL_1,,1511355240515.56c8fd8e42228c3c1ec71f9a4da65f5f. > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id > :1511355240515 > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key > :[] , Key length :0 > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : > [0, 0, 0, 0, 0, 19, -76] , Key length :7 > 18:24:02.755 [main] INFO com.flytxt.HbaseRegionMetaTest - > --- > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : > SAMPLE_TBL_1,\x00\x00\x00\x00\x00\x13\xB4,1511355240515.c06afed17b2a5c4fb54bacf704dd8a9e. > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id > :1511355240515 > 18:24:02.762 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key > :[0, 0, 0, 0, 0, 19, -76] , Key length :7 > 18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : [] > , Key length :0 > 18:24:02.763 [main] INFO com.flytxt.HbaseRegionMetaTest - > --- > 18:24:03.005 [main] INFO com.flytxt.HbaseRegionMetaTest - region split > complete .. Lets verify region infos for table SAMPLE_TBL_2 > 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Name : > SAMPLE_TBL_2,,1511355242363.0679851100e16aad005c743af618452e. > 18:24:03.006 [main] INFO com.flytxt.HbaseRegionMetaTest - Region Id > :1511355242363 > 18:24:03.007 [main] INFO com.flytxt.HbaseRegionMetaTest - Region start key > :[] , Key length :0 > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - Region end key : > [0, 0, 0, 0, 0, 0, 19, -57] , Key length :8 > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - > --- > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest - > === > 18:24:03.008 [main] INFO com.flytxt.HbaseRegionMetaTest -
[jira] [Commented] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265381#comment-16265381
]
Ted Yu commented on HBASE-19159:
Trying to see whether access() is documented in:
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
But I didn't find much detail.
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159-v2.patch, HBASE-19159.patch,
> initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
> to=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/user/root/test-data/fb919a6f-3cb4-4d57-bbcf-561d6e5b3ae8/backupIT/backup_1509646884252/default/IntegrationTestBackupRestore.table2/.hbase-snapshot/.tmp/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.doWork(ExportSnapshot.java:1009)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:386)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.snapshotCopy(FullTableBackupClient.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:175)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTest(IntegrationTestBackupRestore.java:180)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:134)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestFromCommandLine(IntegrationTestBackupRestore.java:263)
> {code}
> It would be more user friendly if the permission is checked before taking the
> snapshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19343:
-
Attachment: Snapshot.jpg
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Attachments: Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537667635, value=1511537511523
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo,
[jira] [Updated] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19343:
-
Description:
Restore snapshot makes parent split region online as shown in the attached
snapshot.
Steps to reproduce
=
1. Create table
2. Insert few records into the table
3. flush the table
4. Split the table
5. Create snapshot before catalog janitor clears the parent region entry from
meta.
6. Restore snapshot
We can see the problem in meta entries,
Meta content before restore snapshot:
{noformat}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
077a12b0b3c91b053fa95223635f9543, NAME =>
't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
'', ENDKEY => '',
OFFLINE => true, SPLIT => true}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:seqnumDuringOpen, timestamp=1511537530107,
value=\x00\x00\x00\x00\x00\x00\x00\x02
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:server, timestamp=1511537530107, value=host-xx:16020
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:splitA, timestamp=1511537565964, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
ENDKEY => 'm'}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:splitB, timestamp=1511537565964, value={ENCODED =>
dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
', ENDKEY => ''}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
'', ENDKEY => 'm'}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:seqnumDuringOpen, timestamp=1511537566075,
value=\x00\x00\x00\x00\x00\x00\x00\x02
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:server, timestamp=1511537566075, value=host-xx:16020
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> 'm', ENDKEY =>
''}
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:seqnumDuringOpen, timestamp=1511537566069,
value=\x00\x00\x00\x00\x00\x00\x00\x08
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:server, timestamp=1511537566069, value=host-xx:16020
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
{noformat}
Meta content after restore snapshot:
{noformat}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
077a12b0b3c91b053fa95223635f9543, NAME =>
't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
'', ENDKEY => ''}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:seqnumDuringOpen, timestamp=1511537667635,
value=\x00\x00\x00\x00\x00\x00\x00\x0A
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:server, timestamp=1511537667635, value=host-xx:16020
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:serverstartcode, timestamp=1511537667635, value=1511537511523
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:regioninfo, timestamp=1511537667598, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
'', ENDKEY => 'm'}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:seqnumDuringOpen, timestamp=1511537667598,
value=\x00\x00\x00\x00\x00\x00\x00\x0B
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:server, timestamp=1511537667598, value=host-xx:16020
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
[jira] [Updated] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19343:
-
Environment: (was: Restore snapshot makes parent split region online as
shown in the attached snapshot.
Steps to reproduce
=
1. Create table
2. Insert few records into the table
3. flush the table
4. Split the table
5. Create snapshot before catalog janitor clears the parent region entry from
meta.
6. Restore snapshot
We can see the problem in meta entries,
Meta content before restore snapshot:
{noformat}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
077a12b0b3c91b053fa95223635f9543, NAME =>
't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
'', ENDKEY => '',
OFFLINE => true, SPLIT => true}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:seqnumDuringOpen, timestamp=1511537530107,
value=\x00\x00\x00\x00\x00\x00\x00\x02
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:server, timestamp=1511537530107, value=host-xx:16020
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:splitA, timestamp=1511537565964, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
ENDKEY => 'm'}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:splitB, timestamp=1511537565964, value={ENCODED =>
dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
', ENDKEY => ''}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
'', ENDKEY => 'm'}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:seqnumDuringOpen, timestamp=1511537566075,
value=\x00\x00\x00\x00\x00\x00\x00\x02
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:server, timestamp=1511537566075, value=host-xx:16020
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> 'm', ENDKEY =>
''}
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:seqnumDuringOpen, timestamp=1511537566069,
value=\x00\x00\x00\x00\x00\x00\x00\x08
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:server, timestamp=1511537566069, value=host-xx:16020
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
{noformat}
Meta content after restore snapshot:
{noformat}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
077a12b0b3c91b053fa95223635f9543, NAME =>
't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
'', ENDKEY => ''}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:seqnumDuringOpen, timestamp=1511537667635,
value=\x00\x00\x00\x00\x00\x00\x00\x0A
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:server, timestamp=1511537667635, value=host-xx:16020
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:serverstartcode, timestamp=1511537667635, value=1511537511523
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:regioninfo, timestamp=1511537667598, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
'', ENDKEY => 'm'}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:seqnumDuringOpen, timestamp=1511537667598,
value=\x00\x00\x00\x00\x00\x00\x00\x0B
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:server, timestamp=1511537667598, value=host-xx:16020
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
[jira] [Created] (HBASE-19343) Restore snapshot makes parent split region online
Pankaj Kumar created HBASE-19343:
Summary: Restore snapshot makes parent split region online
Key: HBASE-19343
URL: https://issues.apache.org/jira/browse/HBASE-19343
Project: HBase
Issue Type: Bug
Components: snapshots
Environment: Restore snapshot makes parent split region online as
shown in the attached snapshot.
Steps to reproduce
=
1. Create table
2. Insert few records into the table
3. flush the table
4. Split the table
5. Create snapshot before catalog janitor clears the parent region entry from
meta.
6. Restore snapshot
We can see the problem in meta entries,
Meta content before restore snapshot:
{noformat}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
077a12b0b3c91b053fa95223635f9543, NAME =>
't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
'', ENDKEY => '',
OFFLINE => true, SPLIT => true}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:seqnumDuringOpen, timestamp=1511537530107,
value=\x00\x00\x00\x00\x00\x00\x00\x02
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:server, timestamp=1511537530107, value=host-xx:16020
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:splitA, timestamp=1511537565964, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
ENDKEY => 'm'}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:splitB, timestamp=1511537565964, value={ENCODED =>
dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
', ENDKEY => ''}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
'', ENDKEY => 'm'}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:seqnumDuringOpen, timestamp=1511537566075,
value=\x00\x00\x00\x00\x00\x00\x00\x02
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:server, timestamp=1511537566075, value=host-xx:16020
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> 'm', ENDKEY =>
''}
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:seqnumDuringOpen, timestamp=1511537566069,
value=\x00\x00\x00\x00\x00\x00\x00\x08
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:server, timestamp=1511537566069, value=host-xx:16020
t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
{noformat}
Meta content after restore snapshot:
{noformat}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
077a12b0b3c91b053fa95223635f9543, NAME =>
't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
'', ENDKEY => ''}
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:seqnumDuringOpen, timestamp=1511537667635,
value=\x00\x00\x00\x00\x00\x00\x00\x0A
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:server, timestamp=1511537667635, value=host-xx:16020
t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
column=info:serverstartcode, timestamp=1511537667635, value=1511537511523
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:regioninfo, timestamp=1511537667598, value={ENCODED =>
3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
'', ENDKEY => 'm'}
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
column=info:seqnumDuringOpen, timestamp=1511537667598,
value=\x00\x00\x00\x00\x00\x00\x00\x0B
t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
[jira] [Created] (HBASE-19342) fix TestTableBasedReplicationSourceManagerImpl#testRemovePeerMetricsCleanup
Chia-Ping Tsai created HBASE-19342: -- Summary: fix TestTableBasedReplicationSourceManagerImpl#testRemovePeerMetricsCleanup Key: HBASE-19342 URL: https://issues.apache.org/jira/browse/HBASE-19342 Project: HBase Issue Type: Test Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai It is number one in [flaky tests|https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265330#comment-16265330 ] Chia-Ping Tsai commented on HBASE-19340: We should backport HBASE-17736 and HBASE-17729 to branch-1.3 and 1.2. [~smartZY] Thanks for the report. Do you want to submit the patch? The patches in HBASE-17736 and HBASE-17729 are good references. I will help to review your patch. Or I can deal with the backport if you have no free cycle. :) > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan > Fix For: 1.2.8 > > Attachments: 19340.branch-1.2.txt > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19281) Snapshot creation failed after splitting table (replica region > 1)
[
https://issues.apache.org/jira/browse/HBASE-19281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265303#comment-16265303
]
Pankaj Kumar commented on HBASE-19281:
--
Adding this Jira under HBASE-18223.
> Snapshot creation failed after splitting table (replica region > 1)
> ---
>
> Key: HBASE-19281
> URL: https://issues.apache.org/jira/browse/HBASE-19281
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 1.3.1
>Reporter: Chandra Sekhar
>Assignee: Pankaj Kumar
>
> Snapshot creation failed with below error when tried on table with multiple
> replica region,
> {noformat}
> hbase(main):025:0> snapshot 't1','t1_snap'
> 2017-11-16 18:04:27,930 DEBUG [main] client.HBaseAdmin: Waiting a max of
> 30 ms for snapshot '{ ss=t1_snap table=t1 type=FLUSH }'' to complete.
> (max 42857 ms per retry)
> 2017-11-16 18:04:27,930 DEBUG [main] client.HBaseAdmin: (#1) Sleeping: 100ms
> while waiting for snapshot completion.
> 2017-11-16 18:04:28,030 DEBUG [main] client.HBaseAdmin: Getting current
> status of snapshot from master...
> 2017-11-16 18:04:28,035 DEBUG [main] client.HBaseAdmin: (#2) Sleeping: 200ms
> while waiting for snapshot completion.
> 2017-11-16 18:04:28,236 DEBUG [main] client.HBaseAdmin: Getting current
> status of snapshot from master...
> 2017-11-16 18:04:28,238 DEBUG [main] client.HBaseAdmin: (#3) Sleeping: 300ms
> while waiting for snapshot completion.
> 2017-11-16 18:04:28,538 DEBUG [main] client.HBaseAdmin: Getting current
> status of snapshot from master...
> ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=t1_snap table=t1 type=FLUSH } had an error. Procedure t1_snap {
> waiting=[] done=[] }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:354)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1091)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2418)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:191)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via
> Failed taking snapshot { ss=t1_snap table=t1 type=FLUSH } due to
> exception:Manifest region info {ENCODED => 3158abebd655fca73cd87b6e84584197,
> NAME => 't1,,1510826577196_0002.3158abebd655fca73cd87b6e84584197.', STARTKEY
> => '', ENDKEY => '', OFFLINE => true, SPLIT => true, REPLICA_ID => 2}doesn't
> match expected region:{ENCODED => 73aa1a133d3344a67afa46ee135e389a, NAME =>
> 't1,,1510826577196.73aa1a133d3344a67afa46ee135e389a.', STARTKEY => '', ENDKEY
> => '', OFFLINE => true, SPLIT =>
> true}:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Manifest
> region info {ENCODED => 3158abebd655fca73cd87b6e84584197, NAME =>
> 't1,,1510826577196_0002.3158abebd655fca73cd87b6e84584197.', STARTKEY => '',
> ENDKEY => '', OFFLINE => true, SPLIT => true, REPLICA_ID => 2}doesn't match
> expected region:{ENCODED => 73aa1a133d3344a67afa46ee135e389a, NAME =>
> 't1,,1510826577196.73aa1a133d3344a67afa46ee135e389a.', STARTKEY => '', ENDKEY
> => '', OFFLINE => true, SPLIT => true}
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:315)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:344)
> ... 6 more
> Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
> Manifest region info {ENCODED => 3158abebd655fca73cd87b6e84584197, NAME =>
> 't1,,1510826577196_0002.3158abebd655fca73cd87b6e84584197.', STARTKEY => '',
> ENDKEY => '', OFFLINE => true, SPLIT => true, REPLICA_ID => 2}doesn't match
> expected region:{ENCODED => 73aa1a133d3344a67afa46ee135e389a, NAME =>
> 't1,,1510826577196.73aa1a133d3344a67afa46ee135e389a.', STARTKEY => '', ENDKEY
> => '', OFFLINE => true, SPLIT => true}
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegionInfo(MasterSnapshotVerifier.java:220)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:198)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:118)
> at
>
[jira] [Commented] (HBASE-17049) Do not issue sync request when there are still entries in ringbuffer
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265278#comment-16265278 ] Duo Zhang commented on HBASE-17049: --- [~zghaobac] Please also try the patch here when testing with PE and ycsb. Thanks. > Do not issue sync request when there are still entries in ringbuffer > > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-17049.patch, delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17049) Do not issue sync request when there are still entries in ringbuffer
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17049: -- Affects Version/s: (was: 2.0.0) Priority: Critical (was: Blocker) Fix Version/s: (was: 2.0.0) 2.0.0-beta-1 > Do not issue sync request when there are still entries in ringbuffer > > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-17049.patch, delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17049) Do not issue sync request when there are still entries in ringbuffer
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17049: -- Status: Patch Available (was: Reopened) > Do not issue sync request when there are still entries in ringbuffer > > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17049.patch, delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17049) Do not issue sync request when there are still entries in ringbuffer
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17049: -- Attachment: HBASE-17049.patch Move the decision up from appendAndSync to consume. > Do not issue sync request when there are still entries in ringbuffer > > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17049.patch, delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17049) Do not issue sync request when there are still entries in ringbuffer
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17049: -- Summary: Do not issue sync request when there are still entries in ringbuffer (was: Find out why AsyncFSWAL issues much more syncs than FSHLog) > Do not issue sync request when there are still entries in ringbuffer > > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19188) Build fails on branch-1 using maven-3.5.2
[
https://issues.apache.org/jira/browse/HBASE-19188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265268#comment-16265268
]
Peter Somogyi commented on HBASE-19188:
---
I see the following solutions:
# Change jasper-runtime's scope to compile in hbase-server
# Change jasper-runtime's scope to compile in main module
# In maven-ant-plugin pass maven.runtime.classpath to JspC
> Build fails on branch-1 using maven-3.5.2
> -
>
> Key: HBASE-19188
> URL: https://issues.apache.org/jira/browse/HBASE-19188
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.0, 1.3.1, 1.2.6, 1.5.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
>
> With maven 3.5.2 the build fails on branch-1-2, branch-1.3, branch-1.4 and
> branch-1. On branch-1.1, branch-2 and master the build succeeds. With older
> maven versions the build finishes.
> {code:title=Maven version}
> $ mvn -v
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=1024m;
> support was removed in 8.0
> Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> 2017-10-18T09:58:13+02:00)
> Maven home: /Users/peter.somogyi/bin/apache-maven-3.5.2
> Java version: 1.8.0_141, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"
> {code}
> {code}
> $ mvn clean install -DskipTests
> ...
> [INFO] --- jamon-maven-plugin:2.4.1:translate (default) @ hbase-server ---
> [INFO]
> [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server ---
> [INFO] Executing tasks
> main:
> log4j:WARN No appenders could be found for logger (org.apache.jasper.JspC).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> [INFO] Logging to org.slf4j.impl.MavenSimpleLogger(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> java.util.MissingResourceException: Can't find bundle for base name
> org.apache.jasper.resources.LocalStrings, locale en_US
> at
> java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1564)
> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1387)
> at java.util.ResourceBundle.getBundle(ResourceBundle.java:773)
> at org.apache.jasper.compiler.Localizer.(Localizer.java:36)
> at
> org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:103)
> at org.apache.jasper.JspC.initServletContext(JspC.java:1242)
> at org.apache.jasper.JspC.execute(JspC.java:1103)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> at org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:154)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> at org.apache.tools.ant.Task.perform(Task.java:348)
> at org.apache.tools.ant.Target.execute(Target.java:390)
> at org.apache.tools.ant.Target.performTasks(Target.java:411)
> at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
> at org.apache.tools.ant.Project.executeTarget(Project.java:1366)
> at
> org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:270)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:353)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:198)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
> at
>
[jira] [Reopened] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-17049: --- Assignee: Duo Zhang Talked with [~chancelq] and [~carp84] offline, and we found a problem that may cause AsyncFSWAL issues more syncs than FSHLog. For AsyncFSWAL, we will first take all the entries out of the ringbuffer, and then do append and sync, but while we do append and sync, there may be more entries inserted to the ringbuffer, but we just ignore them and issue a sync at last even if we have only buffered very small data. For FSHLog, there is no such problem. We just do take, append, take, append. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HBASE-19159:
--
Attachment: HBASE-19159-v2.patch
Updated patch by the review. Instead of passing the fsaction I changed the name
of the function. The purpose of checkBackupCanBeCreated is to check if the
first existing parent of the backup dir is writable -> subdirs can be created.
I am not sure if this is usable for any other fsaction. [~tedyu] what do you
think? Also I am not sure, maybe it should check WRITE_EXECUTE instead of
WRITE...
Ran the following:
{code}
mvn clean install -Dtest=TestBackup\*
{code}
result:
{code}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupSmallTests
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.641 s
- in org.apache.hadoop.hbase.backup.TestBackupSmallTests
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupStatusProgress
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.517 s
- in org.apache.hadoop.hbase.backup.TestBackupStatusProgress
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupShowHistory
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.36 s
- in org.apache.hadoop.hbase.backup.TestBackupShowHistory
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.712 s
- in org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupCommandLineTool
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.672 s
- in org.apache.hadoop.hbase.backup.TestBackupCommandLineTool
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupHFileCleaner
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.072 s
- in org.apache.hadoop.hbase.backup.TestBackupHFileCleaner
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupDeleteWithFailures
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.715
s - in org.apache.hadoop.hbase.backup.TestBackupDeleteWithFailures
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupRepair
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.598 s
- in org.apache.hadoop.hbase.backup.TestBackupRepair
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.887 s
- in org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
[INFO] Running org.apache.hadoop.hbase.backup.master.TestBackupLogCleaner
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.676 s
- in org.apache.hadoop.hbase.backup.master.TestBackupLogCleaner
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupMultipleDeletes
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 153.956
s - in org.apache.hadoop.hbase.backup.TestBackupMultipleDeletes
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupDescribe
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.424 s
- in org.apache.hadoop.hbase.backup.TestBackupDescribe
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupDelete
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.575 s
- in org.apache.hadoop.hbase.backup.TestBackupDelete
[INFO] Running org.apache.hadoop.hbase.backup.TestBackupSystemTable
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.655
s - in org.apache.hadoop.hbase.backup.TestBackupSystemTable
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
{code}
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159-v2.patch, HBASE-19159.patch,
> initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
>
[jira] [Updated] (HBASE-19341) Ensure CP can abort a Server
[ https://issues.apache.org/jira/browse/HBASE-19341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19341: --- Priority: Critical (was: Major) > Ensure CP can abort a Server > > > Key: HBASE-19341 > URL: https://issues.apache.org/jira/browse/HBASE-19341 > Project: HBase > Issue Type: Improvement > Components: Coprocessors >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > > We used to allow a CP pull the Server#abort chain. We removed it in the CP > refactor. At the end of HBASE-18298, [~an...@apache.org] describes a case > where Phoenix needs to kill Server in extreme case to maintain consistency. > This issue is about ensuring that throwing a CPException will indeed kill the > running server Add a test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19341) Ensure CP can abort a Server
[ https://issues.apache.org/jira/browse/HBASE-19341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19341: --- Issue Type: Improvement (was: Bug) > Ensure CP can abort a Server > > > Key: HBASE-19341 > URL: https://issues.apache.org/jira/browse/HBASE-19341 > Project: HBase > Issue Type: Improvement > Components: Coprocessors >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > > We used to allow a CP pull the Server#abort chain. We removed it in the CP > refactor. At the end of HBASE-18298, [~an...@apache.org] describes a case > where Phoenix needs to kill Server in extreme case to maintain consistency. > This issue is about ensuring that throwing a CPException will indeed kill the > running server Add a test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19341) Ensure CP can abort a Server
[ https://issues.apache.org/jira/browse/HBASE-19341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265212#comment-16265212 ] Anoop Sam John commented on HBASE-19341: Just to add : We have way to abort server now also. CP code throwing any non IOE , the core will either abort server or remove this CP. This depends on a config which defaults to a value to say abort server. Still it would be better to explicitly give a special exception type which will be handled by server abort(Whatever be the value of the above said config). Or we should just get rid of this config fully? May be a BC issue. Just we can deprecate now and remove for 3.0 and there only rely on the Exception type? WDYT boss? > Ensure CP can abort a Server > > > Key: HBASE-19341 > URL: https://issues.apache.org/jira/browse/HBASE-19341 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > > We used to allow a CP pull the Server#abort chain. We removed it in the CP > refactor. At the end of HBASE-18298, [~an...@apache.org] describes a case > where Phoenix needs to kill Server in extreme case to maintain consistency. > This issue is about ensuring that throwing a CPException will indeed kill the > running server Add a test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265204#comment-16265204
]
Ted Yu commented on HBASE-19159:
{code}
612 public static void checkBackupAccess(FileSystem fs, Path path) throws
IOException {
{code}
The method name seems general. Please pass the FsAction as parameter.
For TestBackupSmallTests, please add license header.
Add test category: SmallTests.
{code}
+@Test public void testBackupPathIsAccessible() throws Exception {
{code}
Put @Test on its own line.
I ran the new test which passed.
QA bot is malfunctioning.
Please run backup unit tests locally and post result back.
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159.patch, initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
> to=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/user/root/test-data/fb919a6f-3cb4-4d57-bbcf-561d6e5b3ae8/backupIT/backup_1509646884252/default/IntegrationTestBackupRestore.table2/.hbase-snapshot/.tmp/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.doWork(ExportSnapshot.java:1009)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:386)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.snapshotCopy(FullTableBackupClient.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:175)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTest(IntegrationTestBackupRestore.java:180)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:134)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestFromCommandLine(IntegrationTestBackupRestore.java:263)
> {code}
> It would be more user friendly if the permission is checked before taking the
> snapshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19337) AsyncMetaTableAccessor may hang when call ScanController.terminate many times
[
https://issues.apache.org/jira/browse/HBASE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265201#comment-16265201
]
Ted Yu commented on HBASE-19337:
Is the following change intentional ?
{code}
if (LOG.isTraceEnabled()) {
{code}
Is it possible to add a test ?
> AsyncMetaTableAccessor may hang when call ScanController.terminate many times
> -
>
> Key: HBASE-19337
> URL: https://issues.apache.org/jira/browse/HBASE-19337
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19337.master.001.patch
>
>
> Code in ScanControllerImpl.
> {code}
> private void preCheck() {
> Preconditions.checkState(Thread.currentThread() == callerThread,
> "The current thread is %s, expected thread is %s, " +
> "you should not call this method outside onNext or onHeartbeat",
> Thread.currentThread(), callerThread);
> Preconditions.checkState(state.equals(ScanControllerState.INITIALIZED),
> "Invalid Stopper state %s", state);
> }
> @Override
> public void terminate() {
> preCheck();
> state = ScanControllerState.TERMINATED;
> }
> {code}
> So if call terminate on a already terminated scan, it will throw
> IllegalStateException.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265199#comment-16265199 ] Hudson commented on HBASE-19092: FAILURE: Integrated in Jenkins build HBase-2.0 #907 (See [https://builds.apache.org/job/HBase-2.0/907/]) HBASE-19092 Make Tag IA.LimitedPrivate and expose for CPs (Ram) (ramkrishna.s.vasudevan: rev 6ac6ae3fa2c7cdf346e2101318871802b1c5cbb1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALCellCodecWithCompression.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/Tag.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCellBuilderImpl.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/IndividualBytesFieldCell.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/PrivateCellUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/TagUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelReplicationWithExpAsString.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/codec/TestKeyValueCodecWithTags.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/codec/TestCellCodecWithTags.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCellBuilderFactory.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestByteBufferKeyValue.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsReplication.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/RawCell.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionCoprocessorEnvironment.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/DefaultVisibilityLabelServiceImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/IndividualBytesFieldCellBuilder.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileScannerWithTagCompression.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileTestUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/ExpAsStringVisibilityLabelServiceImpl.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TagUsage.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueBuilder.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWithTags.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * (edit)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19340: --- Attachment: 19340.branch-1.2.txt See if this solves the problem for you. > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan > Fix For: 1.2.8 > > Attachments: 19340.branch-1.2.txt > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265186#comment-16265186
]
Chia-Ping Tsai commented on HBASE-19338:
It seems our jenkins config have been changed recently. I have no permission to
access the [change
page|https://builds.apache.org/job/PreCommit-HBASE-Build/jobConfigHistory/showDiffFiles?timestamp1=2017-11-06_13-27-38=2017-11-23_20-16-38]...
ping [~stack]
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265172#comment-16265172
]
Ted Yu commented on HBASE-19336:
QA bot is malfunctioning today.
Can you run rsgroup_shell_test with patch v3 and paste the result ?
Cheers
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch,
> HBASE-19336-master-V3.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_namespaces_rsgroup
> 'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265168#comment-16265168
]
Ted Yu commented on HBASE-19338:
https://builds.apache.org/job/PreCommit-HBASE-Build/9997/console :
{code}
08:50:46 Modes: MultiJDK Jenkins Robot Docker ResetRepo UnitTests
08:50:46 Processing: HBASE-19338
08:50:47 ERROR: Unsure how to process HBASE-19338.
{code}
Looks like we may not get QA back today.
Lijin:
After verifying a few Visibility and AccessController related tests, you can go
with the commit.
Thanks
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19328) Remove asked if splittable log messages
[
https://issues.apache.org/jira/browse/HBASE-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265152#comment-16265152
]
Chia-Ping Tsai commented on HBASE-19328:
+1
> Remove asked if splittable log messages
> ---
>
> Key: HBASE-19328
> URL: https://issues.apache.org/jira/browse/HBASE-19328
> Project: HBase
> Issue Type: Task
> Components: proc-v2
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19328.master.001.patch
>
>
> I have found this log message in HBase log:
> {code}
> 2017-11-22 11:16:54,133 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52586]
> regionserver.HRegion(1309): ASKED IF SPLITTABLE true
> 0a66d6e20801eec2c6cd1204fedde592
> java.lang.Throwable: LOGGING: REMOVE
> at
> org.apache.hadoop.hbase.regionserver.HRegion.isSplittable(HRegion.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1665)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28159)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
> Still we need this?
> It was introduced in commit {{dc1065a85}} by [~stack] and [~mbertozzi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19328) Remove asked if splittable log messages
[
https://issues.apache.org/jira/browse/HBASE-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19328:
---
Fix Version/s: 2.0.0-beta-1
> Remove asked if splittable log messages
> ---
>
> Key: HBASE-19328
> URL: https://issues.apache.org/jira/browse/HBASE-19328
> Project: HBase
> Issue Type: Task
> Components: proc-v2
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19328.master.001.patch
>
>
> I have found this log message in HBase log:
> {code}
> 2017-11-22 11:16:54,133 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52586]
> regionserver.HRegion(1309): ASKED IF SPLITTABLE true
> 0a66d6e20801eec2c6cd1204fedde592
> java.lang.Throwable: LOGGING: REMOVE
> at
> org.apache.hadoop.hbase.regionserver.HRegion.isSplittable(HRegion.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1665)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28159)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
> Still we need this?
> It was introduced in commit {{dc1065a85}} by [~stack] and [~mbertozzi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-19213) Align check and mutate operations in Table and AsyncTable
[ https://issues.apache.org/jira/browse/HBASE-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reassigned HBASE-19213: - Assignee: Peter Somogyi > Align check and mutate operations in Table and AsyncTable > - > > Key: HBASE-19213 > URL: https://issues.apache.org/jira/browse/HBASE-19213 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0-alpha-4 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 2.0.0-beta-2 > > > Check and mutate methods are way different. Table has checkAndx methods (some > of them are deprecated), but AsyncTable has an interface called > CheckAndMutateBuilder and these kind of operations are handled through that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19328) Remove asked if splittable log messages
[
https://issues.apache.org/jira/browse/HBASE-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-19328:
Attachment: HBASE-19328.master.001.patch
> Remove asked if splittable log messages
> ---
>
> Key: HBASE-19328
> URL: https://issues.apache.org/jira/browse/HBASE-19328
> Project: HBase
> Issue Type: Task
> Components: proc-v2
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
> Attachments: HBASE-19328.master.001.patch
>
>
> I have found this log message in HBase log:
> {code}
> 2017-11-22 11:16:54,133 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52586]
> regionserver.HRegion(1309): ASKED IF SPLITTABLE true
> 0a66d6e20801eec2c6cd1204fedde592
> java.lang.Throwable: LOGGING: REMOVE
> at
> org.apache.hadoop.hbase.regionserver.HRegion.isSplittable(HRegion.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1665)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28159)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
> Still we need this?
> It was introduced in commit {{dc1065a85}} by [~stack] and [~mbertozzi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19328) Remove asked if splittable log messages
[
https://issues.apache.org/jira/browse/HBASE-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-19328:
Status: Patch Available (was: Open)
> Remove asked if splittable log messages
> ---
>
> Key: HBASE-19328
> URL: https://issues.apache.org/jira/browse/HBASE-19328
> Project: HBase
> Issue Type: Task
> Components: proc-v2
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
> Attachments: HBASE-19328.master.001.patch
>
>
> I have found this log message in HBase log:
> {code}
> 2017-11-22 11:16:54,133 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52586]
> regionserver.HRegion(1309): ASKED IF SPLITTABLE true
> 0a66d6e20801eec2c6cd1204fedde592
> java.lang.Throwable: LOGGING: REMOVE
> at
> org.apache.hadoop.hbase.regionserver.HRegion.isSplittable(HRegion.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1665)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28159)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
> Still we need this?
> It was introduced in commit {{dc1065a85}} by [~stack] and [~mbertozzi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-19328) Remove asked if splittable log messages
[
https://issues.apache.org/jira/browse/HBASE-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros reassigned HBASE-19328:
---
Assignee: Balazs Meszaros
> Remove asked if splittable log messages
> ---
>
> Key: HBASE-19328
> URL: https://issues.apache.org/jira/browse/HBASE-19328
> Project: HBase
> Issue Type: Task
> Components: proc-v2
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
>
> I have found this log message in HBase log:
> {code}
> 2017-11-22 11:16:54,133 INFO
> [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52586]
> regionserver.HRegion(1309): ASKED IF SPLITTABLE true
> 0a66d6e20801eec2c6cd1204fedde592
> java.lang.Throwable: LOGGING: REMOVE
> at
> org.apache.hadoop.hbase.regionserver.HRegion.isSplittable(HRegion.java:1310)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1665)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28159)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
> Still we need this?
> It was introduced in commit {{dc1065a85}} by [~stack] and [~mbertozzi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19242) Add MOB compact support for AsyncAdmin
[
https://issues.apache.org/jira/browse/HBASE-19242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-19242:
Attachment: HBASE-19242.master.002.patch
> Add MOB compact support for AsyncAdmin
> --
>
> Key: HBASE-19242
> URL: https://issues.apache.org/jira/browse/HBASE-19242
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, mob
>Reporter: Duo Zhang
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19242.master.001.patch,
> HBASE-19242.master.002.patch
>
>
> {code}
> private CompletableFuture compact(TableName tableName, byte[]
> columnFamily, boolean major,
> CompactType compactType) {
> if (CompactType.MOB.equals(compactType)) {
> // TODO support MOB compact.
> return failedFuture(new UnsupportedOperationException("MOB compact does
> not support"));
> }
> {code}
> We need to support it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265100#comment-16265100 ] Hudson commented on HBASE-19092: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4109 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4109/]) HBASE-19092 Make Tag IA.LimitedPrivate and expose for CPs (Ram) (ramkrishna.s.vasudevan: rev 73e3af00e94f0be12dd6e399d5e72966311ae3fe) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/TagUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileTestUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCellBuilderImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/DefaultVisibilityLabelServiceImpl.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelReplicationWithExpAsString.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/Tag.java * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/ExpAsStringVisibilityLabelServiceImpl.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALCellCodecWithCompression.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileScannerWithTagCompression.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/PrivateCellUtil.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/codec/TestKeyValueCodecWithTags.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionCoprocessorEnvironment.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/codec/TestCellCodecWithTags.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWithTags.java * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/RawCell.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsReplication.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityUtils.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestByteBufferKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/IndividualBytesFieldCellBuilder.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/TestTagUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueBuilder.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TagUsage.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCellBuilderFactory.java * (edit)
[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265096#comment-16265096 ] Hudson commented on HBASE-18309: SUCCESS: Integrated in Jenkins build HBase-2.0 #906 (See [https://builds.apache.org/job/HBase-2.0/906/]) HBASE-18309 Support multi threads in CleanerChore (liyu: rev 2838cf3e0506d9be8c001ebbadc0c3acf68d762c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: Reid Chan > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, HBASE-18309.master.010.patch, > HBASE-18309.master.011.patch, HBASE-18309.master.012.patch, > space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. > What's more, currently there's only one thread iterating the archive > directory, and we could use multiple threads cleaning sub directories in > parallel to speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-19112) Suspect methods on Cell to be deprecated
[
https://issues.apache.org/jira/browse/HBASE-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-19112:
--
Assignee: ramkrishna.s.vasudevan
> Suspect methods on Cell to be deprecated
>
>
> Key: HBASE-19112
> URL: https://issues.apache.org/jira/browse/HBASE-19112
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> [~chia7712] suggested on the [mailing
> list|https://lists.apache.org/thread.html/e6de9af26d9b888a358ba48bf74655ccd893573087c032c0fcf01585@%3Cdev.hbase.apache.org%3E]
> that we have some methods on Cell which should be deprecated for removal:
> * {{#getType()}}
> * {{#getTimestamp()}}
> * {{#getTag()}}
> * {{#getSequenceId()}}
> Let's make a pass over these (and maybe the rest) to make sure that there
> aren't others which are either implementation details or methods returning
> now-private-marked classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265069#comment-16265069
]
Chia-Ping Tsai commented on HBASE-19338:
bq. Simply re-attaching the patch will trigger HadoopQA and no need to change
JIRA status, JFYI.
Thanks for your friendly reminder.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[
https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-19092:
---
Release Note:
This JIRA aims at exposing Tags for Coprocessor usage.
Tag interface is now exposed to Coprocessors and CPs can make use of this
interface to create their own Tags.
RawCell is a new interface that is a subtype of Cell and that is exposed to
CPs. RawCell has the following APIs
List getTags()
Optional getTag(byte type)
byte[] cloneTags()
The above APIs helps to read tags from the Cell.
CellUtil#createCell(Cell cell, List tags)
CellUtil#createCell(Cell cell, byte[] tags)
CellUtil#createCell(Cell cell, byte[] value, byte[] tags)
are deprecated.
If CPs want to create a cell with Tags they can use the
RegionCoprocessEnivronment#getCellBuilder() that returns an ExtendedCellBuilder.
Using ExtendedCellBuilder the CP can create Cells with Tags. Other helper
methods to work on Tags are available as static APIs in Tag interface.
was:
This JIRA aims at exposing Tags for Coprocessor usage.
Tag interface is now exposed to Coprocessors and CPs can make use of this
interface to create their own Tags.
RawCell is a new interface that is a subtype of Cell and that is exposed to
CPs. RawCell has the following APIs
{code}
List getTags()
Optional getTag(byte type)
byte[] cloneTags()
{code}
It helps to read tags from the Cell.
CellUtil#createCell(Cell cell, List tags)
CellUtil#createCell(Cell cell, byte[] tags)
CellUtil#createCell(Cell cell, byte[] value, byte[] tags)
are deprecated.
If CPs want to create a cell with Tags they can use the
RegionCoprocessEnivronment#getCellBuilder() that returns an ExtendedCellBuilder.
Using ExtendedCellBuilder the CP can create Cells with Tags. Other helper
methods to work on Tags are available as static APIs in Tag interface.
> Make Tag IA.LimitedPrivate and expose for CPs
> -
>
> Key: HBASE-19092
> URL: https://issues.apache.org/jira/browse/HBASE-19092
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19092-branch-2.patch,
> HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch,
> HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch,
> HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch,
> HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch,
> HBASE-19092_3.patch, HBASE-19092_4.patch
>
>
> We need to make tags as LimitedPrivate as some use cases are trying to use
> tags like timeline server. The same topic was discussed in dev@ and also in
> HBASE-18995.
> Shall we target this for beta1 - cc [~saint@gmail.com].
> So once we do this all related Util methods and APIs should also move to
> LimitedPrivate Util classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[
https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-19092:
---
Hadoop Flags: Incompatible change,Reviewed (was: Reviewed)
Release Note:
This JIRA aims at exposing Tags for Coprocessor usage.
Tag interface is now exposed to Coprocessors and CPs can make use of this
interface to create their own Tags.
RawCell is a new interface that is a subtype of Cell and that is exposed to
CPs. RawCell has the following APIs
{code}
List getTags()
Optional getTag(byte type)
byte[] cloneTags()
{code}
It helps to read tags from the Cell.
CellUtil#createCell(Cell cell, List tags)
CellUtil#createCell(Cell cell, byte[] tags)
CellUtil#createCell(Cell cell, byte[] value, byte[] tags)
are deprecated.
If CPs want to create a cell with Tags they can use the
RegionCoprocessEnivronment#getCellBuilder() that returns an ExtendedCellBuilder.
Using ExtendedCellBuilder the CP can create Cells with Tags. Other helper
methods to work on Tags are available as static APIs in Tag interface.
> Make Tag IA.LimitedPrivate and expose for CPs
> -
>
> Key: HBASE-19092
> URL: https://issues.apache.org/jira/browse/HBASE-19092
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19092-branch-2.patch,
> HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch,
> HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch,
> HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch,
> HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch,
> HBASE-19092_3.patch, HBASE-19092_4.patch
>
>
> We need to make tags as LimitedPrivate as some use cases are trying to use
> tags like timeline server. The same topic was discussed in dev@ and also in
> HBASE-18995.
> Shall we target this for beta1 - cc [~saint@gmail.com].
> So once we do this all related Util methods and APIs should also move to
> LimitedPrivate Util classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265062#comment-16265062
]
Yu Li commented on HBASE-19338:
---
Thanks [~chia7712]. Simply re-attaching the patch will trigger HadoopQA and no
need to change JIRA status, JFYI.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HBASE-19159:
--
Attachment: HBASE-19159.patch
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159.patch, initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
> to=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/user/root/test-data/fb919a6f-3cb4-4d57-bbcf-561d6e5b3ae8/backupIT/backup_1509646884252/default/IntegrationTestBackupRestore.table2/.hbase-snapshot/.tmp/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.doWork(ExportSnapshot.java:1009)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:386)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.snapshotCopy(FullTableBackupClient.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:175)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTest(IntegrationTestBackupRestore.java:180)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:134)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestFromCommandLine(IntegrationTestBackupRestore.java:263)
> {code}
> It would be more user friendly if the permission is checked before taking the
> snapshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19159) Backup should check permission for snapshot copy in advance
[
https://issues.apache.org/jira/browse/HBASE-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janos Gub updated HBASE-19159:
--
Status: Patch Available (was: Open)
> Backup should check permission for snapshot copy in advance
> ---
>
> Key: HBASE-19159
> URL: https://issues.apache.org/jira/browse/HBASE-19159
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
>Priority: Minor
> Attachments: HBASE-19159.patch, initial_patch.txt
>
>
> When the user running the backup doesn't have permission to copy the snapshot
> , he / she would see:
> {code}
> 2017-11-02 18:21:33,654 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the
> snapshot directory:
> from=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/apps/hbase/data/.hbase-snapshot/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
>
> to=hdfs://ctr-e134-1499953498516-263664-01-03.hwx.site:8020/user/root/test-data/fb919a6f-3cb4-4d57-bbcf-561d6e5b3ae8/backupIT/backup_1509646884252/default/IntegrationTestBackupRestore.table2/.hbase-snapshot/.tmp/snapshot_1509646891251_default_IntegrationTestBackupRestore.table2
> at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.doWork(ExportSnapshot.java:1009)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:386)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.snapshotCopy(FullTableBackupClient.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.FullTableBackupClient.execute(FullTableBackupClient.java:175)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTest(IntegrationTestBackupRestore.java:180)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:134)
> at
> org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestFromCommandLine(IntegrationTestBackupRestore.java:263)
> {code}
> It would be more user friendly if the permission is checked before taking the
> snapshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19338:
---
Attachment: 19338.master.003.patch
Let me retry the v3.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19338:
---
Status: Open (was: Patch Available)
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19338:
---
Status: Patch Available (was: Open)
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, 19338.master.003.patch,
> HBASE-19338.master.001.patch, HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265037#comment-16265037 ] binlijin commented on HBASE-19323: -- bq. Yes. But I remember the netty server was not using our pool for sure. Ram is correct, NettyRpcServer do not use hbase's ByteBufferPool. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18298) RegionServerServices Interface cleanup for CP expose
[ https://issues.apache.org/jira/browse/HBASE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265027#comment-16265027 ] stack commented on HBASE-18298: --- [~an...@apache.org] I filed HBASE-19341 to ensure you can still abort by throwing an exception. > RegionServerServices Interface cleanup for CP expose > > > Key: HBASE-18298 > URL: https://issues.apache.org/jira/browse/HBASE-18298 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18298.patch, HBASE-18298_V2.patch, > HBASE-18298_V3.patch, HBASE-18298_V4.patch, HBASE-18298_V5.patch, > HBASE-18298_V6.patch, HBASE-18298_V7.patch, HBASE-18298_V7.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19341) Ensure CP can abort a Server
stack created HBASE-19341: - Summary: Ensure CP can abort a Server Key: HBASE-19341 URL: https://issues.apache.org/jira/browse/HBASE-19341 Project: HBase Issue Type: Bug Components: Coprocessors Reporter: stack Assignee: stack Fix For: 2.0.0-beta-1 We used to allow a CP pull the Server#abort chain. We removed it in the CP refactor. At the end of HBASE-18298, [~an...@apache.org] describes a case where Phoenix needs to kill Server in extreme case to maintain consistency. This issue is about ensuring that throwing a CPException will indeed kill the running server Add a test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyuan updated HBASE-19340: - Description: Recently I wanna try to alter the split policy for a table on my cluster which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute of the HTable so I run the command in hbase shell console below. alter 'tablex',SPLIT_POLICY => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' However, It gave the information like this and I confused Unknown argument ignored: SPLIT_POLICY Updating all regions with the new schema... So I check the source code That admin.rb might miss the setting for this argument . htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if arg[MAX_FILESIZE] htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] ... So I think it may be a bug ,is it? was: Recently I wanna try to alter the split policy for a table on my cluster which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute of the HTable so I run the command in hbase shell console below. alter 'tablex',SPLIT_POLICY => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' However, It gave the information like this and I confused Unknown argument ignored: SPLIT_POLICY Updating all regions with the new schema... So I check the source code That admin.rb might miss the setting for this argument . htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if arg[MAX_FILESIZE] htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] ... So I think it may be a bug in hbase-server ,is it? > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan > Fix For: 1.2.8 > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265024#comment-16265024 ] Yu Li commented on HBASE-19323: --- bq. That test was against old RpcServer in which version? May be test against latest SimpleRpcServer in branch-2 would be ideal >From the JIRA history, should be some version last Dec. Any special >improvements made for SimpleRpcServer later on that worth another round of >perf test? Or simply better to verify? Could you share your main concern on >making NettyRpcServer default so we could narrow down the scope to check >(anything besides DBB pool)? Thanks. [~anoop.hbase] [~ram_krish] bq. Sorry we discussed abt this in that Netty issue I forgot now. Really sorry. Yep I also remember the DBB pool issue was discussed in HBASE-17263 but forgot the details. [~aoxiang] As the author of NettyRpcServer, I prefer to leave the detailed questions for you to answer (smile) > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19163) "Maximum lock count exceeded" from region server's batch processing
[
https://issues.apache.org/jira/browse/HBASE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265022#comment-16265022
]
stack commented on HBASE-19163:
---
+1 then on the patch [~huaxiang] Nice work sir.
> "Maximum lock count exceeded" from region server's batch processing
> ---
>
> Key: HBASE-19163
> URL: https://issues.apache.org/jira/browse/HBASE-19163
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 1.2.7, 2.0.0-alpha-3
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-19163-master-v001.patch,
> HBASE-19163.master.001.patch, HBASE-19163.master.002.patch,
> HBASE-19163.master.004.patch, HBASE-19163.master.005.patch,
> HBASE-19163.master.006.patch, unittest-case.diff
>
>
> In one of use cases, we found the following exception and replication is
> stuck.
> {code}
> 2017-10-25 19:41:17,199 WARN [hconnection-0x28db294f-shared--pool4-t936]
> client.AsyncProcess: #3, table=foo, attempt=5/5 failed=262836ops, last
> exception: java.io.IOException: java.io.IOException: Maximum lock count
> exceeded
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.Error: Maximum lock count exceeded
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1327)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:5163)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3018)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2877)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:715)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2148)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more
> {code}
> While we are still examining the data pattern, it is sure that there are too
> many mutations in the batch against the same row, this exceeds the maximum
> 64k shared lock count and it throws an error and failed the whole batch.
> There are two approaches to solve this issue.
> 1). Let's say there are mutations against the same row in the batch, we just
> need to acquire the lock once for the same row vs to acquire the lock for
> each mutation.
> 2). We catch the error and start to process whatever it gets and loop back.
> With HBASE-17924, approach 1 seems easy to implement now.
> Create the jira and will post update/patch when investigation moving forward.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
