[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937468#comment-16937468 ] Francis Liu commented on HBASE-11288: - Let me write one up tomorrow [~stack] and [~ram_krish]. Apologies I decided it was better to push stuff out in pieces than taking ages to get everything out in one go. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936585#comment-16936585 ] Francis Liu edited comment on HBASE-11288 at 9/24/19 9:16 AM: -- Here's a WIP branch: [https://github.com/francisliu/hbase/tree/apache_1.3_splitmeta] I'm working on stripping down most of the changes to only what's needed to split meta. The rest can come as follow-ons if needed. At it's core there's actually not much change, the patch is partly big because there's a lot of renames which are touching a lot of files. So with this branch we can get a very good picture of the changes needed. I'm still running the unit tests. Will move this to a feature branch in the hbase repo as soon as most of the tests are passing. I'll add more info tomorrow. Next question is what we want to do. Should I get everything we want working on the 1.3 branch or should I port this to trunk or 1.x or? was (Author: toffer): Here's a WIP branch: [https://github.com/francisliu/hbase/tree/apache_1.3_splitmeta] I'm working on stripping down most of the changes to only what's needed to split meta. The rest can come as follow-ons if needed. At it's core there's actually not much change, the patch is partly big because there's a lot of renames which are touching a lot of files. So with this branch we can get a very good picture of the changes needed. I'm still running the unit tests. Will move this to a feature branch in the hbase repo as soon as most of the tests are passing. Next question is what we want to do. Should I get everything we want working on the 1.3 branch or should I port this to trunk or 1.x or? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reopened HBASE-11288: - > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936585#comment-16936585 ] Francis Liu commented on HBASE-11288: - Here's a WIP branch: [https://github.com/francisliu/hbase/tree/apache_1.3_splitmeta] I'm working on stripping down most of the changes to only what's needed to split meta. The rest can come as follow-ons if needed. At it's core there's actually not much change, the patch is partly big because there's a lot of renames which are touching a lot of files. So with this branch we can get a very good picture of the changes needed. I'm still running the unit tests. Will move this to a feature branch in the hbase repo as soon as most of the tests are passing. Next question is what we want to do. Should I get everything we want working on the 1.3 branch or should I port this to trunk or 1.x or? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934773#comment-16934773 ] Francis Liu commented on HBASE-11288: - [~stack] can we reopen this? I'll put up what we're running with this weekend. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21903) Backport major compaction tool HBASE-19528 from to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-21903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-21903: Fix Version/s: (was: 1.3.4) 1.3.5 > Backport major compaction tool HBASE-19528 from to 1.4 and 1.3 > -- > > Key: HBASE-21903 > URL: https://issues.apache.org/jira/browse/HBASE-21903 > Project: HBase > Issue Type: Task > Components: Client, Compaction, tooling >Affects Versions: 1.3.3, 1.4.9 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.4.10, 1.3.5 > > Attachments: HBASE-21903-branch-1.3-addendum.patch > > > Our internal deployments are based on branch-1.3. We will be using the major > compaction tool HBASE-19528 from [~churromorales] and the enhancements on top > of it HBASE-21883 on our 1.3 clusters. I would like to backport HBASE-19528 > to 1.3 and hence 1.4 as well. Since its a standalone tool without any other > dependency or code changes, I believe that should be ok. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22225) Profiler tab on Master/RS UI not working w/o comprehensive message
[
https://issues.apache.org/jira/browse/HBASE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-5:
Fix Version/s: (was: 1.3.4)
1.3.5
> Profiler tab on Master/RS UI not working w/o comprehensive message
> --
>
> Key: HBASE-5
> URL: https://issues.apache.org/jira/browse/HBASE-5
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Affects Versions: 1.5.0
>Reporter: Yu Li
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0, 1.2.12, 2.1.5, 1.3.5
>
>
> As titled, when checking 1.5.0 RC3 binary package, clicking the "Profiler"
> tab on HMaster/RegionServer web UI, it complains page not found error like
> below:
> {noformat}
> Problem accessing /prof. Reason:
> NOT_FOUND
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-20993:
Fix Version/s: (was: 1.3.4)
1.3.5
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC, security
>Affects Versions: 1.2.6, 1.3.2, 1.2.7, 1.4.7
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 1.5.0, 1.4.10, 1.3.5
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.010.patch, HBASE-20993.branch-1.011.patch,
> HBASE-20993.branch-1.012.patch, HBASE-20993.branch-1.2.001.patch,
> HBASE-20993.branch-1.wip.002.patch, HBASE-20993.branch-1.wip.patch,
> yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
>
[jira] [Updated] (HBASE-16488) Starting namespace and quota services in master startup asynchronously
[ https://issues.apache.org/jira/browse/HBASE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-16488: Fix Version/s: (was: 1.3.4) 1.3.5 > Starting namespace and quota services in master startup asynchronously > -- > > Key: HBASE-16488 > URL: https://issues.apache.org/jira/browse/HBASE-16488 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2, 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Xu Cang >Priority: Major > Fix For: 1.5.0, 1.4.10, 1.3.5 > > Attachments: HBASE-16488.branch-1.012.patch, > HBASE-16488.branch-1.012.patch, HBASE-16488.revisit.v11-branch-1.patch, > HBASE-16488.v1-branch-1.patch, HBASE-16488.v1-master.patch, > HBASE-16488.v10-branch-1.patch, HBASE-16488.v2-branch-1.patch, > HBASE-16488.v2-branch-1.patch, HBASE-16488.v3-branch-1.patch, > HBASE-16488.v3-branch-1.patch, HBASE-16488.v4-branch-1.patch, > HBASE-16488.v5-branch-1.patch, HBASE-16488.v6-branch-1.patch, > HBASE-16488.v7-branch-1.patch, HBASE-16488.v8-branch-1.patch, > HBASE-16488.v9-branch-1.patch > > > From time to time, during internal IT test and from customer, we often see > master initialization failed due to namespace table region takes long time to > assign (eg. sometimes split log takes long time or hanging; or sometimes RS > is temporarily not available; sometimes due to some unknown assignment > issue). In the past, there was some proposal to improve this situation, eg. > HBASE-13556 / HBASE-14190 (Assign system tables ahead of user region > assignment) or HBASE-13557 (Special WAL handling for system tables) or > HBASE-14623 (Implement dedicated WAL for system tables). > This JIRA proposes another way to solve this master initialization fail > issue: namespace service is only used by a handful operations (eg. create > table / namespace DDL / get namespace API / some RS group DDL). Only quota > manager depends on it and quota management is off by default. Therefore, > namespace service is not really needed for master to be functional. So we > could start namespace service asynchronizely without blocking master startup. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817482#comment-16817482
]
Francis Liu commented on HBASE-21492:
-
Cherry-picked to branch-1.3. Looks like it's missing.
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 1.2.10, 2.0.4, 1.4.10, 1.3.4
>
> Attachments: HBASE-21492-branch-1.patch, HBASE-21492.1.patch,
> HBASE-21492.2.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-21492:
Fix Version/s: 1.3.4
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 1.2.10, 2.0.4, 1.4.10, 1.3.4
>
> Attachments: HBASE-21492-branch-1.patch, HBASE-21492.1.patch,
> HBASE-21492.2.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21903) Backport major compaction tool HBASE-19528 from to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-21903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808363#comment-16808363 ] Francis Liu edited comment on HBASE-21903 at 4/3/19 4:49 AM: - [~busbey] are you suggesting we create a branch-1 in operator tools and put this in there? It seems more intuitive to keep this in the hbase repo given that this tool is already in branch-1.5? was (Author: toffer): [~busbey] are you suggesting we create a branch-1 in operator tools and put this in there? > Backport major compaction tool HBASE-19528 from to 1.4 and 1.3 > -- > > Key: HBASE-21903 > URL: https://issues.apache.org/jira/browse/HBASE-21903 > Project: HBase > Issue Type: Task > Components: Client, Compaction, tooling >Affects Versions: 1.3.3, 1.4.9 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-21903-branch-1.3-addendum.patch > > > Our internal deployments are based on branch-1.3. We will be using the major > compaction tool HBASE-19528 from [~churromorales] and the enhancements on top > of it HBASE-21883 on our 1.3 clusters. I would like to backport HBASE-19528 > to 1.3 and hence 1.4 as well. Since its a standalone tool without any other > dependency or code changes, I believe that should be ok. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21903) Backport major compaction tool HBASE-19528 from to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-21903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808363#comment-16808363 ] Francis Liu commented on HBASE-21903: - [~busbey] are you suggesting we create a branch-1 in operator tools and put this in there? > Backport major compaction tool HBASE-19528 from to 1.4 and 1.3 > -- > > Key: HBASE-21903 > URL: https://issues.apache.org/jira/browse/HBASE-21903 > Project: HBase > Issue Type: Task > Components: Client, Compaction, tooling >Affects Versions: 1.3.3, 1.4.9 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-21903-branch-1.3-addendum.patch > > > Our internal deployments are based on branch-1.3. We will be using the major > compaction tool HBASE-19528 from [~churromorales] and the enhancements on top > of it HBASE-21883 on our 1.3 clusters. I would like to backport HBASE-19528 > to 1.3 and hence 1.4 as well. Since its a standalone tool without any other > dependency or code changes, I believe that should be ok. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Summary: upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2 (was: upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3) > upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4, 1.2.12 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801031#comment-16801031 ] Francis Liu commented on HBASE-22058: - Pushed to branch-1.2 as well. > upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801031#comment-16801031 ] Francis Liu edited comment on HBASE-22058 at 3/25/19 8:26 PM: -- Cherry-picked to branch-1.2 as well. was (Author: toffer): Pushed to branch-1.2 as well. > upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4, 1.2.12 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Resolution: Fixed Status: Resolved (was: Patch Available) > upgrade thrift dependency to 0.9.3.1 on branches 1.4, 1.3 and 1.2 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4, 1.2.12 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Fix Version/s: 1.2.12 > upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4, 1.2.12 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800900#comment-16800900 ] Francis Liu commented on HBASE-22058: - Pushed to 1.4 and cherry-picked to 1.3. I went with the pom-only change since that was the intent of the thrift release. Thanks for the review guys! [~busbey] since the change is fairly small do you want this backported to 1.2 as well? > upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Summary: upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3 (was: backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3) > upgrade thrift dependency to 0.9.3.1 on branch-1.4 and branch-1.3 > -- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Attachment: HBASE-22058.branch-1.4.002.patch > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797742#comment-16797742 ] Francis Liu commented on HBASE-22058: - Decided to attach a patch instead to make things clear. > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch, HBASE-22058.branch-1.4.002.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797736#comment-16797736
]
Francis Liu edited comment on HBASE-22058 at 3/21/19 2:25 AM:
--
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) with only the
security fix as the change, hence no need to regenerate thrift classes and from
what I understand the source release tarball is not expected to build properly.
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I did manage to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
was (Author: toffer):
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) with only the
security fix as the change, hence no need to regenerate thrift classes and from
what I understand the package is not expected to build properly.
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I did manage to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
> backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and
> 1.3
> ---
>
> Key: HBASE-22058
> URL: https://issues.apache.org/jira/browse/HBASE-22058
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Major
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-22058-branch-1.4.patch,
> HBASE-22058.branch-1.4.001.patch
>
>
> Creating a separate Jira to do the backport since the .thrift files differ
> between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from
> branch-1 and regenerated the thrift configs.
> cc [~apurtell]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797736#comment-16797736
]
Francis Liu edited comment on HBASE-22058 at 3/21/19 2:24 AM:
--
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) with only the
security fix as the change, hence no need to regenerate thrift classes and from
what I understand the package is not expected to build properly.
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I did manage to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
was (Author: toffer):
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) hence no need
to regenerate thrift classes and from what I understand the package is not
expected to build properly.
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I did manage to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
> backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and
> 1.3
> ---
>
> Key: HBASE-22058
> URL: https://issues.apache.org/jira/browse/HBASE-22058
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Major
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-22058-branch-1.4.patch,
> HBASE-22058.branch-1.4.001.patch
>
>
> Creating a separate Jira to do the backport since the .thrift files differ
> between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from
> branch-1 and regenerated the thrift configs.
> cc [~apurtell]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797736#comment-16797736
]
Francis Liu edited comment on HBASE-22058 at 3/21/19 2:19 AM:
--
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) hence no need
to regenerate thrift classes and from what I understand the package is not
expected to build properly.
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I did manage to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
was (Author: toffer):
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) hence no need
to regenerate thrift classes and from what I understand the package is not
expected to build properly. F
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I managed to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
> backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and
> 1.3
> ---
>
> Key: HBASE-22058
> URL: https://issues.apache.org/jira/browse/HBASE-22058
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Major
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-22058-branch-1.4.patch,
> HBASE-22058.branch-1.4.001.patch
>
>
> Creating a separate Jira to do the backport since the .thrift files differ
> between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from
> branch-1 and regenerated the thrift configs.
> cc [~apurtell]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797736#comment-16797736
]
Francis Liu commented on HBASE-22058:
-
[~apurtell] I took a look at the relevant thrift jira. The release in essence
is a java language only release (release thrift library to maven) hence no need
to regenerate thrift classes and from what I understand the package is not
expected to build properly. F
Here is the comment that seems to sum things up:
https://issues.apache.org/jira/browse/THRIFT-4506?focusedCommentId=16788697=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788697
If this sounds good to you I can update the thrift version dependency and
hardcode the version being checked in fgrep to 0.9.3 instead of
${thrift.version}. FWIW after some hacking I managed to create the thrift
generator binary but the thrift version wasn't changed as part of the 0.9.3-1
release so it is still shows "Thrift version 0.9.3" when you run "thrift
-version".
> backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and
> 1.3
> ---
>
> Key: HBASE-22058
> URL: https://issues.apache.org/jira/browse/HBASE-22058
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Major
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-22058-branch-1.4.patch,
> HBASE-22058.branch-1.4.001.patch
>
>
> Creating a separate Jira to do the backport since the .thrift files differ
> between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from
> branch-1 and regenerated the thrift configs.
> cc [~apurtell]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796646#comment-16796646 ] Francis Liu edited comment on HBASE-22058 at 3/20/19 12:21 AM: --- I had trouble building the thrift binaries for 0.9.3-1 (the build seems broken) but looking at the changes that went into this release: [https://github.com/apache/thrift/commits/a9b748bb0e02a2b6aaa3f39d09ec7f1fa47a0cf4] Theres only 1 relevant commit and it does not seem to be involved in the actual code generation. If you guys agree, then we can commit this patch without regenerating the thrift classes? I've ran the hbase-thrift unit tests and they ran fine. cc [~apurtell] was (Author: toffer): I had trouble building the thrift binaries for 0.9.3-1 (the build seems broken) but looking at the changes that went into this release: [https://github.com/apache/thrift/commits/a9b748bb0e02a2b6aaa3f39d09ec7f1fa47a0cf4] Theres only 1 relevant commit and it does not seem to be involved in the actual code generation. If you guys are okay with this then we can commit this patch without regenerating the thrift classes. I've ran the hbase-thrift unit tests and they ran fine. cc [~apurtell] > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Status: Patch Available (was: Open) > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796646#comment-16796646 ] Francis Liu commented on HBASE-22058: - I had trouble building the thrift binaries for 0.9.3-1 (the build seems broken) but looking at the changes that went into this release: [https://github.com/apache/thrift/commits/a9b748bb0e02a2b6aaa3f39d09ec7f1fa47a0cf4] Theres only 1 relevant commit and it does not seem to be involved in the actual code generation. If you guys are okay with this then we can commit this patch without regenerating the thrift classes. I've ran the hbase-thrift unit tests and they ran fine. cc [~apurtell] > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Attachment: (was: HBASE-22058-branch-1.4.001.patch) > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Attachment: HBASE-22058.branch-1.4.001.patch > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch, > HBASE-22058.branch-1.4.001.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Attachment: HBASE-22058-branch-1.4.001.patch > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.001.patch, > HBASE-22058-branch-1.4.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796488#comment-16796488
]
Francis Liu commented on HBASE-22058:
-
{quote}
why not just go to 0.9.3-1 in these maintenance branches? going to 0.12 seems
high risk to me.
{quote}
Good idea wasn't aware that there was a release recently.
> backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and
> 1.3
> ---
>
> Key: HBASE-22058
> URL: https://issues.apache.org/jira/browse/HBASE-22058
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Major
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-22058-branch-1.4.patch
>
>
> Creating a separate Jira to do the backport since the .thrift files differ
> between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from
> branch-1 and regenerated the thrift configs.
> cc [~apurtell]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-22058: Attachment: HBASE-22058-branch-1.4.patch > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22058-branch-1.4.patch > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
Francis Liu created HBASE-22058: --- Summary: backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3 Key: HBASE-22058 URL: https://issues.apache.org/jira/browse/HBASE-22058 Project: HBase Issue Type: Bug Components: Thrift Reporter: Francis Liu Fix For: 1.4.10, 1.3.4 Creating a separate Jira to do the backport since the .thrift files differ between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from branch-1 and regenerated the thrift configs. cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-22058) backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and 1.3
[ https://issues.apache.org/jira/browse/HBASE-22058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-22058: --- Assignee: Francis Liu > backport HBASE-HBASE-21791 (Upgrade thrift dependency to 0.12.0) to 1.4 and > 1.3 > --- > > Key: HBASE-22058 > URL: https://issues.apache.org/jira/browse/HBASE-22058 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.4.10, 1.3.4 > > > Creating a separate Jira to do the backport since the .thrift files differ > between branch-1 and 1.4, 1.3. I backported the change in the pom.xml from > branch-1 and regenerated the thrift configs. > cc [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21791) Upgrade thrift dependency to 0.12.0
[ https://issues.apache.org/jira/browse/HBASE-21791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763918#comment-16763918 ] Francis Liu commented on HBASE-21791: - Thanks [~apurtell]. I'll do that. > Upgrade thrift dependency to 0.12.0 > --- > > Key: HBASE-21791 > URL: https://issues.apache.org/jira/browse/HBASE-21791 > Project: HBase > Issue Type: Task > Components: Thrift >Affects Versions: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.9, 2.1.2, 1.2.10, 2.0.4 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.3, 2.0.5, 2.3.0 > > Attachments: HBASE-21791-branch-1.patch, > HBASE-21791-branch-2.1.patch, HBASE-21791.patch > > > As somebody have already known, that there is a CVE for thrift from 0.5.0 to > 0.11.0. > https://nvd.nist.gov/vuln/detail/CVE-2018-1320 > As the CVE is already public, let's upgrade our thrift dependency and release > new versions ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21791) Upgrade thrift dependency to 0.12.0
[ https://issues.apache.org/jira/browse/HBASE-21791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761038#comment-16761038 ] Francis Liu commented on HBASE-21791: - I'd like to backport this to 1.3. Do I need to wait for this to be backported to 1.4? > Upgrade thrift dependency to 0.12.0 > --- > > Key: HBASE-21791 > URL: https://issues.apache.org/jira/browse/HBASE-21791 > Project: HBase > Issue Type: Task > Components: Thrift >Affects Versions: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.9, 2.1.2, 1.2.10, 2.0.4 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.3, 2.0.5, 2.3.0 > > Attachments: HBASE-21791-branch-1.patch, > HBASE-21791-branch-2.1.patch, HBASE-21791.patch > > > As somebody have already known, that there is a CVE for thrift from 0.5.0 to > 0.11.0. > https://nvd.nist.gov/vuln/detail/CVE-2018-1320 > As the CVE is already public, let's upgrade our thrift dependency and release > new versions ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21667) Move to latest ASF Parent POM
[ https://issues.apache.org/jira/browse/HBASE-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754494#comment-16754494 ] Francis Liu commented on HBASE-21667: - I'm fine as well. Same sentiments as Andy. > Move to latest ASF Parent POM > - > > Key: HBASE-21667 > URL: https://issues.apache.org/jira/browse/HBASE-21667 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21667.branch-2.0.001.patch, > HBASE-21667.branch-2.0.002.patch, HBASE-21667.branch-2.001.patch, > HBASE-21667.master.001.patch, HBASE-21667.master.002.patch > > > Currently HBase depends on version 18 which was released on 2016-05-18. > Version 21 was released in August 2018. > Relevant dependency upgrades > > ||Name||Currently used version||New version||Notes|| > |surefire.version|2.21.0|2.22.0| | > |maven-compiler-plugin|3.6.1|3.7| | > |maven-dependency-plugin|3.0.1|3.1.1| | > |maven-jar-plugin|3.0.0|3.0.2| | > |maven-javadoc-plugin|3.0.0|3.0.1| | > |maven-resources-plugin|2.7|3.1.0| | > |maven-site-plugin|3.4|3.7.1|Currently not relying on ASF version. See: > HBASE-18333| > |maven-source-plugin|3.0.0|3.0.1| | > |maven-shade-plugin|3.0.0|3.1.1|Newly added to ASF pom| > |maven-clean-plugin|3.0.0|3.1.0| | > |maven-project-info-reports-plugin |2.9|3.0.0| | > Version 21 added net.nicoulaj.maven.plugins:checksum-maven-plugin which > introduced SHA512 checksum instead of SHA1. Should verify if we can rely on > that for releases or breaks our current processes. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-12233) Bring back root table
[ https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722337#comment-16722337 ] Francis Liu commented on HBASE-12233: - Working on this. > Bring back root table > - > > Key: HBASE-12233 > URL: https://issues.apache.org/jira/browse/HBASE-12233 > Project: HBase > Issue Type: Sub-task >Reporter: Virag Kothari >Assignee: Francis Liu >Priority: Major > Attachments: HBASE-12233.patch > > > First step towards splitting meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-12233) Bring back root table
[ https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-12233: --- Assignee: Francis Liu (was: Virag Kothari) > Bring back root table > - > > Key: HBASE-12233 > URL: https://issues.apache.org/jira/browse/HBASE-12233 > Project: HBase > Issue Type: Sub-task >Reporter: Virag Kothari >Assignee: Francis Liu >Priority: Major > Attachments: HBASE-12233.patch > > > First step towards splitting meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Fix Version/s: 1.4.8 1.3.3 1.5.0 > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch, HBASE-20704.007.patch, HBASE-20704.branch-1.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16619988#comment-16619988 ] Francis Liu commented on HBASE-20704: - [~apurtell] attached branch-1 patch. It's simpler but a bit different note that I had to backport a change in FSDataInputStreamWrapper so an IOException is thrown. Let me know if this looks good. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch, HBASE-20704.007.patch, HBASE-20704.branch-1.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.branch-1.001.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch, HBASE-20704.007.patch, HBASE-20704.branch-1.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616975#comment-16616975 ] Francis Liu commented on HBASE-20704: - Failing tests passed locally doesn't seem related. Committed to master, branch-2, branch-2.1 and branch-2.0. Need a slightly simpler patch for 1.x > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch, HBASE-20704.007.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609791#comment-16609791 ] Francis Liu commented on HBASE-20704: - Uploading exaclty the same file with bumped up rev number to kick of buildbot > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch, HBASE-20704.007.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.007.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch, HBASE-20704.007.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597696#comment-16597696 ] Francis Liu commented on HBASE-20704: - Sounds good the 1.x patch will be much simpler will post one for that as soon as this passes buildbot > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Status: Patch Available (was: Open) resubmitting to kick buildbot > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Status: Open (was: Patch Available) > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-12233) Bring back root table
[ https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596821#comment-16596821 ] Francis Liu commented on HBASE-12233: - Will do [~stack]. Let me clean it up a bit and put it up as a feature branch and RB. Yeah that sounds like a good change. Meta regions are few tho so it's more reducing complexity than perf? Internally we'd have to be able to migrate to the change. > Bring back root table > - > > Key: HBASE-12233 > URL: https://issues.apache.org/jira/browse/HBASE-12233 > Project: HBase > Issue Type: Sub-task >Reporter: Virag Kothari >Assignee: Virag Kothari >Priority: Major > Attachments: HBASE-12233.patch > > > First step towards splitting meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596815#comment-16596815 ] Francis Liu commented on HBASE-20704: - Thanks for the review! Addressed comments tho for one comment instead of elevating warn to error I threw an exception instead. The situation seems important enough for the server to fail (possible consistency issue). If buildbot passes will commit end of this week. Let me know if you have any concerns [~apurtell]. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.006.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, > HBASE-20704.006.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-12233) Bring back root table
[
https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590673#comment-16590673
]
Francis Liu commented on HBASE-12233:
-
{quote}New clients would look for a new root znode. It'd point to a root table,
a table that would never split. It'd have the list of meta regions as it did in
the past. We'd try and avoid mistakes of the past regards stuff like key scheme
and wacky comparator to interpret it. We'd not disturb the current meta znodes.
{quote}
Oh and we used the original root comparators. Would be interested to know what
you meant by using a different key scheme.
> Bring back root table
> -
>
> Key: HBASE-12233
> URL: https://issues.apache.org/jira/browse/HBASE-12233
> Project: HBase
> Issue Type: Sub-task
>Reporter: Virag Kothari
>Assignee: Virag Kothari
>Priority: Major
> Attachments: HBASE-12233.patch
>
>
> First step towards splitting meta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12233) Bring back root table
[ https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590617#comment-16590617 ] Francis Liu commented on HBASE-12233: - Thanks for the interest in this [~stack]. We have been running with root+split meta since 0.98 and now 1.3. We've done all the work you mentioned except for the facade you mentioned (and prolly more). We went with the approach [~apurtell] suggested (a fork in the code to detect wether root or meta is the starting table something we also talked about many moons ago. If you are interested we can push up the 1.3 code into a feature branch. I've have a partially working master branch port but it's I got distractraced getting 1.3 stable internally. Now I believe I have time, would you guys have time to look at a master patch? > Bring back root table > - > > Key: HBASE-12233 > URL: https://issues.apache.org/jira/browse/HBASE-12233 > Project: HBase > Issue Type: Sub-task >Reporter: Virag Kothari >Assignee: Virag Kothari >Priority: Major > Attachments: HBASE-12233.patch > > > First step towards splitting meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Fix Version/s: 2.0.2 2.1.1 2.2.0 3.0.0 > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.2 > > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-12233) Bring back root table
[ https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590617#comment-16590617 ] Francis Liu edited comment on HBASE-12233 at 8/23/18 6:06 PM: -- Thanks for the interest in this [~stack]. We have been running with root+split meta since 0.98 and now 1.3. We've done all the work you mentioned except for the facade you mentioned (and prolly more). We went with the approach [~apurtell] suggested (a fork in the code to detect wether root or meta is the starting table something you and I also talked about many moons ago. If you are interested we can push up the 1.3 code into a feature branch. I've have a partially working master branch port but it's I got distractraced getting 1.3 stable internally. Now I believe I have time, would you guys have time to look at a master patch? was (Author: toffer): Thanks for the interest in this [~stack]. We have been running with root+split meta since 0.98 and now 1.3. We've done all the work you mentioned except for the facade you mentioned (and prolly more). We went with the approach [~apurtell] suggested (a fork in the code to detect wether root or meta is the starting table something we also talked about many moons ago. If you are interested we can push up the 1.3 code into a feature branch. I've have a partially working master branch port but it's I got distractraced getting 1.3 stable internally. Now I believe I have time, would you guys have time to look at a master patch? > Bring back root table > - > > Key: HBASE-12233 > URL: https://issues.apache.org/jira/browse/HBASE-12233 > Project: HBase > Issue Type: Sub-task >Reporter: Virag Kothari >Assignee: Virag Kothari >Priority: Major > Attachments: HBASE-12233.patch > > > First step towards splitting meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577823#comment-16577823 ] Francis Liu commented on HBASE-20704: - [~apurtell] Apologies for the delay was thinking through possible race conditions. Updated patch as discussed and uploaded it to RB link as well. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.005.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571977#comment-16571977
]
Francis Liu commented on HBASE-20704:
-
{quote}The v4 patch looks like the right thing to do, IMHO.
{quote}
Sounds good will go with this then.
{quote}This test isn't actually waiting for anything to happen, right?
{{store.closeAndArchiveCompactedFiles(true);}}completed all actions at that
step. Or otherwise this could flake.
{quote}
If you're talking about that particular section. It's just making sure that
we're getting IOExceptions (ie isntead of NPEs or not at all) which confirms
that the listener hook is working and our expectations on the type of exception
(IOException is thrown).
Let me clean this up then and put it on RB. Thanks for getting back to it.
> Sometimes some compacted storefiles are not archived on region close
>
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch,
> HBASE-20704.003.patch, HBASE-20704.004.draft.patch
>
>
> During region close compacted files which have not yet been archived by the
> discharger are archived as part of the region closing process. It is
> important that these files are wholly archived to insure data consistency. ie
> a storefile containing delete tombstones can be archived while older
> storefiles containing cells that were supposed to be deleted are left
> unarchived thereby undeleting those cells.
> On region close a compacted storefile is skipped from archiving if it has
> read references (ie open scanners). This behavior is correct for when the
> discharger chore runs but on region close consistency is of course more
> important so we should add a special case to ignore any references on the
> storefile and go ahead and archive it.
> Attached patch contains a unit test that reproduces the problem and the
> proposed fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553681#comment-16553681 ] Francis Liu commented on HBASE-20704: - [~apurtell] any chance you have time to have a look? > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545873#comment-16545873
]
Francis Liu edited comment on HBASE-20704 at 7/17/18 12:14 AM:
---
{quote}expecting eventual GC to call a finalizer that cleans things up
{quote}
AFAIK it should get cleaned up either via another next() rpc (that fails
bacause the region is cloased) or scanner lease expiration processing. The
readers won't be garbage until the scanner state is cleaned up. In any case it
would objects that would give gc more work, tho it doesn't sounds like it's
going to be significant and generally just part of normal operation. ie scan
lease expiring and pauses between next() rpc calls.
The trade off is tho now we have to have concurrent threads access a map during
storefilescanner creation and and close for streaming scans. The overhead may
be negligible assuming streaming scans are meant for doing large scans. I've
attached a rough patch on how it would look. Let me know what you think.
was (Author: toffer):
{quote}expecting eventual GC to call a finalizer that cleans things up
{quote}
AFAIK it should get cleaned up either via another next() rpc (that fails
bacause the region is cloased) or scanner lease expiration processing. The
readers won't be garbage until the scanner state is cleaned up. In any case it
would objects that would give gc more work.
The trade off is tho now we have to have concurrent threads access a map during
storefilescanner creation and and close for streaming scans. The overhead may
be negligible assuming streaming scans are meant for doing large scans. I've
attached a rough patch on how it would look. Let me know what you think.
> Sometimes some compacted storefiles are not archived on region close
>
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch,
> HBASE-20704.003.patch, HBASE-20704.004.draft.patch
>
>
> During region close compacted files which have not yet been archived by the
> discharger are archived as part of the region closing process. It is
> important that these files are wholly archived to insure data consistency. ie
> a storefile containing delete tombstones can be archived while older
> storefiles containing cells that were supposed to be deleted are left
> unarchived thereby undeleting those cells.
> On region close a compacted storefile is skipped from archiving if it has
> read references (ie open scanners). This behavior is correct for when the
> discharger chore runs but on region close consistency is of course more
> important so we should add a special case to ignore any references on the
> storefile and go ahead and archive it.
> Attached patch contains a unit test that reproduces the problem and the
> proposed fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545873#comment-16545873
]
Francis Liu commented on HBASE-20704:
-
{quote}expecting eventual GC to call a finalizer that cleans things up
{quote}
AFAIK it should get cleaned up either via another next() rpc (that fails
bacause the region is cloased) or scanner lease expiration processing. The
readers won't be garbage until the scanner state is cleaned up. In any case it
would objects that would give gc more work.
The trade off is tho now we have to have concurrent threads access a map during
storefilescanner creation and and close for streaming scans. The overhead may
be negligible assuming streaming scans are meant for doing large scans. I've
attached a rough patch on how it would look. Let me know what you think.
> Sometimes some compacted storefiles are not archived on region close
>
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch,
> HBASE-20704.003.patch, HBASE-20704.004.draft.patch
>
>
> During region close compacted files which have not yet been archived by the
> discharger are archived as part of the region closing process. It is
> important that these files are wholly archived to insure data consistency. ie
> a storefile containing delete tombstones can be archived while older
> storefiles containing cells that were supposed to be deleted are left
> unarchived thereby undeleting those cells.
> On region close a compacted storefile is skipped from archiving if it has
> read references (ie open scanners). This behavior is correct for when the
> discharger chore runs but on region close consistency is of course more
> important so we should add a special case to ignore any references on the
> storefile and go ahead and archive it.
> Attached patch contains a unit test that reproduces the problem and the
> proposed fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.004.draft.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch, HBASE-20704.004.draft.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.003.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540972#comment-16540972 ] Francis Liu commented on HBASE-20704: - OK I did some testing NPE only actually occurs in the 1.x. For the master branch it's one of two scenarios depending on wether a pread was used or not as stream scanners open their own inputstream, reader, etc. # pread - you'll get a "stream closed" IOException. In 1.x it's an NPE because the stream references are set to null after the streams are closed. # not pread - Since the HStore has no knowledge of the created stream it does not close them. What happens is either the current running scan request is processed or it will get an IOException (replica not found), since the region close operation may have archived and the cleaner chore deleted the file. It sounds to me that #1 is good and #2 is acceptable? Let me know. If #2 is not acceptable I'd have to add a map to keep track of the new readers created when a stream scanner is created and close those on region close so exceptions will be like #1. I've attached another patch which add another test method to surface the exceptions. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528095#comment-16528095 ] Francis Liu commented on HBASE-20704: - [~apurtell] Thanks for the pointers. Sounds good I can look at these for long term solutions. In the meantime I'll work on a fix that we can apply to all affected branches given this is a critical bug. Does that sound ok? > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526933#comment-16526933 ] Francis Liu edited comment on HBASE-20704 at 6/29/18 12:05 AM: --- Hmm actually even if we add the list of parent storefiles in there's still a corner case during regionserver failover that we won't cover (ie a a dead RS commits a compacted storefile before aborting and the region has already been opened elsewhere). It seems the more straightforward and intuitive way to solve this is the current proposed way of closing and cleaning up the compacted storefiles on close. And make sure the still relevant compaction markers are replayed on the region to address HBASE-20724. Let me go down this route and see how that goes. Long term tho I think the better solution is to keep the list of active (and possibly compacted) storefiles in meta. That way we can update the changes atomically. Tho that will probably only be a viable option once splitting meta is available. was (Author: toffer): Hmm actually even if at add the list of parent storefiles in there's still a corner case during regionserver failover that we won't cover (ie a a dead RS commits a compacted storefile before aborting and the region has already been opened elsewhere). It seems the more straightforward and intuitive way to solve this is the current proposed way of closing and cleaning up the compacted storefiles on close. And make sure the still relevant compaction markers are replayed on region for HBASE-20724. Let me go down this route and see how that goes. Long term tho I think the better solution is to keep the list of active (and possibly compacted) storefiles in meta. That way we can update the changes atomically. Tho that will probably only be a viable option once splitting meta is available. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526933#comment-16526933 ] Francis Liu commented on HBASE-20704: - Hmm actually even if at add the list of parent storefiles in there's still a corner case during regionserver failover that we won't cover (ie a a dead RS commits a compacted storefile before aborting and the region has already been opened elsewhere). It seems the more straightforward and intuitive way to solve this is the current proposed way of closing and cleaning up the compacted storefiles on close. And make sure the still relevant compaction markers are replayed on region for HBASE-20724. Let me go down this route and see how that goes. Long term tho I think the better solution is to keep the list of active (and possibly compacted) storefiles in meta. That way we can update the changes atomically. Tho that will probably only be a viable option once splitting meta is available. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525832#comment-16525832 ] Francis Liu commented on HBASE-20704: - OK let me see what I can do without adding to much complexity. For scanners the current expectation is that storefile readers won't get closed on them while they are using it. My initial look on this was that it would require checks in a bunch of places wether the stream has been closed. Just a bit concerned that would affect performance or make things even more complex. Tho thinking about this a bit more. I think we could address this and HBASE-20724 in the same manner. If we add in each created compacted storefile metadata about which storefiles were it's parents. So on the next region open we can remove the compacted storefiles as part of the region open operation. What do you guys think? > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515429#comment-16515429 ] Francis Liu commented on HBASE-20704: - For 1.x in the small window the region hasn't be set a closed by the RS yet the client will get an NPE when the scan tries to access the fs stream. The client will retry. For master branch it depends if the read is a pread or not. If it is a pread it will be the same as 1.x. If not then it has it's own reader in which case the file will be removed while the reader is open. I have not tried this but I believe it will end up getting a file not found exception once it hits the end of the hdfs block currently being read. In both cases the client should retry. Are these acceptable outcomes? Or do you want the situation to be explicitly handled? > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
[ https://issues.apache.org/jira/browse/HBASE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20724: Description: It is important that compacted storefiles of a given compaction execution are wholly opened or archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived thereby undeleting those cells. When a server fails compaction markers (in the wal edit) are used to determine which storefiles are compacted and should be excluded during region open (during failover). But the WALs containing compaction markers can be prematurely archived even though there are still compacted storefiles for that particular compaction event that hasn't been archived yet. Thus losing compaction information that needs to be replayed in the event of an RS crash. This is because hlog archiving logic only keeps track of flushed storefiles and not compacted ones. https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680 was: It is important that compacted storefiles of a given compaction execution are wholly opened or archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived thereby undeleting those cells. When a server fails compaction markers (in the wal edit) are used to determine which storefiles are compacted and should be excluded during region open (during failover). But the WALs containing compaction markers can be prematurely archived even though there are still compacted storefiles for that particular compaction event that hasn't been archived yet. Thus losing compaction information that needs to be replayed. This is because hlog archiving logic only keeps track of flushed storefiles and not archived ones. https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680 > Sometimes some compacted storefiles are still opened after region failover > -- > > Key: HBASE-20724 > URL: https://issues.apache.org/jira/browse/HBASE-20724 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > > It is important that compacted storefiles of a given compaction execution are > wholly opened or archived to insure data consistency. ie a storefile > containing delete tombstones can be archived while older storefiles > containing cells that were supposed to be deleted are left unarchived thereby > undeleting those cells. > When a server fails compaction markers (in the wal edit) are used to > determine which storefiles are compacted and should be excluded during region > open (during failover). But the WALs containing compaction markers can be > prematurely archived even though there are still compacted storefiles for > that particular compaction event that hasn't been archived yet. Thus losing > compaction information that needs to be replayed in the event of an RS crash. > This is because hlog archiving logic only keeps track of flushed storefiles > and not compacted ones. > https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510579#comment-16510579 ] Francis Liu edited comment on HBASE-20704 at 6/13/18 3:48 AM: -- Filed HBASE-20724 should we go ahead and independently review/merge this patch? was (Author: toffer): Filed https://issues.apache.org/jira/browse/HBASE-20724 > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510579#comment-16510579 ] Francis Liu commented on HBASE-20704: - Filed https://issues.apache.org/jira/browse/HBASE-20724 > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
Francis Liu created HBASE-20724: --- Summary: Sometimes some compacted storefiles are still opened after region failover Key: HBASE-20724 URL: https://issues.apache.org/jira/browse/HBASE-20724 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0 Reporter: Francis Liu It is important that compacted storefiles of a given compaction execution are wholly opened or archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived thereby undeleting those cells. When a server fails compaction markers (in the wal edit) are used to determine which storefiles are compacted and should be excluded during region open (during failover). But the WALs containing compaction markers can be prematurely archived even though there are still compacted storefiles for that particular compaction event that hasn't been archived yet. Thus losing compaction information that needs to be replayed. This is because hlog archiving logic only keeps track of flushed storefiles and not archived ones. https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
[ https://issues.apache.org/jira/browse/HBASE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-20724: --- Assignee: Francis Liu > Sometimes some compacted storefiles are still opened after region failover > -- > > Key: HBASE-20724 > URL: https://issues.apache.org/jira/browse/HBASE-20724 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > > It is important that compacted storefiles of a given compaction execution are > wholly opened or archived to insure data consistency. ie a storefile > containing delete tombstones can be archived while older storefiles > containing cells that were supposed to be deleted are left unarchived thereby > undeleting those cells. > When a server fails compaction markers (in the wal edit) are used to > determine which storefiles are compacted and should be excluded during region > open (during failover). But the WALs containing compaction markers can be > prematurely archived even though there are still compacted storefiles for > that particular compaction event that hasn't been archived yet. Thus losing > compaction information that needs to be replayed. This is because hlog > archiving logic only keeps track of flushed storefiles and not archived ones. > https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507680#comment-16507680
]
Francis Liu edited comment on HBASE-20704 at 6/11/18 5:03 AM:
--
{quote}Are we ensuring this always after this patch? What if the RS going down
in between a close so not all the compacted files are archived? This issue is
there with old impl also right (Archive immediately and not by the Discharger
thread)
{quote}
In both cases if the RS is not gracefully shutdown then the WAL will get
replayed and the compaction marker gets replayed thus making sure the compacted
files are never accessed. Or so I'd like to confidently say. But it seems that
even that part has a bug wherein the WAL containing the compaction marker thats
needs to be replayed can get archived as sequence id tracking for WAL is only
tied to memstore flushed, ignoring wether compaction archival for a given
compaction even has completed. The same can be said for when edits are replayed
on region open.
I can think of a few reasons why this was not observed (or not as much) during
pre-discharger versions. 1. Since we archive soon after compacting the window
for exposure is pretty small. 2. At least for the delete case assuming the
common case that the user does not mess with the timestamps. Since the
compacted storefiles are sorted by seq id and removed in sequence, the
storefiles containing rows that were deleted are removed first before the
storefiles containing the corresponding tombstones for those rows. With the
discharger we skip storefiles if they still have references.
So to sum things up with this other bug when a server aborts there's a
possibility some compacted storefiles (the ones not removed) can reopened by
the failover RS.
Should we address this issue here? Or create another jira? If another Jira then
in this one we can prolly add a partial fix wherein the discharger only removes
contiguous storefiles?
was (Author: toffer):
{quote}
Are we ensuring this always after this patch? What if the RS going down in
between a close so not all the compacted files are archived? This issue is
there with old impl also right (Archive immediately and not by the Discharger
thread)
{quote}
In both cases if the RS is not gracefully shutdown then the WAL will get
replayed and the compaction marker gets replayed thus making sure the compacted
files are never accessed. Or so I'd like to confidently say. But it seems that
even that part has a bug wherein the WAL containing the compaction marker thats
needs to be replayed can get archived as sequence id tracking for WAL is only
tied to memstore flushed, ignoring wether compaction archival for a given
compaction even has completed. The same can be said for when edits are replayed
on region open.
I can think of a few reasons why this was not observed (or not as much) during
pre-discharger versions. 1. Since we archive soon after compacting the window
for exposure is pretty small. 2. At least for the delete case assuming the
common case that the user does not mess with the timestamps. Since the
compacted storefiles are sorted by seq id and removed in sequence, the
storefiles containing rows that were deleted are removed first before the
storefiles containing the corresponding tombstones for those rows. With the
discharger we skip storefiles if they still have references.
Should we address this issue here? Or create another jira? If another Jira then
in this one we can prolly add a partial fix wherein the discharger only removes
contiguous storefiles?
> Sometimes some compacted storefiles are not archived on region close
>
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch
>
>
> During region close compacted files which have not yet been archived by the
> discharger are archived as part of the region closing process. It is
> important that these files are wholly archived to insure data consistency. ie
> a storefile containing delete tombstones can be archived while older
> storefiles containing cells that were supposed to be deleted are left
> unarchived thereby undeleting those cells.
> On region close a compacted storefile is skipped from archiving if it has
> read references (ie open scanners). This behavior is correct for when the
> discharger chore runs but on region close consistency is of course more
> important so we should add a special case to ignore any references
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507680#comment-16507680
]
Francis Liu commented on HBASE-20704:
-
{quote}
Are we ensuring this always after this patch? What if the RS going down in
between a close so not all the compacted files are archived? This issue is
there with old impl also right (Archive immediately and not by the Discharger
thread)
{quote}
In both cases if the RS is not gracefully shutdown then the WAL will get
replayed and the compaction marker gets replayed thus making sure the compacted
files are never accessed. Or so I'd like to confidently say. But it seems that
even that part has a bug wherein the WAL containing the compaction marker thats
needs to be replayed can get archived as sequence id tracking for WAL is only
tied to memstore flushed, ignoring wether compaction archival for a given
compaction even has completed. The same can be said for when edits are replayed
on region open.
I can think of a few reasons why this was not observed (or not as much) during
pre-discharger versions. 1. Since we archive soon after compacting the window
for exposure is pretty small. 2. At least for the delete case assuming the
common case that the user does not mess with the timestamps. Since the
compacted storefiles are sorted by seq id and removed in sequence, the
storefiles containing rows that were deleted are removed first before the
storefiles containing the corresponding tombstones for those rows. With the
discharger we skip storefiles if they still have references.
Should we address this issue here? Or create another jira? If another Jira then
in this one we can prolly add a partial fix wherein the discharger only removes
contiguous storefiles?
> Sometimes some compacted storefiles are not archived on region close
>
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch
>
>
> During region close compacted files which have not yet been archived by the
> discharger are archived as part of the region closing process. It is
> important that these files are wholly archived to insure data consistency. ie
> a storefile containing delete tombstones can be archived while older
> storefiles containing cells that were supposed to be deleted are left
> unarchived thereby undeleting those cells.
> On region close a compacted storefile is skipped from archiving if it has
> read references (ie open scanners). This behavior is correct for when the
> discharger chore runs but on region close consistency is of course more
> important so we should add a special case to ignore any references on the
> storefile and go ahead and archive it.
> Attached patch contains a unit test that reproduces the problem and the
> proposed fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Status: Patch Available (was: Open) Updated patch to fix small typo in the unit test > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.002.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Description: During region close compacted files which have not yet been archived by the discharger are archived as part of the region closing process. It is important that these files are wholly archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived thereby undeleting those cells. On region close a compacted storefile is skipped from archiving if it has read references (ie open scanners). This behavior is correct for when the discharger chore runs but on region close consistency is of course more important so we should add a special case to ignore any references on the storefile and go ahead and archive it. Attached patch contains a unit test that reproduces the problem and the proposed fix. was: During region close compacted files which have not yet been archived by the discharger are archived as part of the region closing process. It is important that these files are wholly archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived. On region close a compacted storefile is skipped from archiving if it has read references (ie open scanners). This behavior is correct for when the discharger chore runs but on region close consistency is of course more important so we should add a special case to ignore any references on the storefile and go ahead and archive it. Attached patch contains a unit test that reproduces the problem and the proposed fix. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Summary: Sometimes some compacted storefiles are not archived on region close (was: Sometimes compacted storefiles are not archived on region close) > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Description: During region close compacted files which have not yet been archived by the discharger are archived as part of the region closing process. It is important that these files are wholly archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived. On region close a compacted storefile is skipped from archiving if it has read references (ie open scanners). This behavior is correct for when the discharger chore runs but on region close consistency is of course more important so we should add a special case to ignore any references on the storefile and go ahead and archive it. Attached patch contains a unit test that reproduces the problem and the proposed fix. was: During region close compacted files which have not yet been archived by the discharger are archived as part of the region closing process. It is important that these files are wholly archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived. On region close a compacted storefile is skipped from archiving if it has read references (ie open scanners). This behavior is correct for when the discharger chore runs but on region close consistency is of course more important so we should add a special case to ignore any references on the storefile and go ahead and archive it. Attached patch contains a unit test that reproduces the problem. > Sometimes compacted storefiles are not archived on region close > --- > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20704) Sometimes compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-20704: --- Assignee: Francis Liu > Sometimes compacted storefiles are not archived on region close > --- > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.001.patch > Sometimes compacted storefiles are not archived on region close > --- > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Summary: Sometimes compacted storefiles are not archived on region close (was: Sometimes compacted storefiles are archived on region close) > Sometimes compacted storefiles are not archived on region close > --- > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Priority: Critical > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20704) Sometimes compacted storefiles are archived on region close
Francis Liu created HBASE-20704: --- Summary: Sometimes compacted storefiles are archived on region close Key: HBASE-20704 URL: https://issues.apache.org/jira/browse/HBASE-20704 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 2.0.0, 1.4.0, 1.3.0, 3.0.0, 1.5.0 Reporter: Francis Liu During region close compacted files which have not yet been archived by the discharger are archived as part of the region closing process. It is important that these files are wholly archived to insure data consistency. ie a storefile containing delete tombstones can be archived while older storefiles containing cells that were supposed to be deleted are left unarchived. On region close a compacted storefile is skipped from archiving if it has read references (ie open scanners). This behavior is correct for when the discharger chore runs but on region close consistency is of course more important so we should add a special case to ignore any references on the storefile and go ahead and archive it. Attached patch contains a unit test that reproduces the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18864) NullPointerException thrown when adding rows to a table from peer cluster, table with replication factor other than 0 or 1
[ https://issues.apache.org/jira/browse/HBASE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398126#comment-16398126 ] Francis Liu commented on HBASE-18864: - This broke branch-1, branch-1.3 (and prolly branch-1.4). [ERROR] testIllegalTableDescriptor(org.apache.hadoop.hbase.client.TestFromClientSide) Time elapsed: 25.94 s <<< ERROR! java.lang.IllegalArgumentException: Invalid value '-1' for REPLICATION_SCOPE. at org.apache.hadoop.hbase.client.TestFromClientSide.testIllegalTableDescriptor(TestFromClientSide.java:5580) Same for TestFromClientSideWithCoprocessor which is just a subclass. > NullPointerException thrown when adding rows to a table from peer cluster, > table with replication factor other than 0 or 1 > -- > > Key: HBASE-18864 > URL: https://issues.apache.org/jira/browse/HBASE-18864 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 1.3.0 >Reporter: smita >Assignee: Sakthi >Priority: Major > Labels: beginner > Fix For: 1.3.2, 1.2.7, 1.4.3 > > Attachments: hbase-18864.branch-1.2.001.patch, > hbase-18864.branch-1.2.002.patch, hbase-18864.branch-1.2.003.patch, > hbase-18864.branch-1.2.004.patch > > > Scenario: > = > add_peer > create a table > alter table with REPLICATION_SCOPE => '5' > enable table replication > login to peer cluster and try putting data to the table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18864) NullPointerException thrown when adding rows to a table from peer cluster, table with replication factor other than 0 or 1
[ https://issues.apache.org/jira/browse/HBASE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-18864: Fix Version/s: (was: 1.3.3) 1.3.2 > NullPointerException thrown when adding rows to a table from peer cluster, > table with replication factor other than 0 or 1 > -- > > Key: HBASE-18864 > URL: https://issues.apache.org/jira/browse/HBASE-18864 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 1.3.0 >Reporter: smita >Assignee: Sakthi >Priority: Major > Labels: beginner > Fix For: 1.3.2, 1.2.7, 1.4.3 > > Attachments: hbase-18864.branch-1.2.001.patch, > hbase-18864.branch-1.2.002.patch, hbase-18864.branch-1.2.003.patch, > hbase-18864.branch-1.2.004.patch > > > Scenario: > = > add_peer > create a table > alter table with REPLICATION_SCOPE => '5' > enable table replication > login to peer cluster and try putting data to the table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19802) Wrong usage messages on shell commands (grant/revoke namespace syntax)
[ https://issues.apache.org/jira/browse/HBASE-19802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-19802: Fix Version/s: (was: 1.3.3) 1.3.2 > Wrong usage messages on shell commands (grant/revoke namespace syntax) > -- > > Key: HBASE-19802 > URL: https://issues.apache.org/jira/browse/HBASE-19802 > Project: HBase > Issue Type: Bug > Components: shell >Reporter: Csaba Skrabak >Assignee: Csaba Skrabak >Priority: Minor > Fix For: 2.0.0, 2.1.0, 1.3.2, 1.2.7, 1.4.3 > > Attachments: HBASE-19802.patch > > > Formal syntax of grant and revoke as it appears in help text contradicts > examples. Suggests that a <@namespace> must always precede a > parameter. In fact, it must not precede, it is an alternative to specifying > the table. > Also there is some nonsense in the explanation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20164) failed hadoopcheck should add footer link
[ https://issues.apache.org/jira/browse/HBASE-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20164: Fix Version/s: (was: 1.3.3) 1.3.2 > failed hadoopcheck should add footer link > - > > Key: HBASE-20164 > URL: https://issues.apache.org/jira/browse/HBASE-20164 > Project: HBase > Issue Type: Bug > Components: community >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 2.0.0, 1.3.2, 1.5.0, 1.2.8, 1.4.3 > > Attachments: HBASE-20164.patch, HBASE-20164.wip.1.patch, > HBASE-20164.wip.patch > > > thought for sure this already had an issue, [~busbey], but I can't find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20162) [nightly] depending on pipeline execution we sometimes refer to the wrong workspace
[ https://issues.apache.org/jira/browse/HBASE-20162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20162: Fix Version/s: (was: 1.3.3) 1.3.2 > [nightly] depending on pipeline execution we sometimes refer to the wrong > workspace > --- > > Key: HBASE-20162 > URL: https://issues.apache.org/jira/browse/HBASE-20162 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.3 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0, 1.3.2, 1.5.0, 1.2.8, 1.4.3 > > Attachments: HBASE-20162.0.patch > > > we set BASEDIR at the top of our pipeline to point at the component checkout > within WORKSPACE. > but! > a) at that point WORKSPACE is the workspace for the launching task > b) sometimes our parallel executions get a task with a different local > WORKSPACE to allow for coexisting on the same build host > c) when this happens our parallel stages are referring to some other absolute > path on the host > d) in most cases we're referring to dev-support files like e.g. the nightly > build script or machine info script that are the same across branches so > things are fine if we aren't running at the same time as a job that's > overwritting them > e) we also refer to the Dockerfile this way, so weird bugs I'm sure. > f) we build the source tarball from here, so that's probably broken subtly > g) sometimes that other directory _doesn't exist at all_ and we fail with > confusing messages about stuff not found -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20174) Fix TestZKLessMergeOnCluster flakiness
[ https://issues.apache.org/jira/browse/HBASE-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20174: Resolution: Fixed Status: Resolved (was: Patch Available) > Fix TestZKLessMergeOnCluster flakiness > -- > > Key: HBASE-20174 > URL: https://issues.apache.org/jira/browse/HBASE-20174 > Project: HBase > Issue Type: Bug >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.5.0, 1.4.3, 1.3.2 > > Attachments: HBASE-20174.branch-1.001.patch > > > TestZKLessMergeOnCluster.testCleanMergeReference() sometimes fails because > the condition used to wait for region compaction to complete is not > comprehensive enough which causes a race sometimes leaving the new compacted > storefile not fully committed preventing the discharger from cleaning up the > storefile references. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20174) Fix TestZKLessMergeOnCluster flakiness
[ https://issues.apache.org/jira/browse/HBASE-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396274#comment-16396274 ] Francis Liu commented on HBASE-20174: - Thanks for the reviews. Addressed nit and committed to branch-1 and cherry picked to branch-1.4 and branch-1.3 > Fix TestZKLessMergeOnCluster flakiness > -- > > Key: HBASE-20174 > URL: https://issues.apache.org/jira/browse/HBASE-20174 > Project: HBase > Issue Type: Bug >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Fix For: 1.3.2, 1.5.0, 1.4.3 > > Attachments: HBASE-20174.branch-1.001.patch > > > TestZKLessMergeOnCluster.testCleanMergeReference() sometimes fails because > the condition used to wait for region compaction to complete is not > comprehensive enough which causes a race sometimes leaving the new compacted > storefile not fully committed preventing the discharger from cleaning up the > storefile references. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20139) NPE in RSRpcServices.get() when getRegion throws an exception
[
https://issues.apache.org/jira/browse/HBASE-20139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-20139:
Fix Version/s: (was: 1.3.3)
1.3.2
> NPE in RSRpcServices.get() when getRegion throws an exception
> -
>
> Key: HBASE-20139
> URL: https://issues.apache.org/jira/browse/HBASE-20139
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
>Priority: Minor
> Fix For: 1.3.2, 1.5.0, 1.4.3
>
> Attachments: HBASE-20139.branch-1.001.patch,
> HBASE-20139.branch-1.3.001.patch, HBASE-20139.branch-1.3.001.patch
>
>
> We can get a NPE in RsRpcServices at
> {code:java}
> } finally {
> if (regionServer.metricsRegionServer != null) {
> regionServer.metricsRegionServer.updateGet(
> -> region.getTableDesc().getTableName(), EnvironmentEdgeManager.currentTime()
> - before);
> }
> if (quota != null) {
> quota.close();
> }{code}
> when region itself is null which might happen when getRegion throws an
> exception, this is then sent back to the client which is not able to handle
> this/make sense of it.
> {code:java}
> 2018-03-06 08:31:25,100 DEBUG [0,queue=4,port=60020] ipc.RpcServer -
> RpcServer.FifoWFPBQ.default.handler=30,queue=4,port=60020: callId: 5605567
> service: ClientService methodName: Get size: 79 connection: xyz:58736
> deadline: 9223372036854775807
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2431)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2246)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35068)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2373)
> ... 3 more{code}
> This has been fixed by [~stack] over at HBASE-18946 for master, backporting
> the same to branch-1, 1.3 and 1.4
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19118) Use SaslUtil to set Sasl.QOP in 'Thrift'
[
https://issues.apache.org/jira/browse/HBASE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-19118:
Fix Version/s: (was: 1.3.3)
1.3.2
> Use SaslUtil to set Sasl.QOP in 'Thrift'
>
>
> Key: HBASE-19118
> URL: https://issues.apache.org/jira/browse/HBASE-19118
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
> Attachments: HBASE-19118.branch-1.3.001.patch,
> HBASE-19118.master.001.patch, HBASE-19118.master.002.patch,
> HBASE-19118.master.003.patch
>
>
> In [Configure the Thrift
> Gateway|http://hbase.apache.org/book.html#security.client.thrift], it says
> "set the property {{hbase.thrift.security.qop}} to one of the following three
> values: {{privacy}}, {{integrity}}, {{authentication}}", which would lead to
> failure of starting up a thrift server.
> In fact, the value of {{hbase.thrift.security.qop}} should be {{auth}},
> {{auth-int}}, {{auth-conf}}, according to the documentation of {{Sasl.QOP}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20134) support scripts use hard-coded /tmp
[
https://issues.apache.org/jira/browse/HBASE-20134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-20134:
Fix Version/s: (was: 1.3.3)
1.3.2
> support scripts use hard-coded /tmp
> ---
>
> Key: HBASE-20134
> URL: https://issues.apache.org/jira/browse/HBASE-20134
> Project: HBase
> Issue Type: Bug
> Components: website
>Reporter: Mike Drob
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 1.3.2, 1.5.0, 1.2.7, 1.4.3
>
> Attachments: HBASE-20134.0.patch
>
>
> {code}
> if [ -z "${working_dir}" ]; then
> echo "[DEBUG] defaulting to creating a directory in /tmp"
> working_dir=/tmp
> while [[ -e ${working_dir} ]]; do
> working_dir=/tmp/hbase-generate-website-${RANDOM}.${RANDOM}
> done
> mkdir "${working_dir}"
> else
> {code}
> This should likely use {{$TMPDIR}} or {{mktemp -d}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19502) Make sure we have closed all StoreFileScanners if we fail to open any StoreFileScanners
[
https://issues.apache.org/jira/browse/HBASE-19502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-19502:
Fix Version/s: (was: 1.3.3)
1.3.2
> Make sure we have closed all StoreFileScanners if we fail to open any
> StoreFileScanners
> ---
>
> Key: HBASE-19502
> URL: https://issues.apache.org/jira/browse/HBASE-19502
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 2.0.0, 1.3.2, 1.4.1, 1.2.7
>
> Attachments: HBASE-19502.branch-1.4.patch, HBASE-19502.v0.patch
>
>
> {code:title=StoreFileScanner.java}
> public static List
> getScannersForStoreFiles(Collection files,
> boolean cacheBlocks, boolean usePread, boolean isCompaction, boolean
> canUseDrop,
> ScanQueryMatcher matcher, long readPt, boolean isPrimaryReplica) throws
> IOException {
> List scanners = new
> ArrayList(files.size());
> List sorted_files = new ArrayList<>(files);
> Collections.sort(sorted_files, StoreFile.Comparators.SEQ_ID);
> for (int i = 0; i < sorted_files.size(); i++) {
> StoreFile.Reader r = sorted_files.get(i).createReader(canUseDrop);
> r.setReplicaStoreFile(isPrimaryReplica);
> StoreFileScanner scanner = r.getStoreFileScanner(cacheBlocks, usePread,
> isCompaction, readPt,
> i, matcher != null ? !matcher.hasNullColumnInQuery() : false);
> scanners.add(scanner);
> }
> return scanners;
> }
> {code}
> The missed decrement of ref count will obstruct the cleanup of compacted
> files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20075) remove logic for branch-1.1 nightly testing
[ https://issues.apache.org/jira/browse/HBASE-20075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20075: Fix Version/s: (was: 1.3.3) 1.3.2 > remove logic for branch-1.1 nightly testing > --- > > Key: HBASE-20075 > URL: https://issues.apache.org/jira/browse/HBASE-20075 > Project: HBase > Issue Type: Task > Components: test >Affects Versions: 1.1.13 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 2.0.0, 3.0.0, 2.1.0, 1.3.2, 1.5.0, 1.2.8, 1.4.3 > > Attachments: HBASE-20075.0.patch > > > since branch-1.1 is EOM, remove the branch-1.1 specific logic from our > Jenkinsfile and delete the Jenkinsfile in branch-1.1 so we'll stop running > nightlies for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20146) Regions are stuck while opening when WAL is disabled
[
https://issues.apache.org/jira/browse/HBASE-20146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francis Liu updated HBASE-20146:
Fix Version/s: (was: 1.3.3)
1.3.2
> Regions are stuck while opening when WAL is disabled
>
>
> Key: HBASE-20146
> URL: https://issues.apache.org/jira/browse/HBASE-20146
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 1.3.2, 1.5.0, 1.2.7, 1.4.3
>
> Attachments: HBASE-20146.patch, HBASE-20146.v1.patch
>
>
> On a running cluster we had set {{hbase.regionserver.hlog.enabled}} to false,
> to disable the WAL for complete cluster, after restarting HBase service,
> regions are not getting opened leading to HMaster abort as Namespace table
> regions are not getting assigned.
> jstack for region open:
> {noformat}
> "RS_OPEN_PRIORITY_REGION-BLR106595:16045-1" #159 prio=5 os_prio=0
> tid=0x7fdfa4341000 nid=0x419d waiting on condition [0x7fdfa0467000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x87554448> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at org.apache.hadoop.hbase.wal.WALKey.getWriteEntry(WALKey.java:98)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:131)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:88)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:1026)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6849)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6803)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6774)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6730)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6681)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This used to work with HBase 1.0.2 version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
