[jira] [Commented] (HBASE-10169) Batch coprocessor
[ https://issues.apache.org/jira/browse/HBASE-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879582#comment-13879582 ] Jingcheng Du commented on HBASE-10169: -- [~te...@apache.org] I have fixed the conflicts and uploaded the latest patch, also update the patch in the review board. > Batch coprocessor > - > > Key: HBASE-10169 > URL: https://issues.apache.org/jira/browse/HBASE-10169 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 0.99.0 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: Batch Coprocessor Design Document.docx, > HBASE-10169-V2.patch, HBASE-10169-V3.patch, HBASE-10169.patch > > > This is designed to improve the coprocessor invocation in the client side. > Currently the coprocessor invocation is to send a call to each region. If > there’s one region server, and 100 regions are located in this server, each > coprocessor invocation will send 100 calls, each call uses a single thread in > the client side. The threads will run out soon when the coprocessor > invocations are heavy. > In this design, all the calls to the same region server will be grouped into > one in a single coprocessor invocation. This call will be spread into each > region in the server side. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9740) A corrupt HFile could cause endless attempts to assign the region without a chance of success
[ https://issues.apache.org/jira/browse/HBASE-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879581#comment-13879581 ] Ping commented on HBASE-9740: - Yes, I just checked the trunk, it goes somehow same way, thanks for your comment Jimmy. Would you plan to merge this feature to branch 0.94? > A corrupt HFile could cause endless attempts to assign the region without a > chance of success > - > > Key: HBASE-9740 > URL: https://issues.apache.org/jira/browse/HBASE-9740 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.16 >Reporter: Aditya Kishore >Assignee: Aditya Kishore > Attachments: patch-9740_0.94.txt > > > As described in HBASE-9737, a corrupt HFile in a region could lead to an > assignment storm in the cluster since the Master will keep trying to assign > the region to each region server one after another and obviously none will > succeed. > The region server, upon detecting such a scenario should mark the region as > "RS_ZK_REGION_FAILED_ERROR" (or something to the effect) in the Zookeeper > which should indicate the Master to stop assigning the region until the error > has been resolved (via an HBase shell command, probably "assign"?) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10169) Batch coprocessor
[ https://issues.apache.org/jira/browse/HBASE-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-10169: - Attachment: HBASE-10169-V3.patch > Batch coprocessor > - > > Key: HBASE-10169 > URL: https://issues.apache.org/jira/browse/HBASE-10169 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 0.99.0 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: Batch Coprocessor Design Document.docx, > HBASE-10169-V2.patch, HBASE-10169-V3.patch, HBASE-10169.patch > > > This is designed to improve the coprocessor invocation in the client side. > Currently the coprocessor invocation is to send a call to each region. If > there’s one region server, and 100 regions are located in this server, each > coprocessor invocation will send 100 calls, each call uses a single thread in > the client side. The threads will run out soon when the coprocessor > invocations are heavy. > In this design, all the calls to the same region server will be grouped into > one in a single coprocessor invocation. This call will be spread into each > region in the server side. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879462#comment-13879462 ] stack commented on HBASE-10336: --- bq. I was not suggesting Eric's patch for 0.98 As per Andrew, the above is a non sequitur > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879460#comment-13879460 ] chunhui shen commented on HBASE-10403: -- Current configuration is complex indeed. [~ndimiduk] Could you upload a patch? Thus, I could understand better about the new configuration > Simplify offheap cache configuration > > > Key: HBASE-10403 > URL: https://issues.apache.org/jira/browse/HBASE-10403 > Project: HBase > Issue Type: Bug > Components: io >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > The BucketCache (HBASE-7404) is a very nice piece of functionality which is > hidden behind complex configuration. Enabling it currently requires manual > calculation of L1 cache. It'd be nice to make this easier to use and conform > better with the existing heap management tools we already have. > Turning it on currently requires explicitly setting LruBlockCache (L1) > instance size and IOEngine (L2) size, making sure that L1 size isn't too big > vs global memstore and total heap. This is further confused by > hbase.bucketcache.size accepting a percentage of total heap OR explicit size > in MB. Enabling SlabCache is slightly easier in that it just accepts whatever > LruBlockCache is provided. > Turning on BucketCache using off-heap mode could look like: > hbase-env.sh: > Set HBASE_REGIONSERVER_OPTS: > -Xmx5000m > -XX:MaxDirectMemorySize=15000m > hbase-site.xml: > - hbase.regionserver.global.memstore.size = 0.7 > - hbase.regionserver.onheap.blockcache.size = 0.1 > - hbase.regionserver.blockcache.impl = BucketCache > - hbase.bucketcache.ioengine = offheap > The result being a CombinedCache instance with 500m LruBlockCache + 15000m > ByteBufferIOEngine running in direct mode. > This example does a couple things (mostly for the admin): > - knows NOT to enable SlabCache > - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ > - maintains the validity of HBaseConfiguration's existing check that global > MemStore + LruBlockCache == 0.8 > - maps "BucketCache" into meaning "a CombinedCache instance with these > implementations for L1 and L2." > - Figures out appropriate values for hbase.bucketcache.size and > hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879429#comment-13879429 ] binlijin commented on HBASE-3909: - bq. Does this imply that this region needs re-opening because many parameters take effect at the time of region open ? No，the region should not re-opening，not all parameters will dynamic take effect, we choose some of them. > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: New Feature >Reporter: stack >Assignee: Subbu M Iyer > Attachments: 3909-102812.patch, 3909-102912.patch, 3909-v1.patch, > 3909.v1, 3909_090712-2.patch, HBASE-3909-backport-from-fb-for-trunk.patch, > HBase Cluster Config Details.xlsx, patch-v2.patch, testMasterNoCluster.stack > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10365: --- Resolution: Fixed Status: Resolved (was: Patch Available) > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0, 0.99.0 > > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10365: --- Fix Version/s: 0.98.0 Integrated to 0.98 as well. Thanks Andy. > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0, 0.99.0 > > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879425#comment-13879425 ] Ted Yu commented on HBASE-3909: --- bq. when region open finished, it can check the configuration Does this imply that this region needs re-opening because many parameters take effect at the time of region open ? > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: New Feature >Reporter: stack >Assignee: Subbu M Iyer > Attachments: 3909-102812.patch, 3909-102912.patch, 3909-v1.patch, > 3909.v1, 3909_090712-2.patch, HBASE-3909-backport-from-fb-for-trunk.patch, > HBase Cluster Config Details.xlsx, patch-v2.patch, testMasterNoCluster.stack > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10401: --- Attachment: hbase-10079.v2.patch v2 fixes bad line lengths. > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10079.v2.patch, hbase-10401.10400.patch, > hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10379) Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN
[
https://issues.apache.org/jira/browse/HBASE-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879420#comment-13879420
]
takeshi.miao commented on HBASE-10379:
--
[~andrew.purt...@gmail.com]
Tks for quick review, do I need to open a new ticket for troubleshooting
section in HBase book ?
> Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level
> ERROR to WARN
> --
>
> Key: HBASE-10379
> URL: https://issues.apache.org/jira/browse/HBASE-10379
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.16
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Minor
>
> Hi All,
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while
> we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException as:takeshi_miao@LAB
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction as:takeshi_miao@LAB
> from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
> {code}
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> ERROR level) in log, but after code study, this is basically *NOT* a data
> loss issue due to HBase client API would try 5 times internally
> (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> failed thread), The HTable API would also throw IOEcxeption to client code if
> any thread still fail after these retries.
> Based on HBase users' viewpoint as us, we think this is better to change the
> logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
> confused us for a while...But this code change may need to change boht HBase
> code and Hadoop code as well; so I am wondering how community think about
> this small thing but may be imp
[jira] [Commented] (HBASE-10379) Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN
[
https://issues.apache.org/jira/browse/HBASE-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879419#comment-13879419
]
takeshi.miao commented on HBASE-10379:
--
I think that we still need a patch for this issue is due to we can not expect
that what environments user may use for HBase.
> Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level
> ERROR to WARN
> --
>
> Key: HBASE-10379
> URL: https://issues.apache.org/jira/browse/HBASE-10379
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.16
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Minor
>
> Hi All,
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while
> we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException as:takeshi_miao@LAB
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction as:takeshi_miao@LAB
> from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
> {code}
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> ERROR level) in log, but after code study, this is basically *NOT* a data
> loss issue due to HBase client API would try 5 times internally
> (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> failed thread), The HTable API would also throw IOEcxeption to client code if
> any thread still fail after these retries.
> Based on HBase users' viewpoint as us, we think this is better to change the
> logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
> confused us for a while...But this code change may need to change boht HBase
> code and Hadoop code as well; so I am wondering how community think about
> this small thing but may be imp
[jira] [Commented] (HBASE-10379) Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN
[
https://issues.apache.org/jira/browse/HBASE-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879418#comment-13879418
]
Andrew Purtell commented on HBASE-10379:
We should put this in the troubleshooting section of the manual.
> Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level
> ERROR to WARN
> --
>
> Key: HBASE-10379
> URL: https://issues.apache.org/jira/browse/HBASE-10379
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.16
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Minor
>
> Hi All,
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while
> we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException as:takeshi_miao@LAB
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction as:takeshi_miao@LAB
> from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
> {code}
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> ERROR level) in log, but after code study, this is basically *NOT* a data
> loss issue due to HBase client API would try 5 times internally
> (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> failed thread), The HTable API would also throw IOEcxeption to client code if
> any thread still fail after these retries.
> Based on HBase users' viewpoint as us, we think this is better to change the
> logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
> confused us for a while...But this code change may need to change boht HBase
> code and Hadoop code as well; so I am wondering how community think about
> this small thing but may be important to the pure HBase users.
> mailing list
> http://
[jira] [Commented] (HBASE-10379) Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN
[
https://issues.apache.org/jira/browse/HBASE-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879416#comment-13879416
]
takeshi.miao commented on HBASE-10379:
--
I did some test on +new version of Krb5-server (1.10.3), and seems this issue
is mitigated+. I have been using 20-threads of HBase Client to put 7 records
into one HTable with corresponding 7 regions scattered on three regionservers,
with three round of tests; this simple test can simply reproduce this 'Request
is a replay' issue on the first environment (HBase-0.94.16 with
krb5-server-1.6.1 on CentOS-5.3); But in second environment (HBase-0.94.16 with
krb5-server-1.10.3 on RHEL-6.4_GA-x86_64-10-Hourly2 on AWS), I did not
reproduce this issue. It seems that this issue was fixed in the later of
krb5-server release.
I did this test further on new version of krb5-server is due to I googled two
tickets ([#5924|http://krbdev.mit.edu/rt/Ticket/Display.html?id=5924] &
[#1201|http://krbdev.mit.edu/rt/Ticket/Display.html?id=1201]) whom were talking
about '_KRB5KRB_AP_ERR_REPEAT_' bug in krb5 community.
> Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level
> ERROR to WARN
> --
>
> Key: HBASE-10379
> URL: https://issues.apache.org/jira/browse/HBASE-10379
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.16
>Reporter: takeshi.miao
>Assignee: takeshi.miao
>Priority: Minor
>
> Hi All,
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while
> we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException as:takeshi_miao@LAB
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction as:takeshi_miao@LAB
> from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
> {code}
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_
[jira] [Commented] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879413#comment-13879413 ] Jonathan Hsieh commented on HBASE-10400: patch applied to all branches (for 94 I applied via 'patch -p1 < xxx') > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 0.98.0, 0.96.2, 0.99.0, 0.94.17 > > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10400: --- Resolution: Fixed Fix Version/s: 0.94.17 0.99.0 0.96.2 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 0.98.0, 0.96.2, 0.99.0, 0.94.17 > > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9740) A corrupt HFile could cause endless attempts to assign the region without a chance of success
[ https://issues.apache.org/jira/browse/HBASE-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879409#comment-13879409 ] Jimmy Xiang commented on HBASE-9740: In trunk, the region will not be assigned before some user does something after it is moved to FAILED_OPEN state, however, the region will be shown in transition from master webUI. For this issue, we need [~lhofhansl] to agree with it. > A corrupt HFile could cause endless attempts to assign the region without a > chance of success > - > > Key: HBASE-9740 > URL: https://issues.apache.org/jira/browse/HBASE-9740 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.16 >Reporter: Aditya Kishore >Assignee: Aditya Kishore > Attachments: patch-9740_0.94.txt > > > As described in HBASE-9737, a corrupt HFile in a region could lead to an > assignment storm in the cluster since the Master will keep trying to assign > the region to each region server one after another and obviously none will > succeed. > The region server, upon detecting such a scenario should mark the region as > "RS_ZK_REGION_FAILED_ERROR" (or something to the effect) in the Zookeeper > which should indicate the Master to stop assigning the region until the error > has been resolved (via an HBase shell command, probably "assign"?) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879408#comment-13879408 ] Jonathan Hsieh commented on HBASE-10400: This was tested on a cluster using running CDH4.x's hadoop2 flavor. > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879407#comment-13879407 ] Jonathan Hsieh commented on HBASE-10400: Split transaction is unrelated. Will be committing to 0.94 and up. > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9740) A corrupt HFile could cause endless attempts to assign the region without a chance of success
[ https://issues.apache.org/jira/browse/HBASE-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879403#comment-13879403 ] Ping commented on HBASE-9740: - Yes, @[~jxiang], you're right for we can't notice it from master's status page. on the other way the application which access this table(region) will find that. I am thinking of put some warning message on the page too if this is fit. I followed this: We just consider 0.94 branch, for this branch, if we move the region to FAILED_OPEN state, the AM will assign it again and again which BLOCKs cluster balancing, and in our product cluster, we can't even disable the table to make a repair(all tools include close_region/hbck repair/disable table/... NOT usable). so we think we can make this region offline for advanced repair or maintain. @[~adityakishore], thanks for your help, I will check my code style and fix it. and resubmit the patch if you and jimmy agree with it. > A corrupt HFile could cause endless attempts to assign the region without a > chance of success > - > > Key: HBASE-9740 > URL: https://issues.apache.org/jira/browse/HBASE-9740 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.16 >Reporter: Aditya Kishore >Assignee: Aditya Kishore > Attachments: patch-9740_0.94.txt > > > As described in HBASE-9737, a corrupt HFile in a region could lead to an > assignment storm in the cluster since the Master will keep trying to assign > the region to each region server one after another and obviously none will > succeed. > The region server, upon detecting such a scenario should mark the region as > "RS_ZK_REGION_FAILED_ERROR" (or something to the effect) in the Zookeeper > which should indicate the Master to stop assigning the region until the error > has been resolved (via an HBase shell command, probably "assign"?) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879398#comment-13879398 ] Liang Xie commented on HBASE-7404: -- [~liyin], so far as i know, most are them related with hdfs layer, the existing unfinished jiras(1. hedged read; 2 two wal writers) should be expected to alleviate somehow, but still have other works need to do, e.g. HDFS io qos or request priority(HDFS-5727). i am a late entry to the hbase+hdfs percentile latency tuning game, but i have an intuitive sense of still having lots of improvement space. > Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE > -- > > Key: HBASE-7404 > URL: https://issues.apache.org/jira/browse/HBASE-7404 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0 > > Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch, > 7404-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch, > 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf, > HBASE-7404-backport-0.94.patch, Introduction of Bucket Cache.pdf, > hbase-7404-94v2.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch > > > First, thanks @neil from Fusion-IO share the source code. > Usage: > 1.Use bucket cache as main memory cache, configured as the following: > –"hbase.bucketcache.ioengine" "heap" (or "offheap" if using offheap memory to > cache block ) > –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of > max heap size) > 2.Use bucket cache as a secondary cache, configured as the following: > –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path > where to store the block data) > –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 > means 1GB) > –"hbase.bucketcache.combinedcache.enabled" false (default value being true) > See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache > What's Bucket Cache? > It could greatly decrease CMS and heap fragment by GC > It support a large cache space for High Read Performance by using high speed > disk like Fusion-io > 1.An implementation of block cache like LruBlockCache > 2.Self manage blocks' storage position through Bucket Allocator > 3.The cached blocks could be stored in the memory or file system > 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), > combined with LruBlockCache to decrease CMS and fragment by GC. > 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to > store block) to enlarge cache space > How about SlabCache? > We have studied and test SlabCache first, but the result is bad, because: > 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds > of block size, especially using DataBlockEncoding > 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache > and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , > it causes CMS and heap fragment don't get any better > 3.Direct heap performance is not good as heap, and maybe cause OOM, so we > recommend using "heap" engine > See more in the attachment and in the patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HBASE-10399) Add documentation for VerifyReplication to refguide
[ https://issues.apache.org/jira/browse/HBASE-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gustavo Anatoly reassigned HBASE-10399: --- Assignee: Gustavo Anatoly > Add documentation for VerifyReplication to refguide > --- > > Key: HBASE-10399 > URL: https://issues.apache.org/jira/browse/HBASE-10399 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Gustavo Anatoly >Priority: Minor > > HBase refguide currently doesn't document how VerifyReplication is used for > comparing local table with remote table. > Document for VerifyReplication should be added so that users know how to use > it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879394#comment-13879394 ] binlijin commented on HBASE-3909: - Q: when configuration is updated, how would regions in transition pick up the changes ? A: when region open finished, it can check the configuration, if configuration changes the region can apply the new configuration. Q:when configuration is updated and meanwhile master fails over, what would happen ? A: The new configuration takes effect must be triggered by admin，if some node failed，admin can do it again. > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: New Feature >Reporter: stack >Assignee: Subbu M Iyer > Attachments: 3909-102812.patch, 3909-102912.patch, 3909-v1.patch, > 3909.v1, 3909_090712-2.patch, HBASE-3909-backport-from-fb-for-trunk.patch, > HBase Cluster Config Details.xlsx, patch-v2.patch, testMasterNoCluster.stack > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10399) Add documentation for VerifyReplication to refguide
[ https://issues.apache.org/jira/browse/HBASE-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879392#comment-13879392 ] Gustavo Anatoly commented on HBASE-10399: - I can do this task. This issue will remain open? > Add documentation for VerifyReplication to refguide > --- > > Key: HBASE-10399 > URL: https://issues.apache.org/jira/browse/HBASE-10399 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > HBase refguide currently doesn't document how VerifyReplication is used for > comparing local table with remote table. > Document for VerifyReplication should be added so that users know how to use > it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879390#comment-13879390 ] Andrew Purtell commented on HBASE-10365: +1 for 0.98. Can commit this if you don't first Ted. > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.99.0 > > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-9845) Add MR support for cell ACLs
[ https://issues.apache.org/jira/browse/HBASE-9845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9845: -- Affects Version/s: (was: 0.98.0) Fix Version/s: (was: 0.98.0) 0.98.1 I started working on this and got to the point where the mapreduce support for labels assumes that labels are the only kind of tags that would be added to a cell, and there is a static method that is specific to labels for doing all of the work. Would need to undo that and introduce some additive way to add tags that is agnostic. The release has been held up long enough, this can wait to a point release. Moving out of 0.98.0. > Add MR support for cell ACLs > > > Key: HBASE-9845 > URL: https://issues.apache.org/jira/browse/HBASE-9845 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.1 > > > We should support adding cell ACLs through TableOutputFormat and > HFileOutputFormat. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-9845) Add MR support for cell ACLs
[ https://issues.apache.org/jira/browse/HBASE-9845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9845: -- Assignee: (was: Andrew Purtell) > Add MR support for cell ACLs > > > Key: HBASE-9845 > URL: https://issues.apache.org/jira/browse/HBASE-9845 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell > Fix For: 0.98.1 > > > We should support adding cell ACLs through TableOutputFormat and > HFileOutputFormat. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10365: --- Fix Version/s: 0.99.0 Hadoop Flags: Reviewed Integrated to trunk Thanks for the review, Jon. > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.99.0 > > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879361#comment-13879361 ] Ted Yu commented on HBASE-10365: Tests passed: https://builds.apache.org/job/PreCommit-HBASE-Build/8501/console > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[
https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879363#comment-13879363
]
Hadoop QA commented on HBASE-10365:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624459/10365-v1.txt
against trunk revision .
ATTACHMENT ID: 12624459
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8501//console
This message is automatically generated.
> HBaseFsck should clean up connection properly when repair is completed
> --
>
> Key: HBASE-10365
> URL: https://issues.apache.org/jira/browse/HBASE-10365
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 10365-v1.txt
>
>
> At the end of exec() method, connections to the cluster are not properly
> released.
> Connections should be released upon completion of repair.
> This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16
> release candidate is available for download'
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879364#comment-13879364 ] Hudson commented on HBASE-10338: FAILURE: Integrated in hbase-0.96-hadoop2 #184 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/184/]) Amend HBASE-10338 Fix NPE if the server is terminated before the RegionServerCoprocessorHost is initialized (Vandana Ayyalasomayajula) (apurtell: rev 1560496) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338.0.patch, HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10375) hbase-default.xml hbase.status.multicast.address.port does not match code
[
https://issues.apache.org/jira/browse/HBASE-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879365#comment-13879365
]
Hudson commented on HBASE-10375:
FAILURE: Integrated in hbase-0.96-hadoop2 #184 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/184/])
HBASE-10375 hbase-default.xml hbase.status.multicast.address.port does not
match code (nkeywal: rev 1560322)
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.96/hbase-common/src/main/resources/hbase-default.xml
> hbase-default.xml hbase.status.multicast.address.port does not match code
> -
>
> Key: HBASE-10375
> URL: https://issues.apache.org/jira/browse/HBASE-10375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0, 0.96.1.1
>Reporter: Jonathan Hsieh
>Assignee: Nicolas Liochon
> Fix For: 0.98.0, 0.96.2, 0.99.0
>
> Attachments: 10375.v1.98-96.patch, 10375.v1.patch,
> 10375.v2.96-98.patch, 10375.v2.trunk.patch
>
>
> In hbase-default.xml
> {code}
> +
> +hbase.status.multicast.address.port
> +6100
> +
> + Multicast port to use for the status publication by multicast.
> +
> +
> {code}
> In HConstants it was 60100.
> {code}
> public static final String STATUS_MULTICAST_PORT =
> "hbase.status.multicast.port";
> public static final int DEFAULT_STATUS_MULTICAST_PORT = 16100;
> {code}
> (it was 60100 in the code for 0.96 and 0.98.)
> I lean towards going with the code as opposed to the config file.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HBASE-10403) Simplify offheap cache configuration
Nick Dimiduk created HBASE-10403: Summary: Simplify offheap cache configuration Key: HBASE-10403 URL: https://issues.apache.org/jira/browse/HBASE-10403 Project: HBase Issue Type: Bug Components: io Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd be nice to make this easier to use and conform better with the existing heap management tools we already have. Turning it on currently requires explicitly setting LruBlockCache (L1) instance size and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever LruBlockCache is provided. Turning on BucketCache using off-heap mode could look like: hbase-env.sh: Set HBASE_REGIONSERVER_OPTS: -Xmx5000m -XX:MaxDirectMemorySize=15000m hbase-site.xml: - hbase.regionserver.global.memstore.size = 0.7 - hbase.regionserver.onheap.blockcache.size = 0.1 - hbase.regionserver.blockcache.impl = BucketCache - hbase.bucketcache.ioengine = offheap The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine running in direct mode. This example does a couple things (mostly for the admin): - knows NOT to enable SlabCache - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ - maintains the validity of HBaseConfiguration's existing check that global MemStore + LruBlockCache == 0.8 - maps "BucketCache" into meaning "a CombinedCache instance with these implementations for L1 and L2." - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879357#comment-13879357 ] Andrew Purtell commented on HBASE-10336: bq. I was not suggesting Eric's patch for 0.98 Could have fooled me because there is a mention of "0.98" on every line of your above comment. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879353#comment-13879353 ] Andrew Purtell commented on HBASE-10336: By the way, what does Hoya have to do with validating an HBase release? If you want to make it part of HBase release criteria, raise a discussion on dev@. Mention on this issue is completely beside the point. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879354#comment-13879354 ] Ted Yu commented on HBASE-10336: @Andy: I was not suggesting Eric's patch for 0.98 > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879349#comment-13879349 ] stack commented on HBASE-10336: --- [~wheat9] Thanks. Am up for testing and review np. [~ted_yu] You are not helping. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879351#comment-13879351 ] Andrew Purtell commented on HBASE-10336: I see you didn't actually check the InfoServers [~yuzhih...@gmail.com] -1 for 0.98. I am tagging RC0 today. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879345#comment-13879345 ] Hudson commented on HBASE-10338: FAILURE: Integrated in hbase-0.96 #268 (See [https://builds.apache.org/job/hbase-0.96/268/]) Amend HBASE-10338 Fix NPE if the server is terminated before the RegionServerCoprocessorHost is initialized (Vandana Ayyalasomayajula) (apurtell: rev 1560496) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338.0.patch, HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879344#comment-13879344 ] Hudson commented on HBASE-10338: SUCCESS: Integrated in HBase-TRUNK #4850 (See [https://builds.apache.org/job/HBase-TRUNK/4850/]) Amend HBASE-10338 Fix NPE if the server is terminated before the RegionServerCoprocessorHost is initialized (Vandana Ayyalasomayajula) (apurtell: rev 1560494) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338.0.patch, HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879335#comment-13879335 ] Ted Yu commented on HBASE-10336: For Eric's patch v5: applied on 0.98 tip built 0.98 tar ball built Hoya with 0.98 dependency ran LoadTestTool using Hoya with 0.98 tar ball - it passed. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10365: --- Attachment: 10365-v1.txt Re-attach for QA. Passed TestHBaseFsck on Linux twice. > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10365) HBaseFsck should clean up connection properly when repair is completed
[ https://issues.apache.org/jira/browse/HBASE-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10365: --- Attachment: (was: 10365-v1.txt) > HBaseFsck should clean up connection properly when repair is completed > -- > > Key: HBASE-10365 > URL: https://issues.apache.org/jira/browse/HBASE-10365 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 10365-v1.txt > > > At the end of exec() method, connections to the cluster are not properly > released. > Connections should be released upon completion of repair. > This was mentioned by Jean-Marc in the thread '[VOTE] The 1st hbase 0.94.16 > release candidate is available for download' -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10277) refactor AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879329#comment-13879329 ] Sergey Shelukhin commented on HBASE-10277: -- Callback for each put call would still need to be maintained separately for each put call, and if you need results (like from get) you'd still need to have place to store them. Current patch bypasses the creation of Object[] when results are not needed, e.g. for streaming puts. Previous code does the same thing, but the Object[] for get was still created inside the callback object. One problem with current callback is that as soon as you have multiple submits the index argument to callback becomes ambiguous, so you no longer know for which action you receive the error or result. So, HCM batch code creates AP for each single submit, that way it knows the index is always from that submit when the callback populates the array; and HTable just doesn't use callback for streaming puts because it doesn't need the results... if HTable were to use current callback for error management (or streaming gets when you need results and there are multiple submit calls), it becomes a real problem. We can instead add /per-call/ callback in the context. It's a hybrid between #3 and #4; AP can avoid global error support; we can add async call with callback to HTable which would use the "regular" path; current streaming put can have the same semantics but maintain the contexts in HTable rather than AP. Let me think more about the latter case. > refactor AsyncProcess > - > > Key: HBASE-10277 > URL: https://issues.apache.org/jira/browse/HBASE-10277 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-10277.01.patch, HBASE-10277.patch > > > AsyncProcess currently has two patterns of usage, one from HTable flush w/o > callback and with reuse, and one from HCM/HTable batch call, with callback > and w/o reuse. In the former case (but not the latter), it also does some > throttling of actions on initial submit call, limiting the number of > outstanding actions per server. > The latter case is relatively straightforward. The former appears to be error > prone due to reuse - if, as javadoc claims should be safe, multiple submit > calls are performed without waiting for the async part of the previous call > to finish, fields like hasError become ambiguous and can be used for the > wrong call; callback for success/failure is called based on "original index" > of an action in submitted list, but with only one callback supplied to AP in > ctor it's not clear to which submit call the index belongs, if several are > outstanding. > I was going to add support for HBASE-10070 to AP, and found that it might be > difficult to do cleanly. > It would be nice to normalize AP usage patterns; in particular, separate the > "global" part (load tracking) from per-submit-call part. > Per-submit part can more conveniently track stuff like initialActions, > mapping of indexes and retry information, that is currently passed around the > method calls. > -I am not sure yet, but maybe sending of the original index to server in > "ClientProtos.MultiAction" can also be avoided.- Cannot be avoided because > the API to server doesn't have one-to-one correspondence between requests and > responses in an individual call to multi (retries/rearrangement have nothing > to do with it) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10402) [docs] Add docs on how to create and tag a point release.
[ https://issues.apache.org/jira/browse/HBASE-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10402: --- Attachment: book.html hbase-10402.patch patch and the html file generated from the docs build. > [docs] Add docs on how to create and tag a point release. > - > > Key: HBASE-10402 > URL: https://issues.apache.org/jira/browse/HBASE-10402 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: book.html, hbase-10402.patch > > > I've added a few steps to the "How to release" section of the ref guide based > on what I learned about doing a quickie release. > This is what I think I should have done -- if you disagree please suggest how > it should be done. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10401) [hbck] perform overlap group merges in parallel
[
https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879322#comment-13879322
]
Hadoop QA commented on HBASE-10401:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624420/hbase-10401.10400.patch
against trunk revision .
ATTACHMENT ID: 12624420
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ protected ExecutorService executor; // threads to retrieve data from
regionservers, and handle region merges.
+LOG.warn("[" + thread + "] HDFS region dir " +
contained.getHdfsRegionDir() + " is missing. " +
+LOG.warn("[" + thread + "] HDFS region dir " +
contained.getHdfsRegionDir() + " already sidelined.");
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8500//console
This message is automatically generated.
> [hbck] perform overlap group merges in parallel
> ---
>
> Key: HBASE-10401
> URL: https://issues.apache.org/jira/browse/HBASE-10401
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-10401.10400.patch, hbase-10401.patch
>
>
> In a recent support case, we encountered a corrupt hbase that had thousands
> of "overlap groups" (regions that had overlapping key ranges). The current
> implementation repairs these by serially taking a group, perorming a merge
> and then moving on to the next group. Because assignments and hdfs nn
> operations are involved each merge could take on the order of seconds. With
> thousands of overlap groups, this could take hours to complete.
> This patch makes it so that these independent merge groups are merged in
> parallel. It uses the same thread pool for other fs info-gathering
> operations.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HBASE-10402) [docs] Add docs on how to create and tag a point release.
Jonathan Hsieh created HBASE-10402: -- Summary: [docs] Add docs on how to create and tag a point release. Key: HBASE-10402 URL: https://issues.apache.org/jira/browse/HBASE-10402 Project: HBase Issue Type: Bug Components: documentation Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh I've added a few steps to the "How to release" section of the ref guide based on what I learned about doing a quickie release. This is what I think I should have done -- if you disagree please suggest how it should be done. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879287#comment-13879287 ] Ted Yu commented on HBASE-3909: --- Skimmed the patch. Some high level questions: when configuration is updated, how would regions in transition pick up the changes ? when configuration is updated and meanwhile master fails over, what would happen ? Thanks > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: New Feature >Reporter: stack >Assignee: Subbu M Iyer > Attachments: 3909-102812.patch, 3909-102912.patch, 3909-v1.patch, > 3909.v1, 3909_090712-2.patch, HBASE-3909-backport-from-fb-for-trunk.patch, > HBase Cluster Config Details.xlsx, patch-v2.patch, testMasterNoCluster.stack > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10169) Batch coprocessor
[ https://issues.apache.org/jira/browse/HBASE-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879283#comment-13879283 ] Ted Yu commented on HBASE-10169: @Jingcheng: Mind rebasing your patch ? There're conflicts in several files. > Batch coprocessor > - > > Key: HBASE-10169 > URL: https://issues.apache.org/jira/browse/HBASE-10169 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 0.99.0 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: Batch Coprocessor Design Document.docx, > HBASE-10169-V2.patch, HBASE-10169.patch > > > This is designed to improve the coprocessor invocation in the client side. > Currently the coprocessor invocation is to send a call to each region. If > there’s one region server, and 100 regions are located in this server, each > coprocessor invocation will send 100 calls, each call uses a single thread in > the client side. The threads will run out soon when the coprocessor > invocations are heavy. > In this design, all the calls to the same region server will be grouped into > one in a single coprocessor invocation. This call will be spread into each > region in the server side. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879281#comment-13879281 ] Lars Hofhansl commented on HBASE-10401: --- This should go into 0.94 as well. > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10401.10400.patch, hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10394) Test for Replication with tags
[ https://issues.apache.org/jira/browse/HBASE-10394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10394: --- Resolution: Fixed Fix Version/s: 0.99.0 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and 0.98. Looped the new test 20 times locally, passed every time. > Test for Replication with tags > -- > > Key: HBASE-10394 > URL: https://issues.apache.org/jira/browse/HBASE-10394 > Project: HBase > Issue Type: Test >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.98.0, 0.99.0 > > Attachments: HBASE-10394.patch > > > Followup task for HBASE-10322 for adding a test to assert Replication works > well and replicate cells with Tags when tags are being used. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879265#comment-13879265 ] Jimmy Xiang commented on HBASE-10336: - If this works, it is great. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879256#comment-13879256
]
Haohui Mai commented on HBASE-10336:
As [~sureshms] suggested in HADOOP-10253, an alternative approach is to
maintain a temporary copy of {{HttpServer}} in hadoop-common for HBase in 2.4.
That way hbase can work with hadoop 2.4 untile HBASE-10336 is landed.
> Remove deprecated usage of Hadoop HttpServer in InfoServer
> --
>
> Key: HBASE-10336
> URL: https://issues.apache.org/jira/browse/HBASE-10336
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Eric Charles
>Assignee: Eric Charles
> Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch,
> HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch
>
>
> Recent changes in Hadoop HttpServer give NPE when running on hadoop
> 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be
> not fixed (see HDFS-5760). We'd better move to the new proposed builder
> pattern, which means we can no more use inheritance to build our nice
> InfoServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879216#comment-13879216 ] Andrew Purtell commented on HBASE-10401: +1 for 0.98 > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10401.10400.patch, hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879215#comment-13879215 ] Andrew Purtell commented on HBASE-10400: +1 for 0.98 > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10395) endTime won't be set in VerifyReplication if startTime is not set
[
https://issues.apache.org/jira/browse/HBASE-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879194#comment-13879194
]
Hudson commented on HBASE-10395:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #97 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/97/])
HBASE-10395 endTime won't be set in VerifyReplication if startTime is not set
(Jianwei Cui) (Tedyu: rev 1560433)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> endTime won't be set in VerifyReplication if startTime is not set
> -
>
> Key: HBASE-10395
> URL: https://issues.apache.org/jira/browse/HBASE-10395
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Replication
>Affects Versions: 0.94.16
>Reporter: cuijianwei
>Assignee: cuijianwei
>Priority: Minor
> Fix For: 0.98.0, 0.99.0
>
> Attachments: HBASE-10395-0.94-v1.patch, HBASE-10395-0.94-v2.patch,
> HBASE-10395-0.94-v3.patch, HBASE-10395-trunk-v1.patch,
> HBASE-10395-trunk-v2.patch, HBASE-10395-trunk-v3.patch
>
>
> In VerifyReplication, we may set startTime and endTime to restrict the data
> to verfiy. However, the endTime won't be set in the program if we only pass
> endTime without startTime in command line argument. The reason is the
> following code:
> {code}
> if (startTime != 0) {
> scan.setTimeRange(startTime,
> endTime == 0 ? HConstants.LATEST_TIMESTAMP : endTime);
> }
> {code}
> The code will ignore endTime setting when not passing startTime in command
> line argument. Another place needs to improvement is the help message as
> follows:
> {code}
> System.err.println(" stoprow end of the row");
> {code}
> However, the program actually use "endrow" to parse the arguments:
> {code}
> final String endTimeArgKey = "--endtime=";
> if (cmd.startsWith(endTimeArgKey)) {
> endTime = Long.parseLong(cmd.substring(endTimeArgKey.length()));
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[
https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879166#comment-13879166
]
Hadoop QA commented on HBASE-10400:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624411/hbase-10400.patch
against trunk revision .
ATTACHMENT ID: 12624411
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8499//console
This message is automatically generated.
> [hbck] Continue if region dir missing on region merge attempt
> -
>
> Key: HBASE-10400
> URL: https://issues.apache.org/jira/browse/HBASE-10400
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 0.92.2, 0.94.16, 0.96.1.1
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-10400.patch
>
>
> In a recent support case, the hbck repair tool would eventually hang because
> we didn't handle the case where merge info is old using hadoop2 fs.listStatus
> semantics (throw exn instead of return null). this is a trivial patch that
> handles the newer hadoop2 semantics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879153#comment-13879153 ] Jonathan Hsieh commented on HBASE-10401: This was tested manually on a test cluster: * created 500 regions with data * moved them off to another dir * drop then create again with 500 regions with different split points. * copy the first table's into region dirs. * run hbck. We end up with 100 or so overlap groups. * run hbase hbck -fixOverlaps. * eventually we get back to a consistent state. > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10401.10400.patch, hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10401: --- Attachment: hbase-10401.10400.patch HBASE-10401 assumes HBASE-10400 is applied. I've attached a version that has the two pieces combined for the qabot to test. > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10401.10400.patch, hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879151#comment-13879151 ] Jimmy Xiang commented on HBASE-10336: - I just tested. My HBase cluster works fine with it. Thanks. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10401: --- Status: Patch Available (was: Open) > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.96.1.1, 0.94.16, 0.92.2, 0.98.0, 0.99.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10401.10400.patch, hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10395) endTime won't be set in VerifyReplication if startTime is not set
[
https://issues.apache.org/jira/browse/HBASE-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879147#comment-13879147
]
Hudson commented on HBASE-10395:
SUCCESS: Integrated in HBase-TRUNK #4849 (See
[https://builds.apache.org/job/HBase-TRUNK/4849/])
HBASE-10395 endTime won't be set in VerifyReplication if startTime is not set
(Jianwei Cui) (Tedyu: rev 1560434)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> endTime won't be set in VerifyReplication if startTime is not set
> -
>
> Key: HBASE-10395
> URL: https://issues.apache.org/jira/browse/HBASE-10395
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Replication
>Affects Versions: 0.94.16
>Reporter: cuijianwei
>Assignee: cuijianwei
>Priority: Minor
> Fix For: 0.98.0, 0.99.0
>
> Attachments: HBASE-10395-0.94-v1.patch, HBASE-10395-0.94-v2.patch,
> HBASE-10395-0.94-v3.patch, HBASE-10395-trunk-v1.patch,
> HBASE-10395-trunk-v2.patch, HBASE-10395-trunk-v3.patch
>
>
> In VerifyReplication, we may set startTime and endTime to restrict the data
> to verfiy. However, the endTime won't be set in the program if we only pass
> endTime without startTime in command line argument. The reason is the
> following code:
> {code}
> if (startTime != 0) {
> scan.setTimeRange(startTime,
> endTime == 0 ? HConstants.LATEST_TIMESTAMP : endTime);
> }
> {code}
> The code will ignore endTime setting when not passing startTime in command
> line argument. Another place needs to improvement is the help message as
> follows:
> {code}
> System.err.println(" stoprow end of the row");
> {code}
> However, the program actually use "endrow" to parse the arguments:
> {code}
> final String endTimeArgKey = "--endtime=";
> if (cmd.startsWith(endTimeArgKey)) {
> endTime = Long.parseLong(cmd.substring(endTimeArgKey.length()));
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HBASE-10401) [hbck] perform overlap group merges in parallel
[ https://issues.apache.org/jira/browse/HBASE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10401: --- Attachment: hbase-10401.patch applies to trunk, should apply to 0.96/0.98. > [hbck] perform overlap group merges in parallel > --- > > Key: HBASE-10401 > URL: https://issues.apache.org/jira/browse/HBASE-10401 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.98.0, 0.94.16, 0.99.0, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10401.patch > > > In a recent support case, we encountered a corrupt hbase that had thousands > of "overlap groups" (regions that had overlapping key ranges). The current > implementation repairs these by serially taking a group, perorming a merge > and then moving on to the next group. Because assignments and hdfs nn > operations are involved each merge could take on the order of seconds. With > thousands of overlap groups, this could take hours to complete. > This patch makes it so that these independent merge groups are merged in > parallel. It uses the same thread pool for other fs info-gathering > operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-9426) Make custom distributed barrier procedure pluggable
[
https://issues.apache.org/jira/browse/HBASE-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879146#comment-13879146
]
Hudson commented on HBASE-9426:
---
SUCCESS: Integrated in HBase-TRUNK #4849 (See
[https://builds.apache.org/job/HBase-TRUNK/4849/])
HBASE-10397 Fix findbugs introduced from HBASE-9426. (anoopsamjohn: rev 1560427)
*
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
> Make custom distributed barrier procedure pluggable
>
>
> Key: HBASE-9426
> URL: https://issues.apache.org/jira/browse/HBASE-9426
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.95.2, 0.94.11
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.99.0
>
> Attachments: HBASE-9426-4.patch, HBASE-9426-4.patch,
> HBASE-9426-6.patch, HBASE-9426-7.patch, HBASE-9426.patch.1,
> HBASE-9426.patch.2, HBASE-9426.patch.3
>
>
> Currently if one wants to implement a custom distributed barrier procedure
> (e.g., distributed log roll or distributed table flush), the HBase core code
> needs to be modified in order for the procedure to work.
> Looking into the snapshot code (especially on region server side), most of
> the code to enable the procedure are generic life-cycle management (i.e.,
> init, start, stop). We can make this part pluggable.
> Here is the proposal. Following the coprocessor example, we define two
> properties:
> {code}
> hbase.procedure.regionserver.classes
> hbase.procedure.master.classes
> {code}
> The values for both are comma delimited list of classes. On region server
> side, the classes implements the following interface:
> {code}
> public interface RegionServerProcedureManager {
> public void initialize(RegionServerServices rss) throws KeeperException;
> public void start();
> public void stop(boolean force) throws IOException;
> public String getProcedureName();
> }
> {code}
> While on Master side, the classes implement the interface:
> {code}
> public interface MasterProcedureManager {
> public void initialize(MasterServices master) throws KeeperException,
> IOException, UnsupportedOperationException;
> public void stop(String why);
> public String getProcedureName();
> public void execProcedure(ProcedureDescription desc) throws IOException;
> IOException;
> }
> {code}
> Where the ProcedureDescription is defined as
> {code}
> message ProcedureDescription {
> required string name = 1;
> required string instance = 2;
> optional int64 creationTime = 3 [default = 0];
> message Property {
> required string tag = 1;
> optional string value = 2;
> }
> repeated Property props = 4;
> }
> {code}
> A generic API can be defined on HMaster to trigger a procedure:
> {code}
> public boolean execProcedure(ProcedureDescription desc) throws IOException;
> {code}
> _SnapshotManager_ and _RegionServerSnapshotManager_ are special examples of
> _MasterProcedureManager_ and _RegionServerProcedureManager_. They will be
> automatically included (users don't need to specify them in the conf file).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10397) Fix findbugs introduced from HBASE-9426
[ https://issues.apache.org/jira/browse/HBASE-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879148#comment-13879148 ] Hudson commented on HBASE-10397: SUCCESS: Integrated in HBase-TRUNK #4849 (See [https://builds.apache.org/job/HBase-TRUNK/4849/]) HBASE-10397 Fix findbugs introduced from HBASE-9426. (anoopsamjohn: rev 1560427) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java > Fix findbugs introduced from HBASE-9426 > --- > > Key: HBASE-10397 > URL: https://issues.apache.org/jira/browse/HBASE-10397 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10397.patch > > > Method org.apache.hadoop.hbase.client.HBaseAdmin.execProcedure(String, > String, Map) makes inefficient use of keySet iterator instead of entrySet > iterator > Method org.apache.hadoop.hbase.client.HBaseAdmin.isProcedureFinished(String, > String, Map) makes inefficient use of keySet iterator instead of entrySet > iterator -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879131#comment-13879131 ] Andrew Purtell commented on HBASE-10338: Committed addendum to trunk, 0.98, and 0.96. Patch applied everywhere with minor fuzz only, failing test completes successfully with no NPE reported in the log. > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338.0.patch, HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879127#comment-13879127 ] Eric Charles commented on HBASE-10336: -- [~jxiang] With HADOOP-10254, TestInfoServer may be ok, but hbase will fail at runtime, at least I would bet on that. On the other hand HADOOP-10252 is not enough, a patch uploaded here (not the last one) would be needed. Btw, what about simply dropping any change to hadoop and concentrate on the hbase decoupling. It was the plan, no? Sorry, i have to leave. Read you tomorrow. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HBASE-10401) [hbck] perform overlap group merges in parallel
Jonathan Hsieh created HBASE-10401: -- Summary: [hbck] perform overlap group merges in parallel Key: HBASE-10401 URL: https://issues.apache.org/jira/browse/HBASE-10401 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.96.1.1, 0.94.16, 0.92.2, 0.98.0, 0.99.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh In a recent support case, we encountered a corrupt hbase that had thousands of "overlap groups" (regions that had overlapping key ranges). The current implementation repairs these by serially taking a group, perorming a merge and then moving on to the next group. Because assignments and hdfs nn operations are involved each merge could take on the order of seconds. With thousands of overlap groups, this could take hours to complete. This patch makes it so that these independent merge groups are merged in parallel. It uses the same thread pool for other fs info-gathering operations. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10392) Correct references to hbase.regionserver.global.memstore.upperLimit
[
https://issues.apache.org/jira/browse/HBASE-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879122#comment-13879122
]
Hadoop QA commented on HBASE-10392:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624388/HBASE-10392.2.patch
against trunk revision .
ATTACHMENT ID: 12624388
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8498//console
This message is automatically generated.
> Correct references to hbase.regionserver.global.memstore.upperLimit
> ---
>
> Key: HBASE-10392
> URL: https://issues.apache.org/jira/browse/HBASE-10392
> Project: HBase
> Issue Type: Bug
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 0.99.0
>
> Attachments: HBASE-10392.0.patch, HBASE-10392.1.patch,
> HBASE-10392.2.patch
>
>
> As part of the awesome new HBASE-5349, a couple references to
> {{hbase.regionserver.global.memstore.upperLimit}} was missed. Clean those up
> to use the new {{hbase.regionserver.global.memstore.size}} instead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879118#comment-13879118 ] Jonathan Hsieh commented on HBASE-10400: also applies to 0.96/0.98. I can do a 0.94 version if requested. > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-7320) Replace calls to KeyValue.getBuffer with appropropriate calls to getRowArray, getFamilyArray(), getQualifierArray, and getValueArray
[ https://issues.apache.org/jira/browse/HBASE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879120#comment-13879120 ] Matt Corgan commented on HBASE-7320: It would be nice to move towards callers of getTimestamp() expecting the "long" value returned by the Cell interface. I'm guessing there is little to negative performance gain operating on the bytes directly. > Replace calls to KeyValue.getBuffer with appropropriate calls to getRowArray, > getFamilyArray(), getQualifierArray, and getValueArray > > > Key: HBASE-7320 > URL: https://issues.apache.org/jira/browse/HBASE-7320 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: stack > Fix For: 0.98.0 > > > In many places this is simple task of just replacing the method name. > There, however, quite a few places where we assume that: > # the entire KV is backed by a single byte array > # the KVs key portion is backed by a single byte array > Some of those can easily be fixed, others will need their own jiras. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879116#comment-13879116 ] Jimmy Xiang commented on HBASE-10336: - [~echarles], you are right HADOOP-10252 is not complete. I filed HADOOP-10254, with which the mini cluster starts. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10400: --- Status: Patch Available (was: Open) > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.96.1.1, 0.94.16, 0.92.2 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
[ https://issues.apache.org/jira/browse/HBASE-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-10400: --- Attachment: hbase-10400.patch Applies to trunk. > [hbck] Continue if region dir missing on region merge attempt > - > > Key: HBASE-10400 > URL: https://issues.apache.org/jira/browse/HBASE-10400 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.92.2, 0.94.16, 0.96.1.1 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-10400.patch > > > In a recent support case, the hbck repair tool would eventually hang because > we didn't handle the case where merge info is old using hadoop2 fs.listStatus > semantics (throw exn instead of return null). this is a trivial patch that > handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879103#comment-13879103 ] Eric Charles commented on HBASE-10336: -- [~jxiang] HADOOP-10252 sounds like a duplicate of HADOOP-10232 for which [1] is a valid patch IMHO. the patch submitted for HADOOP-10252 [2] seems like the first try I made, but had to add more code to make it work [~apurtell] sorry, I should have said "IMHO, it would be safe to commit and release if no other issue is found after unit and integration tests + real deployment and traffic on a real cluster" [1] https://issues.apache.org/jira/secure/attachment/12622867/HDFS-5760-2.patch [2] https://issues.apache.org/jira/secure/attachment/12624372/hadoop-10252.patch > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HBASE-10400) [hbck] Continue if region dir missing on region merge attempt
Jonathan Hsieh created HBASE-10400: -- Summary: [hbck] Continue if region dir missing on region merge attempt Key: HBASE-10400 URL: https://issues.apache.org/jira/browse/HBASE-10400 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.96.1.1, 0.94.16, 0.92.2 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh In a recent support case, the hbck repair tool would eventually hang because we didn't handle the case where merge info is old using hadoop2 fs.listStatus semantics (throw exn instead of return null). this is a trivial patch that handles the newer hadoop2 semantics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10395) endTime won't be set in VerifyReplication if startTime is not set
[
https://issues.apache.org/jira/browse/HBASE-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879100#comment-13879100
]
Hudson commented on HBASE-10395:
SUCCESS: Integrated in HBase-0.98 #103 (See
[https://builds.apache.org/job/HBase-0.98/103/])
HBASE-10395 endTime won't be set in VerifyReplication if startTime is not set
(Jianwei Cui) (Tedyu: rev 1560433)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> endTime won't be set in VerifyReplication if startTime is not set
> -
>
> Key: HBASE-10395
> URL: https://issues.apache.org/jira/browse/HBASE-10395
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Replication
>Affects Versions: 0.94.16
>Reporter: cuijianwei
>Assignee: cuijianwei
>Priority: Minor
> Fix For: 0.98.0, 0.99.0
>
> Attachments: HBASE-10395-0.94-v1.patch, HBASE-10395-0.94-v2.patch,
> HBASE-10395-0.94-v3.patch, HBASE-10395-trunk-v1.patch,
> HBASE-10395-trunk-v2.patch, HBASE-10395-trunk-v3.patch
>
>
> In VerifyReplication, we may set startTime and endTime to restrict the data
> to verfiy. However, the endTime won't be set in the program if we only pass
> endTime without startTime in command line argument. The reason is the
> following code:
> {code}
> if (startTime != 0) {
> scan.setTimeRange(startTime,
> endTime == 0 ? HConstants.LATEST_TIMESTAMP : endTime);
> }
> {code}
> The code will ignore endTime setting when not passing startTime in command
> line argument. Another place needs to improvement is the help message as
> follows:
> {code}
> System.err.println(" stoprow end of the row");
> {code}
> However, the program actually use "endrow" to parse the arguments:
> {code}
> final String endTimeArgKey = "--endtime=";
> if (cmd.startsWith(endTimeArgKey)) {
> endTime = Long.parseLong(cmd.substring(endTimeArgKey.length()));
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10375) hbase-default.xml hbase.status.multicast.address.port does not match code
[
https://issues.apache.org/jira/browse/HBASE-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879099#comment-13879099
]
Hudson commented on HBASE-10375:
SUCCESS: Integrated in HBase-0.98 #103 (See
[https://builds.apache.org/job/HBase-0.98/103/])
HBASE-10375 hbase-default.xml hbase.status.multicast.address.port does not
match code (nkeywal: rev 1560320)
*
/hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.98/hbase-common/src/main/resources/hbase-default.xml
> hbase-default.xml hbase.status.multicast.address.port does not match code
> -
>
> Key: HBASE-10375
> URL: https://issues.apache.org/jira/browse/HBASE-10375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0, 0.96.1.1
>Reporter: Jonathan Hsieh
>Assignee: Nicolas Liochon
> Fix For: 0.98.0, 0.96.2, 0.99.0
>
> Attachments: 10375.v1.98-96.patch, 10375.v1.patch,
> 10375.v2.96-98.patch, 10375.v2.trunk.patch
>
>
> In hbase-default.xml
> {code}
> +
> +hbase.status.multicast.address.port
> +6100
> +
> + Multicast port to use for the status publication by multicast.
> +
> +
> {code}
> In HConstants it was 60100.
> {code}
> public static final String STATUS_MULTICAST_PORT =
> "hbase.status.multicast.port";
> public static final int DEFAULT_STATUS_MULTICAST_PORT = 16100;
> {code}
> (it was 60100 in the code for 0.96 and 0.98.)
> I lean towards going with the code as opposed to the config file.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879091#comment-13879091 ] Liyin Tang commented on HBASE-7404: --- Liang, just curious, what's the top contributor for the p99 latency in your case ? > Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE > -- > > Key: HBASE-7404 > URL: https://issues.apache.org/jira/browse/HBASE-7404 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.95.0 > > Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch, > 7404-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch, > 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf, > HBASE-7404-backport-0.94.patch, Introduction of Bucket Cache.pdf, > hbase-7404-94v2.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch > > > First, thanks @neil from Fusion-IO share the source code. > Usage: > 1.Use bucket cache as main memory cache, configured as the following: > –"hbase.bucketcache.ioengine" "heap" (or "offheap" if using offheap memory to > cache block ) > –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of > max heap size) > 2.Use bucket cache as a secondary cache, configured as the following: > –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path > where to store the block data) > –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 > means 1GB) > –"hbase.bucketcache.combinedcache.enabled" false (default value being true) > See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache > What's Bucket Cache? > It could greatly decrease CMS and heap fragment by GC > It support a large cache space for High Read Performance by using high speed > disk like Fusion-io > 1.An implementation of block cache like LruBlockCache > 2.Self manage blocks' storage position through Bucket Allocator > 3.The cached blocks could be stored in the memory or file system > 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), > combined with LruBlockCache to decrease CMS and fragment by GC. > 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to > store block) to enlarge cache space > How about SlabCache? > We have studied and test SlabCache first, but the result is bad, because: > 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds > of block size, especially using DataBlockEncoding > 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache > and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , > it causes CMS and heap fragment don't get any better > 3.Direct heap performance is not good as heap, and maybe cause OOM, so we > recommend using "heap" engine > See more in the attachment and in the patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879081#comment-13879081 ] Jimmy Xiang commented on HBASE-10336: - FYI HADOOP-10252 hasn't fixed all issues related to the deprecated constructor. I am still looking into it now. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-7320) Replace calls to KeyValue.getBuffer with appropropriate calls to getRowArray, getFamilyArray(), getQualifierArray, and getValueArray
[ https://issues.apache.org/jira/browse/HBASE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879080#comment-13879080 ] Lars Hofhansl commented on HBASE-7320: -- Exactly. Would have to think a bit more about the TS. Do we want to this be backed by byte[] as well, or just completely get rid of the array/offset/length API for TS and just a getTimeStamp method? > Replace calls to KeyValue.getBuffer with appropropriate calls to getRowArray, > getFamilyArray(), getQualifierArray, and getValueArray > > > Key: HBASE-7320 > URL: https://issues.apache.org/jira/browse/HBASE-7320 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: stack > Fix For: 0.98.0 > > > In many places this is simple task of just replacing the method name. > There, however, quite a few places where we assume that: > # the entire KV is backed by a single byte array > # the KVs key portion is backed by a single byte array > Some of those can easily be fixed, others will need their own jiras. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879071#comment-13879071 ] Andrew Purtell commented on HBASE-10336: bq. If Ted Yu confirms patch v5 works well, it is safe to commit and release. Well two HBase RMs disagree with your assessment. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879067#comment-13879067 ] Eric Charles commented on HBASE-10336: -- If [~te...@apache.org] confirms patch v5 works well, it is safe to commit and release. The cleaning is more about class servlet move to http package, gitignore file to remove, hadoop key to rename/document, @InterfaceAudience to be removed. This can be done after. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879055#comment-13879055 ] stack commented on HBASE-10336: --- [~wheat9] All well and good. We are working to undo our dependency. This patch will go in soon but as [~jxiang] says up in HADOOP-5305, we want both the deprecated APIs to remain in place a while. We've just 'learned' of your effort to purge this code in 2.x timeframe. Unless there is some hard reason that the deprecated methods cannot stay in place (other than you have not tested them -- we'll yell and patch if you break something), rather than piss off a downstream project and put us in an awkward spot, I suggest leave them in place deprecated. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879046#comment-13879046 ] Andrew Purtell edited comment on HBASE-10336 at 1/22/14 7:15 PM: - bq. I was about to test patch v5 which applies cleanly on 0.98 in a cluster. It doesn't matter because this change isn't going to make RC0. I'm guessing we won't have just one release candidate, but this is not likely to go in on any subsequent RC because it only addresses a future problem in exchange for risky big changes. was (Author: apurtell): bq. I was about to test patch v5 which applies cleanly on 0.98 in a cluster. It doesn't matter because this change isn't going to make RC0. I'm guessing we won't have just one release candidate, but a change of this magnitude is not likely to go in on any subsequent RC because it addresses a future problem. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879046#comment-13879046 ] Andrew Purtell commented on HBASE-10336: bq. I was about to test patch v5 which applies cleanly on 0.98 in a cluster. It doesn't matter because this change isn't going to make RC0. I'm guessing we won't have just one release candidate, but a change of this magnitude is not likely to go in on any subsequent RC because it addresses a future problem. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879037#comment-13879037
]
Haohui Mai commented on HBASE-10336:
HDFS-5305 and related jiras implement HTTPS support, as a side effect they have
changed a lot how {{HttpServer}} works.
I anticipate that there are more changes for {{HttpServer}} in the near term.
We want to keep it as an private API so that we can keep cleaning it up and
without worrying about compatibility.
For distribution, the approach of this jira might work better. It seems to me
that this is a reliable way to not making the table you've mentioned more
frightening. :-)
> Remove deprecated usage of Hadoop HttpServer in InfoServer
> --
>
> Key: HBASE-10336
> URL: https://issues.apache.org/jira/browse/HBASE-10336
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Eric Charles
>Assignee: Eric Charles
> Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch,
> HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch
>
>
> Recent changes in Hadoop HttpServer give NPE when running on hadoop
> 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be
> not fixed (see HDFS-5760). We'd better move to the new proposed builder
> pattern, which means we can no more use inheritance to build our nice
> InfoServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879039#comment-13879039 ] Ted Yu commented on HBASE-10336: @Andy: I was about to test patch v5 which applies cleanly on 0.98 in a cluster. Somehow there was rollback in local repo. I will proceed and report back. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879038#comment-13879038 ] stack commented on HBASE-10336: --- Is [~ted_yu] not picking up HADOOP-10252 in his testing or does that patch not totally fix the issue? > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879013#comment-13879013 ] Andrew Purtell commented on HBASE-10336: The exception Ted reported above has been reported upstream as HADOOP-10252 looks like. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10399) Add documentation for VerifyReplication to refguide
[ https://issues.apache.org/jira/browse/HBASE-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879012#comment-13879012 ] stack commented on HBASE-10399: --- [~ted_yu] You going to do it? Else suggest closing this issue. If we had an issue for all that was undocumented we'd be at 100k issues. > Add documentation for VerifyReplication to refguide > --- > > Key: HBASE-10399 > URL: https://issues.apache.org/jira/browse/HBASE-10399 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > HBase refguide currently doesn't document how VerifyReplication is used for > comparing local table with remote table. > Document for VerifyReplication should be added so that users know how to use > it. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879002#comment-13879002 ] stack commented on HBASE-10336: --- [~wheat9] it is deprecated... and the implication is? That you will purge in a 2.4 release? Why not in a 3.0? Which hadoop issue is this? Thanks. (For a whiff of why we are disturbed, we would like to avoid there being yet more variants on our hbase x hadoop table here http://hbase.apache.org/book.html#hadoop -- it is frightening enough as it is) > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878999#comment-13878999 ] Andrew Purtell commented on HBASE-10336: Thanks [~e...@apache.org] bq. but for which I miss one hour. my bad It's not you. The solution to this problem looks to need doing upstream. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878989#comment-13878989 ] Eric Charles commented on HBASE-10336: -- [~apurtell] I run hbase-trunk on hadoop-trunk. To run it successfully, I have two options: 1. The tiny patch of HADOOP-10232 rejected by [~wheat9] 2. The large patch of HBASE-10336 which is nearly ready, but for which I miss one hour. my bad, and I wouldn't want to inject incomplete and potentially buggy code in a hbase release. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878983#comment-13878983 ] Andrew Purtell commented on HBASE-10336: bq. There is a contract in place. It says we are allowed access at the head of the hadoop class. Lets fix hadoop branch-2 before it releases. Ok, you heard the man. This is very unlikely to make the 0.98.0 RC 0 because I want to tag it today after a couple more things go in. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878986#comment-13878986 ] Andrew Purtell commented on HBASE-10338: ... and will commit fix to 0.96 also, thanks [~stack] > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338.0.patch, HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10338) Region server fails to start with AccessController coprocessor if installed into RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878982#comment-13878982 ] stack commented on HBASE-10338: --- I need it in 0.96 too boss [~apurtell] Thanks. (Thanks [~avandana] for fixing while traveling). > Region server fails to start with AccessController coprocessor if installed > into RegionServerCoprocessorHost > > > Key: HBASE-10338 > URL: https://issues.apache.org/jira/browse/HBASE-10338 > Project: HBase > Issue Type: Bug > Components: Coprocessors, regionserver >Affects Versions: 0.98.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: 10338.1-0.96.patch, 10338.1-0.98.patch, 10338.1.patch, > 10338.1.patch, HBASE-10338.0.patch, HBASE-10338_addendum.patch > > > Runtime exception is being thrown when AccessController CP is used with > region server. This is happening as region server co processor host is > created before zookeeper is initialized in region server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878980#comment-13878980 ] Andrew Purtell commented on HBASE-10336: bq. h3-snapshot. h3-snapshot is branch-2 HEAD and HBase trunk, that right? > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878979#comment-13878979
]
Haohui Mai commented on HBASE-10336:
The following code has been deprecated:
{code}
+
+ static {
+Configuration conf = new Configuration();
+boolean sslEnabled = conf.getBoolean(
+CommonConfigurationKeysPublic.HADOOP_SSL_ENABLED_KEY,
+CommonConfigurationKeysPublic.HADOOP_SSL_ENABLED_DEFAULT);
+policy = sslEnabled ? Policy.HTTPS_ONLY : Policy.HTTP_ONLY;
+ }
+
+ public static void setPolicy(Policy policy) {
+HttpConfig.policy = policy;
+ }
+
+ public static boolean isSecure() {
+return policy == Policy.HTTPS_ONLY;
+ }
+
+ public static String getSchemePrefix() {
+return (isSecure()) ? "https://"; : "http://";;
+ }
+
+ public static String getScheme(Policy policy) {
+return policy == Policy.HTTPS_ONLY ? "https://"; : "http://";;
+ }
{code}
{code}
+/**
+ * Use setAppDir() instead.
+ */
+@Deprecated
+public Builder setName(String name){
+ this.name = name;
+ return this;
+}
+
+/**
+ * Use addEndpoint() instead.
+ */
+@Deprecated
+public Builder setBindAddress(String bindAddress){
+ this.bindAddress = bindAddress;
+ return this;
+}
+
+/**
+ * Use addEndpoint() instead.
+ */
+@Deprecated
+public Builder setPort(int port) {
+ this.port = port;
+ return this;
+}
+
{code}
And there are significant amount of code in {{HttpServer}} has been marked as
deprecated. You might want to get rid of them as well.
> Remove deprecated usage of Hadoop HttpServer in InfoServer
> --
>
> Key: HBASE-10336
> URL: https://issues.apache.org/jira/browse/HBASE-10336
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Eric Charles
>Assignee: Eric Charles
> Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch,
> HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch
>
>
> Recent changes in Hadoop HttpServer give NPE when running on hadoop
> 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be
> not fixed (see HDFS-5760). We'd better move to the new proposed builder
> pattern, which means we can no more use inheritance to build our nice
> InfoServer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878978#comment-13878978 ] stack commented on HBASE-10336: --- Not for a point release in 0.96. The exception should be carried back up to hadoop. There is a contract in place. It says we are allowed access at the head of the hadoop class. Lets fix hadoop branch-2 before it releases. > Remove deprecated usage of Hadoop HttpServer in InfoServer > -- > > Key: HBASE-10336 > URL: https://issues.apache.org/jira/browse/HBASE-10336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.99.0 >Reporter: Eric Charles >Assignee: Eric Charles > Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, > HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch > > > Recent changes in Hadoop HttpServer give NPE when running on hadoop > 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be > not fixed (see HDFS-5760). We'd better move to the new proposed builder > pattern, which means we can no more use inheritance to build our nice > InfoServer. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
