[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985275#comment-16985275
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #195 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/195/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/195//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/195//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/195//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23337) Several modules missing in nexus for Apache HBase 2.2.2
[ https://issues.apache.org/jira/browse/HBASE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23337: Status: In Progress (was: Patch Available) > Several modules missing in nexus for Apache HBase 2.2.2 > --- > > Key: HBASE-23337 > URL: https://issues.apache.org/jira/browse/HBASE-23337 > Project: HBase > Issue Type: Bug > Components: build, community, scripts >Affects Versions: 2.2.2 >Reporter: Chao >Assignee: Sean Busbey >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HBASE-23337.0.patch, HBASE-23337.1.patch > > > The latest version of hbase-shaded-client is currently 2.2.1. It has been a > while since 2.2.2 release (2019/10/25). See: > [https://search.maven.org/search?q=hbase-shaded-client]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23337) Several modules missing in nexus for Apache HBase 2.2.2
[
https://issues.apache.org/jira/browse/HBASE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985246#comment-16985246
]
Sean Busbey commented on HBASE-23337:
-
bq. Could you please create a PR for this patch?
sure. I found an error I need to fix first.
bq. On the gpg part, it works for me in the past, not sure why it does not work
for you...
The failure mode is just prompting me an extra time for the password I already
gave it. If I were around to notice it I think the rest would go fine. AFAICT
it's not related to this change, so I'm happy to deal with it elsewhere.
{quote}
bq. Click close manually on https://repository.apache.org/ is not a big deal...
Just to note that main value of this script is end-to-end RM'ing. If it don't
do that, it won't get used.
{quote}
Please note that this isn't an issue with the current patch. It successfully
closes the staging repo and presents the staged repo name to the RM.
bq. If the gpg agent thing ain't working, suggest putting in a comment inline
into the script so next person along doesn't think it is something they are
doing wrong. ?
Like where though? AFAICT the gpg agent handling isn't working correctly on my
box with our without this change. If no one else has noticed an issue I'd
rather not confuse things with a warning.
{quote}
Hard to tell... is this section right... We replacing the popd+rm-rf w/ a '}'.
May be just formatting.
{code}
300
301 popd
302 rm -rf "$tmp_repo"
303 cd ..
237 )
{code}
{quote}
yeah I switched to a subshell so that I didn't have to worry about changing out
of the project directory. for example, the current code leaves you dropped in
the project directory when something goes wrong. Let me make sure the temp repo
clean up still happens.
(as an aside, this was the cause of a problem with the current patch because
the name of the repo is scoped to just inside the subshell, so it is blank in
the email that gets generated. but I'll fix that shortly.)
bq. The remove of the GPG define in release-build.sh is because it not needed
anymore now using mvn plugin?
yes. originally I didn't muck with it but shellcheck called me out for having
an unused variable.
> Several modules missing in nexus for Apache HBase 2.2.2
> ---
>
> Key: HBASE-23337
> URL: https://issues.apache.org/jira/browse/HBASE-23337
> Project: HBase
> Issue Type: Bug
> Components: build, community, scripts
>Affects Versions: 2.2.2
>Reporter: Chao
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HBASE-23337.0.patch, HBASE-23337.1.patch
>
>
> The latest version of hbase-shaded-client is currently 2.2.1. It has been a
> while since 2.2.2 release (2019/10/25). See:
> [https://search.maven.org/search?q=hbase-shaded-client].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22075) Potential data loss when MOB compaction fails
[ https://issues.apache.org/jira/browse/HBASE-22075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985239#comment-16985239 ] Bo Cui edited comment on HBASE-22075 at 11/30/19 3:22 AM: -- Is it better to add the HBASE-23353 solution? pls check was (Author: bo cui): Is it better to add the hbase-23353 solution? pls check > Potential data loss when MOB compaction fails > - > > Key: HBASE-22075 > URL: https://issues.apache.org/jira/browse/HBASE-22075 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.1.0, 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4, > 2.1.3 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: compaction, mob > Fix For: 2.0.7, 2.2.3, 2.1.9 > > Attachments: HBASE-22075-v1.patch, HBASE-22075-v2.patch, > HBASE-22075.test-only.0.patch, HBASE-22075.test-only.1.patch, > HBASE-22075.test-only.2.patch, ReproMOBDataLoss.java > > > When MOB compaction fails during last step (bulk load of a newly created > reference file) there is a high chance of a data loss due to partially loaded > reference file, cells of which refer to (now) non-existent MOB file. The > newly created MOB file is deleted automatically in case of a MOB compaction > failure, but some cells with the references to this file might be loaded to > HBase. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22075) Potential data loss when MOB compaction fails
[ https://issues.apache.org/jira/browse/HBASE-22075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985239#comment-16985239 ] Bo Cui commented on HBASE-22075: Is it better to add the hbase-23353 solution? pls check > Potential data loss when MOB compaction fails > - > > Key: HBASE-22075 > URL: https://issues.apache.org/jira/browse/HBASE-22075 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.1.0, 2.0.0, 2.0.1, 2.1.1, 2.0.2, 2.0.3, 2.1.2, 2.0.4, > 2.1.3 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: compaction, mob > Fix For: 2.0.7, 2.2.3, 2.1.9 > > Attachments: HBASE-22075-v1.patch, HBASE-22075-v2.patch, > HBASE-22075.test-only.0.patch, HBASE-22075.test-only.1.patch, > HBASE-22075.test-only.2.patch, ReproMOBDataLoss.java > > > When MOB compaction fails during last step (bulk load of a newly created > reference file) there is a high chance of a data loss due to partially loaded > reference file, cells of which refer to (now) non-existent MOB file. The > newly created MOB file is deleted automatically in case of a MOB compaction > failure, but some cells with the references to this file might be loaded to > HBase. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Description: in PartitionedMobCompactor#bulkloadRefFile If region num is changed from 1 to 20 bulkload will fail and we will get an exception (because the bulkload maxRetries is 10，hbase.bulkload.retries.number) !image-2019-11-30-10-30-33-243.png! and then delete bulkload path and new MobFile, old mob file is not deleted. but some hfiles have been successfully loaded through bulkload and not deleted. !image-2019-11-30-10-40-32-837.png! And then get rowkeythrow filenotfound solution : write multiple hfiles and reduce split hfiles. because the data of scanner is ordered，we just need to open a hfile at the same time, when the current data dose not belong to the current hfile, we can close hfile and create new hfile. !image-2019-11-30-10-33-00-867.png! was: in PartitionedMobCompactor#bulkloadRefFile If region num is changed from 1 to 20 bulkload will fail and we will get an exception !image-2019-11-30-10-30-33-243.png! and then delete bulkload path and new MobFile, old mob file is not deleted. but some hfiles have been successfully loaded through bulkload and not deleted. !image-2019-11-30-10-40-32-837.png! And then get rowkeythrow filenotfound solution : write multiple hfiles and reduce split hfiles. because the data of scanner is ordered，we just need to open a hfile at the same time, when the current data dose not belong to the current hfile, we can close hfile and create new hfile. !image-2019-11-30-10-33-00-867.png! > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-30-33-243.png, > image-2019-11-30-10-33-00-867.png, image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception (because the bulkload > maxRetries is 10，hbase.bulkload.retries.number) > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-40-32-837.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Affects Version/s: (was: 1.3.1) > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-30-33-243.png, > image-2019-11-30-10-33-00-867.png, image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-40-32-837.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Attachment: (was: image-2019-11-30-10-27-57-345.png) > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 1.3.1, 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-30-33-243.png, > image-2019-11-30-10-33-00-867.png, image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-40-32-837.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Attachment: (was: image-2019-11-30-10-23-44-540.png) > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 1.3.1, 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-30-33-243.png, > image-2019-11-30-10-33-00-867.png, image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-40-32-837.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Attachment: (was: image-2019-11-30-10-19-03-625.png) > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 1.3.1, 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-30-33-243.png, > image-2019-11-30-10-33-00-867.png, image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-40-32-837.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Description: in PartitionedMobCompactor#bulkloadRefFile If region num is changed from 1 to 20 bulkload will fail and we will get an exception !image-2019-11-30-10-30-33-243.png! and then delete bulkload path and new MobFile, old mob file is not deleted. but some hfiles have been successfully loaded through bulkload and not deleted. !image-2019-11-30-10-40-32-837.png! And then get rowkeythrow filenotfound solution : write multiple hfiles and reduce split hfiles. because the data of scanner is ordered，we just need to open a hfile at the same time, when the current data dose not belong to the current hfile, we can close hfile and create new hfile. !image-2019-11-30-10-33-00-867.png! was: in PartitionedMobCompactor#bulkloadRefFile If region num is changed from 1 to 20 bulkload will fail and we will get an exception !image-2019-11-30-10-30-33-243.png! and then delete bulkload path and new MobFile, old mob file is not deleted. but some hfiles have been successfully loaded through bulkload and not deleted. !image-2019-11-30-10-27-57-345.png! And then get rowkeythrow filenotfound solution : write multiple hfiles and reduce split hfiles. because the data of scanner is ordered，we just need to open a hfile at the same time, when the current data dose not belong to the current hfile, we can close hfile and create new hfile. !image-2019-11-30-10-33-00-867.png! > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 1.3.1, 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-19-03-625.png, > image-2019-11-30-10-23-44-540.png, image-2019-11-30-10-27-57-345.png, > image-2019-11-30-10-30-33-243.png, image-2019-11-30-10-33-00-867.png, > image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-40-32-837.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23353) MOB throws FileNotFound
[ https://issues.apache.org/jira/browse/HBASE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Cui updated HBASE-23353: --- Attachment: image-2019-11-30-10-40-32-837.png > MOB throws FileNotFound > --- > > Key: HBASE-23353 > URL: https://issues.apache.org/jira/browse/HBASE-23353 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 1.3.1, 2.1.1, 2.2.2 >Reporter: Bo Cui >Priority: Critical > Attachments: image-2019-11-30-10-19-03-625.png, > image-2019-11-30-10-23-44-540.png, image-2019-11-30-10-27-57-345.png, > image-2019-11-30-10-30-33-243.png, image-2019-11-30-10-33-00-867.png, > image-2019-11-30-10-40-32-837.png > > > in PartitionedMobCompactor#bulkloadRefFile > If region num is changed from 1 to 20 > bulkload will fail and we will get an exception > !image-2019-11-30-10-30-33-243.png! > and then delete bulkload path and new MobFile, > old mob file is not deleted. > but some hfiles have been successfully loaded through bulkload and not > deleted. > !image-2019-11-30-10-27-57-345.png! > And then get rowkeythrow filenotfound > solution : write multiple hfiles and reduce split hfiles. because the data of > scanner is ordered，we just need to open a hfile at the same time, when the > current data dose not belong to the current hfile, we can close hfile and > create new hfile. > !image-2019-11-30-10-33-00-867.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23353) MOB throws FileNotFound
Bo Cui created HBASE-23353: -- Summary: MOB throws FileNotFound Key: HBASE-23353 URL: https://issues.apache.org/jira/browse/HBASE-23353 Project: HBase Issue Type: Bug Components: mob Affects Versions: 2.2.2, 2.1.1, 1.3.1 Reporter: Bo Cui Attachments: image-2019-11-30-10-19-03-625.png, image-2019-11-30-10-23-44-540.png, image-2019-11-30-10-27-57-345.png, image-2019-11-30-10-30-33-243.png, image-2019-11-30-10-33-00-867.png in PartitionedMobCompactor#bulkloadRefFile If region num is changed from 1 to 20 bulkload will fail and we will get an exception !image-2019-11-30-10-30-33-243.png! and then delete bulkload path and new MobFile, old mob file is not deleted. but some hfiles have been successfully loaded through bulkload and not deleted. !image-2019-11-30-10-27-57-345.png! And then get rowkeythrow filenotfound solution : write multiple hfiles and reduce split hfiles. because the data of scanner is ordered，we just need to open a hfile at the same time, when the current data dose not belong to the current hfile, we can close hfile and create new hfile. !image-2019-11-30-10-33-00-867.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs URL: https://github.com/apache/hbase/pull/884#discussion_r352258475 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/security/provider/TestCustomSaslAuthenticationProvider.java ## @@ -0,0 +1,534 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.security.provider; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.ByteArrayInputStream; +import java.io.DataInput; +import java.io.DataInputStream; +import java.io.DataOutput; +import java.io.File; +import java.io.IOException; +import java.security.PrivilegedExceptionAction; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.ConcurrentHashMap; +import java.util.stream.Collectors; + +import javax.security.auth.callback.Callback; +import javax.security.auth.callback.CallbackHandler; +import javax.security.auth.callback.NameCallback; +import javax.security.auth.callback.PasswordCallback; +import javax.security.auth.callback.UnsupportedCallbackException; +import javax.security.sasl.AuthorizeCallback; +import javax.security.sasl.RealmCallback; +import javax.security.sasl.RealmChoiceCallback; +import javax.security.sasl.Sasl; +import javax.security.sasl.SaslClient; +import javax.security.sasl.SaslServer; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.HBaseClassTestRule; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.LocalHBaseCluster; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.Admin; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.RetriesExhaustedException; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.TableDescriptorBuilder; +import org.apache.hadoop.hbase.coprocessor.CoprocessorHost; +import org.apache.hadoop.hbase.ipc.BlockingRpcClient; +import org.apache.hadoop.hbase.ipc.RpcClientFactory; +import org.apache.hadoop.hbase.ipc.RpcServerFactory; +import org.apache.hadoop.hbase.ipc.SimpleRpcServer; +import org.apache.hadoop.hbase.security.AccessDeniedException; +import org.apache.hadoop.hbase.security.HBaseKerberosUtils; +import org.apache.hadoop.hbase.security.SaslUtil; +import org.apache.hadoop.hbase.security.token.SecureTestCluster; +import org.apache.hadoop.hbase.security.token.TokenProvider; +import org.apache.hadoop.hbase.shaded.protobuf.generated.RPCProtos.UserInformation; +import org.apache.hadoop.hbase.testclassification.MediumTests; +import org.apache.hadoop.hbase.testclassification.SecurityTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.FSUtils; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.io.WritableUtils; +import org.apache.hadoop.minikdc.MiniKdc; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod; +import org.apache.hadoop.security.token.SecretManager; +import org.apache.hadoop.security.token.SecretManager.InvalidToken; +import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.security.token.TokenIdentifier; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.ClassRule; +import org.junit.Rule; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.rules.TestName; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable
custom authentication methods for RPCs
URL: https://github.com/apache/hbase/pull/884#discussion_r352258561
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/security/provider/SimpleSaslServerAuthenticationProvider.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.security.provider;
+
+import java.io.IOException;
+import java.util.Map;
+
+import javax.security.sasl.SaslServer;
+
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.SecretManager;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.AUTHENTICATION)
+@InterfaceStability.Evolving
+public class SimpleSaslServerAuthenticationProvider extends
SimpleSaslClientAuthenticationProvider
+implements SaslServerAuthenticationProvider {
+
+ @Override
+ public SaslServer createServer(SecretManager secretManager,
+ Map saslProps) throws IOException {
+return null;
Review comment:
Seems odd creating a null server.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable
custom authentication methods for RPCs
URL: https://github.com/apache/hbase/pull/884#discussion_r352258730
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/security/provider/GssSaslServerAuthenticationProvider.java
##
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.security.provider;
+
+import java.io.IOException;
+import java.security.PrivilegedExceptionAction;
+import java.util.Map;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.AuthorizeCallback;
+import javax.security.sasl.Sasl;
+import javax.security.sasl.SaslException;
+import javax.security.sasl.SaslServer;
+
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.security.AccessDeniedException;
+import org.apache.hadoop.hbase.security.SaslUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.SecretManager;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.AUTHENTICATION)
+@InterfaceStability.Evolving
+public class GssSaslServerAuthenticationProvider extends
GssSaslClientAuthenticationProvider implements SaslServerAuthenticationProvider
{
+ private static final Logger LOG =
LoggerFactory.getLogger(GssSaslServerAuthenticationProvider.class);
+
+ @Override
+ public SaslServer createServer(SecretManager secretManager,
+ Map saslProps) throws IOException {
+UserGroupInformation current = UserGroupInformation.getCurrentUser();
+String fullName = current.getUserName();
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Server's Kerberos principal name is " + fullName);
Review comment:
parameterized logging and no need if the if LOG.isDebugEnabled check?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable
custom authentication methods for RPCs
URL: https://github.com/apache/hbase/pull/884#discussion_r352258460
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/security/provider/TestSaslServerAuthenticationProviders.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.security.provider;
+
+import static org.junit.Assert.*;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.junit.Test;
+
+public class TestSaslServerAuthenticationProviders {
Review comment:
Test is missing annotations and a ClassRule IIRC? I think build will alert
you that you've left stuff out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs URL: https://github.com/apache/hbase/pull/884#discussion_r352258502 ## File path: hbase-server/src/main/resources/META-INF/services/org.apache.hadoop.hbase.security.provider.SaslServerAuthenticationProvider ## @@ -0,0 +1,3 @@ +org.apache.hadoop.hbase.security.provider.DigestSaslServerAuthenticationProvider +org.apache.hadoop.hbase.security.provider.GssSaslServerAuthenticationProvider +org.apache.hadoop.hbase.security.provider.SimpleSaslServerAuthenticationProvider Review comment: Unused imports? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs URL: https://github.com/apache/hbase/pull/884#discussion_r352258386 ## File path: hbase-server/src/test/resources/log4j.properties ## @@ -68,3 +68,4 @@ log4j.logger.org.apache.hadoop.metrics2.impl.MetricsSystemImpl=WARN log4j.logger.org.apache.hadoop.metrics2.util.MBeans=WARN # Enable this to get detailed connection error/retry logging. # log4j.logger.org.apache.hadoop.hbase.client.ConnectionImplementation=TRACE +log4j.logger.org.apache.directory=WARN Review comment: o.a.directory emits too much log? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs
saintstack commented on a change in pull request #884: HBASE-23347 Allowable custom authentication methods for RPCs URL: https://github.com/apache/hbase/pull/884#discussion_r352258514 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/security/provider/SimpleSaslServerAuthenticationProvider.java ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.security.provider; Review comment: Should this all be in the rpc package? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] romani edited a comment on issue #783: HBASE-23242: [checkstyle] Make LineLength a child of Checker module.
romani edited a comment on issue #783: HBASE-23242: [checkstyle] Make LineLength a child of Checker module. URL: https://github.com/apache/hbase/pull/783#issuecomment-559896163 > Intellij defaults to v8.25 which is incompatible right now Eclipse and Idea plugins support latest checkstyle (8.26), so update to latest should not problem, but config must be changed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] romani commented on issue #783: HBASE-23242: [checkstyle] Make LineLength a child of Checker module.
romani commented on issue #783: HBASE-23242: [checkstyle] Make LineLength a child of Checker module. URL: https://github.com/apache/hbase/pull/783#issuecomment-559896163 > Intellij defaults to v8.25 which is incompatible right now Eclipse and Idea plugins support latest checkstyle, so update to latest should not problem, but config must be changed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #886: HBASE-23323 Update downloads page for Apaceh HBase 1.4.12.
Apache-HBase commented on issue #886: HBASE-23323 Update downloads page for Apaceh HBase 1.4.12. URL: https://github.com/apache/hbase/pull/886#issuecomment-559873886 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 49s | master passed | | +1 :green_heart: | mvnsite | 18m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 14s | the patch passed | | +1 :green_heart: | mvnsite | 19m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 51m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-886/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/886 | | Optional Tests | dupname asflicense mvnsite xml | | uname | Linux 7565226a8d12 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-886/out/precommit/personality/provided.sh | | git revision | master / d6539350ed | | Max. process+thread count | 93 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-886/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23323) Release 1.4.12
[ https://issues.apache.org/jira/browse/HBASE-23323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985159#comment-16985159 ] Sean Busbey commented on HBASE-23323: - * vote passed: https://s.apache.org/hbase-1.4.12-vote-passes * promoted the staged nexus repo * moved the artifacts from dev to release * updated the stable pointer * updated reporter.a.o * pushed rel/1.4.12 tag * updated branch-1.4 with release date and updated version to next SNAPSHOT * made a pr with website changes: https://github.com/apache/hbase/pull/886 > Release 1.4.12 > -- > > Key: HBASE-23323 > URL: https://issues.apache.org/jira/browse/HBASE-23323 > Project: HBase > Issue Type: Improvement > Components: build, community >Affects Versions: 1.4.12 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.12 > > Attachments: Flaky_report_branch-1.4_build_575.zip > > > it's been about a month and it doesn't seem like we're ready to move the > stable pointer. also we have at least one CVE in branch-1 and the critical > not-cleaning-wals thing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey opened a new pull request #886: HBASE-23323 Update downloads page for Apaceh HBase 1.4.12.
busbey opened a new pull request #886: HBASE-23323 Update downloads page for Apaceh HBase 1.4.12. URL: https://github.com/apache/hbase/pull/886 I don't think mirrors have finished updating yet, but these changes should be correct once they do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-23073: - Attachment: HBASE-23073.branch-1.0002.patch > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.0002.patch, > HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-23073: - Status: In Progress (was: Patch Available) > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on a change in pull request #749: HBASE-23205 Correctly update the position of WALs currently being replicated
wchevreuil commented on a change in pull request #749: HBASE-23205 Correctly
update the position of WALs currently being replicated
URL: https://github.com/apache/hbase/pull/749#discussion_r352175795
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##
@@ -436,6 +439,56 @@ public void testWALKeySerialization() throws Exception {
}
}
+ @Test
+ public void testReplicationSourceWALReaderThreadWithFilter() throws
Exception {
+final byte[] notReplicatedCf = Bytes.toBytes("notReplicated");
+final Map> tableCfs = new HashMap<>();
+tableCfs.put(tableName, Collections.singletonList(Bytes.toString(family)));
+ReplicationPeer peer = mock(ReplicationPeer.class);
+when(peer.getTableCFs()).thenReturn(tableCfs);
+WALEntryFilter filter = new ChainWALEntryFilter(new
TableCfWALEntryFilter(peer));
+
+appendToLogPlus(3, notReplicatedCf);
+
+Path firstWAL = walQueue.peek();
+ReplicationSourceManager mockSourceManager =
mock(ReplicationSourceManager.class);
+when(mockSourceManager.getTotalBufferUsed()).thenReturn(new AtomicLong(0));
+final ReplicationSourceWALReaderThread reader =
+new ReplicationSourceWALReaderThread(mockSourceManager,
getQueueInfo(), walQueue,
+0, fs, conf, filter, new MetricsSource("1"));
+reader.start();
+
+// reader won't put any batch, even if EOF reached.
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future future = executor.submit(new
Callable() {
+ @Override
+ public WALEntryBatch call() throws Exception {
+return reader.take();
+ }
+});
+Thread.sleep(2000);
+assertFalse(future.isDone());
Review comment:
Just a _heads up_ here: we may simplify this part if we decide to make
`ReplicationSourceWALReaderThread.lastReadPosition` exposed via _getter_ method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on a change in pull request #749: HBASE-23205 Correctly update the position of WALs currently being replicated
wchevreuil commented on a change in pull request #749: HBASE-23205 Correctly
update the position of WALs currently being replicated
URL: https://github.com/apache/hbase/pull/749#discussion_r352161498
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestWALEntryStream.java
##
@@ -378,37 +380,35 @@ public void testReplicationSourceWALReaderThread()
throws Exception {
}
@Test
- public void testReplicationSourceUpdatesLogPositionOnFilteredEntries()
throws Exception {
+ public void testReplicationSourceWALReaderThreadRecoveredQueue() throws
Exception {
Review comment:
Actually yeah, this is indeed simulating recovered queue when creating it as
recovered. I think this test is fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on a change in pull request #749: HBASE-23205 Correctly update the position of WALs currently being replicated
wchevreuil commented on a change in pull request #749: HBASE-23205 Correctly
update the position of WALs currently being replicated
URL: https://github.com/apache/hbase/pull/749#discussion_r352168908
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReaderThread.java
##
@@ -135,59 +127,46 @@ public void run() {
try (WALEntryStream entryStream =
new WALEntryStream(logQueue, fs, conf, currentPosition, metrics)) {
while (isReaderRunning()) { // loop here to keep reusing stream while
we can
- if (!checkQuota()) {
+ if (manager.isBufferQuotaReached()) {
+Threads.sleep(sleepForRetries);
continue;
}
- WALEntryBatch batch = null;
- while (entryStream.hasNext()) {
-if (batch == null) {
- batch = new WALEntryBatch(replicationBatchCountCapacity,
entryStream.getCurrentPath());
-}
+ WALEntryBatch batch =
+ new WALEntryBatch(replicationBatchCountCapacity,
replicationBatchSizeCapacity);
+ boolean hasNext;
+ while ((hasNext = entryStream.hasNext()) == true) {
Entry entry = entryStream.next();
entry = filterEntry(entry);
if (entry != null) {
WALEdit edit = entry.getEdit();
if (edit != null && !edit.isEmpty()) {
-long entrySize = getEntrySizeIncludeBulkLoad(entry);
-long entrySizeExlucdeBulkLoad =
getEntrySizeExcludeBulkLoad(entry);
-batch.addEntry(entry);
-replicationSourceManager.setPendingShipment(true);
-updateBatchStats(batch, entry, entryStream.getPosition(),
entrySize);
-boolean totalBufferTooLarge =
acquireBufferQuota(entrySizeExlucdeBulkLoad);
+long entrySizeExcludeBulkLoad = batch.addEntry(entry);
+boolean totalBufferTooLarge =
manager.acquireBufferQuota(entrySizeExcludeBulkLoad);
// Stop if too many entries or too big
-if (totalBufferTooLarge || batch.getHeapSize() >=
replicationBatchSizeCapacity
-|| batch.getNbEntries() >= replicationBatchCountCapacity) {
+if (totalBufferTooLarge || batch.isLimitReached()) {
break;
}
}
-} else {
Review comment:
I guess this is fine for the replication progress problem. One additional
issue, though, is regarding monitoring. IIRC, DumpReplicationQueues relies on
replication info available at ZK, so now it may not show an accurate position
for the log queue. We may need to expose
`ReplicationSourceWALReaderThread.lastReadPosition` via _getter_ method for
monitoring purposes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #832: HBASE-23298 Refactor LogRecoveredEditsOutputSink and BoundedLogWriter…
Apache-HBase commented on issue #832: HBASE-23298 Refactor LogRecoveredEditsOutputSink and BoundedLogWriter… URL: https://github.com/apache/hbase/pull/832#issuecomment-559824836 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 27s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | checkstyle | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 4m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | +0 :ok: | spotbugs | 4m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 4s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | checkstyle | 1m 18s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 53s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | | +1 :green_heart: | findbugs | 4m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 279m 5s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 337m 20s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | | hadoop.hbase.client.TestFromClientSide | | | hadoop.hbase.client.TestFromClientSideWithCoprocessor | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-832/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/832 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b3c3ce76b5ef 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-832/out/precommit/personality/provided.sh | | git revision | master / d6539350ed | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-832/10/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-832/10/testReport/ | | Max. process+thread count | 4911 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-832/10/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23352) Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction
[
https://issues.apache.org/jira/browse/HBASE-23352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985062#comment-16985062
]
HBase QA commented on HBASE-23352:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 5m
7s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-885/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/885 |
| JIRA Issue | HBASE-23352 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cf319aa4c8bf 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-885/out/precommit/personality/provided.sh
|
| git revision | master / d6539350ed |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-885/1/testReport/
|
| Max. process+thread count | 404 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
|
[GitHub] [hbase] Apache-HBase commented on issue #885: HBASE-23352: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction
Apache-HBase commented on issue #885: HBASE-23352: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction URL: https://github.com/apache/hbase/pull/885#issuecomment-559817656 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 37s | master passed | | +1 :green_heart: | compile | 0m 32s | master passed | | +1 :green_heart: | checkstyle | 0m 18s | master passed | | +1 :green_heart: | shadedjars | 4m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | master passed | | +0 :ok: | spotbugs | 5m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 2s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | checkstyle | 0m 18s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 42s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | | +1 :green_heart: | findbugs | 0m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 51s | hbase-it in the patch passed. | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 45m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-885/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/885 | | JIRA Issue | HBASE-23352 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux cf319aa4c8bf 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-885/out/precommit/personality/provided.sh | | git revision | master / d6539350ed | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-885/1/testReport/ | | Max. process+thread count | 404 (vs. ulimit of 1) | | modules | C: hbase-it U: hbase-it | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-885/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-20395) Displaying thrift server type on the thrift page
[
https://issues.apache.org/jira/browse/HBASE-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985048#comment-16985048
]
Hudson commented on HBASE-20395:
Results for branch master
[build #1551 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1551/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1551//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1551//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1551//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Displaying thrift server type on the thrift page
>
>
> Key: HBASE-20395
> URL: https://issues.apache.org/jira/browse/HBASE-20395
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-20395.addendum.patch,
> HBASE-20395.master.001.patch, HBASE-20395.master.002.patch,
> HBASE-20395.master.003.patch, HBASE-20395.master.004.patch,
> HBASE-20395.master.005.patch, HBASE-20395.master.005.patch, result.png
>
>
> HBase supports two types of thrift server: thrift and thrift2.
> But after start the thrift server successfully, we can not get the thrift
> server type conveniently.
> So, displaying thrift server type on the thrift page may provide some
> convenience for the users.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23348) Spark's createTable method throws an exception while the table is being split
[
https://issues.apache.org/jira/browse/HBASE-23348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros resolved HBASE-23348.
-
Fix Version/s: connector-1.0.1
Resolution: Fixed
Thanks [~rabikumar.kc]!
> Spark's createTable method throws an exception while the table is being split
> -
>
> Key: HBASE-23348
> URL: https://issues.apache.org/jira/browse/HBASE-23348
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Balazs Meszaros
>Assignee: Rabi Kumar K C
>Priority: Major
> Labels: beginner
> Fix For: connector-1.0.1
>
>
> {{HBaseRelation.createTable}} checks table existence with
> {{HBaseAdmin.isTableAvailable}} method
> [here|https://github.com/apache/hbase-connectors/blob/master/spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/DefaultSource.scala#L167].
> This is unfortunate, because it can return {{false}} while splitting, so
> {{createTable}} will fail. It should use {{tableExists}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-connectors] iadamcsik-cldr edited a comment on issue #47: HBASE-23295 HBaseContext should use most recent delegation token
iadamcsik-cldr edited a comment on issue #47: HBASE-23295 HBaseContext should use most recent delegation token URL: https://github.com/apache/hbase-connectors/pull/47#issuecomment-559795697 #52 should be merged first as currently test would fail, but not related to this change. The reason the checks aren't failing is that the unit tests are not being run by yetus, as @petersomogyi noticed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] meszibalu merged pull request #50: HBASE-23348 Spark's createTable method throws an exception while the table is being split
meszibalu merged pull request #50: HBASE-23348 Spark's createTable method throws an exception while the table is being split URL: https://github.com/apache/hbase-connectors/pull/50 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] BukrosSzabolcs opened a new pull request #885: HBASE-23352: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction
BukrosSzabolcs opened a new pull request #885: HBASE-23352: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction URL: https://github.com/apache/hbase/pull/885 Instead of using the default properties when checking for monkey properties, now we use the ones already extended with command line params. Change FillDiskCommandAction to try to stop the remote process if the command failed with an exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] asf-ci commented on issue #52: HBASE-23351 updating hbase version to 2.2.2
asf-ci commented on issue #52: HBASE-23351 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/52#issuecomment-559801295 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-CONNECTORS-Build/95/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (HBASE-23351) hbase-connectors Test fail when stating HBase mini cluster
[
https://issues.apache.org/jira/browse/HBASE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
István Adamcsik reassigned HBASE-23351:
---
Assignee: István Adamcsik
> hbase-connectors Test fail when stating HBase mini cluster
> --
>
> Key: HBASE-23351
> URL: https://issues.apache.org/jira/browse/HBASE-23351
> Project: HBase
> Issue Type: Bug
>Affects Versions: connector-1.0.0
>Reporter: István Adamcsik
>Assignee: István Adamcsik
>Priority: Major
>
> Due to a shading issue in hbase-shaded-testing-util, whereby jackson is
> omitted from the fat jar, the hbase-connectors tests fail to start mini
> clusters which fail with the below error.
> {code:java}
> *** RUN ABORTED *** java.lang.NoClassDefFoundError: Could not initialize
> class org.apache.hadoop.hdfs.web.WebHdfsFileSystem at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.initWebHdfs(NameNodeHttpServer.java:78)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:166)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:842)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:693)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:906)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:885)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1162)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1037)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830{code}
> This issue is fixed in hbase 2.2.2 with HBASE-23007.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-connectors] iadamcsik-cldr opened a new pull request #52: HBASE-23351 updating hbase version to 2.2.2
iadamcsik-cldr opened a new pull request #52: HBASE-23351 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/52 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] iadamcsik-cldr commented on issue #52: HBASE-23351 updating hbase version to 2.2.2
iadamcsik-cldr commented on issue #52: HBASE-23351 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/52#issuecomment-559800502 mvn clean install passed successfully on local run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] iadamcsik-cldr commented on issue #51: HBASE-23007 updating hbase version to 2.2.2
iadamcsik-cldr commented on issue #51: HBASE-23007 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/51#issuecomment-559799675 used incorrect ticket number. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] iadamcsik-cldr closed pull request #51: HBASE-23007 updating hbase version to 2.2.2
iadamcsik-cldr closed pull request #51: HBASE-23007 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23351) hbase-connectors Test fail when stating HBase mini cluster
[
https://issues.apache.org/jira/browse/HBASE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
István Adamcsik updated HBASE-23351:
Description:
Due to a shading issue in hbase-shaded-testing-util, whereby jackson is omitted
from the fat jar, the hbase-connectors tests fail to start mini clusters which
fail with the below error.
{code:java}
*** RUN ABORTED *** java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.hdfs.web.WebHdfsFileSystem at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.initWebHdfs(NameNodeHttpServer.java:78)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:166)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:842)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:693)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:906)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:885)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1162)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1037)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830{code}
This issue is fixed in hbase 2.2.2 with HBASE-23007.
was:
Due to a shading issue in hbase-shaded-testing-util, whereby jackson is omitted
from the fat jar, the hbase-connectors tests fail to start mini clusters which
fail with the below error.
{code:java}
*** RUN ABORTED *** java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.hdfs.web.WebHdfsFileSystem at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.initWebHdfs(NameNodeHttpServer.java:78)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:166)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:842)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:693)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:906)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:885)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1162)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1037)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830{code}
This issue is fixed in hbase 2.2.2.
> hbase-connectors Test fail when stating HBase mini cluster
> --
>
> Key: HBASE-23351
> URL: https://issues.apache.org/jira/browse/HBASE-23351
> Project: HBase
> Issue Type: Bug
>Affects Versions: connector-1.0.0
>Reporter: István Adamcsik
>Priority: Major
>
> Due to a shading issue in hbase-shaded-testing-util, whereby jackson is
> omitted from the fat jar, the hbase-connectors tests fail to start mini
> clusters which fail with the below error.
> {code:java}
> *** RUN ABORTED *** java.lang.NoClassDefFoundError: Could not initialize
> class org.apache.hadoop.hdfs.web.WebHdfsFileSystem at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.initWebHdfs(NameNodeHttpServer.java:78)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:166)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:842)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:693)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:906)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:885)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1162)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1037)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830{code}
> This issue is fixed in hbase 2.2.2 with HBASE-23007.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-connectors] iadamcsik-cldr commented on issue #51: HBASE-23007 updating hbase version to 2.2.2
iadamcsik-cldr commented on issue #51: HBASE-23007 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/51#issuecomment-559796825 mvn clean install passed successfully on local run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] asf-ci commented on issue #51: HBASE-23007 updating hbase version to 2.2.2
asf-ci commented on issue #51: HBASE-23007 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/51#issuecomment-559795967 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-CONNECTORS-Build/94/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] iadamcsik-cldr commented on issue #47: HBASE-23295 HBaseContext should use most recent delegation token
iadamcsik-cldr commented on issue #47: HBASE-23295 HBaseContext should use most recent delegation token URL: https://github.com/apache/hbase-connectors/pull/47#issuecomment-559795697 #51 should be merged first as currently test would fail, but not related to this change. The reason the checks aren't failing is that the unit tests are not being run by yetus, as @petersomogyi noticed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] iadamcsik-cldr opened a new pull request #51: HBASE-23007 updating hbase version to 2.2.2
iadamcsik-cldr opened a new pull request #51: HBASE-23007 updating hbase version to 2.2.2 URL: https://github.com/apache/hbase-connectors/pull/51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23352) Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction
Szabolcs Bukros created HBASE-23352: --- Summary: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction Key: HBASE-23352 URL: https://issues.apache.org/jira/browse/HBASE-23352 Project: HBase Issue Type: Improvement Components: integration tests Affects Versions: 2.2.2 Reporter: Szabolcs Bukros Assignee: Szabolcs Bukros When integration tests are run through hbase cli the properties passed as cmd line params does not reach the chaos monkies. It is possible to define a property file, but it would be more flexible if we could also pick up properties from the command line. Also I would like to improve FillDiskCommandAction, to stop the remote process if the call times out before it could have finished or was run without a size parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-23348) Spark's createTable method throws an exception while the table is being split
[
https://issues.apache.org/jira/browse/HBASE-23348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-23348 started by Rabi Kumar K C.
--
> Spark's createTable method throws an exception while the table is being split
> -
>
> Key: HBASE-23348
> URL: https://issues.apache.org/jira/browse/HBASE-23348
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Balazs Meszaros
>Assignee: Rabi Kumar K C
>Priority: Major
> Labels: beginner
>
> {{HBaseRelation.createTable}} checks table existence with
> {{HBaseAdmin.isTableAvailable}} method
> [here|https://github.com/apache/hbase-connectors/blob/master/spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/DefaultSource.scala#L167].
> This is unfortunate, because it can return {{false}} while splitting, so
> {{createTable}} will fail. It should use {{tableExists}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on a change in pull request #837: HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered
wchevreuil commented on a change in pull request #837: HBASE-23309: Adding the
flexibility to ChainWalEntryFilter to filter the whole entry if all cells get
filtered
URL: https://github.com/apache/hbase/pull/837#discussion_r352084573
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/CustomChainWALEntryFilter.java
##
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.wal.WAL;
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import java.util.List;
+
+/**
+ * A {@link ChainWALEntryFilter} for providing more flexible options
+ */
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.REPLICATION)
+public class CustomChainWALEntryFilter extends ChainWALEntryFilter {
+
+ private boolean filterEmptyEntry = false;
+
+ public CustomChainWALEntryFilter(final WALEntryFilter... filters) {
+super(filters);
+ }
+
+ public CustomChainWALEntryFilter(final List filters) {
+super(filters);
+ }
+
+ @Override
+ public WAL.Entry filter(WAL.Entry entry) {
+filterEntry(entry);
+if (entry == null) {
+ return null;
+}
+
+filterCells(entry);
Review comment:
I guess we could user super method for this block?
`super.filter(entry);`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on a change in pull request #837: HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered
wchevreuil commented on a change in pull request #837: HBASE-23309: Adding the
flexibility to ChainWalEntryFilter to filter the whole entry if all cells get
filtered
URL: https://github.com/apache/hbase/pull/837#discussion_r352088255
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/CustomChainWALEntryFilter.java
##
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.wal.WAL;
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import java.util.List;
+
+/**
+ * A {@link ChainWALEntryFilter} for providing more flexible options
+ */
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.REPLICATION)
+public class CustomChainWALEntryFilter extends ChainWALEntryFilter {
Review comment:
`CustomChainWALEntryFilter` name is not very intuitive for a built-in
filter. Maybe something like `ChainWALEmptyEntryFilter` or
`ChainWALNullEntryFilter`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #881: HBASE-23345 Duplicate code about deciding whether a table need replicate
Apache-HBase commented on issue #881: HBASE-23345 Duplicate code about deciding whether a table need replicate URL: https://github.com/apache/hbase/pull/881#issuecomment-559762245 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 12s | master passed | | +1 :green_heart: | compile | 1m 41s | master passed | | +1 :green_heart: | checkstyle | 2m 7s | master passed | | +1 :green_heart: | shadedjars | 4m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 17s | master passed | | +0 :ok: | spotbugs | 4m 16s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 57s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | +1 :green_heart: | checkstyle | 2m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 46s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed | | +1 :green_heart: | findbugs | 6m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 54s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 28s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 162m 40s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 1m 35s | The patch does not generate ASF License warnings. | | | | 231m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-881/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/881 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux fe5c09bade41 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-881/out/precommit/personality/provided.sh | | git revision | master / d6539350ed | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-881/8/testReport/ | | Max. process+thread count | 4291 (vs. ulimit of 1) | | modules | C: hbase-client hbase-replication hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-881/8/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23351) hbase-connectors Test fail when stating HBase mini cluster
István Adamcsik created HBASE-23351:
---
Summary: hbase-connectors Test fail when stating HBase mini cluster
Key: HBASE-23351
URL: https://issues.apache.org/jira/browse/HBASE-23351
Project: HBase
Issue Type: Bug
Affects Versions: connector-1.0.0
Reporter: István Adamcsik
Due to a shading issue in hbase-shaded-testing-util, whereby jackson is omitted
from the fat jar, the hbase-connectors tests fail to start mini clusters which
fail with the below error.
{code:java}
*** RUN ABORTED *** java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.hdfs.web.WebHdfsFileSystem at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.initWebHdfs(NameNodeHttpServer.java:78)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:166)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:842)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:693)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:906)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:885)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1162)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1037)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830{code}
This issue is fixed in hbase 2.2.2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-connectors] asf-ci commented on issue #50: HBASE-23348 Spark's createTable method throws an exception while the table is being split
asf-ci commented on issue #50: HBASE-23348 Spark's createTable method throws an exception while the table is being split URL: https://github.com/apache/hbase-connectors/pull/50#issuecomment-559758083 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-CONNECTORS-Build/93/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-connectors] ravowlga123 opened a new pull request #50: HBASE-23348 Spark's createTable method throws an exception while the table is being split
ravowlga123 opened a new pull request #50: HBASE-23348 Spark's createTable method throws an exception while the table is being split URL: https://github.com/apache/hbase-connectors/pull/50 HBaseRelation.createTable checks table existence with HBaseAdmin.isTableAvailable method in DefaultSource.scala (createTable). This is unfortunate, because it can return false while splitting, so createTable will fail. It should use tableExists. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (HBASE-23348) Spark's createTable method throws an exception while the table is being split
[
https://issues.apache.org/jira/browse/HBASE-23348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rabi Kumar K C reassigned HBASE-23348:
--
Assignee: Rabi Kumar K C
> Spark's createTable method throws an exception while the table is being split
> -
>
> Key: HBASE-23348
> URL: https://issues.apache.org/jira/browse/HBASE-23348
> Project: HBase
> Issue Type: Bug
> Components: hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Balazs Meszaros
>Assignee: Rabi Kumar K C
>Priority: Major
> Labels: beginner
>
> {{HBaseRelation.createTable}} checks table existence with
> {{HBaseAdmin.isTableAvailable}} method
> [here|https://github.com/apache/hbase-connectors/blob/master/spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/DefaultSource.scala#L167].
> This is unfortunate, because it can return {{false}} while splitting, so
> {{createTable}} will fail. It should use {{tableExists}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23312) HBase Thrift SPNEGO configs (HBASE-19852) should be backwards compatible
[ https://issues.apache.org/jira/browse/HBASE-23312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-23312: -- Fix Version/s: (was: 2.1.8) 2.1.9 > HBase Thrift SPNEGO configs (HBASE-19852) should be backwards compatible > > > Key: HBASE-23312 > URL: https://issues.apache.org/jira/browse/HBASE-23312 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 3.0.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, > 2.1.8 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > Attachments: HBASE-23312.master.001.patch > > > HBASE-19852 is not backwards compatible since it now requires the SPNEGO > thrift configs. I haven't seen anything in Apache HBase about changing this > so that the older configs still work with a merged keytab. (fall back to the > non SPNEGO specific principal/keytab configs) > I wrote the original patch in HBASE-19852 and with hindsight being 20/20, I > think this section of could be extended to fall back to not requiring the > additional configs. > https://github.com/apache/hbase/blame/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java#L78 > Supporting the older configs allows upgrade from HBase 1.x to 2.x without > needing to change the configs ahead of time. I'll make sure to log a > deprecation warning if the older configs are used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23183) Patch based precommit job fails
[
https://issues.apache.org/jira/browse/HBASE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984838#comment-16984838
]
Peter Somogyi commented on HBASE-23183:
---
Patch based precommit is failing currently with package installation.
{noformat}
11:04:16 Fetched 84.3 MB in 50s (1666 kB/s)
11:04:16 [91mE[0m[91m: [0m[91mFailed to fetch
https://esm.ubuntu.com/ubuntu/pool/main/b/bzip2/bzip2_1.0.6-5ubuntu0.1~esm2_amd64.deb
HttpError401
11:04:16 [0m[91m
11:04:16 [0m[91mE[0m[91m: [0m[91mFailed to fetch
https://esm.ubuntu.com/ubuntu/pool/main/b/bzip2/libbz2-1.0_1.0.6-5ubuntu0.1~esm2_amd64.deb
HttpError401 {noformat}
> Patch based precommit job fails
> ---
>
> Key: HBASE-23183
> URL: https://issues.apache.org/jira/browse/HBASE-23183
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.11
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-23183.branch-1.4.001.patch
>
>
> PreCommit-HBASE-Build job fails on branch-1.4 with missing JAVA_HOME.
> {noformat}
> WARNING: JAVA_HOME not defined. Disabling java tests. {noformat}
> [https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/962/console]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23183) Patch based precommit job fails
[
https://issues.apache.org/jira/browse/HBASE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984824#comment-16984824
]
HBase QA commented on HBASE-23183:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 1m
18s{color} | {color:red} Docker failed to build yetus/hbase:9bbc30c69b. {color}
|
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-23183 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987128/HBASE-23183.branch-1.4.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1050/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Patch based precommit job fails
> ---
>
> Key: HBASE-23183
> URL: https://issues.apache.org/jira/browse/HBASE-23183
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.11
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-23183.branch-1.4.001.patch
>
>
> PreCommit-HBASE-Build job fails on branch-1.4 with missing JAVA_HOME.
> {noformat}
> WARNING: JAVA_HOME not defined. Disabling java tests. {noformat}
> [https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/962/console]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23183) Patch based precommit job fails
[
https://issues.apache.org/jira/browse/HBASE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984822#comment-16984822
]
HBase QA commented on HBASE-23183:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 1m
17s{color} | {color:red} Docker failed to build yetus/hbase:9bbc30c69b. {color}
|
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-23183 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12987128/HBASE-23183.branch-1.4.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1049/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Patch based precommit job fails
> ---
>
> Key: HBASE-23183
> URL: https://issues.apache.org/jira/browse/HBASE-23183
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.11
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-23183.branch-1.4.001.patch
>
>
> PreCommit-HBASE-Build job fails on branch-1.4 with missing JAVA_HOME.
> {noformat}
> WARNING: JAVA_HOME not defined. Disabling java tests. {noformat}
> [https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/962/console]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-23183) Patch based precommit job fails
[
https://issues.apache.org/jira/browse/HBASE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reassigned HBASE-23183:
-
Assignee: Peter Somogyi
> Patch based precommit job fails
> ---
>
> Key: HBASE-23183
> URL: https://issues.apache.org/jira/browse/HBASE-23183
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.11
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Attachments: HBASE-23183.branch-1.4.001.patch
>
>
> PreCommit-HBASE-Build job fails on branch-1.4 with missing JAVA_HOME.
> {noformat}
> WARNING: JAVA_HOME not defined. Disabling java tests. {noformat}
> [https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/962/console]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23183) Patch based precommit job fails
[
https://issues.apache.org/jira/browse/HBASE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-23183:
--
Status: Patch Available (was: Open)
> Patch based precommit job fails
> ---
>
> Key: HBASE-23183
> URL: https://issues.apache.org/jira/browse/HBASE-23183
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.11
>Reporter: Peter Somogyi
>Priority: Major
> Attachments: HBASE-23183.branch-1.4.001.patch
>
>
> PreCommit-HBASE-Build job fails on branch-1.4 with missing JAVA_HOME.
> {noformat}
> WARNING: JAVA_HOME not defined. Disabling java tests. {noformat}
> [https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/962/console]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23183) Patch based precommit job fails
[
https://issues.apache.org/jira/browse/HBASE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-23183:
--
Attachment: HBASE-23183.branch-1.4.001.patch
> Patch based precommit job fails
> ---
>
> Key: HBASE-23183
> URL: https://issues.apache.org/jira/browse/HBASE-23183
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.11
>Reporter: Peter Somogyi
>Priority: Major
> Attachments: HBASE-23183.branch-1.4.001.patch
>
>
> PreCommit-HBASE-Build job fails on branch-1.4 with missing JAVA_HOME.
> {noformat}
> WARNING: JAVA_HOME not defined. Disabling java tests. {noformat}
> [https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/962/console]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #868: HBASE-23296 Add CompositeBucketCache to support tiered BC
Apache-HBase commented on issue #868: HBASE-23296 Add CompositeBucketCache to support tiered BC URL: https://github.com/apache/hbase/pull/868#issuecomment-559702619 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 44s | master passed | | +1 :green_heart: | compile | 1m 16s | master passed | | +1 :green_heart: | checkstyle | 1m 41s | master passed | | +1 :green_heart: | shadedjars | 4m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | master passed | | +0 :ok: | spotbugs | 4m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 35s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | checkstyle | 1m 31s | hbase-server: The patch generated 0 new + 55 unchanged - 4 fixed = 55 total (was 59) | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-external-blockcache | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 15s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | | +1 :green_heart: | findbugs | 5m 10s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 260m 48s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 0m 32s | hbase-external-blockcache in the patch passed. | | +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. | | | | 327m 38s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestFromClientSide | | | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-868/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/868 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux ea181cb86bb0 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-868/out/precommit/personality/provided.sh | | git revision | master / d6539350ed | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-868/4/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-868/4/testReport/ | | Max. process+thread count | 4768 (vs. ulimit of 1) | | modules | C: hbase-server hbase-external-blockcache U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-868/4/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on issue #754: HBASE-22978 : Online slow response log
virajjasani commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-559696527 @guangxuCheng Could you please review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
