[jira] [Assigned] (HBASE-24102) RegionMover should exclude draining/decommissioning nodes from target RSs
[
https://issues.apache.org/jira/browse/HBASE-24102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani reassigned HBASE-24102:
Assignee: Viraj Jasani
> RegionMover should exclude draining/decommissioning nodes from target RSs
> -
>
> Key: HBASE-24102
> URL: https://issues.apache.org/jira/browse/HBASE-24102
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.4.0
>
>
> When using RegionMover tool to unload the regions from a given RS, it decides
> the list of destination RSs by
> {code}
> List regionServers = new ArrayList<>();
> regionServers.addAll(admin.getRegionServers());
> // Remove the host Region server from target Region Servers list
> ServerName server = stripServer(regionServers, hostname, port);
> .
> // Remove RS present in the exclude file
> stripExcludes(regionServers);
> stripMaster(regionServers);
> {code}
> Ya it is removing the RSs mentioned in the exclude file.
> Better when the RegionMover user is NOT mentioning any exclude list, we can
> exclude the draining/decommissioning RSs
> Admin#listDecommissionedRegionServers()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24102) RegionMover should exclude draining/decommissioning nodes from target RSs
[
https://issues.apache.org/jira/browse/HBASE-24102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-24102:
---
Description:
When using RegionMover tool to unload the regions from a given RS, it decides
the list of destination RSs by
{code}
List regionServers = new ArrayList<>();
regionServers.addAll(admin.getRegionServers());
// Remove the host Region server from target Region Servers list
ServerName server = stripServer(regionServers, hostname, port);
.
// Remove RS present in the exclude file
stripExcludes(regionServers);
stripMaster(regionServers);
{code}
Ya it is removing the RSs mentioned in the exclude file.
Better when the RegionMover user is NOT mentioning any exclude list, we can
exclude the draining/decommissioning RSs
Admin#listDecommissionedRegionServers()
was:
When using RegionMover tool to unload the regions from a given RS, it decides
the list of destination RSs by
{code}
List regionServers = new ArrayList<>();
regionServers.addAll(admin.getRegionServers());
// Remove the host Region server from target Region Servers list
ServerName server = stripServer(regionServers, hostname, port);
.
// Remove RS present in the exclude file
stripExcludes(regionServers);
stripMaster(regionServers);
{code}
Ya it is removing the RSs mentioned in the exclude file.
Better when the RegionMover user is mentioning any exclude list, we can exlcude
the draining/decommissioning RSs
Admin#listDecommissionedRegionServers()
> RegionMover should exclude draining/decommissioning nodes from target RSs
> -
>
> Key: HBASE-24102
> URL: https://issues.apache.org/jira/browse/HBASE-24102
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0, 2.4.0
>
>
> When using RegionMover tool to unload the regions from a given RS, it decides
> the list of destination RSs by
> {code}
> List regionServers = new ArrayList<>();
> regionServers.addAll(admin.getRegionServers());
> // Remove the host Region server from target Region Servers list
> ServerName server = stripServer(regionServers, hostname, port);
> .
> // Remove RS present in the exclude file
> stripExcludes(regionServers);
> stripMaster(regionServers);
> {code}
> Ya it is removing the RSs mentioned in the exclude file.
> Better when the RegionMover user is NOT mentioning any exclude list, we can
> exclude the draining/decommissioning RSs
> Admin#listDecommissionedRegionServers()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24102) RegionMover should exclude draining/decommissioning nodes from target RSs
[
https://issues.apache.org/jira/browse/HBASE-24102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-24102:
---
Description:
When using RegionMover tool to unload the regions from a given RS, it decides
the list of destination RSs by
{code}
List regionServers = new ArrayList<>();
regionServers.addAll(admin.getRegionServers());
// Remove the host Region server from target Region Servers list
ServerName server = stripServer(regionServers, hostname, port);
.
// Remove RS present in the exclude file
stripExcludes(regionServers);
stripMaster(regionServers);
{code}
Ya it is removing the RSs mentioned in the exclude file.
Better when the RegionMover user is mentioning any exclude list, we can exlcude
the draining/decommissioning RSs
Admin#listDecommissionedRegionServers()
was:
When using RegionMover tool to unload the regions from a given RS, it decides
the list of destination RSs by
{code}
List regionServers = new ArrayList<>();
regionServers.addAll(admin.getRegionServers());
// Remove the host Region server from target Region Servers list
ServerName server = stripServer(regionServers, hostname, port);
.
// Remove RS present in the exclude file
stripExcludes(regionServers);
stripMaster(regionServers);
{code}
Ya it is removing the RSs mentioned in the excludes list.
Better when the RegionMover user is mentioning any exclude list, we can exlcude
the draining/decommissioning RSs
Admin#listDecommissionedRegionServers()
> RegionMover should exclude draining/decommissioning nodes from target RSs
> -
>
> Key: HBASE-24102
> URL: https://issues.apache.org/jira/browse/HBASE-24102
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Priority: Major
> Fix For: 3.0.0, 2.4.0
>
>
> When using RegionMover tool to unload the regions from a given RS, it decides
> the list of destination RSs by
> {code}
> List regionServers = new ArrayList<>();
> regionServers.addAll(admin.getRegionServers());
> // Remove the host Region server from target Region Servers list
> ServerName server = stripServer(regionServers, hostname, port);
> .
> // Remove RS present in the exclude file
> stripExcludes(regionServers);
> stripMaster(regionServers);
> {code}
> Ya it is removing the RSs mentioned in the exclude file.
> Better when the RegionMover user is mentioning any exclude list, we can
> exlcude the draining/decommissioning RSs
> Admin#listDecommissionedRegionServers()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24102) RegionMover should exclude draining/decommissioning nodes from target RSs
Anoop Sam John created HBASE-24102:
--
Summary: RegionMover should exclude draining/decommissioning nodes
from target RSs
Key: HBASE-24102
URL: https://issues.apache.org/jira/browse/HBASE-24102
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Fix For: 3.0.0, 2.4.0
When using RegionMover tool to unload the regions from a given RS, it decides
the list of destination RSs by
{code}
List regionServers = new ArrayList<>();
regionServers.addAll(admin.getRegionServers());
// Remove the host Region server from target Region Servers list
ServerName server = stripServer(regionServers, hostname, port);
.
// Remove RS present in the exclude file
stripExcludes(regionServers);
stripMaster(regionServers);
{code}
Ya it is removing the RSs mentioned in the excludes list.
Better when the RegionMover user is mentioning any exclude list, we can exlcude
the draining/decommissioning RSs
Admin#listDecommissionedRegionServers()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun opened a new pull request #1414: HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up co…
huaxiangsun opened a new pull request #1414: HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up co… URL: https://github.com/apache/hbase/pull/1414 …mpaction state. Addendum: For major compaction test, set hbase.hstore.compaction.min to a big number to avoid kicking in minor compactions, which will pollute compaction state and sometimes, cause major compaction cannot happen. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
bharathv commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r402059418
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +349,19 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)
Review comment:
Hmm.. I too think it should just be (fs instanceof LocalFS).. With this
patch, try using the following combination (distributed cluster + localfs), it
fails with the same error when it should work..
```
hbase.cluster.distributed
true
```
Point being that, these two configuration params are disjoint. Meaning a
distributed/non-distributed cluster should work with a local file system.
Stream capability is a property of the FS in use, so just a check on FS should
be good IMO.
I poked around a little bit about RawFS in the hadoop code, didn't fully
understand its use but it appears that FileSystem when parsing the config would
pick LocalFileSystem when the scheme is file:///. So I think just doing it for
LocalFileSystem should be ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607616798 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 37s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 39s | the patch passed | | -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 16s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 40m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux b7ec156f9f8e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607615399 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 52s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 5m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 43s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 21s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 21s | hbase-server in the patch failed. | | | | 34m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b3740f0c119b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/testReport/ | | Max. process+thread count | 672 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607615183 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 18s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 5m 41s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 50s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 5m 33s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 4m 59s | hbase-server in the patch failed. | | | | 33m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2d47b8663001 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/testReport/ | | Max. process+thread count | 882 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/6/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607606230 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 13s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 6m 30s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 44s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 51s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 7m 39s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 59s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 6m 50s | hbase-server in the patch failed. | | | | 42m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4dab40e380b0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/testReport/ | | Max. process+thread count | 601 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607605528 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 59s | master passed | | +1 :green_heart: | checkstyle | 1m 16s | master passed | | +1 :green_heart: | spotbugs | 2m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 39s | the patch passed | | -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 11m 36s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 2m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 38m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 2367dc9760dc 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607604793 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 13s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 5m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 51s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 18s | hbase-server in the patch failed. | | | | 35m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6fad9cafd70b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/testReport/ | | Max. process+thread count | 702 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/5/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode URL: https://github.com/apache/hbase/pull/1408#issuecomment-607597616 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 48s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 18s | master passed | | +1 :green_heart: | compile | 0m 33s | master passed | | +1 :green_heart: | shadedjars | 7m 29s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-common in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 39s | the patch passed | | +1 :green_heart: | compile | 0m 34s | the patch passed | | +1 :green_heart: | javac | 0m 34s | the patch passed | | +1 :green_heart: | shadedjars | 7m 24s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 22s | hbase-common in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 43s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 126m 3s | hbase-server in the patch passed. | | | | 164m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1408 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6b268c8d3563 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 554678ea4c | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/testReport/ | | Max. process+thread count | 4126 (vs. ulimit of 1) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607591778 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 9m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 26s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 6m 28s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 58s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 7m 36s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 1m 4s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 6m 50s | hbase-server in the patch failed. | | | | 50m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df3373bb6675 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/testReport/ | | Max. process+thread count | 625 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607590026 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 9m 51s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 7s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 50s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 21s | hbase-server in the patch failed. | | | | 43m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 432d3ceca74e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/testReport/ | | Max. process+thread count | 712 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile
Apache-HBase commented on issue #1380: HBASE-23584 : Descrease rpc getFileStatus count when open a storefile URL: https://github.com/apache/hbase/pull/1380#issuecomment-607588726 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 53s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | the patch passed | | -0 :warning: | checkstyle | 1m 14s | hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -0 :warning: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 11m 22s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 2m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 38m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1380 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux caba12de419f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4ea819b8f2 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1380/4/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r402015257
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +349,19 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)
Review comment:
You tell me :)
Yes, for running out of the source tree, not modifying any configuration,
this change permits the process to start, master and region server threads all
come online. I performed some additional verification by running `ltt` against
the process, a scant 100k rows.
I don't know well when we choose `LocalFileSystem` vs. `RawLocalFIleSystem`,
so there may be other issues lurking.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] binlijin commented on issue #1379: HBASE-24077 When encounter RowTooBigException, log the row info.
binlijin commented on issue #1379: HBASE-24077 When encounter RowTooBigException, log the row info. URL: https://github.com/apache/hbase/pull/1379#issuecomment-607579969 @HorizonNet have any suggest？ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-24098) Ban imports of com.google.protobuf.*
[ https://issues.apache.org/jira/browse/HBASE-24098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24098: -- Parent: (was: HBASE-23797) Issue Type: Task (was: Sub-task) > Ban imports of com.google.protobuf.* > > > Key: HBASE-24098 > URL: https://issues.apache.org/jira/browse/HBASE-24098 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > At least in our code base we should not depend on them any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode URL: https://github.com/apache/hbase/pull/1408#issuecomment-607578878 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 13s | master passed | | +1 :green_heart: | compile | 0m 24s | master passed | | +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | shadedjars | 4m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 67m 3s | hbase-server in the patch passed. | | | | 93m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1408 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 25f60e4044a8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 554678ea4c | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/testReport/ | | Max. process+thread count | 5532 (vs. ulimit of 1) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-24098) Ban imports of com.google.protobuf.*
[ https://issues.apache.org/jira/browse/HBASE-24098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24098: -- Fix Version/s: (was: 3.0.0) > Ban imports of com.google.protobuf.* > > > Key: HBASE-24098 > URL: https://issues.apache.org/jira/browse/HBASE-24098 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > At least in our code base we should not depend on them any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24098) Ban imports of com.google.protobuf.*
[ https://issues.apache.org/jira/browse/HBASE-24098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073306#comment-17073306 ] Duo Zhang commented on HBASE-24098: --- OK this is not easy, as we have asyncfs implementation which depends on hadoop. Move it out of the sub tasks. > Ban imports of com.google.protobuf.* > > > Key: HBASE-24098 > URL: https://issues.apache.org/jira/browse/HBASE-24098 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > At least in our code base we should not depend on them any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
Apache9 commented on a change in pull request #1408: HBASE-24086 Disable output
stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r402010313
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +349,19 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)
Review comment:
Just test whether this is a LocalFileSystem is enough?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-24097) [Flakey Tests] TestSnapshotScannerHDFSAclController#testRestoreSnapshot
[ https://issues.apache.org/jira/browse/HBASE-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073298#comment-17073298 ] Michael Stack commented on HBASE-24097: --- Looks like there are other issues in this test beyond the one addressed here. Saw a flakey fail here... https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/5950/ Test timed out. > [Flakey Tests] TestSnapshotScannerHDFSAclController#testRestoreSnapshot > --- > > Key: HBASE-24097 > URL: https://issues.apache.org/jira/browse/HBASE-24097 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Fails on occasion, 15% of the time according to flakie report. I can > reproduce it failing locally. A single method fails. I don't follow how it is > supposed to work (what looks wrong to me passes...). I noticed that if I ran > testRestoreSnapshot on its own, it passed but failed when run as part of the > test suite so I broke it out into its own suite. Now both old and new suites > pass for me locally after 20 repeats. Let me push it up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting
[
https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073296#comment-17073296
]
Hudson commented on HBASE-24051:
Results for branch branch-2
[build #2578 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2578/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2578//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> When the data directory is set to EC,the amount of storefiles reaches a
> relatively high level,heap Occupancy Percent is above heap occupancy alarm
> watermark,resulting in frequent gc and eventual failure
> --
>
> Key: HBASE-24051
> URL: https://issues.apache.org/jira/browse/HBASE-24051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.5
>
> Attachments: 1.png
>
>
> When the data directory is set to EC, during the continuous load process,
> regionserver is found down ,to view the log:
> regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below
> the heap occupancy alarm watermark (0.95)
> regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap
> occupancy alarm watermark (0.95)
> [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine
> (eg GC): pause of approximately 1254ms
> GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms
> GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms
> In particular, GC pauses occurred 283 times in half an hour.
> The memory analysis found that most of the memory footprint was generated by
> HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer
> See HDFS-14308，but after the fix, hbase did not return to normal.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[
https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073297#comment-17073297
]
Hudson commented on HBASE-24072:
Results for branch branch-2
[build #2578 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2578/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2578//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Nightlies reporting OutOfMemoryError: unable to create new native thread
>
>
> Key: HBASE-24072
> URL: https://issues.apache.org/jira/browse/HBASE-24072
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Priority: Major
> Attachments:
> 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch
>
>
> Seeing this kind of thing in nightly...
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new
> native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> {code}
> Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing
> our docker container. It is set to 20G (The hosts are 48G). Fork count is
> 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add
> the maven 4G and we could be over the top.
> Play w/ downing the fork size (in earlier study we didn't seem to need this
> much RAM when running a fat long test). Let me also take th ms off the mvn
> allocation to see if that helps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode URL: https://github.com/apache/hbase/pull/1408#issuecomment-607563770 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 15s | master passed | | +1 :green_heart: | checkstyle | 0m 24s | master passed | | +1 :green_heart: | spotbugs | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 20s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | hbase-common: The patch generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 38s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 34m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1408 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 32164a8681a0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 554678ea4c | | Max. process+thread count | 94 (vs. ulimit of 1) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/2/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073279#comment-17073279 ] Michael Stack commented on HBASE-24051: --- [~shenshengli] No problem sir. No harm done. Just suggest that you comment on your actions so its easier for those of us trying to follow along. I pushed your patch because it looked good but on one of the old PRs there was a comment by a committer that we need a test that fails unless you make your change. That possible? Thank you. > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Assignee: shenshengli >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.5 > > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073274#comment-17073274 ] shenshengli commented on HBASE-24051: - There was something wrong with that PR, so I closed it and opened a new one.Sorry，I'm a green hand at this。 > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Assignee: shenshengli >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.5 > > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401985510
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
From debugging, it's `LocalFileSystem` that gets used for wals, so that's
what I've used. This seems very fragile though :/
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
bharathv commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401979773
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
Ya, that'd be ideal.. probably a new Hadoop jira for the FileSystem
interface to expose the capabilities in some form...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401978483
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
I was hoping to avoid pinning the logic to specific `FileSystem`
implementations, rather on capabilities. Fun fact: the only commonality between
`LocalFileSystem` and `RawLocalFileSystem` is `FileSystem` :/
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1368: HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check
Apache-HBase commented on issue #1368: HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check URL: https://github.com/apache/hbase/pull/1368#issuecomment-607543670 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 6m 57s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 6m 16s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 41s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 6m 17s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | hbase-common in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 18s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 37s | hbase-assembly in the patch passed. | | +1 :green_heart: | unit | 100m 43s | hbase-server in the patch passed. | | | | 135m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1368 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6fe64dc3f74c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/testReport/ | | Max. process+thread count | 4119 (vs. ulimit of 1) | | modules | C: hbase-common hbase-assembly U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
bharathv commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401971878
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
I think @saintstack is right. We can run non-distributed on a distributed fs
with stream capabilities. So the check is probably not tight...
> Other suggestions?
One naive way is to just check for instance of LocalFileSystem and override?
(Think this is what Stack was suggesting). This covers the most common case of
testing something by just unpacking a bin release. Any other fancy filesystems,
users may have to figure out themselves?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-24071) [JDK11] Remove `unit` filter from nightly and precommit jobs
[
https://issues.apache.org/jira/browse/HBASE-24071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073257#comment-17073257
]
Hudson commented on HBASE-24071:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> [JDK11] Remove `unit` filter from nightly and precommit jobs
>
>
> Key: HBASE-24071
> URL: https://issues.apache.org/jira/browse/HBASE-24071
> Project: HBase
> Issue Type: Sub-task
> Components: build, test
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Added in HBASE-23946, we can remove this filter once HBASE-24007 lands.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24100) [Flakey Tests] Add test to check we work properly when port clash setting up thriftserver
[
https://issues.apache.org/jira/browse/HBASE-24100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073256#comment-17073256
]
Hudson commented on HBASE-24100:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> [Flakey Tests] Add test to check we work properly when port clash setting up
> thriftserver
> -
>
> Key: HBASE-24100
> URL: https://issues.apache.org/jira/browse/HBASE-24100
> Project: HBase
> Issue Type: Bug
> Components: flakies, Thrift
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-24100-Flakey-Tests-Add-test-to-check-we-work-p.patch
>
>
> My hack to add retry when thrift setup has a port clash over in HBASE-24079
> didn't work. Add a test and fix what it turns up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24092) Fix links to build reports generated by nightly job
[
https://issues.apache.org/jira/browse/HBASE-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073255#comment-17073255
]
Hudson commented on HBASE-24092:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Fix links to build reports generated by nightly job
> ---
>
> Key: HBASE-24092
> URL: https://issues.apache.org/jira/browse/HBASE-24092
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
>
> Links going back to JIRA look like
> {noformat}
> For more information [see jdk8 (hadoop2)
> report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2574//JDK8_Nightly_Build_Report_(Hadoop2)/]
> {noformat}
> But the actual URL to this report is
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2574/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23937) Retrieve online large RPC logs
[
https://issues.apache.org/jira/browse/HBASE-23937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073252#comment-17073252
]
Hudson commented on HBASE-23937:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Retrieve online large RPC logs
> --
>
> Key: HBASE-23937
> URL: https://issues.apache.org/jira/browse/HBASE-23937
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0
>
>
> Similar to online slow RPC response logs, we should also provide get and
> clear of large RPC logs which should be stored in the online RingBuffer. Many
> of slow response logs would also turn out to be large response logs for
> RegionServers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24097) [Flakey Tests] TestSnapshotScannerHDFSAclController#testRestoreSnapshot
[
https://issues.apache.org/jira/browse/HBASE-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073253#comment-17073253
]
Hudson commented on HBASE-24097:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> [Flakey Tests] TestSnapshotScannerHDFSAclController#testRestoreSnapshot
> ---
>
> Key: HBASE-24097
> URL: https://issues.apache.org/jira/browse/HBASE-24097
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Fails on occasion, 15% of the time according to flakie report. I can
> reproduce it failing locally. A single method fails. I don't follow how it is
> supposed to work (what looks wrong to me passes...). I noticed that if I ran
> testRestoreSnapshot on its own, it passed but failed when run as part of the
> test suite so I broke it out into its own suite. Now both old and new suites
> pass for me locally after 20 repeats. Let me push it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23678) Literate builder API for version management in schema
[
https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073251#comment-17073251
]
Hudson commented on HBASE-23678:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Literate builder API for version management in schema
> -
>
> Key: HBASE-23678
> URL: https://issues.apache.org/jira/browse/HBASE-23678
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.7.0, 2.4.0
>
>
> Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and
> KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance
> regarding their usage. Almost all combinations of these four settings make
> sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and
> KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the
> behavior with TTL easier to conceive when creating the schema. This could
> take the form of a literate builder API for ColumnDescriptor or an extension
> to an existing one.
> Let me give you a motivating example: We may want to retain all versions for
> a given TTL, and then only a specific number of versions after that interval
> elapses. This can be achieved with VERSIONS=INT_MAX,
> TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSIONS=_num_versions_
> . This is not intuitive though because VERSIONS has been used to specify the
> number of versions to retain (_num_versions_ in this example) since HBase
> version 0.1, so this is going to be a source of confusion - I've seen it in
> practice.
> A literate builder API, by way of the way we design its method names, could
> let a user describe more or less in speaking language how they want version
> retention to work, and internally the builder API could set the low level
> schema attributes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24075) [Flakey Tests] teardown fails because JmxCacheBuster wants to read ClusterId from closed fs
[
https://issues.apache.org/jira/browse/HBASE-24075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073254#comment-17073254
]
Hudson commented on HBASE-24075:
Results for branch branch-2
[build #2577 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> [Flakey Tests] teardown fails because JmxCacheBuster wants to read ClusterId
> from closed fs
> ---
>
> Key: HBASE-24075
> URL: https://issues.apache.org/jira/browse/HBASE-24075
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0
>
>
> I've seen this a few times. There may be an issue already but couldn't turn
> it up.
> In teardown, we fail because of below access:
> {code}
> 2020-03-28 22:01:47,484 WARN [HBase-Metrics2-1]
> master.CachedClusterId(105): Error fetching cluster ID
> java.io.IOException: Filesystem closed
>at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:468)
>at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1650)
>at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1523)
>at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520)
>at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520)
>at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1627)
>at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:578)
>at
> org.apache.hadoop.hbase.master.CachedClusterId.attemptFetch(CachedClusterId.java:103)
>at
> org.apache.hadoop.hbase.master.CachedClusterId.getFromCacheOrFetch(CachedClusterId.java:140)
>at org.apache.hadoop.hbase.master.HMaster.getClusterId(HMaster.java:3839)
>at
> org.apache.hadoop.hbase.master.MetricsMasterWrapperImpl.getClusterId(MetricsMasterWrapperImpl.java:68)
>at
> org.apache.hadoop.hbase.master.MetricsMasterSourceImpl.getMetrics(MetricsMasterSourceImpl.java:112)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
>at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
>at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
>at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:66)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:223)
>at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
>at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
>at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:239)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
> sun.reflect.Delega
[GitHub] [hbase] Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates
Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates URL: https://github.com/apache/hbase/pull/1407#issuecomment-607538392 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 12s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 5m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 42s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 97m 57s | hbase-server in the patch passed. | | | | 129m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1407 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aed7facccff6 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/testReport/ | | Max. process+thread count | 3873 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[
https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073236#comment-17073236
]
Michael Stack commented on HBASE-24072:
---
I pushed 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch on
branch-2. Sets forkcount down from 0.5C to 0.25C for both first and second
parts. Lets see how it does.
> Nightlies reporting OutOfMemoryError: unable to create new native thread
>
>
> Key: HBASE-24072
> URL: https://issues.apache.org/jira/browse/HBASE-24072
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Priority: Major
> Attachments:
> 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch
>
>
> Seeing this kind of thing in nightly...
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new
> native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> {code}
> Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing
> our docker container. It is set to 20G (The hosts are 48G). Fork count is
> 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add
> the maven 4G and we could be over the top.
> Play w/ downing the fork size (in earlier study we didn't seem to need this
> much RAM when running a fat long test). Let me also take th ms off the mvn
> allocation to see if that helps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[
https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-24072:
--
Attachment: 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch
> Nightlies reporting OutOfMemoryError: unable to create new native thread
>
>
> Key: HBASE-24072
> URL: https://issues.apache.org/jira/browse/HBASE-24072
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Priority: Major
> Attachments:
> 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch
>
>
> Seeing this kind of thing in nightly...
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new
> native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> {code}
> Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing
> our docker container. It is set to 20G (The hosts are 48G). Fork count is
> 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add
> the maven 4G and we could be over the top.
> Play w/ downing the fork size (in earlier study we didn't seem to need this
> much RAM when running a fat long test). Let me also take th ms off the mvn
> allocation to see if that helps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1368: HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check
Apache-HBase commented on issue #1368: HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check URL: https://github.com/apache/hbase/pull/1368#issuecomment-607531928 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 16s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 47s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | compile | 0m 52s | the patch passed | | +1 :green_heart: | javac | 0m 52s | the patch passed | | +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 29s | hbase-assembly in the patch passed. | | +1 :green_heart: | unit | 63m 1s | hbase-server in the patch passed. | | | | 93m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1368 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b990cf5a9495 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/testReport/ | | Max. process+thread count | 5683 (vs. ulimit of 1) | | modules | C: hbase-common hbase-assembly U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[
https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073234#comment-17073234
]
Michael Stack commented on HBASE-24072:
---
Let me down the parallelism and see if that helps; i.e. 0.25C vs 0.5C for fork
count as we currently have.
> Nightlies reporting OutOfMemoryError: unable to create new native thread
>
>
> Key: HBASE-24072
> URL: https://issues.apache.org/jira/browse/HBASE-24072
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Priority: Major
>
> Seeing this kind of thing in nightly...
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new
> native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> {code}
> Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing
> our docker container. It is set to 20G (The hosts are 48G). Fork count is
> 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add
> the maven 4G and we could be over the top.
> Play w/ downing the fork size (in earlier study we didn't seem to need this
> much RAM when running a fat long test). Let me also take th ms off the mvn
> allocation to see if that helps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates
Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates URL: https://github.com/apache/hbase/pull/1407#issuecomment-607529712 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 23s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 5m 48s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 53s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 68m 46s | hbase-server in the patch passed. | | | | 99m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1407 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 318abf56260c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/testReport/ | | Max. process+thread count | 5693 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting i
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reopened HBASE-24051: --- Reopening and reverting. I just saw the @openinx comment on one of the PRs asking for unit test to prove the old way doesn't work. > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Assignee: shenshengli >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.5 > > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #251: HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
Apache-HBase commented on issue #251: HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 URL: https://github.com/apache/hbase/pull/251#issuecomment-607526760 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | -1 :x: | hbaseanti | 0m 0s | The patch appears use Hadoop classification instead of HBase. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 1m 46s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 27s | branch-1 passed | | +1 :green_heart: | compile | 1m 45s | branch-1 passed with JDK v1.8.0_242 | | +1 :green_heart: | compile | 1m 52s | branch-1 passed with JDK v1.7.0_252 | | +1 :green_heart: | checkstyle | 8m 57s | branch-1 passed | | +0 :ok: | refguide | 4m 3s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 40s | branch-1 passed with JDK v1.8.0_242 | | +1 :green_heart: | javadoc | 5m 28s | branch-1 passed with JDK v1.7.0_252 | | +0 :ok: | spotbugs | 2m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +0 :ok: | findbugs | 0m 19s | branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml) | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | -1 :x: | compile | 0m 8s | root in the patch failed with JDK v1.8.0_242. | | -1 :x: | javac | 0m 8s | root in the patch failed with JDK v1.8.0_242. | | +1 :green_heart: | compile | 2m 3s | the patch passed with JDK v1.7.0_252 | | +1 :green_heart: | javac | 2m 3s | the patch passed | | +1 :green_heart: | checkstyle | 5m 58s | root: The patch generated 0 new + 87 unchanged - 11 fixed = 87 total (was 98) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | xml | 0m 1s | The patch has 5 ill-formed XML file(s). | | +0 :ok: | refguide | 3m 8s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 15s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | -1 :x: | javadoc | 0m 8s | root in the patch failed with JDK v1.8.0_242. | | +1 :green_heart: | javadoc | 4m 7s | the patch passed with JDK v1.7.0_252 | | +0 :ok: | findbugs | 0m 11s | hbase-resource-bundle has no data from findbugs | | -1 :x: | findbugs | 0m 7s | hbase-tinylfu-blockcache in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 186m 24s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 295m 27s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | hbase-common/src/main/resources/hbase-default.xml | | | hbase-it/pom.xml | | | hbase-resource-bundle/src/main/resources/supplemental-models.xml | | | hbase-tinylfu-blockcache/pom.xml | | | pom.xml | | Failed junit tests | hadoop.hbase.regionserver.TestPerColumnFamilyFlush | | | hadoop.hbase.master.TestMasterBalanceThrottling | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-251/32/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/251 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml | | uname | Linux bc7139670a57 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-251/out/precommit/personality/provided.sh | | git revision | branch-1 / 96ae39b | | Default Java | 1.7.0_252 | | Multi-JDK versions | /usr/lib/jvm/
[GitHub] [hbase] Apache-HBase commented on issue #1368: HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check
Apache-HBase commented on issue #1368: HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check URL: https://github.com/apache/hbase/pull/1368#issuecomment-607512087 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 18s | master passed | | +1 :green_heart: | checkstyle | 0m 36s | master passed | | +1 :green_heart: | spotbugs | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 8s | the patch passed | | +1 :green_heart: | checkstyle | 0m 35s | the patch passed | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 10m 22s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 0m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 32m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1368 | | Optional Tests | dupname asflicense shellcheck shelldocs xml spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 2c8add5d9b94 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Max. process+thread count | 93 (vs. ulimit of 1) | | modules | C: hbase-common hbase-assembly U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1368/3/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) shellcheck=0.4.6 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates
Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates URL: https://github.com/apache/hbase/pull/1407#issuecomment-607512170 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 9s | master passed | | +1 :green_heart: | checkstyle | 1m 21s | master passed | | +1 :green_heart: | spotbugs | 2m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 37s | the patch passed | | +1 :green_heart: | checkstyle | 1m 17s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 12s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 3m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 45m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1407 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux a215f8df0e7d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Max. process+thread count | 83 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/2/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
Apache-HBase commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat URL: https://github.com/apache/hbase/pull/1413#issuecomment-607510325 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 5m 20s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 43s | master passed | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 33s | the patch passed | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-assembly in the patch passed. | | | | 27m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1413 | | Optional Tests | javac javadoc unit | | uname | Linux 94d122c6a2c5 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 2020-01-14 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/testReport/ | | Max. process+thread count | 71 (vs. ulimit of 1) | | modules | C: hbase-assembly U: hbase-assembly | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401930132
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
> make it an else if? meaning if the user didn't override
UNSAFE_STREAM_CAPABILITY_ENFORCE and its a local fs mode..
I explicitly disable it here because we know it doesn't work with the local
filesystem. There are plenty of places where conf lookups of
`UNSAFE_STREAM_CAPABILITY_ENFORCE` have a default of `true`, so leaving it
unset sometimes doesn't help.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
ndimiduk commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401931301
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
> Can you not ask fs if it is an instance of local fs? If it is, then set
the boolean?
> would we run non-distributed on an fs that supports sync? s3?
My preference would be `if !distributed && !fs.supportsCapabilities()`,
however the only way I see to ask a FS if it supports the capabilities is to
first instantiate a stream, which I didn't want to do here. Other suggestions?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
Apache-HBase commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat URL: https://github.com/apache/hbase/pull/1413#issuecomment-607507537 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 2s | master passed | | +1 :green_heart: | javadoc | 0m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | the patch passed | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 29s | hbase-assembly in the patch passed. | | | | 20m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1413 | | Optional Tests | javac javadoc unit | | uname | Linux f3a5d256449c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/testReport/ | | Max. process+thread count | 62 (vs. ulimit of 1) | | modules | C: hbase-assembly U: hbase-assembly | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting i
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reassigned HBASE-24051: - Assignee: shenshengli > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Assignee: shenshengli >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.5 > > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resul
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073195#comment-17073195 ] Michael Stack edited comment on HBASE-24051 at 4/1/20, 9:43 PM: I see you opened new one instead #1406. Makes sense. I merged it to master, branch-2 and branch-2.2. Where else do you want it? Reopen if this not enough places. was (Author: stack): I see you opened new one instead #1406. Makes sense. I merged it to master, branch-2 and branch-2.3. Where else do you want it? > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Priority: Major > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting i
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-24051. --- Fix Version/s: 2.2.5 2.3.0 3.0.0 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.2+ Thanks for the patch [~shenshengli]. I made you a contributor and assigned you this issue. Thanks. > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.5 > > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073195#comment-17073195 ] Michael Stack commented on HBASE-24051: --- I see you opened new one instead #1406. Makes sense. I merged it to master, branch-2 and branch-2.3. Where else do you want it? > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Priority: Major > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #1406: HBASE-24051 Allows indirect inheritance to CanUnbuffer
saintstack merged pull request #1406: HBASE-24051 Allows indirect inheritance to CanUnbuffer URL: https://github.com/apache/hbase/pull/1406 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1404: HBASE-24051 Allows indirect inheritance to CanUnbuffer
saintstack commented on issue #1404: HBASE-24051 Allows indirect inheritance to CanUnbuffer URL: https://github.com/apache/hbase/pull/1404#issuecomment-607502574 #1406 opened to replace this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
ndimiduk commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat URL: https://github.com/apache/hbase/pull/1413#issuecomment-607502075 For reviewer reference (@jatsakthi in particular), the transitive dependency hull of `org.apache.hadoop:hadoop-common:jar:2.10.0:compile`: ``` [INFO] +- org.apache.hadoop:hadoop-common:jar:2.10.0:compile [INFO] | +- asm:asm:jar:3.1:compile [INFO] | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile [INFO] | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile [INFO] | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile [INFO] | +- com.jcraft:jsch:jar:0.1.54:compile [INFO] | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile [INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile [INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile [INFO] | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | +- commons-logging:commons-logging:jar:1.2:compile [INFO] | +- commons-net:commons-net:jar:3.1:compile [INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.12:compile [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile [INFO] | +- org.apache.avro:avro:jar:1.7.7:compile [INFO] | +- org.apache.commons:commons-compress:jar:1.19:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile [INFO] | +- org.apache.curator:curator-client:jar:4.2.0:compile [INFO] | +- org.apache.curator:curator-framework:jar:4.2.0:compile [INFO] | +- org.apache.curator:curator-recipes:jar:4.2.0:compile [INFO] | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile [INFO] | +- org.apache.directory.api:api-util:jar:1.0.0-M20:compile [INFO] | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile [INFO] | +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.10.0:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.10.0:compile [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.14:compile [INFO] | +- org.codehaus.jettison:jettison:jar:1.3.8:compile [INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile [INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.30:compile [INFO] | +- org.xerial.snappy:snappy-java:jar:1.0.5:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile ``` And of `com.sun.xml.ws:jaxws-ri:pom:2.3.2:compile` ``` [INFO] +- com.sun.xml.ws:jaxws-ri:pom:2.3.2:compile [INFO] | +- com.sun.istack:istack-commons-runtime:jar:3.0.8:compile [INFO] | +- com.sun.xml.bind:jaxb-jxc:jar:2.3.2:compile [INFO] | +- com.sun.xml.bind:jaxb-xjc:jar:2.3.2:compile [INFO] | +- com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile [INFO] | +- com.sun.xml.messaging.saaj:saaj-impl:jar:1.5.1:compile [INFO] | +- com.sun.xml.stream.buffer:streambuffer:jar:1.5.7:compile [INFO] | +- com.sun.xml.ws:jaxws-eclipselink-plugin:jar:2.3.2:compile [INFO] | +- com.sun.xml.ws:jaxws-rt:jar:2.3.2:compile [INFO] | +- com.sun.xml.ws:jaxws-tools:jar:2.3.2:compile [INFO] | +- com.sun.xml.ws:policy:jar:2.7.6:compile [INFO] | +- com.sun.xml.ws:release-documentation:zip:docbook:2.3.2:compile [INFO] | +- com.sun.xml.ws:samples:zip:2.3.2:compile [INFO] | +- com.sun.xml.ws:sdo-eclipselink-plugin:jar:2.3.2:compile [INFO] | +- jakarta.activation:jakarta.activation-api:jar:1.2.1:compile [INFO] | +- jakarta.annotation:jakarta.annotation-api:jar:1.3.4:compile [INFO] | +- jakarta.jws:jakarta.jws-api:jar:1.1.1:compile [INFO] | +- jakarta.mail:jakarta.mail-api:jar:1.6.3:compile [INFO] | +- jakarta.persistence:jakarta.persistence-api:jar:2.2.2:compile [INFO] | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile [INFO] | +- jakarta.xml.soap:jakarta.xml.soap-api:jar:1.4.1:compile [INFO] | +- jakarta.xml.ws:jakarta.xml.ws-api:jar:2.3.2:compile [INFO] | +- org.eclipse.persistence:commonj.sdo:jar:2.1.1:compile [INFO] | +- org.eclipse.persistence:org.eclipse.persistence.asm:jar:2.7.4:compile [INFO] |
[GitHub] [hbase] Apache-HBase commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
Apache-HBase commented on issue #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat URL: https://github.com/apache/hbase/pull/1413#issuecomment-607500592 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 2m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1413 | | Optional Tests | dupname asflicense xml | | uname | Linux 3931444a8385 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Max. process+thread count | 51 (vs. ulimit of 1) | | modules | C: hbase-assembly U: hbase-assembly | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1413/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1188: HBASE-23843 Removed deprecated Scan(byte[]) from Scan
Apache-HBase commented on issue #1188: HBASE-23843 Removed deprecated Scan(byte[]) from Scan URL: https://github.com/apache/hbase/pull/1188#issuecomment-607498286 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 43s | master passed | | +1 :green_heart: | compile | 3m 44s | master passed | | +1 :green_heart: | shadedjars | 5m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 32s | master passed | | -0 :warning: | patch | 8m 47s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 33s | the patch passed | | +1 :green_heart: | compile | 3m 46s | the patch passed | | +1 :green_heart: | javac | 3m 46s | the patch passed | | +1 :green_heart: | shadedjars | 5m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 52s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 95m 8s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 6m 56s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 2m 39s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 7s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 3m 2s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 1m 33s | hbase-examples in the patch passed. | | | | 156m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1188 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9861b9973662 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/testReport/ | | Max. process+thread count | 4396 (vs. ulimit of 1) | | modules | C: hbase-client hbase-server hbase-mapreduce hbase-thrift hbase-shell hbase-rest hbase-examples U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk opened a new pull request #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
ndimiduk opened a new pull request #1413: HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat URL: https://github.com/apache/hbase/pull/1413 ### sort lines and cleanup whitespace in hbase-assembly/hadoop-two-compat.xml Simply cleanup the whitespace and readability of these include, exclude lists before making any subsequent changes. To be squished into the subsequent commit on merge. ### HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat After HBASE-23986, running on JDK8/Hadoop2 from the binary tarball results in ``` Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062) Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more ``` The missing class is from `com.fasterxml.woodstox:woodstox-core`. This and `org.codehaus.woodstox:stax2-api` are explicit dependencies of `org.apache.hadoop:hadoop-common:jar:2.10.0`. Thus, include them in the default jars provided under `lib`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
saintstack commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401913995
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
Can you not ask fs if it is an instance of local fs? If it is, then set the
boolean?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-24084) Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
[
https://issues.apache.org/jira/browse/HBASE-24084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24084:
-
Summary: Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
(was: master fails to start out of binary tarball)
Subject was "master fails to start out of binary tarball".
> Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
> -
>
> Key: HBASE-24084
> URL: https://issues.apache.org/jira/browse/HBASE-24084
> Project: HBase
> Issue Type: Bug
> Components: build, master
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> Trying to launch a standalone master out of binary tarball, doesn't start.
> {noformat}
> $
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
> ~/tmp/packaging_and_integration/hbase-install/bin/hbase master start
> 2020-03-30 17:14:21,831 INFO [main] master.HMaster: STARTING service HMaster
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: HBase 3.0.0-SNAPSHOT
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Source code repository
> file:///Users/ndimiduk/tmp/packaging_and_integration/unpacked_src_tarball
> revision=Unknown
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Compiled by ndimiduk
> on Mon Mar 30 14:59:17 PDT 2020
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: From source with
> checksum
> c13d259fe1832de6fef3f5d192e6651d3ca0b52599c64ef19eaa87c6bdbd3956cc51fda3c3dc917b47add5a633c9058a6968df24ddce88f85c345681763cd568
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)
> Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
saintstack commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401915111
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
Hmm Looking here, the distributed flag is good to check [1] but would we
run non-distributed on an fs that supports sync? s3?
1. http://hbase.apache.org/book.html#standalone
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1377: HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module
Apache-HBase commented on issue #1377: HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module URL: https://github.com/apache/hbase/pull/1377#issuecomment-607493893 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 2s | No case conflicting files found. | | +0 :ok: | prototool | 0m 2s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 44s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 14s | master passed | | +1 :green_heart: | checkstyle | 0m 56s | master passed | | +1 :green_heart: | spotbugs | 7m 21s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 36s | the patch passed | | +1 :green_heart: | checkstyle | 0m 39s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 10s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | hbaseprotoc | 2m 25s | the patch passed | | +1 :green_heart: | spotbugs | 5m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 56m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1377 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux 39e0cd8dc7fb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Max. process+thread count | 94 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-endpoint hbase-examples U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates
Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates URL: https://github.com/apache/hbase/pull/1407#issuecomment-607493258 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 17s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 55s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 7m 58s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 1m 5s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -0 :warning: | unit | 157m 42s | hbase-server in the patch failed. | | | | 194m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1407 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 58a4acee5f67 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/testReport/ | | Max. process+thread count | 3894 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1188: HBASE-23843 Removed deprecated Scan(byte[]) from Scan
Apache-HBase commented on issue #1188: HBASE-23843 Removed deprecated Scan(byte[]) from Scan URL: https://github.com/apache/hbase/pull/1188#issuecomment-607492518 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 6m 15s | master passed | | +1 :green_heart: | compile | 4m 12s | master passed | | +1 :green_heart: | shadedjars | 5m 33s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 19s | hbase-examples in master failed. | | -0 :warning: | javadoc | 0m 21s | hbase-mapreduce in master failed. | | -0 :warning: | javadoc | 0m 20s | hbase-rest in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 49s | hbase-thrift in master failed. | | -0 :warning: | patch | 9m 59s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 51s | the patch passed | | +1 :green_heart: | compile | 4m 11s | the patch passed | | +1 :green_heart: | javac | 4m 11s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 33s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 51s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 24s | hbase-mapreduce in the patch failed. | | -0 :warning: | javadoc | 1m 7s | hbase-thrift in the patch failed. | | -0 :warning: | javadoc | 0m 24s | hbase-rest in the patch failed. | | -0 :warning: | javadoc | 0m 24s | hbase-examples in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 11s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 75m 25s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 6m 46s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 2m 33s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 11s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 3m 12s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 1m 27s | hbase-examples in the patch passed. | | | | 142m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1188 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a3a0ce8ea8b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-examples.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-mapreduce.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-rest.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://
[GitHub] [hbase] bharathv commented on a change in pull request #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
bharathv commented on a change in pull request #1408: HBASE-24086 Disable
output stream capability enforcement when running in standalone mode
URL: https://github.com/apache/hbase/pull/1408#discussion_r401910697
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/util/CommonFSUtils.java
##
@@ -350,11 +350,18 @@ public static void setWALRootDir(final Configuration c,
final Path root) {
public static FileSystem getWALFileSystem(final Configuration c) throws
IOException {
Path p = getWALRootDir(c);
FileSystem fs = p.getFileSystem(c);
-// hadoop-core does fs caching, so need to propogate this if set
+// hadoop-core does fs caching, so need to propagate this if set
String enforceStreamCapability = c.get(UNSAFE_STREAM_CAPABILITY_ENFORCE);
if (enforceStreamCapability != null) {
fs.getConf().set(UNSAFE_STREAM_CAPABILITY_ENFORCE,
enforceStreamCapability);
}
+if (!c.getBoolean(HConstants.CLUSTER_DISTRIBUTED, false)) {
Review comment:
nit: make it an else if? meaning if the user didn't override
UNSAFE_STREAM_CAPABILITY_ENFORCE and its a local fs mode..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode URL: https://github.com/apache/hbase/pull/1408#issuecomment-607490076 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 4s | master passed | | +1 :green_heart: | compile | 0m 26s | master passed | | +1 :green_heart: | shadedjars | 6m 13s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 38s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | shadedjars | 6m 10s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 21s | hbase-common in the patch passed. | | -0 :warning: | unit | 114m 28s | hbase-server in the patch failed. | | | | 146m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1408 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 19045a0e8c32 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/testReport/ | | Max. process+thread count | 4180 (vs. ulimit of 1) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates
Apache-HBase commented on issue #1407: HBASE-24099 Use a fair ReentrantReadWriteLock for the locks used to guard region closing and updates URL: https://github.com/apache/hbase/pull/1407#issuecomment-607489711 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 14s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 37s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 155m 26s | hbase-server in the patch failed. | | | | 185m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1407 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d2d0e2e27123 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/testReport/ | | Max. process+thread count | 3688 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1407/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1377: HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module
Apache-HBase commented on issue #1377: HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module URL: https://github.com/apache/hbase/pull/1377#issuecomment-607486440 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 6m 13s | master passed | | +1 :green_heart: | compile | 1m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 39s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-examples in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 56s | the patch passed | | +1 :green_heart: | compile | 1m 57s | the patch passed | | +1 :green_heart: | javac | 1m 57s | the patch passed | | +1 :green_heart: | shadedjars | 5m 38s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-examples in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 2m 45s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 1m 12s | hbase-examples in the patch passed. | | | | 38m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1377 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fdcc4d9c6046 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-examples.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-examples.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/testReport/ | | Max. process+thread count | 3809 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-endpoint hbase-examples U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (HBASE-24084) master fails to start out of binary tarball
[
https://issues.apache.org/jira/browse/HBASE-24084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reassigned HBASE-24084:
Assignee: Nick Dimiduk
> master fails to start out of binary tarball
> ---
>
> Key: HBASE-24084
> URL: https://issues.apache.org/jira/browse/HBASE-24084
> Project: HBase
> Issue Type: Bug
> Components: build, master
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> Trying to launch a standalone master out of binary tarball, doesn't start.
> {noformat}
> $
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
> ~/tmp/packaging_and_integration/hbase-install/bin/hbase master start
> 2020-03-30 17:14:21,831 INFO [main] master.HMaster: STARTING service HMaster
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: HBase 3.0.0-SNAPSHOT
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Source code repository
> file:///Users/ndimiduk/tmp/packaging_and_integration/unpacked_src_tarball
> revision=Unknown
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Compiled by ndimiduk
> on Mon Mar 30 14:59:17 PDT 2020
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: From source with
> checksum
> c13d259fe1832de6fef3f5d192e6651d3ca0b52599c64ef19eaa87c6bdbd3956cc51fda3c3dc917b47add5a633c9058a6968df24ddce88f85c345681763cd568
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)
> Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-24084) master fails to start out of binary tarball
[
https://issues.apache.org/jira/browse/HBASE-24084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-24084 started by Nick Dimiduk.
> master fails to start out of binary tarball
> ---
>
> Key: HBASE-24084
> URL: https://issues.apache.org/jira/browse/HBASE-24084
> Project: HBase
> Issue Type: Bug
> Components: build, master
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> Trying to launch a standalone master out of binary tarball, doesn't start.
> {noformat}
> $
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
> ~/tmp/packaging_and_integration/hbase-install/bin/hbase master start
> 2020-03-30 17:14:21,831 INFO [main] master.HMaster: STARTING service HMaster
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: HBase 3.0.0-SNAPSHOT
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Source code repository
> file:///Users/ndimiduk/tmp/packaging_and_integration/unpacked_src_tarball
> revision=Unknown
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Compiled by ndimiduk
> on Mon Mar 30 14:59:17 PDT 2020
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: From source with
> checksum
> c13d259fe1832de6fef3f5d192e6651d3ca0b52599c64ef19eaa87c6bdbd3956cc51fda3c3dc917b47add5a633c9058a6968df24ddce88f85c345681763cd568
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)
> Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24084) master fails to start out of binary tarball
[
https://issues.apache.org/jira/browse/HBASE-24084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24084:
-
Affects Version/s: 2.3.0
> master fails to start out of binary tarball
> ---
>
> Key: HBASE-24084
> URL: https://issues.apache.org/jira/browse/HBASE-24084
> Project: HBase
> Issue Type: Bug
> Components: build, master
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Major
>
> Trying to launch a standalone master out of binary tarball, doesn't start.
> {noformat}
> $
> JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
> ~/tmp/packaging_and_integration/hbase-install/bin/hbase master start
> 2020-03-30 17:14:21,831 INFO [main] master.HMaster: STARTING service HMaster
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: HBase 3.0.0-SNAPSHOT
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Source code repository
> file:///Users/ndimiduk/tmp/packaging_and_integration/unpacked_src_tarball
> revision=Unknown
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: Compiled by ndimiduk
> on Mon Mar 30 14:59:17 PDT 2020
> 2020-03-30 17:14:21,833 INFO [main] util.VersionInfo: From source with
> checksum
> c13d259fe1832de6fef3f5d192e6651d3ca0b52599c64ef19eaa87c6bdbd3956cc51fda3c3dc917b47add5a633c9058a6968df24ddce88f85c345681763cd568
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)
> Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1377: HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module
Apache-HBase commented on issue #1377: HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module URL: https://github.com/apache/hbase/pull/1377#issuecomment-607483949 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 19s | master passed | | +1 :green_heart: | compile | 1m 38s | master passed | | +1 :green_heart: | shadedjars | 4m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 4m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 43s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 2m 46s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 1m 8s | hbase-examples in the patch passed. | | | | 32m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1377 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4efdcd52fe78 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 040a0a7ac5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/testReport/ | | Max. process+thread count | 3572 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-endpoint hbase-examples U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1377/4/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode
Apache-HBase commented on issue #1408: HBASE-24086 Disable output stream capability enforcement when running in standalone mode URL: https://github.com/apache/hbase/pull/1408#issuecomment-607469079 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | master passed | | +1 :green_heart: | compile | 0m 24s | master passed | | +1 :green_heart: | shadedjars | 4m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 14s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | shadedjars | 4m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 72m 48s | hbase-server in the patch passed. | | | | 98m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1408 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1580c68550b6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/testReport/ | | Max. process+thread count | 5359 (vs. ulimit of 1) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1408/1/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1188: HBASE-23843 Removed deprecated Scan(byte[]) from Scan
Apache-HBase commented on issue #1188: HBASE-23843 Removed deprecated Scan(byte[]) from Scan URL: https://github.com/apache/hbase/pull/1188#issuecomment-607463382 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 43s | master passed | | +1 :green_heart: | checkstyle | 4m 8s | master passed | | +1 :green_heart: | spotbugs | 7m 57s | master passed | | -0 :warning: | patch | 1m 49s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 7m 51s | the patch passed | | +1 :green_heart: | checkstyle | 4m 7s | the patch passed | | -0 :warning: | rubocop | 0m 11s | The patch generated 1 new + 110 unchanged - 1 fixed = 111 total (was 111) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 16m 33s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. | | +1 :green_heart: | spotbugs | 9m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 39s | The patch does not generate ASF License warnings. | | | | 75m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1188 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 766f1395d3fc 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0595200fb5 | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 84 (vs. ulimit of 1) | | modules | C: hbase-client hbase-server hbase-mapreduce hbase-thrift hbase-shell hbase-rest hbase-examples U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1188/9/console | | versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-24101) Correct snapshot handling
[ https://issues.apache.org/jira/browse/HBASE-24101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073152#comment-17073152 ] Sean Busbey commented on HBASE-24101: - I'll try to get my WIP test up on HBASE-23723 this week so that we can avoid some duplicative work here. > Correct snapshot handling > - > > Key: HBASE-24101 > URL: https://issues.apache.org/jira/browse/HBASE-24101 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24051) When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting
[ https://issues.apache.org/jira/browse/HBASE-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073140#comment-17073140 ] Michael Stack commented on HBASE-24051: --- Why close the PR [~shenshengli] ? Thanks. > When the data directory is set to EC,the amount of storefiles reaches a > relatively high level,heap Occupancy Percent is above heap occupancy alarm > watermark,resulting in frequent gc and eventual failure > -- > > Key: HBASE-24051 > URL: https://issues.apache.org/jira/browse/HBASE-24051 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: shenshengli >Priority: Major > Attachments: 1.png > > > When the data directory is set to EC, during the continuous load process, > regionserver is found down ,to view the log: > regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below > the heap occupancy alarm watermark (0.95) > regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap > occupancy alarm watermark (0.95) > [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine > (eg GC): pause of approximately 1254ms > GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms > GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms > In particular, GC pauses occurred 283 times in half an hour. > The memory analysis found that most of the memory footprint was generated by > HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer > See HDFS-14308，but after the fix, hbase did not return to normal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24101) Correct snapshot handling
Vladimir Rodionov created HBASE-24101: - Summary: Correct snapshot handling Key: HBASE-24101 URL: https://issues.apache.org/jira/browse/HBASE-24101 Project: HBase Issue Type: Sub-task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23723) Add tests for MOB compaction on a table created from snapshot
[ https://issues.apache.org/jira/browse/HBASE-23723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073132#comment-17073132 ] Vladimir Rodionov commented on HBASE-23723: --- Reopening this umbrella to address correct snapshot handling. Particularly, the following scenario must be verified: #load data to a table #take snapshot #major compact table #run mob file cleaner chore #load data to table #restore table from snapshot into another table #verify data integrity #restore table from snapshot into original table #verify data integrity > Add tests for MOB compaction on a table created from snapshot > - > > Key: HBASE-23723 > URL: https://issues.apache.org/jira/browse/HBASE-23723 > Project: HBase > Issue Type: Sub-task > Components: Compaction, mob >Reporter: Vladimir Rodionov >Assignee: Sean Busbey >Priority: Blocker > > How does code handle snapshot naming convention for MOB files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23723) Add tests for MOB compaction on a table created from snapshot
[ https://issues.apache.org/jira/browse/HBASE-23723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073132#comment-17073132 ] Vladimir Rodionov edited comment on HBASE-23723 at 4/1/20, 7:38 PM: Reopening this umbrella to address correct snapshot handling. Particularly, the following scenario must be verified: # load data to a table # take snapshot # major compact table # run mob file cleaner chore # load data to table # restore table from snapshot into another table # verify data integrity # restore table from snapshot into original table # verify data integrity was (Author: vrodionov): Reopening this umbrella to address correct snapshot handling. Particularly, the following scenario must be verified: #load data to a table #take snapshot #major compact table #run mob file cleaner chore #load data to table #restore table from snapshot into another table #verify data integrity #restore table from snapshot into original table #verify data integrity > Add tests for MOB compaction on a table created from snapshot > - > > Key: HBASE-23723 > URL: https://issues.apache.org/jira/browse/HBASE-23723 > Project: HBase > Issue Type: Sub-task > Components: Compaction, mob >Reporter: Vladimir Rodionov >Assignee: Sean Busbey >Priority: Blocker > > How does code handle snapshot naming convention for MOB files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073122#comment-17073122 ] Vladimir Rodionov edited comment on HBASE-22749 at 4/1/20, 7:37 PM: Reopening this umbrella to address correct snapshot handling. Particularly, the following scenario must be verified: # load data to a table # take snapshot # major compact table # run mob file cleaner chore # load data to table # restore table from snapshot into another table # verify data integrity # restore table from snapshot into original table # verify data integrity was (Author: vrodionov): Reopening this umbrella to address correct snapshot handling > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-22749-branch-2.2-v4.patch, > HBASE-22749-master-v1.patch, HBASE-22749-master-v2.patch, > HBASE-22749-master-v3.patch, HBASE-22749-master-v4.patch, > HBASE-22749_nightly_Unit_Test_Results.csv, > HBASE-22749_nightly_unit_test_analyzer.pdf, HBase-MOB-2.0-v3.0.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #1411: Backport "HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs" to branch-2
ndimiduk merged pull request #1411: Backport "HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs" to branch-2 URL: https://github.com/apache/hbase/pull/1411 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk merged pull request #1412: Backport "HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs" to branch-2.3
ndimiduk merged pull request #1412: Backport "HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs" to branch-2.3 URL: https://github.com/apache/hbase/pull/1412 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk opened a new pull request #1411: HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs
ndimiduk opened a new pull request #1411: HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs URL: https://github.com/apache/hbase/pull/1411 Signed-off-by: Bharath Vissapragada Signed-off-by: Jan Hentschel Signed-off-by: Duo Zhang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-24071) [JDK11] Remove `unit` filter from nightly and precommit jobs
[ https://issues.apache.org/jira/browse/HBASE-24071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24071. -- Fix Version/s: 2.3.0 3.0.0 Resolution: Fixed > [JDK11] Remove `unit` filter from nightly and precommit jobs > > > Key: HBASE-24071 > URL: https://issues.apache.org/jira/browse/HBASE-24071 > Project: HBase > Issue Type: Sub-task > Components: build, test >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Added in HBASE-23946, we can remove this filter once HBASE-24007 lands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1412: Backport "HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs" to branch-2.3
ndimiduk opened a new pull request #1412: Backport "HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs" to branch-2.3 URL: https://github.com/apache/hbase/pull/1412 Signed-off-by: Bharath Vissapragada Signed-off-by: Jan Hentschel Signed-off-by: Duo Zhang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #1385: HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs
ndimiduk commented on issue #1385: HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs URL: https://github.com/apache/hbase/pull/1385#issuecomment-607446323 Thanks for the reviews folks, and your patience while I work through these JDK11 tasks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk merged pull request #1385: HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs
ndimiduk merged pull request #1385: HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs URL: https://github.com/apache/hbase/pull/1385 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-24100) [Flakey Tests] Add test to check we work properly when port clash setting up thriftserver
[ https://issues.apache.org/jira/browse/HBASE-24100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-24100. --- Fix Version/s: 2.3.0 3.0.0 Assignee: Michael Stack Resolution: Fixed What I pushed to branch-2.3+ > [Flakey Tests] Add test to check we work properly when port clash setting up > thriftserver > - > > Key: HBASE-24100 > URL: https://issues.apache.org/jira/browse/HBASE-24100 > Project: HBase > Issue Type: Bug > Components: flakies, Thrift >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-24100-Flakey-Tests-Add-test-to-check-we-work-p.patch > > > My hack to add retry when thrift setup has a port clash over in HBASE-24079 > didn't work. Add a test and fix what it turns up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24100) [Flakey Tests] Add test to check we work properly when port clash setting up thriftserver
[ https://issues.apache.org/jira/browse/HBASE-24100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-24100: -- Attachment: 0001-HBASE-24100-Flakey-Tests-Add-test-to-check-we-work-p.patch > [Flakey Tests] Add test to check we work properly when port clash setting up > thriftserver > - > > Key: HBASE-24100 > URL: https://issues.apache.org/jira/browse/HBASE-24100 > Project: HBase > Issue Type: Bug > Components: flakies, Thrift >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24100-Flakey-Tests-Add-test-to-check-we-work-p.patch > > > My hack to add retry when thrift setup has a port clash over in HBASE-24079 > didn't work. Add a test and fix what it turns up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-22749: --- Reopening this umbrella to address correct snapshot handling > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-22749-branch-2.2-v4.patch, > HBASE-22749-master-v1.patch, HBASE-22749-master-v2.patch, > HBASE-22749-master-v3.patch, HBASE-22749-master-v4.patch, > HBASE-22749_nightly_Unit_Test_Results.csv, > HBASE-22749_nightly_unit_test_analyzer.pdf, HBase-MOB-2.0-v3.0.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24071) [JDK11] Remove `unit` filter from nightly and precommit jobs
[ https://issues.apache.org/jira/browse/HBASE-24071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24071: - Affects Version/s: 2.3.0 3.0.0 > [JDK11] Remove `unit` filter from nightly and precommit jobs > > > Key: HBASE-24071 > URL: https://issues.apache.org/jira/browse/HBASE-24071 > Project: HBase > Issue Type: Sub-task > Components: build, test >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Added in HBASE-23946, we can remove this filter once HBASE-24007 lands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24092) Fix links to build reports generated by nightly job
[
https://issues.apache.org/jira/browse/HBASE-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-24092.
--
Resolution: Fixed
> Fix links to build reports generated by nightly job
> ---
>
> Key: HBASE-24092
> URL: https://issues.apache.org/jira/browse/HBASE-24092
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
>
> Links going back to JIRA look like
> {noformat}
> For more information [see jdk8 (hadoop2)
> report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2574//JDK8_Nightly_Build_Report_(Hadoop2)/]
> {noformat}
> But the actual URL to this report is
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2574/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24100) [Flakey Tests] Add test to check we work properly when port clash setting up thriftserver
[ https://issues.apache.org/jira/browse/HBASE-24100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-24100: -- Description: My hack to add retry when thrift setup has a port clash over in HBASE-24079 didn't work. Add a test and fix what it turns up. (was: My hack to retry when thrift setup has a port clash didn't work. Add a test and fix what it turns up.) > [Flakey Tests] Add test to check we work properly when port clash setting up > thriftserver > - > > Key: HBASE-24100 > URL: https://issues.apache.org/jira/browse/HBASE-24100 > Project: HBase > Issue Type: Bug > Components: flakies, Thrift >Reporter: Michael Stack >Priority: Major > > My hack to add retry when thrift setup has a port clash over in HBASE-24079 > didn't work. Add a test and fix what it turns up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
